Interpreting the judgment of surgical faculty regarding resident competence

A novel simulator-based checklist for evaluating residents' competence in cerebral angiography in China

Background

Nowadays, with the fast-increasing demand for neuro-endovascular therapy, surgeons in this field are in urgent need. Unfortunately, there is still no formal skill assessment in neuro-endovascular therapy in China.

Methods

We used a Delphi method to design a newly objective checklist for standards of cerebrovascular angiography in China and evaluated its validity and reliability. A total of 19 neuro-residents with no interventional experience and 19 neuro-endovascular surgeons from two centers (Guangzhou and Tianjin) were recruited; they were divided into two groups: residents and surgeons. Residents completed a simulation-based cerebrovascular angiography operation training before assessment. Assessments were under live and video record forms with two tools: the existing global rating scale (GRS) of endovascular performance and the new checklist.

Results

The average scores of residents were significantly increased after training in two centers (p < 0.05). There is good consistency between GRS and the checklist (p = 0.856). Intra-rater reliability (Spearman's rho) of the checklist was >0.9, and the same result was also observed in raters between different centers and different assessment forms (p < 0.001, rho > 0.9). The reliability of the checklist was higher than that of the GRS (Kendall's harmonious coefficient is 0.849, while GRS is 0.684).

Conclusion

The newly developed checklist appears reliable and valid for evaluating the technical performance of cerebral angiography and differentiating between trained and untrained trainees' performance well. For its efficiency, our method has been proven to be a feasible tool for resident angiography examination in certification nationwide.

The Role of Mixed Reality Simulation for Surgical Training in Spine: Phase 1 Validation

STUDY DESIGN

This study shows the first phase of validation of a new model for realistic training on spine surgery, conducted from January 2014 to November 2015.

OBJECTIVE

To propose and validate a new tool for neurosurgical education, associating virtual and realistic simulation (mixed reality), for spine surgery.

SUMMARY OF BACKGROUND DATA

Surgical simulation is a relatively new filed that has a lot to offer to neurosurgical education. Training a new surgeon may take years of hands-on procedures, increasing the risk to patient's safety. The development of surgical simulation platforms is therefore essential to reducing the risk of potentially serious risks and improving outcome.

METHODS

Sixteen experienced spinal surgeons evaluated these simulators and answered the questionnaire regarding the simulation as a beneficial education tool. They evaluated the simulators in regard to dissection by planes, identification of pathology (lumbar canal stenosis), instrumentation and simulation of cerebrospinal fluid (CSF) leak, and the relevant aspects of the computerized tomography (CT) imaging.

RESULTS

The virtual and physical simulators for spine surgery were approved by an expert surgery team, and considered adequate for educational purposes. The proportion of the answers was estimated by the confidence intervals.

CONCLUSION

The surgery team considered that this virtual simulation provides a highly effective training environment, and it significantly enhances teaching of surgical anatomy and operative strategies in the neurosurgical field. A mixture of physical and virtual simulation provided the desired results of enhancing the requisite psychomotor and cognitive skills, previously acquired only during a surgical apprenticeship. The combination of these tools may potentially improve and abbreviate the learning curve for trainees, in a safe environment.

LEVEL OF EVIDENCE

3.

Conventional Medical Education and the History of Simulation in Radiology

Simulation is a promising method for improving clinician performance, enhancing team training, increasing patient safety, and preventing errors. Training scenarios to enrich medical student and resident education, and apply toward competency assessment, recertification, and credentialing are important applications of simulation in radiology. This review will describe simulation training for procedural skills, interpretive and noninterpretive skills, team-based training and crisis management, professionalism and communication skills, as well as hybrid and in situ applications of simulation training. A brief overview of current simulation equipment and software and the barriers and strategies for implementation are described. Finally, methods of measuring competency and assessment are described, so that the interested reader can successfully implement simulation training into their practice.

Robotic sacrocolpopexy performance and cumulative summation analysis

OBJECTIVES

This study aimed to apply Cumulative Summation (CUSUM) analysis as a tool to monitor robotic sacrocolpopexy (RSCP) proficiency over time.

METHODS

A retrospective analysis of all women who underwent RSCP between September of 2008 and December of 2011 at the University of North Carolina at Chapel Hill. The performance for 2 attending surgeons was analyzed sequentially over time. Intraoperative complications such as genitourinary or gastrointestinal tract injury, conversion to laparotomy, pulmonary embolus, hemorrhage, and blood transfusion, were identified by International Classification of Diseases, Ninth Revision and Current Procedural Terminology codes. A successful outcome was defined as no intraoperative complications. The target value of success was set at less than 10% complications. CUSUM analysis was then sequentially applied to all RSCP cases for 2 attending surgeons.

RESULTS

Over 27 months, 169 RSCPs were performed. The first surgeon performed 107 RSCPs and the second surgeon performed 62 RSCPs with 8 (7.4%) and 3 (4.9%) intraoperative complications, respectively. Total complications included 7 (4.1%) cystotomies, 2 (1.2%) vaginal lacerations, 1 (0.6%) blood transfusion, and 1 (0.6%) bowel perforation. A CUSUM graph was created for each surgeon.

CONCLUSIONS

CUSUM analysis was successfully applied to monitor RSCP proficiency. Such testing of individual successive procedural outcomes with CUSUM may offer an objective tool to aid in physician self-assessment.

Pilot study of an objective structured clinical examination (“the Six Pack”) for evaluating clinical competencies

OBJECTIVE: The purpose of this study was to evaluate an Objective Structured Clinical Examination (OSCE) designed to examine all six competencies and provide meaningful results to improve resident performance.

STUDY DESIGN: A prospective, observational study.

METHODS: The OSCE consists of 1 hour for examination of a simulated patient, documentation, and feedback, with 30 minutes for an evidence-based medicine (EBM) test. Eight otolaryngology residents participated. Digital recordings and written documentation were evaluated on all competencies except practice-based learning. An EBM test was scored to assess practice-based learning.

RESULTS: Overall, senior residents scored better. Seniors scored better on performing a focused history, whereas juniors scored better for thoroughness. Seniors coded better than juniors, although seniors tended to undercode and juniors tended to over-code. Two cases of “insurance fraud” were also discovered.

CONCLUSIONS: The “Six Pack” successfully evaluated all competencies while providing valuable information. A clinical practicum was added to improve history and physical examination skills. The discussion of coding and a billing seminar also resulted. Residents believed the OSCE was valuable. Faculty members were impressed with the useful information obtained.

When to Cut? Using an Objective Structured Clinical Examination to Evaluate Surgical Decision-Making

OBJECTIVES

To create an objective structured clinical examination (OSCE) tailored to surgical residents that uses surgical case scenarios appropriate for the experience level of the resident to evaluate whether residents understand the indications for a specific surgery, identify when indications are met, and make the appropriate decision to proceed with an operation.

METHODS

The surgical OSCE consisted of two case scenarios: a junior case created for postgraduate year (PGY) 2 to 3 level residents and a senior case created for PGY 4 to 5 level residents. Four senior residents and four junior residents in the otolaryngology department participated. Residents were instructed to proceed as if they had all the capabilities of the otolaryngology clinic. Recordings were made of each encounter and were reviewed by two separate faculty member with expertise in each case. Faculty evaluated whether surgical indications were elicited by history, whether certain physical examination or radiographic findings were recognized, whether residents made the correct diagnosis, and whether residents not only made the decision to proceed with surgery but also indicated the correct surgery.

RESULTS

Seniors (100%) were better at obtaining needed surgical information and indications than juniors (25%). Seniors were more willing to make the decision to proceed with surgery (100%), whereas no juniors actually scheduled surgery. The reasons for this ranged from failure to elicit surgical indications to lack of surety in proceeding with surgery. All seniors recommended the appropriate surgery, but only 75% correctly identified the need for emergent intervention. Results of evaluations were reviewed with each resident individually.

CONCLUSIONS

The surgical OSCE was successful in evaluating resident surgical decision-making. Juniors were not as capable as seniors in eliciting surgical indications and in "making the jump" to proceed with surgery. These results were used by our faculty to work with juniors on surgical history-taking and decision-making. These results were also useful in identifying seniors who could recognize an emergency situation.

Karl Storz Lecture. Skills training in pediatric minimal access surgery

Learning the craft of surgery is central to every surgical program. Although effective, acquisition of skills, particularly that of minimal access surgery in the operating theater, is becoming increasingly difficult. Published data indicate that the early phase of the learning curve could be achieved outside the operating room. However, there is no consensus regarding the optimal training schemes and assessment tools. With an increase in the number of operations performed endoscopically and the number of surgeons performing them, the importance of well-defined and validated training programs cannot be overemphasized.

Surgical simulation: a current review

BACKGROUND

Simulation tools offer the opportunity for the acquisition of surgical skill in the preclinical setting. Potential educational, safety, cost, and outcome benefits have brought increasing attention to this area in recent years. Utility in ongoing assessment and documentation of surgical skill, and in documenting proficiency and competency by standardized metrics, is another potential application of this technology. Significant work is yet to be done in validating simulation tools in the teaching of endoscopic, laparoscopic, and other surgical skills. Early data suggest face and construct validity, and the potential for clinical benefit, from simulation-based preclinical skills development. The purpose of this review is to highlight the status of simulation in surgical education, including available simulator options, and to briefly discuss the future impact of these modalities on surgical training.

Utilization of standardized patients to evaluate clinical and interpersonal skills of surgical residents

BACKGROUND

This project was designed to determine the growth of interpersonal skills during the first year of a surgical residency.

METHODS

All categorical surgical residents were given a clinical skills examination of abdominal pain using standardized patients during their orientation (T1). The categorical residents were retested after 11 months (T2). The assessment tool was based on a 12-item modified version of the 5-point Likert Interpersonal Scale (IP) used on the National Board of Medical Examiners prototype Clinical Skills Examination and a 24-item, done-or-not-done, history-taking checklist. Residents' self-evaluation scores were compared to standardized patients' assessment scores. Data were analyzed using the Pearson correlation coefficient, Wilcoxon signed rank test, Student t test, and Cronbach alpha.

RESULTS

Thirty-eight categorical residents were evaluated at T1 and T2. At T1, in the history-taking exercise, the scores of the standardized patients and residents correlated (Pearson = .541, P = .000). In the interpersonal skills exercise, the scores of the standardized patients and residents did not correlate (Pearson = -0.238, P = .150). At T2, there was a significant improvement in the residents' self-evaluation scores in both the history-taking exercise (t = -3.280, P = .002) and the interpersonal skills exercise (t = 2.506, P = 0.017). In the history-taking exercise, the standardized patients' assessment scores correlated with the residents' self-evaluation scores (Pearson = 0.561, P = .000). In the interpersonal skills exercise, the standardized patients' assessment scores did not correlate with the residents' self-evaluation scores (Pearson = 0.078, P = .646).

CONCLUSIONS

Surgical residents demonstrate a consistently low level of self-awareness regarding their interpersonal skills. Observed improvement in resident self-evaluation may be a function of growth in self-confidence.

Assessment of psychomotor skills acquisition during laparoscopic cholecystectomy courses

BACKGROUND

Standardized short courses in laparoscopic cholecystectomy aim to teach laparoscopic skills to surgical trainees, although end-of-course assessments of performance remain subjective. The current study aims to objectively assess psychomotor skills acquisition of trainees attending laparoscopic cholecystectomy courses.

METHODS

Thirty-seven junior surgical trainees had their laparoscopic skills assessed before and after attending 1 of 3 separate 2-day courses (A, B, and C), all with identical format. Assessments were comprised of a standardized simulated laparoscopic task, with performance measured using a valid electromagnetic hand-motion tracking device.

RESULTS

Overall, trainees made significant improvements in path length (P=.006), number of movements (P<.001), and time taken (P<.001). Analyzing the 3 courses separately, only trainees attending courses A and C made significant improvements.

DISCUSSION

Objective validated methods can be used to assess learning of psychomotor skills on courses. In addition to providing participants with an insight into their skills, these data can be used to demonstrate course efficacy.

Laparoscopic skills training and assessment

BACKGROUND

The introduction of laparoscopic techniques to general surgery was associated with many unnecessary complications, which led to the development of skills laboratories to train novice laparoscopic surgeons. This article reviews the tools currently available for training and assessment in laparoscopic surgery.

METHODS

Medline searches were performed to identify articles with combinations of the following key words: laparoscopy, training, curriculum, virtual reality and assessment. Further articles were obtained by manually searching the reference lists of identified papers.

RESULTS

Current training involves the use of box trainers with either innate models or animal tissues; it lacks objective assessment of skill acquisition. Virtual reality simulators have the ability to teach laparoscopic psychomotor skills, and objective assessment is now possible using dexterity-based and video analysis systems.

CONCLUSION

The tools are now available for the development of a structured, competency-based, laparoscopic surgical training programme.

Perceived Educational Benefits of Objective Structured Clinical Examination (OSCE) Development and Implementation by Resident Learners

OBJECTIVE

To assess the perceived learning outcomes from the various roles involved in the development and implementation of an objective structured clinical examination (OSCE) by residents for residents.

METHODS

Final-year residents in Obstetrics and Gynaecology at the University of Toronto created OSCE stations in preparation for their certification examination. They evaluated their experience using a survey designed to assess the learning effectiveness of the roles played in an OSCE. Residents were asked to compare resident-created stations with faculty-created stations.

RESULTS

The students found every aspect of OSCE development to be of educational benefit. Residents rated the candidate role as more beneficial than other roles. Residents perceived the benefits of the OSCE sessions to be greater than equivalent lengths of time spent in traditional group study sessions.

CONCLUSIONS

A self-directed learning approach, based on OSCE development and implementation, shows promise as a learning aid at the senior residency level. We suggest a controlled trial designed to objectively measure outcomes of this learner-centred approach.

When can I be proficient in laparoscopic surgery? A systematic review of the evidence

PURPOSE

The aim of this study was to quantify the learning curve in laparoscopic surgery.

METHODS

A systematic review of the evidence using a defined search strategy (PubMed, Medline, OVID, Embase, ERIC, Cochrane databases) was performed. Studies without statistical evaluation of the learning curve and opinion articles were excluded. The authors analysed 7 common laparoscopic procedures: cholecystectomy, fundoplication, colectomy, herniorrhaphy, splenectomy, appendicectomy, and pyloromyotomy. The "initial" and "late" stages of experience were compared with regards to the following outcome measures: operating time, conversion rate, complication rate, and length of stay in hospital.

RESULTS

A total of 3,641 articles were reviewed, of which, 37 (25,777 patients) fulfilled the entry criteria (5 in children). In all articles, the definition of proficiency was subjective, and the number of operations required to reach it was highly variable. There were improvements in all 4 outcome measures for cholecystectomy, fundoplication, colectomy, herniorrhaphy, and splenectomy between the "initial" and "late" experience. No data were available for the learning curves in appendicectomy or pyloromyotomy.

CONCLUSIONS

The number of procedures required to reach proficiency in laparoscopic surgery has not been defined clearly. These findings are important for training, ethical and medico-legal issues.

Ability of the objective structured clinical examination to differentiate surgical residents, medical students, and physician assistant students

BACKGROUND

The Objective Structured Clinical Examination (OSCE) has been used extensively to evaluate the clinical abilities of medical students and residents. The purpose of this study was to investigate whether the standard OSCE would differentiate performance of subjects with different levels and/or types of training.

METHODS

We conducted a blinded OSCE, during which we simultaneously evaluated surgical residents from all 5 years of the general surgery training program, third-year medical students, and second-year physician assistant students. All examinees went through the same clinical evaluation stations, which consisted of history-taking, physical examination, technical skills, trauma management, and X-ray interpretation. The students and residents were rated at each station by a trained standardized patient evaluator or a faculty evaluator using a checklist for performance evaluation. All subjects wore surgical scrubs without name tags or identification of program or year of training.

RESULTS

Overall mean performance scores (P = 0.09, NS) were for surgical residents 71.2% (+/-9.7); for medical students 66.9% (+/-5.7); for physician assistant students 64.7% (+/-5.8). This shows a significant trend toward higher scores with more training. Surgical residents scored higher on technical stations, history-taking, and X-ray interpretation. Medical students scored higher in performance of physical examination. Physician assistant students scored quite close to the other two groups.

CONCLUSIONS

The differences among group performance appeared to reflect the level of experience of the learners. Some components of the OSCE appear to better differentiate levels of training.

Assessing laparoscopic manipulative skills

BACKGROUND

The rapid emergence of laparoscopic surgery over the past decade has highlighted the need for teaching and assessing laparoscopic manipulative skills. Most analyses consist of a measurement of the time to complete a specified task and a subjective assessment by an observer. Objective assessments of accuracy in the performance of manipulative skills have been lacking. To quantify both speed and accuracy during laparoscopic skill performance, we have developed a skills assessment device (SAD) using a laparoscopic simulator platform and computer-based skills assessment software that precisely measures an instrument's movement during performance of timed laparoscopic manipulations.

STUDY DESIGN

The SAD device measures the time necessary for an operator to complete a task, and the movements of the working end of laparoscopic instruments in three dimensions. Ten nonsurgeons performed 10 repetitions of a standardized laparoscopic manipulation. Data were captured in real time for both hands on a personal computer. Accuracy was determined by calculating the sum of all distances traveled. Duration was measured in seconds. Results are group means.

RESULTS

The time necessary to perform defined laparoscopic manipulative skills improved dramatically during the first 3 repetitions and then stabilized. However, accuracy of manipulations continued to improve over all 10 repetitions.

CONCLUSIONS

When untrained subjects are learning a laparoscopic manipulative task, measurement of time alone fails to account for the more protracted learning curve for accuracy. Therefore, devices and training programs that fail to consider objective assessments of accuracy may overestimate laparoscopic proficiency.

The Objective Structured Clinical Exam - practical aspects

Assessment of clinical skills in addition to factual knowledge has become part of the examination system in both undergraduate and postgraduate examinations. There is an increasing interest in the concept of competence and how it can be measured, particularly for the process of revalidation of practitioners. The Objective Structured Clinical Examination (OSCE) has been introduced for the assessment of clinical skills into postgraduate anaesthetic examinations in the United Kingdom. As with any examination, the issues regarding setting a standard are under continual review. The application of this exam format to the measurement of competency as well as skills has been the subject of scrutiny during the period reviewed. The literature suggests that competence is a much more complex construct than skill and that OSCE exams will be a contributory part but more work is required to establish assessment methods which will produce a global rating of competence. The role of the OSCE format in education as well as assessment has been examined and it is suggested that formative assessment can be made, and peer review learning brought about using an OSCE performed in teams.

The relative contributions of interpersonal and specific clinical skills to the perception of global clinical competence

BACKGROUND

The objective structured clinical examination (OSCE), an established instrument for evaluating resident competence, was used to test the hypothesis that faculty assessment of clinical competence in residents at various levels of training may be influenced more by general skills as a physician and less by competency in the actual skills being specifically tested. In this study, advantage was taken of the anticipated observation that general surgery residents did not demonstrate improvement in their ability to perform a focused neurological assessment over time.

METHODS

An OSCE, which was administered to 56 general surgery residents at all levels of training, included the assessment of a specific clinical neurosurgical problem (sciatica). Univariate and multivariate analyses were used to evaluate the relationship between the global faculty judgment of competent or noncompetent and the other performance measures that were applied.

RESULTS

At different levels of training, there was no observed difference in the specific skills being tested; nevertheless, junior and senior residents were more likely than incoming interns to be judged "competent" and received better evaluations of how well they introduced themselves to the patient. The competence judgment correlated significantly with all of the other performance measures, including the skills being tested.

CONCLUSIONS

The perception of competence is not solely dependent upon the particular skills under scrutiny. General competence does not guarantee competence in each specific skill set of a medical specialty.