Assessing the surgeon's technical skills: analysis of the available tools

Review of the validity of DOPS as an assessment tool for the procedural skills of surgical trainees

INTRODUCTION

Surgical trainees at all stages are mandated to use workplace-based assessments (WBAs) to gain feedback from their trainers. Direct observation of procedural skills (DOPS) is a tool to assess hands-on surgical skills. This review of the literature seeks to ascertain how valid DOPS are as an assessment tool for the procedural skills of surgical trainees according to the American Psychology Association (APA) validity framework.

METHODS

Relevant literature was identified through a structured search of Medline, CINAHL and Web of Science databases, with further papers included on citation review. Following this, papers meeting the inclusion and exclusion criteria were included in the final review specifying those investigating WBAs including DOPS, assessments in practice rather than simulation and specifically for postgraduate surgical trainees.

FINDINGS

Sixteen papers were included in the final analysis. Extracted data from the returned papers were assessed for evidence of validity in each of the five domains on the APA framework: validity based on consequence, response process, reliability, content and relationship to other variables. There are studies that show good parameters for reliability and validity for specific DOPS used in endoscopy and otolaryngology. However, there is confusion over the purpose of DOPS among trainers and trainees as to whether they should be used formatively or summatively. Recent changes to the surgical curriculum have sought to address this, and further work into the impact of this needs to be done.

Core procedural skills competencies and the maintenance of procedural skills for medical students: a Delphi study

BACKGROUND

It is well recognised that medical students need to acquire certain procedural skills during their medical training, however, agreement on the level and acquisition of competency to be achieved in these skills is under debate. Further, the maintenance of competency of procedural skills across medical curricula is often not considered. The purpose of this study was to identify core procedural skills competencies for Australian medical students and to establish the importance of the maintenance of such skills.

METHODS

A three-round, online Delphi method was used to identify consensus on competencies of procedural skills for graduating medical students in Australia. In Round 1, an initial structured questionnaire was developed using content identified from the literature. Respondents were thirty-six experts representing medical education and multidisciplinary clinicians involved with medical students undertaking procedural skills, invited to rate their agreement on the inclusion of teaching 74 procedural skills and 11 suggested additional procedures. In Round 2, experts re-appraised the importance of 85 skills and rated the importance of maintenance of competency (i.e., Not at all important to Extremely important). In Round 3, experts rated the level of maintenance of competence (i.e., Observer, Novice, Competent, Proficient) in 46 procedures achieving consensus.

RESULTS

Consensus, defined as > 80% agreement, was established with 46 procedural skills across ten categories: cardiovascular, diagnostic/measurement, gastrointestinal, injections/intravenous, ophthalmic/ENT, respiratory, surgical, trauma, women's health and urogenital procedures. The procedural skills that established consensus with the highest level of agreement included cardiopulmonary resuscitation, airway management, asepsis and surgical scrub, gown and gloving. The importance for medical students to demonstrate maintenance of competency in all procedural skills was assessed on the 6-point Likert scale with a mean of 5.03.

CONCLUSIONS

The findings from the Delphi study provide critical information about procedural skills for the Clinical Practice domain of Australian medical curricula. The inclusion of experts from medical faculty and clinicians enabled opportunities to capture a range of experience independent of medical speciality. These findings demonstrate the importance of maintenance of competency of procedural skills and provides the groundwork for further investigations into monitoring medical students' skills prior to graduation.

A cost-effective IoT learning environment for the training and assessment of surgical technical skills with visual learning analytics

BACKGROUND

Surgeons need to train and certify their technical skills. This is usually done with the intervention of experts who monitor and assess trainees. Nevertheless, this is a time-consuming task that is subject to variations among evaluators. In recent decades, subjectivity has been significantly reduced through 1) the introduction of standard curricula, such as the Fundamentals of Laparoscopic Surgery (FLS) program, which measures students' performance in specific exercises, and 2) rubrics, which are widely accepted in the literature and serve to provide feedback about the overall technical skills of the trainees. Although these two elements reduce subjectivity, they do not, however, eliminate the figure of the expert evaluator, and so the process remains time consuming.

OBJECTIVES

The objective of this work is to automate those parts of the work of the expert evaluator that the technology can measure objectively, using sensors to collect evidence, and visualizations to provide feedback. We designed and developed 1) a cost-effective IoT (Internet of Things) learning environment for the training and assessment of surgical technical skills and 2) visualizations supported by the literature on visual learning analytics (VLA) to provide feedback about the exercises (in real time) and overall performance (at the end of the training) of the trainee.

METHODS

A hybrid approach was followed based on previous research for the design of the sensor based IoT learning environment. Previous studies were used as the basis for getting best practices on the tracking of surgical instruments and on the detection of the force applied to the tissue, with a focus on reducing the costs of data collection. The monitoring of the specific exercises required the design of sensors and collection mechanisms from scratch as there is little existing research on this subject. Moreover, it was necessary to design the overall architecture to collect, process, synchronize and communicate the data coming from the different sensors to provide high-level information relevant to the end user. The information to be presented was already validated by the literature and the focus was on how to visualize this information and the optimal time for its presentation to end users. The visualizations were validated with 18 VLA experts assessing the technical aspects of the visualizations and 4 medical experts assessing their functional aspects.

RESULTS

This IoT learning environment amplifies the evaluation mechanisms already validated by the literature, allowing automatic data collection. First, it uses IoT sensors to automatically correct two of the exercises defined in the FLS (peg transfer and precision cutting), providing real-time visualizations. Second it monitors the movement of the surgical instruments and the force applied to the tissues during the exercise, computing 6 of the high-level indicators used by expert evaluators in their rubrics (efficiency, economy of movement, hand tremor, depth perception, bimanual dexterity, and respect for tissue), providing feedback about the technical skills of the trainee using a radar chart with these six indicators at the end of the training (summative visualizations).

CONCLUSIONS

The proposed IoT learning environment is a promising and cost-effective alternative to help in the training and assessment of surgical technical skills. The system shows the trainees' progress and presents new indicators about the correctness of each specific exercise through real-time visualizations, as well as their general technical skills through summative visualizations, aligned with the 6 more frequent indicators in standardized scales. Early results suggest that although both types of visualizations are useful, it is necessary to reduce the cognitive load of the graphs presented in real time during training. Nevertheless, an additional evaluation is needed to confirm these results.

Scoping Evidence Review on Training and Skills Assessment for Open Emergency Surgery

OBJECTIVE

Scope evidence on technical performance metrics for open emergency surgery. Identify surgical performance metrics and procedures used in trauma training courses.

DESIGN

Structured literature searches of electronic databases were conducted from January 2010 to December 2019 to identify systematic reviews of tools to measure surgical skills employed in vascular or trauma surgery evaluation and training.

SETTING AND PARTICIPANTS

Faculty of Shock Trauma Anesthesiology Research Center, University of Maryland School of Medicine, Uniformed Services University of Health Sciences, Bethesda, Maryland and Implementation Science, King's College, London.

RESULTS

The evidence from 21 systematic reviews including over 54,000 subjects enrolled into over 840 eligible studies, identified that the Objective Structured Assessment of Technical Skill was used for elective surgery not for emergency trauma and vascular control surgery procedures. The Individual Procedure Score (IPS), used to evaluate emergency trauma procedures performed before and after training, distinguished performance of residents from experts and practicing surgeons. IPS predicted surgeons who make critical errors and need remediation interventions. No metrics showed Kirkpatrick's Level 4 evidence of technical skills training benefit to emergency surgery outcomes.

CONCLUSIONS

Expert benchmarks, errors, complication rates, task completion time, task-specific checklists, global rating scales, Objective Structured Assessment of Technical Skills, and IPS were found to identify surgeons, at all levels of seniority, who are in need of remediation of technical skills for open surgical hemorrhage control. Large-scale, multicenter studies are needed to evaluate any benefit of trauma technical skills training on patient outcomes.

Assessing Competence in Central Venous Catheter Placement by Pediatric Critical Care Fellows: A National Survey Study

OBJECTIVES

To describe the current approach to initial training, ongoing skill maintenance, and assessment of competence in central venous catheter placement by pediatric critical care medicine fellows, a subset of trainees in whom this skill is required.

DESIGN

Cross-sectional internet-based survey with deliberate sampling.

SETTING

United States pediatric critical care medicine fellowship programs.

SUBJECTS

Pediatric critical care medicine program directors of Accreditation Council for Graduate Medical Education-accredited fellowship programs.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

A working group of the Education in Pediatric Intensive Care Investigators research collaborative conducted a national study to assess the degree of standardization of training and competence assessment of central venous catheter placement across pediatric critical care medicine fellowship programs. After piloting, the survey was sent to all program directors (n = 67) of Accreditation Council for Graduate Medical Education-accredited pediatric critical care medicine programs between July 2017 and September 2017. The response rate was 85% (57/67). Although 98% of programs provide formalized central venous catheter placement training for first-year fellows, only 42% of programs provide ongoing maintenance training as part of fellowship. Over half (55%) of programs use a global assessment tool and 33% use a checklist-based tool when evaluating fellow central venous catheter placement competence under direct supervision. Only two programs (4%) currently use an assessment tool previously published and validated by the Education in Pediatric Intensive Care group. A majority (82%) of responding program directors believe that a standardized approach to assessment of central venous catheter competency across programs is important.

CONCLUSIONS

Despite national mandates for skill competence by many accrediting bodies, no standardized system currently exists across programs for assessing central venous catheter placement. Most pediatric critical care medicine programs use a global assessment and decisions around the ability of a fellow to place a central venous catheter under indirect supervision are largely based upon subjective assessment of performance. Further investigation is needed to determine if this finding is consistent in other specialties/subspecialties, if utilization of standardized assessment methods can improve program directors' abilities to ensure trainee competence in central venous catheter insertion in the setting of variable training approaches, and if these findings are consistent with other procedures across critical care medicine training programs, adult and pediatric.

Evaluation of Urology Residency Training and Perceived Resident Abilities in the United States

OBJECTIVE

To identify differences and potential deficiencies in urology residency training programs in the United States as they are perceived by residents/recent graduates and program directors.

MATERIALS AND METHODS

A 45-question and 38-question survey was sent to chief residents/recent graduates and program directors, respectively, at all 120 US urology programs regarding prior medical education, urologic training curricula, and perceived surgical proficiency, among other topics.

RESULTS

Survey response rate was 58% and 52% for residents and program directors, respectively. Responses regarding program characteristics (e.g., salary, vacation) and research training were similar between program directors and residents. However, their responses regarding skills training and subspecialty training (e.g., robotics and pediatrics) differed substantially. Program directors reported the availability of advanced skills trainers (robot-88%, laparoscopic-86%), whereas fewer residents felt they were available (robot 54% and laparoscopic 72%). The same discrepancies persisted with questions about subspecialty exposure (e.g., program directors reported 48% renal transplant experience vs. 13% reported by residents). Most residents felt comfortable performing essential urology procedures (e.g., cystoscopy/ureteroscopy, open nephrectomy). In contrast, the majority expressed a lack of confidence in performing unsupervised advanced minimally invasive procedures (e.g., laparoscopic and robotic partial nephrectomy, endopyelotomy). Among the responding residents, 72% pursued fellowship training; nearly two-thirds of these residents chose to enter fellowship in order to overcome perceived training deficiencies.

CONCLUSIONS

Program directors and residents have differing perceptions regarding the education and resources associated with US urology residency training programs. US graduates of urology residency programs express a perceived lack of confidence in several procedures that are commonly encountered in a general urologic practice.

Assessment of progress and competence in tonsillectomy surgery using cumulative sum analysis

OBJECTIVE

To assess the feasibility of using cumulative sum analysis to show trainees' performance curves and highlight concerns in tonsillectomy surgery.

METHODS

In this prospective study, the performance of eight otolaryngology trainees (in their first 6-12 months in the specialty) was compared to that of experts (over 150 tonsillectomies performed) in terms of operative time and post-operative complications. Cumulative sum analysis curves were generated to highlight trainees' performance, and were updated after each performance.

RESULTS

The average operative time was 23 minutes (standard deviation = 11) for experts and 38 minutes (standard deviation = 16) for trainees (p < 0.0001). Cumulative sum analysis charts for trainees' operative time initially rose and then started to plateau after a number of cases (range, 25-30), while that of experts remained low. Cumulative sum analysis charts for complications can be used to monitor performance. In this study, complications were combined (because of low incidence) to allow use of this outcome measure.

CONCLUSION

The flexibility of the cumulative sum analysis makes it adaptable to any outcome. It is a strong adjunct in surgical training to monitor progress and competence. Its sensitivity also allows early detection of poor performance, to instigate intervention.

Assessment of Surgical Skills and Competency

Evaluation of surgical skills and competency are important aspects of the medical education process. Measurable and reproducible methods of assessment with objective feedback are essential components of surgical training. Objective Structured Assessment of Technical Skills (OSATS) is widely used across the medical specialties and otolaryngology-specific tools have been developed and validated for sinus and mastoid surgery. Although assessment of surgical skills can be time-consuming and requires human and financial resources, new evaluation methods and emerging technology may alleviate these barriers while also improving data collection practices.

Cadaver-Based Trauma Procedural Skills Training: Skills Retention 30 Months after Training among Practicing Surgeons in Comparison to Experts or More Recently Trained Residents

BACKGROUND

Long-term retention of trauma procedural core-competency skills and need for re-training after a 1-day cadaver-based course remains unknown. We measured and compared technical skills for trauma core competencies at mean 14 months (38 residents), 30 months (35 practicing surgeons), and 46 months (10 experts) after training to determine if skill degradation occurs with time. Technical performance during extremity vascular exposures and lower-extremity fasciotomy in fresh cadavers measured by validated individual procedure score (IPS) was the primary outcome.

STUDY DESIGN

We performed a prospective study between May 2013 and September 2016.

RESULTS

Practicing surgeons had lower IPS and IPS component scores (p = 0.02 to 0.001) than residents (p < 0.05) and experts (p < 0.002) for vascular procedures. Frequencies of errors were no different among residents and experts. Practicing surgeons made more critical errors (p < 0.05) than experts or residents. Experts had shortest time to proximal vascular control. Fasciotomy procedural errors occurred in all participants. Cluster analysis of anatomy vs procedural steps identified tertiles of performance and wide variance (32.5% practicing surgeons, 26.5% residents vs 13% experts) for vascular procedures. Vascular control duration > 20 minutes (n = 21) and failure to decompress fasciotomy compartments were correlated with incorrect landmarks and skin incisions. Modeling found interval trauma skills experience, not time since training, was associated with lower IPS.

CONCLUSIONS

Practicing surgeons with low trauma skills experience since training had lower IPS and component scores (p = 0.02 to 0.001) and more errors compared with experts and residents (p < 0.05). Surgeons, including experts with low interval experience performing trauma procedures, may benefit from refreshing of correct landmarks and skin incision placement identification.

Simulation in paediatric urology and surgery. Part 1: An overview of educational theory

Surgical training has changed radically in the last few decades. The traditional Halstedian model of time-bound apprenticeship has been replaced with competency-based training. Advanced understanding of mastery learning principles has vastly altered educational methodology in surgical training, in terms of instructional design, delivery of educational content, assessment of learning, and programmatic evaluation. As part of this educational revolution, fundamentals of simulation-based education have been adopted into all levels and aspects of surgical training, requiring an understanding of concepts of fidelity and realism and the impact they have on learning. There are many educational principles and theories that can help clinical teachers understand the way that their trainees learn. In the acquisition of surgical expertise, concepts of mastery learning, deliberate practice, and experiential learning are particularly important. Furthermore, surgical teachers need to understand the principles of effective feedback, which is essential to all forms of skills learning. This article, the first of two papers, presents an overview of relevant learning theory for the busy paediatric surgeon and urologist. Seeking to introduce the concepts underpinning current changes in surgical education and training, providing practical tips to optimise teaching endeavours.

Usefulness of Three-Dimensional Modeling in Surgical Planning, Resident Training, and Patient Education

BACKGROUND

Imaging has a critical impact on surgical decision making and three-dimensional (3D) digital models of patient pathology can now be made commercially. We developed a 3D digital model of a cancer of the head of the pancreas by integrating actual CT data with 3D modeling process. After this process, the virtual pancreatic model was also produced using a high-quality 3D printer.

PATIENTS AND METHODS

A 56-year-old female with pancreatic head adenocarcinoma presented with biliary obstruction and jaundice. The CT scan showed a borderline resectable tumor with a clear involvement of the gastroduodenal artery but doubtful relationships with the hepatic artery. Our team in collaboration with the Immersive Touch team used multiple series from the CT and segmented the relevant anatomy to understand the physical location of the tumor. An STL file was then developed and printed.

RESULTS

Reconstructing and compositing the different series together enhanced the imaging, which allowed clearer observations of the relationship between the mass and the blood vessels, and evidence that the tumor was unresectable. Data files were converted for printing a 100% size rendering model, used for didactic purposes and to discuss with the patient.

CONCLUSIONS

This study showed that (1) reconstructing enhanced traditional imaging by merging and modeling different series together for a 3D view with diverse angles and transparency, allowing the observation of previously unapparent anatomical details; (2) with this new technology surgeons and residents can preobserve their planned surgical intervention, explore the patient-specific anatomy, and sharpen their procedure choices; (3) high-quality 3D printed models are increasingly useful not only in the clinical realm but also for personalized patient education.

Modular emergency general surgery training: A pilot study of a novel programme

Introduction Pan-speciality consensus guidance advocates mandatory emergency general surgery (EGS) training modules for specialist registrars (StRs). This pilot study evaluated the impact of EGS modules aimed at StRs over 1 year. Methods Eleven StRs were allocated a focused 4-week EGS module, in addition to the standard 1:12 on-call duty rota, in a tertiary surgical centre. Primary outcome measures included the number of indicative emergency operations and validated Procedure Based Assessments (PBAs) performed, both during the EGS module and over the training year. Results StRs performed a median of 11 (range 5-15) laparotomies during the EGS module versus 31 (range 9-49) over the whole training year. StRs attended 43.7% of available laparotomies during the module (range 24.1-63.7%). EGS modules provided more than one-third of the total emergency laparotomy experience, and a quarter of the emergency colectomy, appendicectomy and Hartmann's procedure experience. There were no differences in EGS module-related outcomes between junior and senior StRs. Significantly more PBAs related to laparotomy and segmental colectomy were completed during EGS modules than the on-call duty rota, at 32% versus 14% (p<0.001) and 48% versus 22% (p=0.019), respectively. Performance levels were maintained following module completion. Conclusions These findings provide an important baseline when considering future modular EGS training.

The Future of Plastic Surgery Resident Education

The challenge of the current graduate medical education environment requires in plastic surgery acceptance of those contemporary pressures that cannot be substantially modified and address of those that can be successfully met. To do so implies an examination of conference didactics, intraoperative teaching, and a valid assessment of resident performance.

3D Printout Models vs. 3D-Rendered Images: Which Is Better for Preoperative Planning?

INTRODUCTION

Correct interpretation of a patient's anatomy and changes that occurs secondary to a disease process are crucial in the preoperative process to ensure optimal surgical treatment. In this study, we presented 3 different pancreatic cancer cases to surgical residents in the form of 3D-rendered images and 3D-printed models to investigate which modality resulted in the most appropriate preoperative plan.

METHODS

We selected 3 cases that would require significantly different preoperative plans based on key features identifiable in the preoperative computed tomography imaging. 3D volume rendering and 3D printing were performed respectively to create 2 different training ways. A total of 30, year 1 surgical residents were randomly divided into 2 groups. Besides traditional 2D computed tomography images, residents in group A (n = 15) reviewed 3D computer models, whereas in group B, residents (n = 15) reviewed 3D-printed models. Both groups subsequently completed an examination, designed in-house, to assess the appropriateness of their preoperative plan and provide a numerical score of the quality of the surgical plan.

RESULTS

Residents in group B showed significantly higher quality of the surgical plan scores compared with residents in group A (76.4 ± 10.5 vs. 66.5 ± 11.2, p = 0.018). This difference was due in large part to a significant difference in knowledge of key surgical steps (22.1 ± 2.9 vs. 17.4 ± 4.2, p = 0.004) between each group. All participants reported a high level of satisfaction with the exercise.

CONCLUSION

Results from this study support our hypothesis that 3D-printed models improve the quality of surgical trainee's preoperative plans.

Training Veterinary Students to Perform Ovariectomy Using theMOOSE Spay Model with Traditional Method versus the Dowling Spay Retractor

Educators understand the importance of developing safe and effective methods to teach veterinary students basic surgical skills. Ovariectomy (OVE) is a procedure that employs many of the skills agreed to be vital for a newly graduated veterinarian. This study endeavored to compare two methods of teaching OVE on a model based on assessment of procedure time and skill performance scores. Students' opinions regarding their experience are also reported. Students performed the Dowling Spay Retractor (DSR) method more quickly (p<.001) but with performance scores similar to the traditional (T) method depicted in textbooks. Students responded positively when surveyed regarding their experience with the training and the DSR method.

Model for Team Training Using the Advanced Trauma Operative Management Course: Pilot Study Analysis

BACKGROUND

Education and training of surgeons has traditionally focused on the development of individual knowledge, technical skills, and decision making. Team training with the surgeon's operating room staff has not been prioritized in existing educational paradigms, particularly in trauma surgery. We aimed to determine whether a pilot curriculum for surgical technicians and nurses, based on the American College of Surgeons' Advanced Trauma Operative Management (ATOM) course, would improve staff knowledge if conducted in a team-training environment.

METHODS

Between December 2012 and December 2014, 22 surgical technicians and nurses participated in a curriculum complementary to the ATOM course, consisting of 8 individual 8-hour training sessions designed by and conducted at our institution. Didactic and practical sessions included educational content, hands-on instruction, and alternating role play during 5 system-specific injury scenarios in a simulated operating room environment. A pre- and postcourse examination was administered to participants to assess for improvements in team members' didactic knowledge.

RESULTS

Course participants displayed a significant improvement in didactic knowledge after working in a team setting with trauma surgeons during the ATOM course, with a 9-point improvement on the postcourse examination (83%-92%, p = 0.0008). Most participants (90.5%) completing postcourse surveys reported being "highly satisfied" with course content and quality after working in our simulated team-training setting.

CONCLUSIONS

Team training is critical to improving the knowledge base of surgical technicians and nurses in the trauma operative setting. Improved communication, efficiency, appropriate equipment use, and staff awareness are the desired outcomes when shifting the paradigm from individual to surgical team training so that improved patient outcomes, decreased risk, and cost savings can be achieved.

OBJECTIVE

Determine whether a pilot curriculum for surgical technicians and nurses, based on the American College of Surgeons' ATOM course, improves staff knowledge if conducted in a team-training environment.

DESIGN

Surgical technicians and nurses participated in a curriculum complementary to the ATOM course. In all, 8 individual 8-hour training sessions were conducted at our institution and contained both didactic and practical content, as well as alternating role play during 5 system-specific injury scenarios. A pre- and postcourse examination was administered to assess for improvements in didactic knowledge.

SETTING

The course was conducted in a simulated team-training setting at the Legacy Institute for Surgical Education and Innovation (Portland, OR), an American College of Surgeons Accredited Educational Institute.

PARTICIPANTS

In all, 22 surgical technicians and operating room nurses participated in 8 separate ATOM(s) courses and had at least 1 year of surgical scrubbing experience in general surgery with little or no exposure to Level I trauma surgical care. Of these participants, 16 completed the postcourse examination.

RESULTS

Participants displayed a significant improvement in didactic knowledge (83%-92%, p = 0.0008) after the ATOM(s) course. Of the 14 participants who completed postcourse surveys, 90.5% were "highly satisfied" with the course content and quality.

CONCLUSIONS

Team training is critical to improving the knowledge base of surgical technicians and nurses in the trauma operative setting and may contribute to improved patient outcomes, decreased risk, and hospital cost savings.

Development and content validation of performance assessments for endoscopic third ventriculostomy

PURPOSE

This study aims to develop and establish the content validity of multiple expert rating instruments to assess performance in endoscopic third ventriculostomy (ETV), collectively called the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT).

METHODS

The important aspects of ETV were identified through a review of current literature, ETV videos, and discussion with neurosurgeons, fellows, and residents. Three assessment measures were subsequently developed: a procedure-specific checklist (CL), a CL of surgical errors, and a global rating scale (GRS). Neurosurgeons from various countries, all identified as experts in ETV, were then invited to participate in a modified Delphi survey to establish the content validity of these instruments. In each Delphi round, experts rated their agreement including each procedural step, error, and GRS item in the respective instruments on a 5-point Likert scale.

RESULTS

Seventeen experts agreed to participate in the study and completed all Delphi rounds. After item generation, a total of 27 procedural CL items, 26 error CL items, and 9 GRS items were posed to Delphi panelists for rating. An additional 17 procedural CL items, 12 error CL items, and 1 GRS item were added by panelists. After three rounds, strong consensus (>80% agreement) was achieved on 35 procedural CL items, 29 error CL items, and 10 GRS items. Moderate consensus (50-80% agreement) was achieved on an additional 7 procedural CL items and 1 error CL item. The final procedural and error checklist contained 42 and 30 items, respectively (divided into setup, exposure, navigation, ventriculostomy, and closure). The final GRS contained 10 items.

CONCLUSIONS

We have established the content validity of three ETV assessment measures by iterative consensus of an international expert panel. Each measure provides unique assessment information and thus can be used individually or in combination, depending on the characteristics of the learner and the purpose of the assessment. These instruments must now be evaluated in both the simulated and operative settings, to determine their construct validity and reliability. Ultimately, the measures contained in the NEVAT may prove suitable for formative assessment during ETV training and potentially as summative assessment measures during certification.

Test re-test reliability and construct validity of the star-track test of manual dexterity

Objectives. We wished to determine test re-test reliability and construct validity of the star-track test of manual dexterity. Design. Test re-test reliability was examined in a controlled study. Construct validity was tested in a blinded randomized crossover study. Setting. The study was performed at a university hospital in Denmark. Participants. A total of 11 subjects for test re-test and 20 subjects for the construct validity study were included. All subjects were healthy volunteers. Intervention. The test re-test trial had two measurements with 2 days pause in between. The interventions in the construct validity study included baseline measurement, intervention 1: fatigue, intervention 2: stress, and intervention 3: fatigue and stress. There was a 2 day pause between each intervention. Main outcome measure. An integrated measure of completion time and number of errors was used. Results. All participants completed the study (test re-test n = 11; construct validity n = 20). The test re-testshowed a strong Pearson product-moment correlation (r = 0.90, n = 11, P < 0.01) with no sign of learning effect. The 20 subjects in the construct validity trial were randomized to the order of the four interventions, so that all subjects completed each intervention once. A repeated measures ANOVA determined that mean integrated measure differed between interventions (p = 0.002). Post hoc tests using Bonferroni correction revealed that compared with baseline all interventions had significantly higher integrated scores ranging from 47-59% difference in mean. Conclusion. The star track test of manual dexterity had a strong test re-test reliability, and was able to discriminate between a subject's normal manual dexterity and dexterity after exposure to fatigue and/or stress.

Use of a machine learning algorithm to classify expertise: analysis of hand motion patterns during a simulated surgical task

PURPOSE

To test the hypothesis that machine learning algorithms increase the predictive power to classify surgical expertise using surgeons' hand motion patterns.

METHOD

In 2012 at the University of North Carolina at Chapel Hill, 14 surgical attendings and 10 first- and second-year surgical residents each performed two bench model venous anastomoses. During the simulated tasks, the participants wore an inertial measurement unit on the dorsum of their dominant (right) hand to capture their hand motion patterns. The pattern from each bench model task performed was preprocessed into a symbolic time series and labeled as expert (attending) or novice (resident). The labeled hand motion patterns were processed and used to train a Support Vector Machine (SVM) classification algorithm. The trained algorithm was then tested for discriminative/predictive power against unlabeled (blinded) hand motion patterns from tasks not used in the training. The Lempel-Ziv (LZ) complexity metric was also measured from each hand motion pattern, with an optimal threshold calculated to separately classify the patterns.

RESULTS

The LZ metric classified unlabeled (blinded) hand motion patterns into expert and novice groups with an accuracy of 70% (sensitivity 64%, specificity 80%). The SVM algorithm had an accuracy of 83% (sensitivity 86%, specificity 80%).

CONCLUSIONS

The results confirmed the hypothesis. The SVM algorithm increased the predictive power to classify blinded surgical hand motion patterns into expert versus novice groups. With further development, the system used in this study could become a viable tool for low-cost, objective assessment of procedural proficiency in a competency-based curriculum.

Low-fidelity bench models for basic surgical skills training during undergraduate medical education

It is remarkable the reduction in the number of medical students choosing general surgery as a career. In this context, new possibilities in the field of surgical education should be developed to combat this lack of interest. In this study, a program of surgical training based on learning with models of low-fidelity bench is designed as a complementary alternative to the various methodologies in the teaching of basic surgical skills during medical education, and to develop personal interests in career choice.

Tools for the direct observation and assessment of psychomotor skills in medical trainees: a systematic review

CONTEXT

The Accreditation Council for Graduate Medical Education (ACGME) Milestone Project mandates programmes to assess the attainment of training outcomes, including the psychomotor (surgical or procedural) skills of medical trainees. The objectives of this study were to determine which tools exist to directly assess psychomotor skills in medical trainees on live patients and to identify the data indicating their psychometric and edumetric properties.

METHODS

An electronic search was conducted for papers published from January 1948 to May 2011 using the PubMed, Education Resource Information Center (ERIC), Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Web of Science electronic databases and the review of references in article bibliographies. A study was included if it described a tool or instrument designed for the direct observation of psychomotor skills in patient care settings by supervisors. Studies were excluded if they referred to tools that assessed only clinical or non-technical skills, involved non-medical health professionals, or assessed skills performed on a simulator. Overall, 4114 citations were screened, 168 (4.1%) articles were reviewed for eligibility and 51 (1.2%) manuscripts were identified as meeting the study inclusion criteria. Three authors abstracted and reviewed studies using a standardised form for the presence of key psychometric and edumetric elements as per ACGME and American Psychological Association (APA) recommendations, and also assigned an overall grade based on the ACGME Committee on Educational Outcome Assessment grading system.

RESULTS

A total of 30 tools were identified. Construct validity based on associations between scores and training level was identified in 24 tools, internal consistency in 14, test-retest reliability in five and inter-rater reliability in 20. The modification of attitudes, knowledge or skills was reported using five tools. The seven-item Global Rating Scale and the Procedure-Based Assessment received an overall Class 1 ACGME grade and are recommended based on Level A ACGME evidence.

CONCLUSIONS

Numerous tools are available for the assessment of psychomotor skills in medical trainees, but evidence supporting their psychometric and edumetric properties is limited.

The Arthroscopic Surgical Skill Evaluation Tool (ASSET)

BACKGROUND

Surgeries employing arthroscopic techniques are among the most commonly performed in orthopaedic clinical practice; however, valid and reliable methods of assessing the arthroscopic skill of orthopaedic surgeons are lacking.

HYPOTHESIS

The Arthroscopic Surgery Skill Evaluation Tool (ASSET) will demonstrate content validity, concurrent criterion-oriented validity, and reliability when used to assess the technical ability of surgeons performing diagnostic knee arthroscopic surgery on cadaveric specimens.

STUDY DESIGN

Cross-sectional study; Level of evidence, 3.

METHODS

Content validity was determined by a group of 7 experts using the Delphi method. Intra-articular performance of a right and left diagnostic knee arthroscopic procedure was recorded for 28 residents and 2 sports medicine fellowship-trained attending surgeons. Surgeon performance was assessed by 2 blinded raters using the ASSET. Concurrent criterion-oriented validity, interrater reliability, and test-retest reliability were evaluated.

RESULTS

Content validity: The content development group identified 8 arthroscopic skill domains to evaluate using the ASSET. Concurrent criterion-oriented validity: Significant differences in the total ASSET score (P < .05) between novice, intermediate, and advanced experience groups were identified. Interrater reliability: The ASSET scores assigned by each rater were strongly correlated (r = 0.91, P < .01), and the intraclass correlation coefficient between raters for the total ASSET score was 0.90. Test-retest reliability: There was a significant correlation between ASSET scores for both procedures attempted by each surgeon (r = 0.79, P < .01).

CONCLUSION

The ASSET appears to be a useful, valid, and reliable method for assessing surgeon performance of diagnostic knee arthroscopic surgery in cadaveric specimens. Studies are ongoing to determine its generalizability to other procedures as well as to the live operating room and other simulated environments.

The objective assessment of experts' and novices' suturing skills using an image analysis program

PURPOSE

To objectively assess suturing performance using an image analysis program and to provide validity evidence for this assessment method by comparing experts' and novices' performance.

METHOD

In 2009, the authors used an image analysis program to extract objective variables from digital images of suturing end products obtained during a previous study involving third-year medical students (novices) and surgical faculty and residents (experts). Variables included number of stitches, stitch length, total bite size, travel, stitch orientation, total bite-size-to-travel ratio, and symmetry across the incision ratio. The authors compared all variables between groups to detect significant differences and two variables (total bite-size-to-travel ratio and symmetry across the incision ratio) to ideal values.

RESULTS

Five experts and 15 novices participated. Experts' and novices' performances differed significantly (P < .05) with large effect sizes attributable to experience (Cohen d > 0.8) for total bite size (P = .009, d = 1.5), travel (P = .045, d = 1.1), total bite-size-to-travel ratio (P < .0001, d = 2.6), stitch orientation (P = .014,d = 1.4), and symmetry across the incision ratio (P = .022, d = 1.3).

CONCLUSIONS

The authors found that a simple computer algorithm can extract variables from digital images of a running suture and rapidly provide quantitative summative assessment feedback. The significant differences found between groups confirm that this system can discriminate between skill levels. This image analysis program represents a viable training tool for objectively assessing trainees' suturing, a foundational skill for many medical specialties.

Teaching and assessing technical proficiency in surgical subspecialty fellowships

OBJECTIVES

To determine how programs are teaching and assessing procedural skills, and their perceived success.

DESIGN

Cross-sectional survey.

SETTING

Accreditation Council for Graduate Medical Education (ACGME) approved training programs in pediatric urology and colorectal surgery.

PARTICIPANTS

Program directors and recent graduates (2007-2009).

RESULTS

Thirty-nine program directors (60%), and 57 graduates (64%) responded; 89.5% of graduates and 94.9% of program directors felt training occurred successfully for the procedures that trainees were performing in their present practice. Nearly 90% of trainees and all program directors reported that there was no formal assessment of procedural competency at the beginning of training, although 66.7% of program directors reported that trainees were assessed "informally." Both program directors and trainees reported dialogue with faculty was the most frequent method used in preparing for operative procedures. Other methods (textbook/atlas, journals, web-based programs, videos) were used less frequently. Program directors with shorter tenure were more likely to use web-based and video methods; younger trainees were less likely to use textbooks/atlases. Faculty feedback on clinical decision-making and postprocedural review were perceived by both program directors and trainees as the most effective assessment methods for improving performance; however, trainees were more likely than program directors to report that postprocedure reviews were not included in their training (15.8% vs 9%, p = 0.045). Patient outcomes, written feedback from peers, and self-assessment were included in most programs, but valued less. Simulation was used in only about half the programs and was valued more highly by trainees than program directors (p = 0.011).

CONCLUSIONS

Training in procedural proficiency was viewed as successful by both program directors and graduates. Dialogue with, assessment by, and feedback from faculty were frequently used and most valued; stressing the importance of the facilitator role of faculty in the education of the trainee. These findings provide guidance for the development of newer methods of teaching and assessment.

Computer-aided feedback of surgical knot tying using optical tracking

BACKGROUND

Quantifying the information content of hand motion during surgical knot tying using information theory based entropy measures enables the comparison of different groups: novice and expert. We hypothesized that complexity would differ between the 2 groups and predicted based on motor learning models that complexity/information would reduce with increased expertise.

METHODS

Six degrees of freedom hand-motion data during surgical knot tying were acquired using an infrared optical hand tracking device. Multiple data samples were obtained from 2 groups: novice (third-year medical students) and expert (attending surgeons). After preprocessing each knot tying data sample into a binary symbolic time series, 3 nonlinear complexity measures were calculated: Lempel Ziv complexity, Shannon entropy, and Renyi entropy. The Shannon and Renyi entropies were calculated using a word length of 6. A Student t test was used to test whether the 2 groups were from the same population when using these entropy measures, applying a p value of 0.05 to reject the null hypothesis.

RESULTS

The expert surgeons were found to have less complex patterns of motion compared with the novice group. This finding was statistically significant using Lempel Ziv complexity (p = 0.004), Shannon entropy (p = 0.006), and Renyi entropy with q = 2 (p = 0.006). Using Renyi entropy with q = 0.5, the 2 groups were not significantly different (p = 0.26).

CONCLUSIONS

The ability to separate novice from expert populations during surgical knot tying using information theory entropy measures could form the basis of a low-cost educational tool to provide feedback and to assess skill acquisition using low-fidelity bench models.

The Concept of Surgical Assessment: Part 3 – Characteristics of the Ideal Assessment Tool

Adequate documentation of technical competence in surgery has come under increased scrutiny in recent years. The stipulation in the European Working Time Regulations of a 48-hour working week, an emphasis on operating room efficiency and concerns about medical errors and patient safety may limit the ability of a teaching faculty to provide graded responsibility with adequate skill acquisition in a surgical training programme.