An objective scoring system for laparoscopic cholecystectomy

Automated analysis of intraoperative phase in laparoscopic cholecystectomy: A comparison of one attending surgeon and their residents

OBJECTIVE

This study compares the intraoperative phase times in laparoscopic cholecystectomy performed by an attending surgeon and supervised residents over 10-years to assess operative times as a marker of performance and any impact of case severity on times.

DESIGN

Laparoscopic cholecystectomy videos were uploaded to Touch Surgery™ Enterprise, a combined software and hardware solution for securely recording, storing, and analysing surgical videos, which provide analytics of intraoperative phase times. Case severity and visualisation of the critical view of safety (CVS) were manually assessed using modified 10-point intraoperative gallbladder scoring system (mG10) and CVS scores, respectively. Attending and residents' times were compared unmatched and matched by mG10.

SETTING

Secondary analysis of anonymized laparoscopic cholecystectomy video, recorded as standard of care.

PARTICIPANTS

Adult patients who underwent elective laparoscopic cholecystectomy a single UK hospital. Cases were performed by one attending and their residents.

RESULTS

159 (attending=96, resident=63) laparoscopic cholecystectomy videos and intraoperative phase times were reviewed on Touch Surgery™ Enterprise and analyzed. Attending cases were more challenging (p=0.037). Residents achieved higher CVS scores (p=0.034) and showed longer dissection of hepatocystic triangle (HCT) times (p=0.012) in more challenging cases. Residents' total operative time (p=0.001) and dissection of HCT (p=0.002) times exceeded the attending's in low-severity matched cases (mG10=1). Residents' total operative times (p<0.001), port insertion/gallbladder exposure (p=0.032), and dissection of HCT (p<0.001) exceeded the attending's in matched cases (mG10=2). Residents' total operative (p<0.001), dissection of HCT (p<0.001), and gallbladder dissection (p=0.010) times exceeded the attendings in unmatched cases.

CONCLUSIONS

Residents' total operative and dissection of HCT times significantly exceeded the attending's unmatched cases and low-severity matched cases which could suggest training need, however, also reflects an expected assessment of competence, and validates time as a marker of performance.

Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review

BACKGROUND

Quality of surgery has substantial impact on both short- and long-term clinical outcomes. This stresses the need for objective surgical quality assessment (SQA) for education, clinical practice and research purposes. The aim of this systematic review was to provide a comprehensive overview of all video-based objective SQA tools in laparoscopic procedures and their validity to objectively assess surgical performance.

METHODS

PubMed, Embase.com and Web of Science were systematically searched by two reviewers to identify all studies focusing on video-based SQA tools of technical skills in laparoscopic surgery performed in a clinical setting. Evidence on validity was evaluated using a modified validation scoring system.

RESULTS

Fifty-five studies with a total of 41 video-based SQA tools were identified. These tools were used in 9 different fields of laparoscopic surgery and were divided into 4 categories: the global assessment scale (GAS), the error-based assessment scale (EBAS), the procedure-specific assessment tool (PSAT) and artificial intelligence (AI). The number of studies focusing on these four categories were 21, 6, 31 and 3, respectively. Twelve studies validated the SQA tool with clinical outcomes. In 11 of those studies, a positive association between surgical quality and clinical outcomes was found.

CONCLUSION

This systematic review included a total of 41 unique video-based SQA tools to assess surgical technical skills in various domains of laparoscopic surgery. This study suggests that validated SQA tools enable objective assessment of surgical performance with relevance for clinical outcomes, which can be used for training, research and quality improvement programs.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

OBJECTIVE

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

SUMMARY OF BACKGROUND DATA

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

METHODS

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

RESULTS

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

CONCLUSIONS

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

Supporting laparoscopic general surgery training with digital technology: The United Kingdom and Ireland paradigm

Surgical training in the UK and Ireland has faced challenges following the implementation of the European Working Time Directive and postgraduate training reform. The health services are undergoing a digital transformation; digital technology is remodelling the delivery of surgical care and surgical training. This review aims to critically evaluate key issues in laparoscopic general surgical training and the digital technology such as virtual and augmented reality, telementoring and automated workflow analysis and surgical skills assessment. We include pre-clinical, proof of concept research and commercial systems that are being developed to provide solutions. Digital surgical technology is evolving through interdisciplinary collaboration to provide widespread access to high-quality laparoscopic general surgery training and assessment. In the future this could lead to integrated, context-aware systems that support surgical teams in providing safer surgical care.

Implementation of anisotropic soft pads in a surgical gripper for secure and gentle grip on vulnerable tissues

Current surgical grippers rely on friction grip, where normal loads (i.e. pinch forces) are translated into friction forces. Operating errors with surgical grippers are often force-related, including tissue slipping out of the gripper because of too low pinch forces and tissue damaging due to too high pinch forces. Here, we prototyped a modular surgical gripper with elastomeric soft pads reinforced in the shear direction with a carbon-fiber fabric. The elastomeric component provides low normal stiffness to maximize contact formation without the need of applying high normal loads (i.e. pinch forces), whereas the carbon-fiber fabric offers high shear stiffness to preserve the formed contact under the lateral loads (i.e. shear forces) that occur during tissue lifting. Additionally, we patterned the pads with a sub-surface micropattern, to further reduce the normal stiffness and increase shear stiffness. The body of the prototype gripper, including shaft, joints, and gripper tips, was fabricated in a single step using 3D printing, followed by manual attachment of the soft pads to the gripper. The gripping performance of the newly developed soft gripper on soft tissues was experimentally compared to reference grippers equipped with metal patterned pads. The soft-pad gripper generated similar gripping forces but significantly lower pinch forces than metal-pad grippers. We conclude that grippers with anisotropic-stiffness pads are promising for secure and gentle tissue grip.

The initiation, standardization and proficiency (ISP) phases of the learning curve for minimally invasive liver resection: comparison of a fellowship-trained surgeon with the pioneers and early adopters

BACKGROUND

Using the Ideal Development Exploration Assessment and Long-term study (IDEAL) paradigm, Halls et al. created risk-adjusted cumulative sum (RA-CUSUM) curves concluding that Pioneers (P) and Early Adopters (EA) of minimally invasive (MI) liver resection obtained similar results after fewer cases. In this study, we applied this framework to a MI Hepatic-Pancreatic and Biliary fellowship-trained surgeon (FT) in order to assess where along the curves this generation fell.

METHODS

The term FT was used to designate surgeons without previous independent operative experience who went from surgical residency directly into fellowship. Three phases of the learning curve were defined using published data on EAs and Ps of MI Hepatectomy, including phase 1 (initiation) (i.e., the first 17 or 50), phase 2 (standardization) (i.e., cases 18-46 or 1-50) and phase 3 (proficiency) (i.e., cases after 46, 50 or 135). Data analysis was performed using the Social Science Statistics software ( www.socscistatistics.com ). Statistical significance was defined as p < .05.

RESULTS

From November 2007 until April 2018, 95 MI hepatectomies were performed by a FT. During phase 1, the FT approached larger tumors than the EA group (p = 0.002), that were more often malignant (94.1%) when compared to the P group (52.5%) (p < 0.001). During phase 2, the FT operated on larger tumors and more malignancies (93.1%) when compared to the Ps (p = 0.004 and p = 0.017, respectively). However, there was no difference when compared to the EA. In the phase 3, the EAs tended to perform more major hepatectomies (58.7) when compared to either the FT (30.6%) (p = 0.002) or the P's cases 51-135 and after 135 (35.3% and 44.3%, respectively) (both p values < 0.001). When compared to the Ps cases from 51-135, the FT operated on more malignancies (p = 0.012), but this was no longer the case after 135 cases by the Ps (p = 0.164). There were no statistically significant differences when conversions; major complications or 30- and 90-day mortality were compared among these 3 groups.

DISCUSSION

Using the IDEAL framework and RA-CUSUM curves, a FT surgeon was found to have curves similar to EAs despite having no previous independent experience operating on the liver. As in our study, FTs may tend to approach larger and more malignant tumors and do more concomitant procedures in patients with higher ASA classifications than either of their predecessors, without statistically significant increases in major morbidity or mortality.

CONCLUSION

It is possible that the ISP (i.e., initiation, standardization, proficiency) model could apply to other innovative surgical procedures, creating different learning curves depending on where along the IDEAL paradigm surgeons fall.

Development and implementation of an assessment tool to evaluate technical skills in the insertion of implantable venous access devices, a Prospective Cohort Study

OBJECTIVE

Based on the Competency Assessment Tool, herein we developed an assessment instrument suitable to evaluate the implantation of central intravenous devices.

BACKGROUND

Surgical assessment is based mainly on the subjective impressions of the teacher. Based on the "Competency Assessment Tool" (CAT) developed for the evaluation of technical surgical skills in minimally invasive colorectal resection, we designed an assessment tool suitable to evaluate the implantation of central venous access devices performed by junior surgical trainees.

METHODS

Four major assessments during the different steps of the intervention were used in this evaluation. Each of these tasks was divided into four sub-domains according to surgical skill. In addition to the CAT score, the apprentices' skills were evaluated using a visual assessment that was quantified using an analogue scale (value from 1 to 10). The candidates were classified into junior and senior trainees depending on the number of procedures they had already performed and on their surgical experience.

RESULTS

71 procedures were evaluated during the study period. Seven senior trainees conducted 43 procedures and five junior trainees performed 28 interventions. The senior trainees had significantly higher CAT scores than junior candidates, and the scores fluctuated according to surgical experience, usually reaching their peak after 10 procedures.

CONCLUSIONS

The CAT model is well suited for the assessment of surgical trainees during central venous access device implantation. It enables a close assessment of the learning process and the technical skills of trainees, which helps them improving in a safe, standardized manner.

Development of a program for teaching practical skills in visceral and digestive surgery by simulation

Up until 2013 in France, practical training for DES/DESC (advanced level) residents in visceral and digestive surgery was not standardized. Since 2017, the third cycle of medical studies has been restructured around three major thematic axes: academic knowledge, and technical and non-technical skills. The curriculum now includes a practical training program by means of simulation outside the operating theater, and it is structured, uniformized and standardized nationwide. Development of this training program is derived from the deliberations of a national consensus panel working under the umbrella of the French college of visceral and digestive surgery, program presenting a training guide to all future surgeons in the specialty. Four consensus conference sessions bringing together an eight-member commission have led to the drafting of a "Resident's manual for practical teaching in visceral and digestive surgery". As a reference document, the manual details in 272 pages the objectives (phase I), the learning resources for each skill (phase II) and, lastly, the means of evaluation for the cornerstone phases as well as the in-depth phases of an advanced degree (DES) in visceral and digestive surgery. As a complement to the manual, we have conducted a review of the structuring and implementation of the program as of November 2017 on a nationwide scale; the conclusions of the review are detailed at the end of this article.

Performance assessment - The knowledge, skills and attitudes of surgical performance

Pediatric surgical education has traditionally focused on the hard cognitive and psychomotor (technical) skills. While more and more attention is being paid to softer skills such as communication, collaboration, leadership, health advocacy, professionalism and scholarship, the bulk of curricula remain focused on the hard skills. An intricate part of education is the assessment of performance. This article reviews the current literature on the assessment of cognitive and psychomotor skills, focusing particularly on the assessment of technical skills in the realm of simulation.

Validity Evidence for the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT)

BACKGROUND

Growing demand for transparent and standardized methods for evaluating surgical competence prompted the construction of the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT).

OBJECTIVE

To provide validity evidence of the NEVAT by reporting on the tool's internal structure and its relationship with surgical expertise during simulation-based training.

METHODS

The NEVAT was used to assess performance of trainees and faculty at an international neuroendoscopy workshop. All participants performed an endoscopic third ventriculostomy (ETV) on a synthetic simulator. Participants were simultaneously scored by 2 raters using the NEVAT procedural checklist and global rating scale (GRS). Evidence of internal structure was collected by calculating interrater reliability and internal consistency of raters' scores. Evidence of relationships with other variables was collected by comparing the ETV performance of experts, experienced trainees, and novices using Jonckheere's test (evidence of construct validity).

RESULTS

Thirteen experts, 11 experienced trainees, and 10 novices participated. The interrater reliability by the intraclass correlation coefficient for the checklist and GRS was 0.82 and 0.94, respectively. Internal consistency (Cronbach's α) for the checklist and the GRS was 0.74 and 0.97, respectively. Median scores with interquartile range on the checklist and GRS for novices, experienced trainees, and experts were 0.69 (0.58-0.86), 0.85 (0.63-0.89), and 0.85 (0.81-0.91) and 3.1 (2.5-3.8), 3.7 (2.2-4.3) and 4.6 (4.4-4.9), respectively. Jonckheere's test showed that the median checklist and GRS score increased with performer expertise ( P = .04 and .002, respectively).

CONCLUSION

This study provides validity evidence for the NEVAT to support its use as a standardized method of evaluating neuroendoscopic competence during simulation-based training.

A Randomized Controlled Trial of Skills Transfer: From Touch Surgery to Laparoscopic Cholecystectomy

BACKGROUND

Surgical training has traditionally involved teaching trainees in the operating room. However, intraoperative training is time-intensive and exposes patients to greater risks. Touch Surgery (TS) is an application that uses animation to provide simulation training via cognitive task analysis as an adjunct to intraoperative training.

METHODS

Forty students were recruited and randomly allocated to either a control or intervention group. Each group received the same preparation before intervention, including a 10-min introduction to laparoscopic equipment and a 15-min educational tutorial on laparoscopic cholecystectomies. The participants then received training via either TS (intervention) or written information (control). Their performance was compared using a validated scoring tool on a porcine laparoscopic cholecystectomy model. Significance was defined as P < 0.050.

RESULTS

In total, n = 22 and n = 18 participants were randomly assigned to intervention and control groups, respectively. There was no significant difference between age (P = 0.320), year of medical school (P = 0.322), handedness (P = 1.000), or gender (P = 0.360) of the groups. The overall mean performance score was higher for intervention (mean ± SD = 41.9 ± 22.5) than control (mean ± SD = 24.7 ± 19.6; P = 0.016). There was no significant difference between scores for each intraoperative segment between the intervention and control group (P > 0.050).

CONCLUSIONS

This study demonstrates that TS is effective for providing cognitive training in laparoscopic cholecystectomies to medical students. It is likely that this effect will be seen across modules and other platforms that use cognitive task analysis alongside high-fidelity animation. Further work is necessary to extend this to other surgical procedures for evaluating its longitudinal effectiveness.

Development of a technical checklist for the assessment of suturing in robotic surgery

BACKGROUND

With the increased use of simulation for surgical training, there is a need for objective forms of assessment to evaluate trainees. The Global Evaluative Assessment of Robotic Skills (GEARS) is widely used for assessing skills in robotic surgery, but there are no recognised checklist scoring systems. This study aimed to develop a checklist for suturing in robotic surgery.

METHODS

A suturing checklist for needle driving and knot tying was constructed following evaluation of participants performing urethrovesical anastomoses. Key procedural steps were identified from expert videos, while assessing novice videos allowed identification of common technical errors. 22 novice and 13 expert videos were marked on needle driving, while 18 novices and 10 experts were assessed on knot tying. Validation of the finalised checklist was performed with the assessment of 39 separate novices by an expert surgeon and compared to GEARS scoring.

RESULTS

The internal consistency of the preliminary checklist was high (Cronbach's alpha = 0.870 for needle driving items; 0.736 for knot tying items), and after removal of poorly correlating items, the final checklist contained 23 steps. Both the needle driving and knot tying categories discriminated between novices and experts, p < 0.005. While the GEARS score demonstrated construct validity for needle driving, it could not significantly differentiate between novices and experts for knot tying, p = 0.286. The needle driving category significantly correlated with the corresponding GEARS scores (r_s = 0.613, p < 0.005), but the correlation for knot tying was insignificant (r_s = 0.296, p = 0.127). The pilot data indicates the checklist significantly correlated with the GEARS score (p < 0.005).

CONCLUSION

This study reports the development of a valid assessment tool for suturing in robotic surgery. Given that checklists are simple to use, there is significant scope for this checklist to be used in surgical training.

The Dimensionless Squared Jerk: An Objective Parameter That Improves Assessment of Hand Motion Analysis during Simulated Shoulder Arthroscopy

PURPOSE

Attempts to quantify hand movements of surgeons during arthroscopic surgery faced limited progress beyond motion analysis of hands and/or instruments. Surrogate markers such as procedure time have been used. The dimensionless squared jerk (DSJ) is a measure of deliberate hand movements. This study tests the ability of DSJ to differentiate novice and expert surgeons (construct validity) whilst performing simulated arthroscopic shoulder surgical tasks.

METHODS

Six residents (novice group) and six consultants (expert group) participated in this study. Participants performed three validated tasks sequentially under the same experimental setup (one performance). Each participant had ten performances assessed. Hand movements were recorded with optical tracking system. The DSJ, time taken, total path length, multiple measures of acceleration, and number of movements were recorded.

RESULTS

There were significant differences between novices and experts when assessed using time, number of movements with average and minimal acceleration threshold, and DSJ. No significant differences were observed in maximum acceleration, total path length, and number of movements with 10m/s2 acceleration threshold.

CONCLUSION

DSJ is an objective parameter that can differentiate novice and expert surgeons' simulated arthroscopic performances. We propose DSJ as an adjunct to more conventional parameters for arthroscopic surgery skills assessment.

Structured assessment of laparoscopic camera navigation skills: the SALAS score

BACKGROUND

Tools are needed to assess laparoscopic camera navigation (LCN) in the operating room. Here, we aimed to develop an objective rating scale for LCN.

STUDY DESIGN

We defined the following key aspects of LCN: operational field centering, correct angle of the horizon, correct instrument visualization, verbal commands from the operating surgeon, and manual corrections from the operating surgeon. We then developed a score based on intraoperative error evaluation from intraoperative recordings of 80 procedures. Finally, the newly developed score was validated by four different raters using video-based analysis of 20 elective laparoscopic cholecystectomies.

RESULTS

We developed and validated a tool for the structured assessment of laparoscopic assistant skills (SALAS). This score showed good internal consistency, with a Cronbach's alpha of > 0.7. Intraclass correlation revealed a low interrater variability (ICC 0.866) for the total score. Comparison of experienced and inexperienced camera assistants revealed significantly better SALAS scores for experienced assistants (p < 0.05).

CONCLUSION

Our present results show that SALAS score is valid, reliable, and practicable. This score can be used for future investigations of camera navigation efficiency and training.

Assessment of Technical Skills Competence in the Operating Room: A Systematic and Scoping Review

PURPOSE

While academic accreditation bodies continue to promote competency-based medical education (CBME), the feasibility of conducting regular CBME assessments remains challenging. The purpose of this study was to identify evidence pertaining to the practical application of assessments that aim to measure technical competence for surgical trainees in a nonsimulated, operative setting.

METHOD

In August 2016, the authors systematically searched Medline, Embase, and the Cochrane Database of Systematic Reviews for English-language, peer-reviewed articles published in or after 1996. The title, abstract, and full text of identified articles were screened. Data regarding study characteristics, psychometric and measurement properties, implementation of assessment, competency definitions, and faculty training were extracted. The findings from the systematic review were supplemented by a scoping review to identify key strategies related to faculty uptake and implementation of CBME assessments.

RESULTS

A total of 32 studies were included. The majority of studies reported reasonable scores of interrater reliability and internal consistency. Seven articles identified minimum scores required to establish competence. Twenty-five articles mentioned faculty training. Many of the faculty training interventions focused on timely completion of assessments or scale calibration.

CONCLUSIONS

There are a number of diverse tools used to assess competence for intraoperative technical skills and a lack of consensus regarding the definition of technical competence within and across surgical specialties. Further work is required to identify when and how often trainees should be assessed and to identify strategies to train faculty to ensure timely and accurate assessment.

A New Needle Holder Design to Facilitate Intracorporeal Knot Tying

PURPOSE

Intracorporeal knot tying in laparoscopic surgery continues to be a problem especially for beginners and inexperienced surgeons. A wide-angle needle holder was designed to make the knot maneuver easier while also ensuring that the knot does not come out of the needle holder. In this study, it was planned to compare the wide-angle needle holder with the classic needle holder in regard to knot tying time.

MATERIAL AND METHOD

A total of 11 male volunteers were randomly selected from freshmen students of the faculty of medicine, who had no experience of surgery or laparoscopic surgery. After the required training and practice, candidates were asked to tie 3 knots each in the training box using a classic needle holder and a wide-angle needle holder. Their knot tying times were recorded.

RESULTS

Although the students had no experience, it was observed that they tied knots more easily and more comfortably using the wide-angle needle holder. It was found that the knot tying times with the wide-angle needle holder were quite short compared with the classic needle holder in all candidates. This difference was also statistically significant ( P = .01).

CONCLUSION

We believe and claim that the use of a wide-angle needle holder during knot tying in laparoscopic surgery can facilitate knot tying and shorten the duration of the knotting, especially for inexperienced surgeons.

What Are the Principles That Guide Behaviors in the Operating Room?: Creating a Framework to Define and Measure Performance

OBJECTIVE

To identify the core principles that guide expert intraoperative behaviors and to use these principles to develop a universal framework that defines intraoperative performance.

BACKGROUND

Surgical outcomes are associated with intraoperative cognitive skills. Yet, our understanding of factors that control intraoperative judgment and decision-making are limited. As a result, current methods for training and measuring performance are somewhat subjective-more task rather than procedure-oriented-and usually not standardized. They thus provide minimal insight into complex cognitive processes that are fundamental to patient safety.

METHODS

Cognitive task analyses for 6 diverse surgical procedures were performed using semistructured interviews and field observations to describe the thoughts, behaviors, and actions that characterize and guide expert performance. Verbal data were transcribed, supplemented with content from published literature, coded, thematically analyzed using grounded-theory by 4 independent reviewers, and synthesized into a list of items.

RESULTS

A conceptual framework was developed based on 42 semistructured interviews lasting 45 to 120 minutes, 5 expert panels and 51 field observations involving 35 experts, and 135 sources from the literature. Five domains of intraoperative performance were identified: psychomotor skills, declarative knowledge, advanced cognitive skills, interpersonal skills, and personal resourcefulness. Within the advanced cognitive skills domain, 21 themes were perceived to guide the behaviors of surgeons: 18 for surgical planning and error prevention, and 3 for error/injury recognition, rescue, and recovery. The application of these thought patterns was highly case-specific and variable amongst subspecialties, environments, and individuals.

CONCLUSIONS

This study provides a comprehensive definition of intraoperative expertise, with greater insight into the complex cognitive processes that seem to underlie optimal performance. This framework provides trainees and other nonexperts with the necessary information to use in deliberate practice and the creation of effective thought habits that characterize expert performance. It may help to identify gaps in performance, and to isolate root causes of surgical errors with the ultimate goal of improving patient safety.

The learning curve of laparoscopic holecystectomy in general surgery resident training: old age of the patient may be a risk factor?

A well-designed learning curve is essential for the acquisition of laparoscopic skills: but, are there risk factors that can derail the surgical method? From a review of the current literature on the learning curve in laparoscopic surgery, we identified learning curve components in video laparoscopic cholecystectomy; we suggest a learning curve model that can be applied to assess the progress of general surgical residents as they learn and master the stages of video laparoscopic cholecystectomy regardless of type of patient.

Electronic databases were interrogated to better define the terms “surgeon”, “specialized surgeon”, and “specialist surgeon”; we surveyed the literature on surgical residency programs outside Italy to identify learning curve components, influential factors, the importance of tutoring, and the role of reference centers in residency education in surgery. From the definition of acceptable error, self-efficacy, and error classification, we devised a learning curve model that may be applied to training surgical residents in video laparoscopic cholecystectomy.

Based on the criteria culled from the literature, the three surgeon categories (general, specialized, and specialist) are distinguished by years of experience, case volume, and error rate; the patients were distinguished for years and characteristics. The training model was constructed as a series of key learning steps in video laparoscopic cholecystectomy. Potential errors were identified and the difficulty of each step was graded using operation-specific characteristics. On completion of each procedure, error checklist scores on procedure-specific performance are tallied to track the learning curve and obtain performance indices of measurement that chart the trainee’s progress.

Conclusions. The concept of the learning curve in general surgery is disputed. The use of learning steps may enable the resident surgical trainee to acquire video laparoscopic cholecystectomy skills proportional to the instructor’s ability, the trainee’s own skills, and the safety of the surgical environment. There were no patient characteristics that can derail the methods. With this training scheme, resident trainees may be provided the opportunity to develop their intrinsic capabilities without the loss of basic technical skills.

A Team Performance Data Collection and Analysis System

At present there exists no commercially available product capable of capturing and broadcasting multimedia audiovisual data from teams performing high-risk work. We have developed such a recording and analysis system for the purpose of studying team behavior, which we currently use to observe and record up to 8 people working as a co-located team in a hospital operating room. The system has four data collection computers, each recording one video stream and up to 2 audio feeds. A separate software package is used to synchronize and view the audio/video streams together on a fifth computer. This software has several annotation and scoring features which can be used either for data analysis or for team debriefing purposes. Although currently being used for patient safety research in the operating room, this system could also be adapted to collect and analyze team behavior in other domains, even for participants who are distributed.

Error tolerance: an evaluation of residents' repeated motor coordination errors

BACKGROUND

The study investigates the relationship between motor coordination errors and total errors using a human factors framework. We hypothesize motor coordination errors will correlate with total errors and provide validity evidence for error tolerance as a performance metric.

METHODS

Residents' laparoscopic skills were evaluated during a simulated laparoscopic ventral hernia repair for motor coordination errors when grasping for intra-abdominal mesh or suture. Tolerance was defined as repeated, failed attempts to correct an error and the time required to recover.

RESULTS

Residents (N = 20) committed an average of 15.45 (standard deviation [SD] = 4.61) errors and 1.70 (SD = 2.25) motor coordination errors during mesh placement. Total errors correlated with motor coordination errors (r[18] = .572, P = .008). On average, residents required 5.09 recovery attempts for 1 motor coordination error (SD = 3.15). Recovery approaches correlated to total error load (r[13] = .592, P = .02).

CONCLUSIONS

Residents' motor coordination errors and recovery approaches predict total error load. Error tolerance proved to be a valid assessment metric relating to overall performance.

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

Background

There is no widely used method to evaluate procedure-specific laparoscopic skills. The first aim of this study was to develop a procedure-based assessment method. The second aim was to compare its validity, reliability and feasibility with currently available global rating scales (GRSs).

Methods

An independence-scaled procedural assessment was created by linking the procedural key steps of the laparoscopic cholecystectomy to an independence scale. Subtitled and blinded videos of a novice, an intermediate and an almost competent trainee, were evaluated with GRSs (OSATS and GOALS) and the independence-scaled procedural assessment by seven surgeons, three senior trainees and six scrub nurses. Participants received a short introduction to the GRSs and independence-scaled procedural assessment before assessment. The validity was estimated with the Friedman and Wilcoxon test and the reliability with the intra-class correlation coefficient (ICC). A questionnaire was used to evaluate user opinion.

Results

Independence-scaled procedural assessment and GRS scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, independence-scaled procedural assessment p < 0.001). The ICCs of the OSATS, GOALS and independence-scaled procedural assessment were 0.78, 0.74 and 0.84, respectively, among surgeons. The ICCs increased when the ratings of scrub nurses were added to those of the surgeons. The independence-scaled procedural assessment was not considered more of an administrative burden than the GRSs (p = 0.692).

Discussion/conclusion

A procedural assessment created by combining procedural key steps to an independence scale is a valid, reliable and acceptable assessment instrument in surgery. In contrast to the GRSs, the reliability of the independence-scaled procedural assessment exceeded the threshold of 0.8, indicating that it can also be used for summative assessment. It furthermore seems that scrub nurses can assess the operative competence of surgical trainees.

Design and Validation of an Open-Source, Partial Task Trainer for Endonasal Neuro-Endoscopic Skills Development: Indian Experience

BACKGROUND

Box trainers are ideal simulators, given they are inexpensive, accessible, and use appropriate fidelity.

OBJECTIVE

The development and validation of an open-source, partial task simulator that teaches the fundamental skills necessary for endonasal skull-base neuro-endoscopic surgery.

METHODS

We defined the Neuro-Endo-Trainer (NET) SkullBase-Task-GraspPickPlace with an activity area by analyzing the computed tomography scans of 15 adult patients with sellar suprasellar parasellar tumors. Four groups of participants (Group E, n = 4: expert neuroendoscopists; Group N, n =19: novice neurosurgeons; Group R, n = 11: neurosurgery residents with multiple iterations; and Group T, n = 27: neurosurgery residents with single iteration) performed grasp, pick, and place tasks using NET and were graded on task completion time and skills assessment scale score.

RESULTS

Group E had lower task completion times and greater skills assessment scale scores than both Group N and R (P ≤ 0.03, 0.001). The performance of Groups N and R was found to be equivalent; in self-assessing neuro-endoscopic skill, the participants in these groups were found to have equally low pretraining scores (4/10) with significant improvement shown after NET simulation (6, 7 respectively). Angled scopes resulted in decreased scores with tilted plates compared with straight plates (30° P ≤ 0.04, 45° P ≤ 0.001). With tilted plates, decreased scores were observed when we compared the 0° with 45° endoscope (right, P ≤ 0.008; left, P ≤ 0.002).

CONCLUSIONS

The NET, a face and construct valid open-source partial task neuroendoscopic trainer, was designed. Presimulation novice neurosurgeons and neurosurgical residents were described as having insufficient skills and preparation to practice neuro-endoscopy. Plate tilt and endoscope angle were shown to be important factors in participant performance. The NET was found to be a useful partial-task trainer for skill building in neuro-endoscopy.

Development and content validation of performance assessments for endoscopic third ventriculostomy

PURPOSE

This study aims to develop and establish the content validity of multiple expert rating instruments to assess performance in endoscopic third ventriculostomy (ETV), collectively called the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT).

METHODS

The important aspects of ETV were identified through a review of current literature, ETV videos, and discussion with neurosurgeons, fellows, and residents. Three assessment measures were subsequently developed: a procedure-specific checklist (CL), a CL of surgical errors, and a global rating scale (GRS). Neurosurgeons from various countries, all identified as experts in ETV, were then invited to participate in a modified Delphi survey to establish the content validity of these instruments. In each Delphi round, experts rated their agreement including each procedural step, error, and GRS item in the respective instruments on a 5-point Likert scale.

RESULTS

Seventeen experts agreed to participate in the study and completed all Delphi rounds. After item generation, a total of 27 procedural CL items, 26 error CL items, and 9 GRS items were posed to Delphi panelists for rating. An additional 17 procedural CL items, 12 error CL items, and 1 GRS item were added by panelists. After three rounds, strong consensus (>80% agreement) was achieved on 35 procedural CL items, 29 error CL items, and 10 GRS items. Moderate consensus (50-80% agreement) was achieved on an additional 7 procedural CL items and 1 error CL item. The final procedural and error checklist contained 42 and 30 items, respectively (divided into setup, exposure, navigation, ventriculostomy, and closure). The final GRS contained 10 items.

CONCLUSIONS

We have established the content validity of three ETV assessment measures by iterative consensus of an international expert panel. Each measure provides unique assessment information and thus can be used individually or in combination, depending on the characteristics of the learner and the purpose of the assessment. These instruments must now be evaluated in both the simulated and operative settings, to determine their construct validity and reliability. Ultimately, the measures contained in the NEVAT may prove suitable for formative assessment during ETV training and potentially as summative assessment measures during certification.

Peroral endoscopic myotomy: an evolving treatment for achalasia

Peroral endoscopic myotomy (POEM) was first performed in Japan in 2008 for uncomplicated achalasia. With excellent results, it was adopted by highly skilled endoscopists around the world and the indications for POEM were expanded to include advanced sigmoid achalasia, failed surgical myotomy, patients with previous endoscopic treatments and even other spastic oesophageal motility disorders. With increased uptake and performance of POEM, variations in technique and improved management of adverse events have been developed. Now, 6 years since the first case and with >3,000 procedures performed worldwide, long-term data has shown the efficacy of POEM to be long-lasting. A growing body of literature also exists pertaining to the learning curve, application of novel technologies, extended indications and physiologic changes with POEM. Ultimately, this once experimental procedure is evolving towards becoming the preferred treatment for achalasia and other spastic oesophageal motility disorders.

Assessing Arthroscopic Skills Using Wireless Elbow-Worn Motion Sensors

BACKGROUND

Assessment of surgical skill is a critical component of surgical training. Approaches to assessment remain predominantly subjective, although more objective measures such as Global Rating Scales are in use. This study aimed to validate the use of elbow-worn, wireless, miniaturized motion sensors to assess the technical skill of trainees performing arthroscopic procedures in a simulated environment.

METHODS

Thirty participants were divided into three groups on the basis of their surgical experience: novices (n = 15), intermediates (n = 10), and experts (n = 5). All participants performed three standardized tasks on an arthroscopic virtual reality simulator while wearing wireless wrist and elbow motion sensors. Video output was recorded and a validated Global Rating Scale was used to assess performance; dexterity metrics were recorded from the simulator. Finally, live motion data were recorded via Bluetooth from the wireless wrist and elbow motion sensors and custom algorithms produced an arthroscopic performance score.

RESULTS

Construct validity was demonstrated for all tasks, with Global Rating Scale scores and virtual reality output metrics showing significant differences between novices, intermediates, and experts (p < 0.001). The correlation of the virtual reality path length to the number of hand movements calculated from the wireless sensors was very high (p < 0.001). A comparison of the arthroscopic performance score levels with virtual reality output metrics also showed highly significant differences (p < 0.01). Comparisons of the arthroscopic performance score levels with the Global Rating Scale scores showed strong and highly significant correlations (p < 0.001) for both sensor locations, but those of the elbow-worn sensors were stronger and more significant (p < 0.001) than those of the wrist-worn sensors.

CONCLUSIONS

A new wireless assessment of surgical performance system for objective assessment of surgical skills has proven valid for assessing arthroscopic skills. The elbow-worn sensors were shown to achieve an accurate assessment of surgical dexterity and performance.

CLINICAL RELEVANCE

The validation of an entirely objective assessment of arthroscopic skill with wireless elbow-worn motion sensors introduces, for the first time, a feasible assessment system for the live operating theater with the added potential to be applied to other surgical and interventional specialties.

A systematic review of performance assessment tools for laparoscopic cholecystectomy

BACKGROUND

Multiple tools are available to assess clinical performance of laparoscopic cholecystectomy (LC), but there are no guidelines on how best to implement and interpret them in educational settings. The purpose of this systematic review was to identify and critically appraise LC assessment tools and their measurement properties, in order to make recommendations for their implementation in surgical training.

METHODS

A systematic search (1989-2013) was conducted in MEDLINE, Embase, Scopus, Cochrane, and grey literature sources. Evidence for validity (content, response process, internal structure, relations to other variables, and consequences) and the conditions in which the evidence was obtained were evaluated.

RESULTS

A total of 54 articles were included for qualitative synthesis. Fifteen technical skills and two non-technical skills assessment tools were identified. The 17 tools were used for either: recorded procedures (nine tools, 60%), direct observation (five tools, 30%), or both (three tools, 18%). Fourteen (82%) tools reported inter-rater reliability and one reported a Generalizability Theory coefficient. Nine (53%) had evidence for validity based on clinical experience and 11 (65%) compared scores to other assessments. Consequences of scores, educational impact, applications to residency training, and how raters were trained were not clearly reported. No studies mentioned cost.

CONCLUSIONS

The most commonly reported validity evidence was inter-rater reliability and relationships to other known variables. Consequences of assessments and rater training were not clearly reported. These data and the evidence for validity should be taken into consideration when deciding how to select and implement a tool to assess performance of LC, and especially how to interpret the results.

Verification of accurate technical insight: a prerequisite for self-directed surgical training

Simulation-based surgical skills training during preclinical education is a persistent challenge due to time constraints of trainees and instructors alike. Self-directed practice is resource-efficient and flexible; however, insight into technical proficiency among trainees is often lacking. The purpose of this study is to prospectively assess the accuracy of self-assessments among medical students learning basic surgical suturing. Over seven weekly practice sessions, preclinical medical students performed serial repetitions of a simulation-based suturing task under one-on-one observation by one of four trainers. Following each task repetition, self- and trainer-assessments (SA-TA) were performed using a 36-point weighted checklist of technical standards developed a priori by expert consensus. Upon study completion, agreement between SA and TA was measured using weighted Cohen's kappa coefficients. Twenty-nine medical students each performed a median of 25 suture task repetitions (IQR 21.5-28). Self-assessments tended to overestimate proficiency during the first tertile of practice attempts. Agreement between SA and TA improved with experience, such that the weighted kappa statistics for the two-handed and instrument ties were >0.81 after 18-21 task attempts. Inexperienced trainees frequently overestimate technical proficiency through self-assessments. However, this bias diminishes with repetitive practice. Only after trainees have attained the capacity to accurately self-assess can effective self-directed learning take place.

Development of a standardized training course for laparoscopic procedures using Delphi methodology

BACKGROUND

Content, evaluation, and certification of laparoscopic skills and procedure training lack uniformity among different hospitals in The Netherlands. Within the process of developing a new regional laparoscopic training curriculum, a uniform and transferrable curriculum was constructed for a series of laparoscopic procedures. The aim of this study was to determine regional expert consensus regarding the key steps for laparoscopic appendectomy and cholecystectomy using Delphi methodology.

METHODS

Lists of suggested key steps for laparoscopic appendectomy and cholecystectomy were created using surgical textbooks, available guidelines, and local practice. A total of 22 experts, working for teaching hospitals throughout the region, were asked to rate the suggested key steps for both procedures on a Likert scale from 1-5. Consensus was reached with Crohnbach's α ≥ 0.90.

RESULTS

Of the 22 experts, 21 completed and returned the survey (95%). Data analysis already showed consensus after the first round of Delphi on the key steps for laparoscopic appendectomy (Crohnbach's α = 0.92) and laparoscopic cholecystectomy (Crohnbach's α = 0.90). After the second round, 15 proposed key steps for laparoscopic appendectomy and 30 proposed key steps for laparoscopic cholecystectomy were rated as important (≥4 by at least 80% of the expert panel). These key steps were used for the further development of the training curriculum.

CONCLUSION

By using the Delphi methodology, regional consensus was reached on the key steps for laparoscopic appendectomy and cholecystectomy. These key steps are going to be used for standardized training and evaluation purposes in a new regional laparoscopic curriculum.

Are we getting the critical view? A prospective study of photographic documentation during laparoscopic cholecystectomy

BACKGROUND

At laparoscopic cholecystectomy, most surgeons have adopted the operative approach where the 'critical view of safety' (CVS) is achieved prior to dividing the cystic duct and artery. This prospective study evaluated whether an adequate critical view was achieved by scoring standardized intra-operative photographic views and whether there were other factors that might impact on the ability to obtain an adequate critical view.

METHODS

One hundred consecutive patients undergoing a laparoscopic cholecystectomy were studied. At each operation, two photographs were taken. Two independent experienced hepatobiliary surgeons scored the photographs on whether a critical view of safety was achieved. Inter-observer agreement was calculated using the weighted kappa coefficient. The Cochran-Mantel-Haenszel test was used to analyse the scores with potential confounding clinical factors.

RESULTS

The kappa coefficient for adequate display of the cystic duct and artery was 0.49; 95% confidence interval (CI) 0.33 to 0.64; P = 0.001. No bias was detected in the overall scorings between the two observers (χ(2) 1.33; P = 0.312). Other clinical factors including surgeon seniority did not alter the outcome [odds ratio (OR) 0.902; 95% confidence interval 0.622 to 1.264].

CONCLUSION

Heightened awareness of the CVS through mandatory documentation may improve both trainee and surgeon technique.

Reliability of laparoscopic skills assessment on video: 8-year results of the endoscopic surgical skill qualification system in Japan

BACKGROUND AND PURPOSE

The Japanese Urological Association and Japanese Society of Endourology established a urologic laparoscopic skills qualification system called the Endoscopic Surgical Skill Qualification (ESSQ) System in Urological Laparoscopy in 2004. The reliability of video assessments by referees was evaluated.

MATERIALS AND METHODS

Videos of nephrectomies or adrenalectomies performed by the applicants were assessed by two referees selected among a pool of 42 referees. From 2004 to 2011, 1308 urologists applied and 60.2% were qualified after video assessments. The results of skills assessments on 1220 videos that had fixed points by two referees were analyzed statistically.

RESULTS

The average number of videos that each referee assessed was 58.1, with a range of 16 to 87. The accordance rate of the results of the video assessment, pass or fail, by the two referees was 68.9%. The scores of the video assessment by each referee averaged 62.7±2.4 (standard deviation) (full score was set at 75 points and ≥60 points was needed to pass). There was a statistically significant difference in the average video assessment score among the referees (P<0.001), and five referees showed significantly higher or lower average scores than the other referees. The percentage qualification of the final decision made by the Referee Committee on the videos originally assessed by each referee showed no significant differences among the 42 referees. The accordance rate of the results from the video assessment by each referee with the final decision by the committee showed a statistically significant positive correlation with the number of videos assessed by each referee (r=0.404, P=0.0080).

CONCLUSIONS

The ESSQ system showed moderate reliability for the video assessments by the referees. It was concluded that the video assessments by the referees were fair for all applicants, because the final qualification rates showed no significant differences among the referees.

Tools for the direct observation and assessment of psychomotor skills in medical trainees: a systematic review

CONTEXT

The Accreditation Council for Graduate Medical Education (ACGME) Milestone Project mandates programmes to assess the attainment of training outcomes, including the psychomotor (surgical or procedural) skills of medical trainees. The objectives of this study were to determine which tools exist to directly assess psychomotor skills in medical trainees on live patients and to identify the data indicating their psychometric and edumetric properties.

METHODS

An electronic search was conducted for papers published from January 1948 to May 2011 using the PubMed, Education Resource Information Center (ERIC), Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Web of Science electronic databases and the review of references in article bibliographies. A study was included if it described a tool or instrument designed for the direct observation of psychomotor skills in patient care settings by supervisors. Studies were excluded if they referred to tools that assessed only clinical or non-technical skills, involved non-medical health professionals, or assessed skills performed on a simulator. Overall, 4114 citations were screened, 168 (4.1%) articles were reviewed for eligibility and 51 (1.2%) manuscripts were identified as meeting the study inclusion criteria. Three authors abstracted and reviewed studies using a standardised form for the presence of key psychometric and edumetric elements as per ACGME and American Psychological Association (APA) recommendations, and also assigned an overall grade based on the ACGME Committee on Educational Outcome Assessment grading system.

RESULTS

A total of 30 tools were identified. Construct validity based on associations between scores and training level was identified in 24 tools, internal consistency in 14, test-retest reliability in five and inter-rater reliability in 20. The modification of attitudes, knowledge or skills was reported using five tools. The seven-item Global Rating Scale and the Procedure-Based Assessment received an overall Class 1 ACGME grade and are recommended based on Level A ACGME evidence.

CONCLUSIONS

Numerous tools are available for the assessment of psychomotor skills in medical trainees, but evidence supporting their psychometric and edumetric properties is limited.

Requirements for the structured recording of surgical device data in the digital operating room

PURPOSE

Due to the increasing complexity of the surgical working environment, increasingly technical solutions must be found to help relieve the surgeon. This objective is supported by a structured storage concept for all relevant device data.

METHODS

In this work, we present a concept and prototype development of a storage system to address intraoperative medical data. The requirements of such a system are described, and solutions for data transfer, processing, and storage are presented. In a subsequent study, a prototype based on the presented concept is tested for correct and complete data transmission and storage and for the ability to record a complete neurosurgical intervention with low processing latencies. In the final section, several applications for the presented data recorder are shown.

RESULTS

The developed system based on the presented concept is able to store the generated data correctly, completely, and quickly enough even if much more data than expected are sent during a surgical intervention.

CONCLUSIONS

The Surgical Data Recorder supports automatic recognition of the interventional situation by providing a centralized data storage and access interface to the OR communication bus. In the future, further data acquisition technologies should be integrated. Therefore, additional interfaces must be developed. The data generated by these devices and technologies should also be stored in or referenced by the Surgical Data Recorder to support the analysis of the OR situation.

Arthroscopic proficiency: methods in evaluating competency

BACKGROUND

The current paradigm of arthroscopic training lacks objective evaluation of technical ability and its adequacy is concerning given the accelerating complexity of the field. To combat insufficiencies, emphasis is shifting towards skill acquisition outside the operating room and sophisticated assessment tools. We reviewed (1) the validity of cadaver and surgical simulation in arthroscopic training, (2) the role of psychomotor analysis and arthroscopic technical ability, (3) what validated assessment tools are available to evaluate technical competency, and (4) the quantification of arthroscopic proficiency.

METHODS

The Medline and Embase databases were searched for published articles in the English literature pertaining to arthroscopic competence, arthroscopic assessment and evaluation and objective measures of arthroscopic technical skill. Abstracts were independently evaluated and exclusion criteria included articles outside the scope of knee and shoulder arthroscopy as well as original articles about specific therapies, outcomes and diagnoses leaving 52 articles cited in this review.

RESULTS

Simulated arthroscopic environments exhibit high levels of internal validity and consistency for simple arthroscopic tasks, however the ability to transfer complex skills to the operating room has not yet been established. Instrument and force trajectory data can discriminate between technical ability for basic arthroscopic parameters and may serve as useful adjuncts to more comprehensive techniques. There is a need for arthroscopic assessment tools for standardized evaluation and objective feedback of technical skills, yet few comprehensive instruments exist, especially for the shoulder. Opinion on the required arthroscopic experience to obtain proficiency remains guarded and few governing bodies specify absolute quantities.

CONCLUSIONS

Further validation is required to demonstrate the transfer of complex arthroscopic skills from simulated environments to the operating room and provide objective parameters to base evaluation. There is a deficiency of validated assessment tools for technical competencies and little consensus of what constitutes a sufficient case volume within the arthroscopy community.

Defining technical errors in laparoscopic surgery: a systematic review

BACKGROUND

Technical errors, a distinct subcomponent of surgical proficiency, have a significant impact on patient safety and clinical outcomes. To date, only a few studies have been designed to describe and evaluate these errors. This review was performed to assess technical errors described in laparoscopic surgery.

METHODS

A literature search of Medline, Cochrane, EMBASE, and OVID databases (1946-2012, week 14) using the terms "technical/medical error," "technical skill," and "adverse event" in combination with the terms "laparoscopy/laparoscopic surgery" was conducted. English language peer review articles with a description of technical errors were included. Opinion papers, reviews, and articles not addressing laparoscopic surgery were excluded.

RESULTS

The search returned 2,282 articles. Application of the inclusion criteria reduced the number of articles to 21. Of these 21 articles, 14 (67 %) were observational studies, 3 (14 %) were randomized trials, 2 (10 %) were prospective interventional studies, and 2 (10 %) were retrospective analyses. Eight articles (38 %) applied error analysis as an approach to determine error rates within routine procedures. The remaining 13 articles (62 %) used the assessment of errors to describe and quantify surgical skill in an educational setting.

CONCLUSIONS

A number of approaches for the assessment of surgical technical errors exist. The error definitions vary greatly, making a comparison of error rates between groups impossible. Complexity of scale design and subjectivity in ratings have resulted in limited use of these scores outside the experimental setting. To facilitate error analysis as a self-assessment method of continuous learning and quality control, further research and better tools are required.

Is motion analysis a valid tool for assessing laparoscopic skill?

BACKGROUND

The use of simulation for laparoscopic training has led to the development of objective tools for skills assessment. Motion analysis represents one area of focus. This study was designed to assess the evidence for the use of motion analysis as a valid tool for laparoscopic skills assessment.

METHODS

Embase, MEDLINE and PubMed were searched using the following domains: (1) motion analysis, (2) validation and (3) laparoscopy. Studies investigating motion analysis as a tool for assessment of laparoscopic skill in general surgery were included. Common endpoints in motion analysis metrics were compared between studies according to a modified form of the Oxford Centre for Evidence-Based Medicine levels of evidence and recommendation.

RESULTS

Thirteen studies were included from 2,039 initial papers. Twelve (92.3 %) reported the construct validity of motion analysis across a range of laparoscopic tasks. Of these 12, 5 (41.7 %) evaluated the ProMIS Augmented Reality Simulator, 3 (25 %) the Imperial College Surgical Assessment Device (ICSAD), 2 (16.7 %) the Hiroshima University Endoscopic Surgical Assessment Device (HUESAD), 1 (8.33 %) the Advanced Dundee Endoscopic Psychomotor Tester (ADEPT) and 1 (8.33 %) the Robotic and Video Motion Analysis Software (ROVIMAS). Face validity was reported by 1 (7.7 %) study each for ADEPT and ICSAD. Concurrent validity was reported by 1 (7.7 %) study each for ADEPT, ICSAD and ProMIS. There was no evidence for predictive validity.

CONCLUSIONS

Evidence exists to validate motion analysis for use in laparoscopic skills assessment. Valid parameters are time taken, path length and number of hand movements. Future work should concentrate on the conversion of motion data into competency-based scores for trainee feedback.

High-fidelity, low-cost, automated method to assess laparoscopic skills objectively

BACKGROUND

We sought to define the extent to which a motion analysis-based assessment system constructed with simple equipment could measure technical skill objectively and quantitatively.

METHODS

An "off-the-shelf" digital video system was used to capture the hand and instrument movement of surgical trainees (beginner level = PGY-1, intermediate level = PGY-3, and advanced level = PGY-5/fellows) while they performed a peg transfer exercise. The video data were passed through a custom computer vision algorithm that analyzed incoming pixels to measure movement smoothness objectively.

RESULTS

The beginner-level group had the poorest performance, whereas those in the advanced group generated the highest scores. Intermediate-level trainees scored significantly (p < 0.04) better than beginner trainees. Advanced-level trainees scored significantly better than intermediate-level trainees and beginner-level trainees (p < 0.04 and p < 0.03, respectively).

CONCLUSIONS

A computer vision-based analysis of surgical movements provides an objective basis for technical expertise-level analysis with construct validity. The technology to capture the data is simple, low cost, and readily available, and it obviates the need for expert human assessment in this setting.

The Endoscopic Surgical Skill Qualification System for gastric surgery in Japan

INTRODUCTION

Laparoscopic surgery has been increasing in popularity in recent years. In 2004, the Japan Society for Endoscopic Surgery developed its Endoscopic Surgical Skill Qualification System (ESSQS) to assess surgeons.

METHODS

To earn the ESSQS accreditation, applicants must submit an unedited operative video in which they perform either a distal gastrectomy or pylorus-preserving gastrectomy with lymph node dissection for gastric cancer. The videos are assessed by two separate judges based on detailed criteria for common and procedure-specific technical-grade slips. Common criteria from all fields of gastrointestinal and general surgery are used to evaluate the basic laparoscopic surgical skills and autonomy of the operator. The target organ determines the procedure-specific criteria are set to assess whether or not adequate oncological clearance has been achieved.

RESULTS

Between 2004 and 2009, 154 (44.6%) out of 345 applicant surgeons assessed under the ESSQS for gastric surgery have been accredited. Interrater agreement was acceptable and ranged between 0.21 and 0.59.

CONCLUSION

The ESSQS system may facilitate improvement in surgical technique and the standardization of laparoscopic surgery in Japan.

Observational tools for assessment of procedural skills: a systematic review

BACKGROUND

Assessment by direct observation of procedural skills is an important source of constructive feedback. The aim of this study was to identify observational tools for technical skill assessment, to assess characteristics of these tools, and to assess their usefulness for assessment.

METHODS

Included studies reported tools for observational assessment of technical skills. A total of 106 articles were included.

RESULTS

Three main categories included global assessment scales evaluating generic skills (n = 29), task-specific methods assessing procedure-specific skills (n = 30), and combinations of tools evaluating both generic and task-specific skills (n = 47). In most studies, content validity was not evaluated using an accepted scientific method. All tools were assessed for inter-rater reliability and construct validity. Data on feasibility, acceptability, and educational impact were sparse.

CONCLUSIONS

There is evidence of validity and reliability for observational assessment tools at the trainee level. In most studies a comprehensive analysis of the tools was not achieved. Evaluation of technical skill using current observational assessment tools is not reliable and valid at the specialist level. Future research needs to focus on further systematic tool development and analysis, especially at the specialist level.

Objective assessment, selection, and certification in surgery

Historically, surgical competence has been evaluated subjectively. Fundamental changes in surgical technology and training have focused attention on the use of objective measurement of performance to improve patient safety and reduce errors. Surgical performance can be measured using a variety of tools, both in the clinical and simulated environments. Objective assessments can play a role in training by improving the evaluation and feedback. At the end of training or when a new skill is acquired, objective assessments may be used to ensure that a proficiency level has been reached and potentially as a condition for independent practice. When assessments are used for high-stakes evaluations like certification, they must be demonstrably reliable and valid. The definition of assessment, and the necessary components of a valid instrument, will be summarized. An overview of practical applications of objective assessment as it applies to training, selection, and certification of surgeons will be presented.

Operative performance in laparoscopic cholecystectomy using the Procedural-Based Assessment tool

AIMS

The Intercollegiate Surgical Curriculum Project (ISCP) has devised assessment tools for index operations to assess trainee technical skills. In this study we used the Procedural-Based Assessment (PBA) tool to evaluate operations performed by trainees.

METHODS

Live and simulated laparoscopic cholecystectomies were performed by trainees. Two experienced surgeons assessed each operation blindly and independently.

RESULTS

Eighty-four live (supervised) and 112 simulated (unsupervised) operations were performed by 28 trainees. Mean inter-rater reliability was kappa = .86 and .84 for live and simulated operations, respectively. Construct validity using Mann-Whitney for generic technical skills was significant for live and simulated operations, P < or = .05. Assessing specific technical skills showed construct validity for simulated unsupervised operations only, Mann-Whitney P < .05, but not for supervised live operations, Mann-Whitney P > .05.

CONCLUSIONS

The PBA showed good inter-rater reliability. Assessing generic technical skills, PBA showed construct validity for both types of operations and for specific technical skills in the unsupervised simulated operations. We conclude that the PBA seems to be a reliable and valid assessment tool for generic technical skills in unsupervised simulated and live supervised laparoscopic cholecystectomies.