Objective assessment, selection, and certification in surgery

Using neuroimaging to assess brain activity and areas associated with surgical skills: a systematic review

BACKGROUND

Surgical skills acquisition is under continuous development due to the emergence of new technologies, and there is a need for assessment tools to develop along with these. A range of neuroimaging modalities has been used to map the functional activation of brain networks while surgeons acquire novel surgical skills. These have been proposed as a method to provide a deeper understanding of surgical expertise and offer new possibilities for the personalized training of future surgeons. With studies differing in modalities, outcomes, and surgical skills there is a need for a systematic review of the evidence. This systematic review aims to summarize the current knowledge on the topic and evaluate the potential use of neuroimaging in surgical education.

METHODS

We conducted a systematic review of neuroimaging studies that mapped functional brain activation while surgeons with different levels of expertise learned and performed technical and non-technical surgical tasks. We included all studies published before July 1st, 2023, in MEDLINE, EMBASE and WEB OF SCIENCE.

RESULTS

38 task-based brain mapping studies were identified, consisting of randomized controlled trials, case-control studies, and observational cohort or cross-sectional studies. The studies employed a wide range of brain mapping modalities, including electroencephalography, functional magnetic resonance imaging, positron emission tomography, and functional near-infrared spectroscopy, activating brain areas involved in the execution and sensorimotor or cognitive control of surgical skills, especially the prefrontal cortex, supplementary motor area, and primary motor area, showing significant changes between novices and experts.

CONCLUSION

Functional neuroimaging can reveal how task-related brain activity reflects technical and non-technical surgical skills. The existing body of work highlights the potential of neuroimaging to link task-related brain activity patterns with the individual level of competency or improvement in performance after training surgical skills. More research is needed to establish its validity and usefulness as an assessment tool.

Sustainable Approach to Certification of Persons: Ensuring Reliability and Quality

Nowadays, sustainability issues are gaining more and more topicality in the context of improving organizational processes, including in the field of conformity assessment. In the field of certification of persons, competence assessment institutions have also become interested in using new approaches as the quality of assessment execution does not fully meet the requirements of stakeholders regarding the ability of a person to apply the acquired knowledge and skills in situations related to the professional activity. The aim of the research is to analyze the aspects of professional competence assessment related to the certification of persons and its performance according to the stakeholder requirements in order to develop a new sustainable approach to the process of certification of persons that would ensure the quality and reliability of its execution. Qualitative and quantitative research methods have been used to analyze the elements of the concept of certification of persons and the main elements of the concept of professional competence. As a result of the research, a new sustainable approach to certification of persons is offered, where certification of persons transforms from a conformity assessment procedure into a professional competence assessment process and ensures that a person’s ability is assessed not only in accordance with the industry requirements, but also with the stakeholder needs.

Assessment of surgical performance of laparoscopic benign hiatal surgery: a systematic review

BACKGROUND

Operative skills correlate with patient outcomes, yet at the completion of training or after learning a new procedure, these skills are rarely formally evaluated. There is interest in the use of summative video assessment of laparoscopic benign foregut and hiatal surgery (LFS). If this is to be used to determine competency, it must meet the robust criteria established for high-stakes assessments. The purpose of this review is to identify tools that have been used to assess performance of LFS and evaluate the available validity evidence for each instrument.

METHODS

A systematic search was conducted up to July 2017. Eligible studies reported data on tools used to assess performance in the operating room during LFS. Two independent reviewers considered 1084 citations for eligibility. The characteristics and testing conditions of each assessment tool were recorded. Validity evidence was evaluated using five sources of validity (content, response process, internal structure, relationship to other variables, and consequences).

RESULTS

There were six separate tools identified. Two tools were generic to laparoscopy, and four were specific to LFS [two specific to Nissen fundoplication (NF), one heller myotomy (HM), and one paraesophageal hernia repair (PEH)]. Overall, only one assessment was supported by moderate evidence while the others had limited or unknown evidence. Validity evidence was based mainly on internal structure (all tools reporting reliability and item analysis) and content (two studies referencing previous papers for tool development in the context of clinical assessment, and four listing items without specifying the development procedures). There was little or no evidence supporting test response process (one study reporting rater training), relationship to other variables (two comparing scores in subjects with different clinical experience), and consequences (no studies). Two tools were identified to have evidence for video assessment, specific to NF.

CONCLUSION

There is limited evidence supporting the validity of assessment tools for laparoscopic foregut surgery. This precludes their use for summative video-based assessment to verify competency. Further research is needed to develop an assessment tool designed for this purpose.

Skill Acquisition, Assessment, and Simulation in Minimal Access Surgery: An Evolution of Technical Training in Surgery

Diminishing resources and expanding technologies, such as minimal access surgery, have complicated the acquisition and assessment of technical skills in surgical training programs. However, these challenges have been met with both innovation and an evolution in our understanding of how learners develop technical competence and how to better measure it. As these skills continue to grow in breadth and complexity, so too must the surgical education systems’ ability. This literature review examines and describes the pressures placed on surgical education programs and the development of methods to ameliorate them with a focus on surgical simulation.

A novel assessment tool for evaluating competence in video-assisted thoracoscopic surgery lobectomy

BACKGROUND

Specific assessment tools can accelerate trainees' learning through structured feedback and ensure that trainees attain the knowledge and skills required to practice as competent, independent surgeons (competency-based surgical education). The objective was to develop an assessment tool for video-assisted thoracoscopic surgery (VATS) lobectomy by achieving consensus within an international group of VATS experts.

METHOD

The Delphi method was used as a structured process for collecting and distilling knowledge from a group of internationally recognized VATS experts. Opinions were obtained in an iterative process involving answering repeated rounds of questionnaires. Responses to one round were summarized and integrated into the next round of questionnaires until consensus was reached.

RESULTS

Thirty-one VATS experts were included and four Delphi rounds were conducted. The response rate for each round were 68.9% (31/45), 100% (31/31), 96.8% (30/31), and 93.3% (28/30) for the final round where consensus was reached. The first Delphi round contained 44 items and the final VATS lobectomy Assessment Tool (VATSAT) comprised eight items with rating anchors: (1) localization of tumor and other pathological tissue, (2) dissection of the hilum and veins, (3) dissection of the arteries, (4) dissection of the bronchus, (5) dissection of lymph nodes, (6) retrieval of lobe in bag, (7) respect for tissue and structures, and (8) technical skills in general.

CONCLUSION

A novel and dedicated assessment tool for VATS lobectomy was developed based on VATS experts' consensus. The VATSAT can support the learning of VATS lobectomy by providing structured feedback and help supervisors make the important decision of when trainees have acquired VATS lobectomy competencies for independent performance.

Early Performance on an Eye Surgery Simulator Predicts Subsequent Resident Surgical Performance

OBJECTIVE

To examine early performance on an eye surgery simulator and its relationship to subsequent live surgical performance in a single large residency program.

DESIGN

Retrospective study.

SETTING

Massachusetts Eye and Ear, Harvard Medical School, Department of Ophthalmology.

METHODS

In a retrospective study, we compared performance of 30 first-year ophthalmology residents on an eye surgery simulator to their surgical skills as third-year residents. Variables collected from the eye surgery simulator included scores on the following modules of the simulator (Eyesi, VRmagic, Mannheim, Germany): antitremor training level 1, bimanual training level 1, capsulorhexis level 1 (configured), forceps training level 1, and navigation training level 1. Subsequent surgical performance was assessed using the total number of phacoemulsification cataract surgery cases for each resident, as well as the number performed as surgeon during residency and scores on global rating assessment of skills in intraocular surgery (GRASIS) scales during the third year of residency. Spearman correlation coefficients were calculated between each of the simulator performance and subsequent surgical performance variables. We also compared variables in a small group of residents who needed extra help in learning cataract surgery to the other residents in the study.

MAIN OUTCOME MEASURES

Relationships between Eyesi scores early in residency and surgical performance measures in the final year of residency.

RESULTS

A total of 30 residents had Eyesi data from their first year of residency and had already graduated so that all subsequent surgical performance data were available. There was a significant correlation between capsulorhexis task score on the simulator and total surgeries (r = 0.745, p = 0.008). There was a significant correlation between antitremor training level 1 (r = 0.554, p = 0.040), and forceps training level 1 (r = 0.622, p = 0.023) with primary surgery numbers. There was a significant correlation between forceps training level 1 (r = 0.811, p = 0.002), and navigation training level 1 (r = 0.692, p = 0.013) with total GRASIS score. There was a significant inverse correlation between total GRASIS score and residents in need of extra help (r = -0.358, p = 0.003).

CONCLUSION

Module scores on an eye surgery simulator early in residency may predict a resident׳s future performance in the operating room. These scores may allow early identification of residents in need of supplemental training in cataract surgery.

What errors make a laparoscopic cancer surgery unsafe? An ad hoc analysis of competency assessment in the National Training Programme for laparoscopic colorectal surgery in England

BACKGROUND

The National Training Programme for laparoscopic colorectal surgery in England was implemented to ensure training was supervised, structured, safe and effective. Delegates were required to pass a competency assessment (sign-off) before undertaking independent practice. This study described the types of errors identified and associated these errors with competency to progress to independent laparoscopic colorectal practice.

METHODS

All sign-off submissions from the start of the process in January 2008 until July 2013 were included. Content analysis was used to categorise errors. Bayes factor (BF) was used to measure the impact of individual error on assessment outcome. A smaller BF indicates that an error has stronger associations with unsuccessful assessments. Bayesian network was employed to graphically represent the reasoning process whereby the chance of successful assessment diminished with the identification of each error. Quality of the error feedback was measured by the area under the ROC curve which linked the predictions from the Bayesian model to the expert verdict.

RESULTS

Among 370 assessments analysed, 240 passed and 130 failed. On average, 2.5 different types of error were identified in each assessment. Cases that were more likely to fail had three or more different types of error (χ(2) = 72, p < 0.0001) and demonstrated poorer technical skills (CAT score <2.7, χ(2) = 164, p < 0.0001). Case complexity or right- versus left-sided resection did not have a significant impact. Errors associated with dissection (BF = 0.18), anastomosis (BF = 0.23) and oncological quality (BF = 0.19) were critical determinants of surgical competence, each reducing the odds of pass by at least fourfold. The area under the ROC curve was 0.84.

CONCLUSIONS

Errors associated with dissection, anastomosis and oncological quality were critical determinants of surgical competency. The detailed error analysis reported in this study can guide the design of future surgical education and clinical training programmes.

A systematic review of performance assessment tools for laparoscopic cholecystectomy

BACKGROUND

Multiple tools are available to assess clinical performance of laparoscopic cholecystectomy (LC), but there are no guidelines on how best to implement and interpret them in educational settings. The purpose of this systematic review was to identify and critically appraise LC assessment tools and their measurement properties, in order to make recommendations for their implementation in surgical training.

METHODS

A systematic search (1989-2013) was conducted in MEDLINE, Embase, Scopus, Cochrane, and grey literature sources. Evidence for validity (content, response process, internal structure, relations to other variables, and consequences) and the conditions in which the evidence was obtained were evaluated.

RESULTS

A total of 54 articles were included for qualitative synthesis. Fifteen technical skills and two non-technical skills assessment tools were identified. The 17 tools were used for either: recorded procedures (nine tools, 60%), direct observation (five tools, 30%), or both (three tools, 18%). Fourteen (82%) tools reported inter-rater reliability and one reported a Generalizability Theory coefficient. Nine (53%) had evidence for validity based on clinical experience and 11 (65%) compared scores to other assessments. Consequences of scores, educational impact, applications to residency training, and how raters were trained were not clearly reported. No studies mentioned cost.

CONCLUSIONS

The most commonly reported validity evidence was inter-rater reliability and relationships to other known variables. Consequences of assessments and rater training were not clearly reported. These data and the evidence for validity should be taken into consideration when deciding how to select and implement a tool to assess performance of LC, and especially how to interpret the results.

Clinical validity of consultant technical skills assessment in the English National Training Programme for Laparoscopic Colorectal Surgery

Background

The English National Training Programme for Laparoscopic Colorectal Surgery introduced a validated objective competency assessment tool to accredit surgeons before independent practice. The aim of this study was to determine whether this technical skills assessment predicted clinical outcomes.

Methods

Established consultants, training in laparoscopic colorectal surgery, were asked to submit two operative videos for evaluation by two blinded assessors using the competency assessment tool. A mark of 2·7 or above was considered a pass. Clinical and oncological outcomes were compared above and below this mark, including regression analysis.

Results

Eighty-five consultant surgeons submitted 171 videos. Of these, 44 (25·7 per cent) were in the fail group (score less than 2·7). This low scoring group had more postoperative morbidity (25 versus 8·7 per cent; P = 0·005), including surgical complications (18 versus 6·3 per cent; P = 0·020) and fewer lymph nodes harvested (median 13 versus 18; P = 0·004). A score of less than 2·7 was an independent predictor of surgical complication, lymph node yield and distal resection margin clearance. Consultants with higher scores had performed similar numbers of laparoscopic colorectal operations (median 37 versus 40; P = 0·373) but more structured training operations (18 versus 9; P &lt; 0·001).

Conclusion

An objective technical skills assessment provided a discriminatory tool with which to accredit laparoscopic colorectal surgeons.

Training faculty in nontechnical skill assessment: national guidelines on program requirements

OBJECTIVE

To develop guidelines for a faculty training program in nontechnical skill assessment in surgery.

BACKGROUND

Nontechnical skills in the operating room are critical for patient safety. The successful integration of these skills into workplace-based assessment is dependent upon the availability of faculty who are able to teach and assess them. At present, no guidelines exist regarding the training requirements for such faculty in surgical contexts.

METHODS

The development of the guidelines was carried out in several stages: stage 1-a detailed literature review on current training for nontechnical skill assessors; stage 2-semistructured interviews with a multidisciplinary panel (consisting of clinicians and psychologists/human factors specialists) of experts in surgical nontechnical skills; and stage 3-interview findings fed into an Expert Consensus Panel (ECP) Delphi approach to establish consensus regarding training requirements for faculty assessing nontechnical skills in surgery.

RESULTS

The ECP agreed that training in nontechnical skill assessment should be delivered by a multidisciplinary team consisting of clinicians and psychologists/human factors specialists. The ECP reached consensus regarding who should be targeted to be trained as faculty (including proficiency and revalidation requirements). Consensus was reached on 7 essential training program content elements (including training in providing feedback/debriefing) and 8 essential methods of evaluating the effectiveness of a "train-the-trainers" program.

CONCLUSIONS

This study provides evidence-based guidelines that can be used to guide the development and evaluation of programs to educate faculty in the training and assessment of nontechnical skills. Uptake of these guidelines could accelerate the development of surgical expertise required for safe and high-quality patient care.

Simulation-based cutaneous surgical-skill training on a chicken-skin bench model in a medical undergraduate program

BACKGROUND

Because of ethical and medico-legal aspects involved in the training of cutaneous surgical skills on living patients, human cadavers and living animals, it is necessary the search for alternative and effective forms of training simulation.

AIMS

To propose and describe an alternative methodology for teaching and learning the principles of cutaneous surgery in a medical undergraduate program by using a chicken-skin bench model.

MATERIALS AND METHODS

One instructor for every four students, teaching materials on cutaneous surgical skills, chicken trunks, wings, or thighs, a rigid platform support, needled threads, needle holders, surgical blades with scalpel handles, rat-tooth tweezers, scissors, and marking pens were necessary for training simulation.

RESULTS

A proposal for simulation-based training on incision, suture, biopsy, and on reconstruction techniques using a chicken-skin bench model distributed in several sessions and with increasing levels of difficultywas structured. Both feedback and objective evaluations always directed to individual students were also outlined.

CONCLUSION

The teaching of a methodology for the principles of cutaneous surgery using a chicken-skin bench model versatile, portable, easy to assemble, and inexpensive is an alternative and complementary option to the armamentarium of methods based on other bench models described.

A simple framework for assessing technical skills in a resident observed structured clinical examination (OSCE): vaginal laceration repair

OBJECTIVES

Educators of trainees in procedure-based specialties need focused assessment tools that are valid, objective, and assess technical skills in a realistic context. A framework for hybrid assessment using standardized patient scenarios and bench skills testing might facilitate evaluation of competency.

METHODS

Seven PGY-1 obstetrics and gynecology residents participated in a hybrid assessment that used observed structured clinical examination (OSCE) by a standardized patient who had sustained a vaginal laceration during vaginal delivery. The residents elicited a history and counseled the patient, and then completed a laceration repair on a pelvic model. The residents were rated on their performance in the scenario, which included issues of cultural competency, rapport-building, patient counseling. The technical skills were videotaped and rated using a modified global assessment form by 2 faculty members on a 3-point scale from "not done" to "partly done" to "well-done." Residents also completed a subjective assessment of the station.

RESULTS

Mean technical performance of the residents on the technical skills was 55% "well-done," with a range of 20%-90%. The assessment identified 3 residents as below the mean, and 1 resident with areas of deficiency. Subjective assessment by the residents was that juggling the technical, cognitive, and affective components of the examination was challenging.

CONCLUSIONS

Technical skills can be included in a case-based assessment using scenarios that address a range of cognitive and affective skills required of physicians. Results may help training programs assess individuals' abilities as well as identify program needs for curricular improvement. This framework might be useful in setting standards for competency and identifying poor performers.