Comprehensive assessment of thyroidectomy skills development: A pilot project

Development and Validation of Instrument for Operative Competency Assessment in Selective Neck Dissection

BACKGROUND

Instruments to assess surgical skills have been validated for several key indicator procedures in otolaryngology. Selective neck dissection is a core procedure for which trainees must integrate knowledge of complex head and neck anatomy with technical surgical skills. An instrument for assessment of surgical performance in selective neck dissection has not been previously developed. The objective of the current study is to develop and validate an instrument for assessing surgical competency for level II-IV selective neck dissection.

DESIGN

A Delphi working group comprised of 23 fellowship trained head and neck surgeons from 17 institutions was assembled. The modified Delphi method encompassed a 3-step process, including 2 anonymous voting rounds to successively refine individual items and establish levels of consensus. Thresholds for achieving strong consensus, at >80% agreement, were determined a priori. The resulting instrument was subsequently validated in a prospective cohort of 17 resident surgeons, spanning postgraduate year 1 to 5 training experience. Participants were asked to perform a level II-IV selective neck dissection on fresh-frozen cadaveric specimens. Performance was scored by 2 independent, blinded observers using the devised instrument and construct validity was assessed.

RESULTS

Through the modified Delphi process a final list of 30 items, considered to be the most essential items for achieving the goals of a level II-IV selective neck dissection, was developed. Construct validity was supported by a positive association between instrument scores compared to both resident postgraduate year level and number of head and neck rotations completed.

CONCLUSION

The development and validation of a novel instrument for assessment of surgical competency in level II-IV selective neck dissection, a key indicator case in otolaryngology, is described. This new instrument may be used to provide objective feedback on overall and task-specific competency to identify surgical deficiencies and offer granular feedback to enhance surgical training.

Resident evaluations in the age of competency-based medical education: faculty perspectives on minimizing burdens

OBJECTIVE

Competency-based medical education (CBME), an outcomes-based approach to medical education, continues to be implemented across many postgraduate medical education programs worldwide, including a recent introduction into Canadian neurosurgical training programs (July 2019). The success of this educational paradigm shift requires frequent faculty observation and evaluation of residents performing defined tasks of the specialty. A main challenge involves providing residents with frequent performance evaluations and feedback that are feasible for faculty to complete. This study aims to define what is currently happening and what changes are needed to make CBME successful for the certification of neurosurgeons' competence.

METHODS

A 55-item questionnaire was emailed nationwide to survey Canadian neurosurgical faculty.

RESULTS

Fifty-two complete responses were received and achieved a distribution highly correlated with the number of faculty neurosurgeons practicing in each Canadian province (Pearson's r = 0.94). Two-thirds (35/52) of faculty reported currently taking a median of 10 minutes to complete evaluation forms at the end of a resident's rotation block. Regardless of the faculty's province of practice (p = 0.50) or years of experience (p = 0.06), they reported 3 minutes (minimum 1 minute, maximum 10 minutes, interquartile range [IQR] 3 minutes) as a feasible amount of time to spend completing an evaluation form following an observation of a resident's performance of an entrustable professional activity (EPA). If evaluation forms took 3 minutes to complete, 85% of respondents (44/52) would complete EPA evaluations weekly or daily. The faculty recommended 5 minutes as a feasible amount of time to provide oral feedback (minimum 1 minute, maximum 20 minutes, IQR 3.25 minutes), which was significantly higher (p = 0.00099) than their recommended amount of time for completing evaluation forms. The majority of faculty (71%) stated they would prefer to access resident evaluation forms through a mobile application compared to a paper form (12%), an evaluation website (8%), or through a URL link sent via email (10%; p = 0.0032).

CONCLUSIONS

To facilitate the successful implementation of CBME into a neurosurgical training curriculum, resident EPA assessment forms should take 3 minutes or less to complete and be accessible through a mobile application.

Competency-Based Education in Minimally Invasive and Robotic Colorectal Surgery

Minimally invasive and robotic techniques have become increasingly implemented into surgical practice and are now an essential part of the foundational skills of training colorectal surgeons. Over the past 5 years there has been a shift in the surgical educational paradigm toward competency-based education (CBE). CBE recognizes that trainees learn at different rates but regardless, are required to meet a competent threshold of performance prior to independent practice. Thus, CBE attempts to replace the traditional "time" endpoint of training with "performance." Although conceptually sensible, implementing CBE has proven challenging. This article will define competence, outline appropriate assessment tools to assess technical skill, and review the literature on the number of cases required to achieve competence in colorectal procedures while outlining the barriers to implementing CBE.

Enhancing Operative Feedback: A Descriptive Trajectory for Surgical Development in Otolaryngology

OBJECTIVE

Developmental frameworks are narrative descriptions of learner performance that may be useful to provide operative feedback tools in surgical education. The authors previously proposed the Technician, Anatomist, Anticipator, Strategist, Executive (TAASE) framework utilizing data collected from general surgeons and conjectured its applicability in other surgical specialties. Here, we construct a developmental framework in otolaryngology using similar techniques, then compare the results to TAASE.

DESIGN

Ten otolaryngology educators participated in semi-structured, audio-recorded interviews, to explore how otolaryngology faculty characterize surgical learning at different levels of training. Researchers analyzed the transcripts using a thematic analysis approach to build a developmental framework. Results were then qualitatively compared to the TAASE trajectory.

SETTING

Single tertiary academic medical center.

PARTICIPANTS

Faculty within the Department of Otolaryngology-Head and Neck Surgery.

RESULTS

Otolaryngology faculty characterized surgical development as an integrated trajectory of technical and cognitive skills progressing from specific operative tasks to a global understanding of patients and procedures largely consistent with stages outlined in the Technician, Anatomist, Anticipator, Strategist, and Executive model. A new theme identified was learner emotions, which may impact learning and hinder or enhance progression along the trajectory.

CONCLUSIONS

Otolaryngology and general surgeons agree upon a common arc to operative maturity reflected by the TAASE trajectory. The TAASE framework may be useful as a tool for intraoperative assessment in otolaryngology to help promote quality feedback. Further research is needed to understand how emotions may influence operative skill development.

Assessment of Technical Skills Competence in the Operating Room: A Systematic and Scoping Review

PURPOSE

While academic accreditation bodies continue to promote competency-based medical education (CBME), the feasibility of conducting regular CBME assessments remains challenging. The purpose of this study was to identify evidence pertaining to the practical application of assessments that aim to measure technical competence for surgical trainees in a nonsimulated, operative setting.

METHOD

In August 2016, the authors systematically searched Medline, Embase, and the Cochrane Database of Systematic Reviews for English-language, peer-reviewed articles published in or after 1996. The title, abstract, and full text of identified articles were screened. Data regarding study characteristics, psychometric and measurement properties, implementation of assessment, competency definitions, and faculty training were extracted. The findings from the systematic review were supplemented by a scoping review to identify key strategies related to faculty uptake and implementation of CBME assessments.

RESULTS

A total of 32 studies were included. The majority of studies reported reasonable scores of interrater reliability and internal consistency. Seven articles identified minimum scores required to establish competence. Twenty-five articles mentioned faculty training. Many of the faculty training interventions focused on timely completion of assessments or scale calibration.

CONCLUSIONS

There are a number of diverse tools used to assess competence for intraoperative technical skills and a lack of consensus regarding the definition of technical competence within and across surgical specialties. Further work is required to identify when and how often trainees should be assessed and to identify strategies to train faculty to ensure timely and accurate assessment.

A High-Definition Video Teaching Module for Thyroidectomy Surgery

OBJECTIVE

With the changing landscape of postgraduate surgical education to competency-based curricula, there emerges a need for alternative forms of training. Video teaching modules have been shown to be effective tools in surgical education, complementing traditional postgraduate curricula. There is a lack of validated modules described in the literature, specifically for teaching thyroidectomy. The primary objective of this study was to develop and validate a high definition video-based teaching module instructing thyroidectomy surgery to Otolaryngology-Head and Neck Surgery trainees.

DESIGN

This prospective study included intermediate to senior Otolaryngology-Head and Neck Surgery residents. Each participant first performed a thyroid lobectomy, serving as the initial assessment. After a washout period of at least 3 weeks, each participant was given the teaching module. The 15-minute module was developed using a 3-camera system and detailed a step-by-step approach to the surgery. After exposure to the module, each trainee performed the same procedure. Recordings of both procedures were deidentified and reviewed by a blinded, independent evaluator. Scoring was done using the Observational Clinical Human Reliability Assessment (OCHRA) system.

SETTING

University of Alberta Hospital and Royal Alexandra Hospital, Edmonton, Alberta, Canada.

PARTICIPANTS

A total of 6 intermediate to senior Otolaryngology-Head and Neck Surgery residents entered and completed the study.

RESULTS

The mean error rate was 8.8 errors per procedure before module exposure and 4.5 errors per procedure after exposure, representing a 49% decrease in error occurrence (p < 0.05). The mean staff takeover event rate was 10.5 takeovers per procedure prior to module exposure and 5.0 takeovers per procedure after exposure, representing a 52% decrease in error occurrence (p < 0.05).

CONCLUSION

High-definition video teaching modules are a useful complement to traditional surgical training. In a climate where new innovations for teaching thyroid surgery are needed, properly constructed and validated video teaching modules can serve as important tools in supplementing traditional surgical training.

Objective Assessment of Technical Skills in Otorhinolaryngology-Head and Neck Surgery Residents: A Systematic Review

Objectives The primary goal is the indexation of validated methods used to assess surgical competency in otorhinolaryngology-head and neck surgery (ORL-HNS) residents. Secondary goals include assessment of the reliability and validity of these tools, as well as the documentation of specific procedures in ORL-HNS involved. Data Sources MEDBASE, OVID, Medline, CINAHL, and EBM, as well as the printed references, available through the Université de Montréal library. Review Methods The PRISMA method was used to review digital and printed databases. Publications were reviewed by 2 independent reviewers, and selected articles were fully analyzed to classify evaluation methods and categorize them by procedure and subspecialty of ORL-HNS involved. Reliability and validity were assessed and scored for each assessment tool. Results Through the review of 30 studies, 5 evaluation methods were described and validated to assess the surgical competency of ORL-HNS residents. The evaluation method most often described was the combined Global Rating Scale and Task-Specific Checklist tool. Reliability and validity for this tool were overall high; however, considerable data were unavailable. Eleven distinctive surgical procedures were studied, encompassing many subspecialties of ORL-HNS: facial plastics, general ear-nose-throat, laryngology, otology, pediatrics, and rhinology. Conclusions Although assessment tools have been developed for an array of surgical procedures, involving most ORL-HNS subspecialties, the use of combined checklists has been repeatedly validated in the literature and shown to be easily applicable in practice. It has been applied to many ORL-HNS procedures but not in oncologic surgery to date.

Development and testing for an operative competency assessment tool for nasal septoplasty surgery

BACKGROUND

Assessing surgical competency in otolaryngology is challenging, and residency programs are now responsible for ensuring the surgical competency of their graduates. Therefore, more objective assessment tools are being incorporated into the evaluation process. Objective structured assessment of technical skills (OSATSs) tools have been developed for multiple otolaryngology procedures. These include tonsillectomy, endoscopic sinus surgery, thyroidectomy, mastoidectomy, direct laryngoscopy, and rigid bronchoscopy. The purpose of this study was to develop and test a new assessment tool for septoplasty surgery and ensuring its feasibility, reliability, and construct validity. This study was designed to develop and test a valid, reliable, and feasible evaluation tool designed to measure the development of trainees' surgical skills in the operating room for septoplasty surgery.

METHODS

A new OSATSs-based instrument form for septoplasty was developed. During the study period of 2 years, 21 otolaryngology-head and neck surgery residents (ranging from postgraduate year 2 to 5) were evaluated intraoperatively by one faculty member obtaining a total of 175 evaluations. Surgical performance was rated using a seven-item task-specific checklist (TSC) and a global rating scale (GRS). The TSC assessed specific septoplasty technical skills, and the GRS assessed the overall surgical performance.

RESULTS

Our tool showed construct validity for both components of the assessment instrument, with increasing mean scores with advancing clinical levels. Cronbach's α, a measure of internal consistency, was 0.911 for TSC and 0.898 for GRS. Strong correlation between the TSC and GRS was established (r = 0.955; p < 0.01).

CONCLUSION

This study proved our educational tool to be a valid, reliable, and feasible method for assessing competency in septoplasty surgery. It can be integrated into surgical training programs to facilitate direct formative feedback. Assessing trainees' learning curves enables insight into their progression, ensuring their appropriate development.