Assessment of Surgical Competency

Global assessment of surgical skills (GASS): validation of a new instrument to measure global technical safety in surgical procedures

BACKGROUND

Broad implementation of the American Board of Surgery's entrustable professional activities initiative will require assessment instruments that are reliable and easy to use. Existing assessment instruments of general laparoscopic surgical skills have limited reliability, efficiency, and validity across the spectrum of formative (low-stakes) and summative (high-stakes) assessments. A novel six-item global assessment of surgical skills (GASS) instrument was developed and evaluated with a focus upon safe versus unsafe surgical practice scoring rubric.

METHODS

The GASS was developed by iterative engagement with expert laparoscopic surgeons and includes six items (economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, achievement of hemostasis, overall performance) with a uniform three-point scoring rubric ("poor-unsafe", "adequate-safe", "good-safe"). To test inter-rater reliability, a cross-sectional study of four bariatric surgeons with experience ranging from 4 to 28 years applied the GASS and the global operative assessment of laparoscopic skills (GOALS) to 30 consecutive Roux-en-Y gastric bypass procedure operative videos. Inter-rater reliability was assessed for a simplified dichotomous "safe" versus "unsafe" scoring rubric using Gwet's AC2.

RESULTS

The GASS inter-rater reliability was very high across all six domains (0.88-1.00). The GASS performed comparably to the GOALS inter-rater reliability scores (0.96-1.00). The economy of motion and bimanual dexterity items had the highest percentage of unsafe ratings (9.2% and 5.8%, respectively).

CONCLUSION

The GASS, a novel six-item instrument of general laparoscopic surgical skills, was designed with a simple scoring rubric (poor-safe, adequate-safe, good-safe) to minimize rater burden and focus feedback to trainees and promotion evaluations on safe surgical performance. Initial evaluation of the GASS is promising, demonstrating high inter-rater reliability. Future research will seek to assess the GASS against a broader spectrum of laparoscopic procedures.

Assessment of operative competency for thyroidectomy: Comparison of resident self-assessment vs attending surgeon assessment

BACKGROUND

Self-awareness of skill, essential for progression as a surgeon, has direct bearing on postresidency practice; however, studies have supported that residents achieve self-perceived competence later than believed by program directors. This study compares residents' self-perception of operative competency to attending surgeon's evaluation using Computer Enhanced Visual Learning, a validated online hemithyroidectomy-specific assessment tool.

METHODS

Eleven otolaryngology-Head and Neck Surgery (HNS) residents completed a preoperative module and postoperative survey, later reviewed by an attending surgeon. Eighty-three performances were assessed for inter-rater reliabilities of key surgical steps.

RESULTS

Almost perfect agreement (Kappa = 0.81-1.00) was shown in 11 of 18 parameters. Substantial agreement (Kappa = 0.61-0.80) was demonstrated in the remaining seven parameters.

CONCLUSIONS

Otolaryngology-HNS trainees have high self-awareness of their performance at each step in a hemithyroidectomy. Standardized assessment tools can allow for documentation of procedural performance and serve as guides for improvement. This is the only study to examine otolaryngology-HNS trainees' self-perceived skill compared to an attending surgeon's assessment for hemithyroidectomy.

Wrist Motion Variation between Novices and Experienced Surgeons Performing Simulated Airway Surgery

Objective

To determine whether wrist motion measured by a smartphone application can be used as a performance metric for a simulated airway procedure requiring both wrist and finger dexterity. We hypothesized that this accelerometer application could detect differences between novices and experienced surgeons performing simulated cricothyrotomy.

Setting

Academic medical center.

Study Design

Prospective pilot cohort study.

Methods

Voluntary surgeons and nonsurgeons were recruited. After viewing a training video, smartphones with accelerometer applications were attached to both wrists while subjects performed a cricothyrotomy on a validated task trainer. Procedure time and motion parameters, including average resultant acceleration (ARA), total resultant acceleration (TRA), and suprathreshold acceleration events (STAEs), were collected for dominant and nondominant hands. Subjects were stratified by prior experience. Blinded experts scored each performance using Objective Structured Assessment of Technical Skills (OSATS), and t tests were used to compare performance.

Results

Thirty subjects were enrolled. Median age was 26 years, and 20 subjects were male. In the dominant hand, significant differences were seen between novice and experienced surgeons in TRA (P = .005) and procedure time (P = .006), while no significant differences were seen in STAEs (P = .42) and ARA (P = .33). In the nondominant hand, all variables were significantly different between the 2 groups: STAEs (P = .012), ARA (P = .007), TRA (P = .004), and procedure time (P = .006).

Conclusions

Wrist motion measured by a low-cost smartphone application can distinguish between novice and experienced surgeons performing simulated airway surgery. This tool provides cost-effective and objective performance feedback.

Evaluation of the training capacity of the Spanish Resident Book of Otolaryngology (FORMIR) as an electronic portfolio

BACKGROUND

and objectives We have evaluated the training capacity of the Spanish resident training book as an electronic portfolio to achieve the learning objectives of otorhinolaryngology (ENT) residents.

METHODS

A multi-method qualitative investigation with transversal characteristics, temporal and retrospective guidance was performed on Spanish ENT residents using a structured questionnaire, a semi-structured interview, and a computer application on the FORMIR website.

RESULTS

A 56.5% of ENT-residents specialising in one of the 63 accredited Spanish hospitals between 2009-2012 participated in the study. The results obtained show that the ENT residents who used the e-portfolio were better able to implement self-guided study, were more aware of their learning objectives, fulfilled the training programme more efficiently, identified the causes of learning gaps more clearly, and considered FORMIR in e-portfolio format to be an ideal training tool to replace the resident training book in paper format.

CONCLUSIONS

The ENT residents greatly appreciated the training benefits of FORMIR as an e-portfolio, especially its simple and intuitive interface, the ease and comfort with which they could record their activities, the automatic and numeric feedback on the acquisition of their competencies (which facilitates self-guided learning), its storage capacity for evidence, and its ability to be used as UEMS logbook as well as a standard curriculum vitae. All these features make FORMIR a training and evaluation tool that outperforms similar instruments available to ENT residents. They do not hesitate to identify it as the ideal resident training book for facilitating their specialised training.

Evaluating Coding Accuracy in General Surgery Residents' Accreditation Council for Graduate Medical Education Procedural Case Logs

INTRODUCTION

The Accreditation Council for Graduate Medical Education (ACGME) case log captures resident operative experience based on Current Procedural Terminology (CPT) codes and is used to track operative experience during residency. With increasing emphasis on resident operative experiences, coding is more important than ever. It has been shown in other surgical specialties at similar institutions that the residents' ACGME case log may not accurately reflect their operative experience. What barriers may influence this remains unclear. As the only objective measure of resident operative experience, an accurate case log is paramount in representing one's operative experience. This study aims to determine the accuracy of procedural coding by general surgical residents at a single institution.

METHODS

Data were collected from 2 consecutive graduating classes of surgical residents' ACGME case logs from 2008 to 2014. A total of 5799 entries from 7 residents were collected. The CPT codes entered by residents were compared to departmental billing records submitted by the attending surgeon for each procedure. Assigned CPT codes by institutional American Academy of Professional Coders certified abstract coders were considered the "gold standard." A total of 4356 (75.12%) of 5799 entries were identified in billing records. Excel 2010 and SAS 9.3 were used for analysis. In the event of multiple codes for the same patient, any match between resident codes and billing record codes was considered a "correct" entry. A 4-question survey was distributed to all current general surgical residents at our institution for feedback on coding habits, limitations to accurate coding, and opinions on ACGME case log representation of their operative experience.

RESULTS

All 7 residents had a low percentage of correctly entered CPT codes. The overall accuracy proportion for all residents was 52.82% (range: 43.32%-60.07%). Only 1 resident showed significant improvement in accuracy during his/her training (p = 0.0043). The survey response rate was 100%. Survey results indicated that inability to find the precise code within the ACGME search interface and unfamiliarity with available CPT codes were by far the most common perceived barriers to accuracy. Survey results also indicated that most residents (74%) believe that they code accurately most of the time and agree that their case log would accurately represent their operative experience (66.6%).

CONCLUSION

This is the first study to evaluate correctness of residents' ACGME case logs in general surgery. The degree of inaccuracy found here necessitates further investigation into the etiology of these discrepancies. Instruction on coding practices should also benefit the residents after graduation. Optimizing communication among attendings and residents, improving ACGME coding search interface, and implementing consistent coding practices could improve accuracy giving a more realistic view of residents' operative experience.

A Systematic Review of Simulators in Otolaryngology

Objective

To conduct a systematic review of published articles that describe simulators that could be used in otolaryngology for education, skill acquisition, and/or skill improvement.

Data Sources

Ovid and Embase databases searched July 14, 2011.

Review Methods

Three hundred fifty-three abstracts were independently reviewed by both authors, then 154 eligible articles were reviewed by both authors, and 95 articles were categorized by organ system (eg, otologic); type of simulator (eg, physical, virtual); whether the simulator was a prototype, could be purchased, or was constructed; validation; and level of learning assessment. Discrepancies were resolved by re-review and discussion.

Results

In addition to 11 overview articles, 28 articles described 16 otology simulators, most of which are virtual and prototypes. Ninteen articles described 10 sinus/rhinology simulators; most are virtual surgery simulators and prototypes. Eight articles described 8 oral cavity simulators, and 8 articles described neck simulators. Seventeen articles described 13 bronchoscopy simulators; several are full-body high-technology manikins adapted from other purposes. Five articles described eclectic simulators, including some for learning nontechnical and teamwork skills. Half of the simulators have been validated. Learning levels were often not assessed or assessment was limited to the learners’ perceptions.

Conclusion

A wide variety of simulators are available or under development. Lack of unified validation concepts and limited descriptions restricted our ability to assess model characteristics, availability, and validation. Simulators are emerging as powerful tools to facilitate learning; this review may provide a platform for discussion and refinement of the information reported and analyzed in evaluating simulators.

Extremes in Otolaryngology Resident Surgical Case Numbers

Objective

Understand (1) significance between minimum, mean, and maximum case numbers for otolaryngology resident surgical procedures; (2) weaknesses of Accreditation Council for Graduate Medical Education (ACGME) resident case log system contributing to inaccurate data/case inflation; and (3) when excessive case load may transform education into service.

Study Design

Cross-sectional survey using a national database.

Setting

Academic otolaryngology residency programs.

Subjects/Methods

Review of otolaryngology resident national data reports from ACGME resident case log system performed from 2004 to 2010. Minimum, mean, standard deviation, and maximum values for total number of resident surgeon cases and for specific surgical procedures were compared. Case unbundling, resident data entry habits, and tracked vs untracked Current Procedural Terminology ( CPT) codes were reviewed.

Results

Mean total number of resident surgeon cases remained constant (1699.5 ± 424.2 to 1772.2 ± 517). Minimum total number of cases ranged from 730 to 811 (approaching 2 standard deviations below the mean). Zero was reported as the minimum case number for some procedures. Maximum for total number of cases increased (3559 to 4857) and surpassed the mean by 4 to 5 standard deviations. Some procedures have maximums greater than 11 standard deviations above the mean. The authors identified untracked CPT codes and variations in resident case log documentation habits.

Conclusions

Large differences between the minimum, mean, and maximum resident surgeon case numbers exist. Establishing minimum case number requirements for otolaryngology residents should be considered. Educational benefit derived from excessive case load is unclear. Critical examination of the ACGME resident case log system and resident documentation habits is needed to improve accuracy of reporting.

Construct validity of a simulator for myringotomy with ventilation tube insertion

Objectives: To establish construct validity of an anatomic model as a simulator for myringotomy with ventilation tube insertion and to assess its subjective appeal.

Study Design: Cross-sectional, repeated-measures comparative evaluation of simulator.

Setting: University academic otolaryngology residency program.

Subjects and Methods: Using an anatomic model of the human auricle, ear canal, eardrum, and middle ear space, 18 otolaryngologists of various levels of training performed 10 timed procedures: myringotomy with ventilation tube insertion. Errors were recorded, and participants reported the quality of their experience.

Results: Both time-to-completion and errors per trial discriminated novices from non-novice participants; novices (02:23, 95% confidence interval [CI], 01:42-03:04) were 3.6 times slower than non-novices (00:39, 95% CI, 00:35-00:43) and 6.5 times more error prone (novices 2.16 errors/trial, 95% CI, 1.68-2.64; non-novices 0.33 errors/trial, 95% CI, 0.21-0.45). Errors were strongly correlated with prior surgical experience. All participants required more time to complete the first trial, and their performance stabilized thereafter. Overall, the simulation was perceived as a valuable experience.

Conclusion: Our model is a valid platform for simulating myringotomy with ventilation tube insertion. The model discriminates novices from non-novices, has a learning curve, and is perceived to be a valuable and realistic teaching tool by users.

Assessing vaginal surgical skills using video motion analysis

OBJECTIVE

To demonstrate the feasibility of using video motion analysis to quantitate a key step of vaginal hysterectomy and define measurable differences between novice and experienced surgical trainees during vaginal hysterectomy.

METHODS

Analyses focused on clamping, transecting, and suturing the left uterosacral ligament. Using a cutoff of 25, trainees were grouped as experienced (n=14) and novice (n=9) based on the total number of vaginal hysterectomies performed by each trainee. Contrasting-groups analysis was used to determine cutoff values that separated novices from experts.

RESULTS

Novice trainees took longer (112 seconds compared with 84 seconds) and had greater cumulative translational motion (92 cm compared with 49 cm, P=.05) while performing the task. Experienced trainees placed the Heaney clamp closer to a right angle to the vertical axis (experienced 96 degrees compared with novice 109 degrees , P=.03) while passing the needle through the uterosacral ligament. Trainees move from novice to experts when the steps occur in 112 or fewer seconds, cumulative translational motion is at or less than 75 cm, and the angle between the clamp to bladder retractor is at or below 105 degrees .

CONCLUSION

Video motion analysis is a feasible technique to quantify and compare surgical skills objectively during vaginal surgery. There are measurable differences between novice and more experienced surgical trainees performing vaginal hysterectomy that can be quantified using motion analysis.

LEVEL OF EVIDENCE

III.

The rhinology experience in otolaryngology residency: a survey of chief residents

OBJECTIVES

The development of rhinology as a distinct subspecialty is based on recent advances in the research, clinical, and surgical aspects of the field. The impact of this evolution on the rhinologic experience in otolaryngology residency programs is not currently understood.

METHODS

An anonymous, web-based survey of chief residents in otolaryngology residency programs was performed. Participants were asked to rate their residency experience in rhinology in terms of didactics, research opportunities, mentorship availability, clinical care, and surgical experience on an ordinal 5-point Likert scale with higher scores representing more favorable responses.

RESULTS

Forty-eight respondents completed the survey, yielding a response rate of 17.6%. The overall experience in rhinology was reported as positive (4.3 points). The individual items with statistically lower scores included availability of basic science research (P = .04), comfort level with the medical management of unusual rhinologic issues (P = .0006), and comfort level with advanced sinus procedures (P < .0001). However, overall ratings were high, and even the poorer-scoring items were still rated favorably. In programs with rhinology fellowships, basic science opportunities were reported as being higher (P = .04) than programs without fellowships. There were no other statistically significant differences in programs with rhinology fellowships. There was no correlation between responses and future career choice.

CONCLUSION

The overall residency training experience in rhinology appears to be positive, and presence of a rhinology fellowship appears not to have a negative impact on resident rhinologic training.