Assessing competency in surgery: where to begin?

Competency Assessment of General Surgery Trainees: A Perspective From the Global South, in a CBME-Naive Context

OBJECTIVE

Before proceeding with local implementation of competency-based medical education-related assessment practices designed and evaluated in the Global North, we sought to challenge the assumption that this would be perceived as both necessary and acceptable in our context where training and assessment is based on a traditional, knowledge-focused approach. The aim of this study was to determine the perspectives of general surgery trainees and consultants towards the assessment of competence, how this has been achieved previously, and how it should be performed in the future at the University of Cape Town (UCT), South Africa.

DESIGN

Semi-structured interviews were conducted with consultants and trainees. Interviews were transcribed and then analyzed using a Reflexive Thematic Analysis approach.

SETTING AND PARTICIPANTS

Ten consultants (5 senior and 5 junior) and 10 trainees (5 South African and 5 international) from the Division of General Surgery at UCT in August 2022.

RESULTS

Five unique themes were developed: (1) Assessment of competence is essential, (2) competence includes multiple domains of practice, (3) a surgeon must be able to operate, (4) previously used methods were inadequate to assess competence, and (5) frequent assessment with feedback is desired. The themes were considered in the context of Situated Learning Theory, particularly Communities of Practice and their role in the training for, and authentic assessment of, competence in general surgery trainees.

CONCLUSIONS

Participants described a need to develop and implement a new competency assessment program for general surgery training in this context, which is aligned with described competency-based medical education principles. Thoughtful integration of the formative and summative use of direct observation in the workplace, with a clear emphasis on procedural ability and the provision of high-quality feedback, may enhance the successful implementation of a strategy for competency-based assessment in general surgery training programs.

Formative objective structured clinical examination with immediate feedback improves surgical clerks' self-confidence and clinical competence

OBJECTIVE

Clerkship is crucial for fourth-year medical students before entering the clinical environment. However, lack of confidence impairs clerks' performance during the clinical rotation. We assess the impact of formative Objective Structured Clinical Examination (OSCE) with immediate feedback on surgical clerks' self-confidence and clinical competence.

METHODS

This is a prospective randomized controlled study. Thirty-eight fourth-year medical students starting their surgical clerkship were randomly divided into the control group (n = 19) and the OSCE group (n = 19), where an extra 6-station formative OSCE was given prior to the surgical rotation with immediate feedback on the participant's performance. Self-confidence assessment (SCA) was collected from each participant before, right after the formative OSCE and one month later. Clinical competence was assessed using a mini-clinical evaluation exercise (mini-CEX) with a case of acute abdominal pain and direct observation of procedural skills (DOPS) with incision and suture one month later.

RESULTS

The SCAs were significantly improved in the OSCE group right after the training, and a month later, compared to the control group. The mini-CEX score was significantly higher in the OSCE group compared to the control group, but not the DOPS score of incision and suture.

CONCLUSION

The formative OSCE with immediate feedback could significantly enhance surgical clerks' self-confidence and their clinical competence when taking the history, performing the physical examination, and in clinical reasoning; however, the formative OSCE did not improve their dexterity in performing the procedural skills.

Assessing clinical reasoning ability in fourth-year medical students via an integrative group history-taking with an individual reasoning activity

BACKGROUND

The most important factor in evaluating a physician's competence is strong clinical reasoning ability, leading to correct principal diagnoses. The process of clinical reasoning includes history taking, physical examinations, validating medical records, and determining a final diagnosis. In this study, we designed a teaching activity to evaluate the clinical reasoning competence of fourth-year medical students.

METHODS

We created five patient scenarios for our standardized patients, including hemoptysis, abdominal pain, fever, anemia, and chest pain. A group history-taking with individual reasoning principles was implemented to teach and evaluate students' abilities to take histories, document key information, and arrive at the most likely diagnosis. Residents were trained to act as teachers, and a post-study questionnaire was employed to evaluate the students' satisfaction with the training activity.

RESULTS

A total of 76 students, five teachers, and five standardized patients participated in this clinical reasoning training activity. The average history-taking score was 64%, the average key information number was 7, the average diagnosis number was 1.1, and the average correct diagnosis rate was 38%. Standardized patients presenting with abdominal pain (8.3%) and anemia (18.2%) had the lowest diagnosis rates. The scenario of anemia presented the most difficult challenge for students in history taking (3.5/5) and clinical reasoning (3.5/5). The abdominal pain scenario yielded even worse results (history taking: 2.9/5 and clinical reasoning 2.7/5). We found a correlation in the clinical reasoning process between the correct and incorrect most likely diagnosis groups (group history-taking score, p = 0.045; key information number, p = 0.009 and diagnosis number, p = 0.004). The post-study questionnaire results indicated significant satisfaction with the teaching program (4.7/5) and the quality of teacher feedback (4.9/5).

CONCLUSIONS

We concluded that the clinical reasoning skills of fourth-year medical students benefited from this training course, and the lower correction of the most likely diagnosis rate found with abdominal pain, anemia, and fever might be due to a system-based teaching modules in fourth-year medical students; cross-system remedial reasoning auxiliary training is recommended for fourth-year medical students in the future.

Becoming virtually real: using the Virtual Environment for Radiotherapy Training (VERT™) platform for the summative assessment of performance in a palliative radiotherapy treatment technique

Introduction:

Direct observation in the radiotherapy clinic remains the primary method for the assessment of procedural skills. But with increasing automation and reductions in clinical placement time during the recent pandemic, the role of summative, simulation-based assessment is being revisited. The aim of this article is to share early experiences of using Virtual Environment for Radiotherapy Training (VERT) in the assessment of a palliative, parallel pair, external beam treatment delivery technique in a new pre-registration undergraduate programme.

Methods:

Eight first-year students completed a campus-based individual virtual assessment using virtual patient plans representing metastatic disease in the brain and pelvis and late-stage primary lung tumours. Performance was logged on a 25-item checklist for pre and post-procedure checks and treatment delivery tasks.

Results:

All eight students participated in the assessment with seven students (87·5%) achieving a pass grade at first attempt. The course team observed that participants demonstrated a range of skills and abilities and were able to compare and contrast individual approaches to patient positioning. Specific feed-forward action points were also highlighted as areas for students’ to focus on during their second placement. The project also identified logistical benefits for assessment teams.

Conclusions:

While these experiences are a single snapshot, a VERT simulation-based summative assessment is feasible and identified benefits included controlled observation and structured feedback on individual performance and scheduling pressure reductions for clinical teams. However, more work is needed to determine the psychometric qualities and predictability of performance in more complex techniques.

Analysis of Surgical Volume in Military Medical Treatment Facilities and Clinical Combat Readiness of US Military Surgeons

Importance

Low surgical volume in the US Military Health System (MHS) has been identified as a challenge to military surgeon readiness. The Uniformed Services University of Health Sciences, in partnership with the American College of Surgeons, developed the Knowledge, Skills, and Abilities (KSA) Clinical Readiness Program that includes a tool for quantifying the clinical readiness value of surgeon workload, known as the KSA metric.

Objective

To describe changes in US military general surgeon procedural volume and readiness using the KSA metric.

Design, Setting, and Participants

This cohort study analyzed general surgery workload performed across the MHS, including military and civilian facilities, between fiscal year 2015 and 2019 and the calculated KSA metric value. The surgeon-level readiness among military general surgeons was calculated based on the KSA metric readiness threshold. Data were obtained from TRICARE, the US Department of Defense health insurance product.

Main Outcomes and Measures

The main outcomes were general surgery procedural volumes and the KSA metric point value of those procedures across the MHS as well as the number of military general surgeons meeting the KSA metric readiness threshold. Aggregate facility and regional market-level claims data were used to calculate the procedural volumes and KSA metric readiness value of those procedures. Annual adjusted KSA metric points earned were used to determine the number of individual US military general surgeons meeting the readiness threshold.

Results

The number of general surgery procedures generating KSAs in military hospitals decreased 25.6%, from 128 377 in 2015 to 95 461 in 2019, with a 19.1% decrease in the number of general surgeon KSA points (from 7 155 563 to 5 790 001). From 2015 to 2019, there was a 3.2% increase in both the number of procedures (from 419 980 to 433 495) and KSA points (from 21 071 033 to 21 748 984) in civilian care settings. The proportion of military general surgeons meeting the KSA metric readiness threshold decreased from 16.7% (n = 97) in 2015 to 10.1% (n = 68) in 2019.

Conclusions and Relevance

This study noted that the number of KSA metric points and procedural volume in military hospitals has been decreasing since 2015, whereas both measures have increased in civilian facilities. The findings suggest that loss of surgical workload has resulted in further decreases in military surgeon readiness and may require substantial changes in patient care flow in the MHS to reverse the change.

Best Approaches to Evaluation and Feedback in Post-Graduate Medical Education

PURPOSE OF REVIEW

The objectives of this literature review are to appraise current approaches and assess new technologies that have been utilized for evaluation and feedback of residents, with focus on surgical trainees.

RECENT FINDINGS

In 1999, the Accreditation Council for Graduate Medical Education introduced the Milestone system as a tool for summative evaluation. The organization allows individual program autonomy on how evaluation and feedback are performed. In the past, questionnaire evaluations and informal verbal feedback were employed. However, with the advent of technology, they have taken a different shape in the form of crowdsourcing, mobile platforms, and simulation. Limited data is available on new methods but studies show promise citing low cost and positive impact on resident education. No one "best approach" exists for evaluation and feedback. However, it is apparent that a multimodal approach that is based on the ACGME Milestones can be effective and aid in guiding programs.

Educational Impact Drives Feasibility of Implementing Daily Assessment in the Workplace

Construct: Authors investigated the perspectives of stakeholders on feasibility elements of workplace-based assessments (WBA) with varying designs. Background: In the transition to competency-based medical education, WBA are taking a more prominent role in assessment programs. However, the increased demand for WBA leads to new challenges for implementing suitable WBA tools with published validity evidence, while also being feasible and useful in practice. Despite the availability of published WBA tools, implementation does not necessarily occur; a more fulsome understanding of the perspectives of stakeholders who are ultimately the end-users of these tools, as well as the system factors that both deter or support their use, could help to explain why evidence-based assessment tools may not be incorporated into residency programs. Approach: We examined the perspectives of two groups of stakeholders, surgical teachers and resident learners, during an assessment intervention that varied the assessment tools while keeping the assessment process constant. We chose diverse exemplars from published assessment tools that each represented a different response format: global rating scales, step-by-step surgical rubrics, and an entrustability scale. The primary purpose was to investigate how stakeholders are impacted by WBA tools with varying response formats to better understand their feasibility for assessment of cataract surgery. Secondarily, we were able to explore the culture of assessment in cataract surgery education including stakeholders' perceptions of WBA unrelated to assessment form design. Semi-structured interviews with teachers and a focus group with the residents enabled discussion of their perspectives on dimensions of the tools such as acceptability, demand, implementation, practicality, adaptation, and integration. Findings: Three themes summarize teachers' and residents' experiences with the assessment tools: (1) Feedback is the priority; (2) Forms informing coaching; and (3) Forcing the conversation. The tools helped to facilitate the feedback conversation by serving as a reminder to initiate the conversation, a framework to structure the conversation, and a memory aid for providing detailed feedback. Surgical teachers preferred the assessment tool with a design that best aligned with their approach to teaching and how they wanted to provide feedback. Orientation to the tools, combined with established remediation pathways, may help preceptors to better use assessment tools and improve their ability to give critical feedback. Conclusions: Feedback, more so than assessment, dominated the comments provided by both teachers and residents after using the various WBA tools. Our typical assessment design efforts focus on the creation or selection of a robust assessment tool according to good design and measurement principles, but the current findings would encourage us to also prioritize the coaching relationship and include efforts to design WBA tools to function as a mediator to augment teaching, learning, and feedback exchange within that relationship in the workplace.

Development of the Diabetic Wound Assessment Learning Tool (DiWALT) and validity evidence

OBJECTIVE

Diabetic foot wounds account for up to one-third of diabetes-related health care expenditure and are the greatest cause of extremity amputation in Canada. Physicians encounter patients with such wounds in all specialties, particularly as generalists in medical wards and emergency departments. However, there is a dearth of literature on the optimal way to teach and to assess the management of these patients. Given the importance of assessment for learning in the shift toward competency-based medical education, we aimed to develop an assessment tool and to build validity evidence for its use in this context.

METHODS

A consensus process involving nine Canadian experts in diabetic wound management was used to develop the Diabetic Wound Assessment Learning Tool (DiWALT) items and two 10-minute simulation-based testing scenarios. The simulators used were modified from commercially available models to serve the testing scenarios. Validity evidence for the DiWALT was subsequently evaluated by assessing 24 physician participants' performance during the two scenarios. All participants were novices (<50 cases managed). Two assessors independently rated participants using the DiWALT. Evidence was organized using Kane's validity framework and included Cronbach α for interitem consistency as well as test-retest and inter-rater reliability using the intra-class correlation coefficient (ICC).

RESULTS

Cronbach α was 0.92, implying high internal consistency. Test-retest reliability was also excellent with ICC of 0.89 (confidence interval [CI], 0.76-0.95) for single measures and ICC of 0.94 (CI, 0.86-0.98) for average measures. Inter-rater reliability was fair for single measures with ICC of 0.68 (CI, 0.65-0.71) and good for average measures with ICC of 0.81 (CI, 0.79-0.83).

CONCLUSIONS

These results demonstrate that the DiWALT consistently and reliably evaluates competence in diabetic wound management during simulated cases using a small, homogeneous sample of physicians. Further work is necessary to quantify sources of error in the assessment scores, to establish validity evidence when it is used to assess larger and more heterogeneous participants, and to identify how well the DiWALT differentiates between different experience levels.

Educational Alternatives for the Maintenance of Educational Competencies in Surgical Training Programs Affected by the COVID-19 Pandemic

Along with the socio-economic burden the COVID-19 pandemic carried, the strain it brought upon our health care system is unparalleled. In an attempt to conserve much needed personal protective equipment (PPE) as well as to free up available hospital beds to accommodate the significant influx of COVID-19 patients, many elective surgical cases were essentially put on hold. Furthermore, to taper the spread of this highly contagious virus and to protect the medical staff, surgical clinics were limited to urgent care that could not be managed through virtual platforms. Surgical trainees, such as residents and fellows, who solemnly rely on clinical and surgical exposure to hone their operative and clinical skills, were evidently left deprived. As the pandemic rapidly progressed, medical staff in the emergency departments and what is now known as the COVID wards and COVID ICUs quickly became overwhelmed and overworked. This new reality required surgical trainees to rapidly redeploy to help meet the rising hospital needs. With no clear end to this pandemic, surgical trainees worry they will not reach the appropriate milestones and acquire the amount of surgical experience required to become competent surgeons. As a result, a rapid solution should be found and applied to remedy this newly created gap in surgical education. The measures we recommend include access to regular webinars from world-renowned experts, increased implementation of surgical simulation, selective redeployment of residents to favor level-appropriate learning opportunities and lastly, the active participation of trainees in telemedicine with an increase in surgical exposure as soon as the restrictions are lifted.

Development and evaluation of the General Surgery Objective Structured Assessment of Technical Skill (GOSATS)

Background

Technical skill acquisition is important in surgery specialty training. Despite an emphasis on competency-based training, few tools are currently available for direct technical skills assessment at the completion of training. The aim of this study was to develop and validate a simulated technical skill examination for graduating (postgraduate year (PGY)5) general surgery trainees.

Methods

A simulated eight-station, procedure-based general surgery technical skills examination was developed. Board-certified general surgeons blinded to the level of training rated performance of PGY3 and PGY5 trainees by means of validated scoring. Cronbach's α was used to calculate reliability indices, and a conjunctive model to set a pass score with borderline regression methodology. Subkoviak methodology was employed to assess the reliability of the pass–fail decision. The relationship between passing the examination and PGY level was evaluated using χ2 analysis.

Results

Ten PGY3 and nine PGY5 trainees were included. Interstation reliability was 0·66, and inter-rater reliability for three stations was 0·92, 0·97 and 0·76. A pass score of 176·8 of 280 (63·1 per cent) was set. The pass rate for PGY5 trainees was 78 per cent (7 of 9), compared with 30 per cent (3 of 10) for PGY3 trainees. Reliability of the pass–fail decision had an agreement coefficient of 0·88. Graduating trainees were significantly more likely to pass the examination than PGY3 trainees (χ2 = 4·34, P = 0·037).

Conclusion

A summative general surgery technical skills examination was developed with reliability indices within the range needed for high-stakes assessments. Further evaluation is required before the examination can be used in decisions regarding certification.

The Resident Report Card: A Tool for Operative Feedback and Evaluation of Technical Skills

BACKGROUND

Competency-based medical education surgical curriculums will require frequent, recorded trainee performance evaluations. It is our hypothesis that written feedback after each operation can be used to chart surgical progress, can identify underperforming trainees, and will prove beneficial for resident learning.

METHODS

The resident report card (RRC) is an online, easy-to-use evaluation tool designed to facilitate the creation and distribution of resident technical assessments. RRC data were collected from urologic trainees and analyzed using ANOVA and post hoc testing to confirm our hypothesis. A standardized survey was sent to residents, gauging their views on the RRC.

RESULTS

Over a 5-y period, 958 RRCs with the resident listed as the primary operator were collected across 29 different procedures. Resident cohort and individual performance scores stratified by postgraduate year (PGY) were shown to significantly improve when all procedures (cohort, 6.5 ± 1.9 [PGY-1] to 9.1 ± 1.0 [PGY-5]; individual [resident M], 8.8 ± 1.8 [PGY-3] to 9.4 ± 0.7 [PGY-5], P < 0.01) and specific procedures (laparoscopic donor nephrectomy: cohort, 7.3 ± 1.3 [PGY-3] to 8.9 ± 1.0 [PGY-5]; individual [resident I], 7.2 ± 1.3 [PGY-3] to 9.5 ± 0.6 [PGY-5], P < 0.01) were analyzed. Individual residents were able to be compared to their own peer group and to the average scores across all evaluated residents. Surveyed residents were overwhelmingly positive about the RRC.

CONCLUSIONS

The RRC adds further evidence to the fact that standardized, formative, and timely assessment can capture trainee performance over time and against comparator cohorts in an acceptable format to residents and academic training programs.

Augmented Reality to Improve Surgical Simulation. Lessons Learned Towards the Design of a Hybrid Laparoscopic Simulator for Cholecystectomy

Hybrid surgical simulators based on Augmented Reality (AR) solutions benefit from the advantages of both the box trainers and the Virtual Reality simulators. This paper reports on the results of a long development stage of a hybrid simulator for laparoscopic cholecystectomy that integrates real and the virtual components. We first outline the specifications of the AR simulator and then we explain the strategy adopted for implementing it based on a careful selection of its simulated anatomical components, and characterized by a real-time tracking of both a target anatomy and of the laparoscope. The former is tracked by means of an electromagnetic field generator, while the latter requires an additional camera for video tracking. The new system was evaluated in terms of AR visualization accuracy, realism and hardware robustness. Obtained results show that the accuracy of AR visualization is adequate for training purposes. The qualitative evaluation confirms the robustness and the realism of the simulator. The AR simulator satisfies all the initial specifications in terms of anatomical appearance, modularity, reusability, minimization of spare parts cost, and ability to record surgical errors and to track in real-time the Calot's triangle and the laparoscope. The proposed system could be an effective training tool for learning the task of identification and isolation of Calot's triangle in laparoscopic cholecystectomy. Moreover, the presented strategy could be applied to simulate other surgical procedures involving the task of identification and isolation of generic tubular structures, such as blood vessels, biliary tree and nerves, which are not directly visible.

Effect of rater training on the reliability of technical skill assessments: a randomized controlled trial

BACKGROUND

Rater training improves the reliability of observational assessment tools but has not been well studied for technical skills. This study assessed whether rater training could improve the reliability of technical skill assessment.

METHODS

Academic and community surgeons in Royal College of Physicians and Surgeons of Canada surgical subspecialties were randomly allocated to either rater training (7-minute video incorporating frame-of-reference training elements) or no training. Participants then assessed trainees performing a suturing and knot-tying task using 3 assessment tools: a visual analogue scale, a task-specific checklist and a modified version of the Objective Structured Assessment of Technical Skill global rating scale (GRS). We measured interrater reliability (IRR) using intraclass correlation type 2.

RESULTS

There were 24 surgeons in the training group and 23 in the no-training group. Mean assessment tool scores were not significantly different between the 2 groups. The training group had higher IRR than the no-training group on the visual analogue scale (0.71 v. 0.46), task-specific checklist (0.46 v. 0.33) and GRS (0.71 v. 0.61). However, confidence intervals were wide and overlapping for all 3 tools.

CONCLUSION

For education purposes, the reliability of the visual analogue scale and GRS would be considered "good" for the training group but "moderate" for the no-training group. However, a significant difference in IRR was not shown, and reliability remained below the desired level of 0.8 for high-stakes testing. Training did not significantly improve assessment tool reliability. Although rater training may represent a way to improve reliability, further study is needed to determine effective training methods.

The Dimensionless Squared Jerk: An Objective Parameter That Improves Assessment of Hand Motion Analysis during Simulated Shoulder Arthroscopy

PURPOSE

Attempts to quantify hand movements of surgeons during arthroscopic surgery faced limited progress beyond motion analysis of hands and/or instruments. Surrogate markers such as procedure time have been used. The dimensionless squared jerk (DSJ) is a measure of deliberate hand movements. This study tests the ability of DSJ to differentiate novice and expert surgeons (construct validity) whilst performing simulated arthroscopic shoulder surgical tasks.

METHODS

Six residents (novice group) and six consultants (expert group) participated in this study. Participants performed three validated tasks sequentially under the same experimental setup (one performance). Each participant had ten performances assessed. Hand movements were recorded with optical tracking system. The DSJ, time taken, total path length, multiple measures of acceleration, and number of movements were recorded.

RESULTS

There were significant differences between novices and experts when assessed using time, number of movements with average and minimal acceleration threshold, and DSJ. No significant differences were observed in maximum acceleration, total path length, and number of movements with 10m/s2 acceleration threshold.

CONCLUSION

DSJ is an objective parameter that can differentiate novice and expert surgeons' simulated arthroscopic performances. We propose DSJ as an adjunct to more conventional parameters for arthroscopic surgery skills assessment.

Residents' Impressions of the Impact of Advanced Practice Providers on Surgical Training

BACKGROUND

To ensure that residents are appropriately trained in the era of the 80-hour work-week, training programs have restructured resident duties and hired advanced practice providers (APPs). However, the effect of APPs on surgical training remains unknown.

STUDY DESIGN

We created a survey using a modified Delphi technique to examine the interaction between residents and APPs across practice settings (inpatient, outpatient, and operating room). We identified the following domains: administrative tasks, clinical experience, operative experience, and overall impressions. We administered the survey to residents across 7 surgical training programs at a single institution and assessed internal reliability with Cronbach's α.

RESULTS

Fifty residents responded (77% participation rate). The majority reported APPs reduced the time spent on administrative tasks, such as completing documentation (96%) and answering pages (88%). For clinical experience, 62% of residents thought that APPs had no impact on the amount of time spent evaluating consult patients, and 80% reported no difference in the number of bedside procedures performed. However, 77% of residents reported a reduction in the time spent counseling patients. When APPs worked in the inpatient setting, 90% of residents reported leaving the operating room less frequently to manage patients. When APPs were present in the operating room, 34% of residents thought they were less likely to perform key parts of the case. Cronbach's α showed excellent to good reliability for the administrative tasks (0.96), clinical experience (0.76), operative experience (0.69), and overall impressions (0.66) domains.

CONCLUSIONS

Most residents report that the integration of APPs has decreased the administrative burden. The reduction in patient counseling might be an unrecognized and unintended consequence of implementing APPs. The perceived effect on operative experience is dependent on the role of the APPs.

Automatically rating trainee skill at a pediatric laparoscopic suturing task

BACKGROUND

Minimally invasive surgeons must acquire complex technical skills while minimizing patient risk, a challenge that is magnified in pediatric surgery. Trainees need realistic practice with frequent detailed feedback, but human grading is tedious and subjective. We aim to validate a novel motion-tracking system and algorithms that automatically evaluate trainee performance of a pediatric laparoscopic suturing task.

METHODS

Subjects (n = 32) ranging from medical students to fellows performed two trials of intracorporeal suturing in a custom pediatric laparoscopic box trainer after watching a video of ideal performance. The motions of the tools and endoscope were recorded over time using a magnetic sensing system, and both tool grip angles were recorded using handle-mounted flex sensors. An expert rated the 63 trial videos on five domains from the Objective Structured Assessment of Technical Skill (OSATS), yielding summed scores from 5 to 20. Motion data from each trial were processed to calculate 280 features. We used regularized least squares regression to identify the most predictive features from different subsets of the motion data and then built six regression tree models that predict summed OSATS score. Model accuracy was evaluated via leave-one-subject-out cross-validation.

RESULTS

The model that used all sensor data streams performed best, achieving 71% accuracy at predicting summed scores within 2 points, 89% accuracy within 4, and a correlation of 0.85 with human ratings. 59% of the rounded average OSATS score predictions were perfect, and 100% were within 1 point. This model employed 87 features, including none based on completion time, 77 from tool tip motion, 3 from tool tip visibility, and 7 from grip angle.

CONCLUSIONS

Our novel hardware and software automatically rated previously unseen trials with summed OSATS scores that closely match human expert ratings. Such a system facilitates more feedback-intensive surgical training and may yield insights into the fundamental components of surgical skill.

Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation

PURPOSE

The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model.

METHODS

Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review.

RESULTS

The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05).

CONCLUSIONS

The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching.

CLINICAL RELEVANCE

As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.

Performance Assessment for Mastoidectomy

Objective The aim of this report is to provide a review of the current literature for assessment of performance for mastoidectomy, to identify the current assessment tools available in the literature, and to summarize the evidence for their validity. Data Sources The MEDLINE database was accessed via PubMed. Review Methods Inclusion criteria consisted of English-language published articles that reported use of a mastoidectomy performance assessment tool. Studies ranged from 2007 to November 2015 and were divided into 2 groups: intraoperative assessments and those performed with simulation (cadaveric laboratory or virtual reality). Studies that contained specific reliability analyses were also highlighted. For each publication, validity evidence data were analyzed and interpreted according to conceptual definitions provided in a recent systematic review on the modern framework of validity evidence. Conclusions Twenty-three studies were identified that met our inclusion criteria for review, including 4 intraoperative objective assessment studies, 5 cadaveric studies, 10 virtual reality simulation studies, and 4 that used both cadaveric assessment and virtual reality. Implications for Practice A review of the literature revealed a wide variety of mastoidectomy assessment tools and varying levels of reliability and validity evidence. The assessment tool developed at Johns Hopkins possesses the most validity evidence of those reviewed. However, a number of agreed-on specific metrics could be integrated into a standardized assessment instrument to be used nationally. A universally agreed-on assessment tool will provide a means for developing standardized benchmarks for performing mastoid surgery.

Assessment of Surgical Competence: Development and Validation of Rating Scales Used for Laparoscopic Supracervical Hysterectomy

OBJECTIVE

To develop a procedure-specific rating scale for laparoscopic supracervical hysterectomy (LSH), and to compare the construct validity and reliability with a general rating scale in laparoscopic surgery, global operative assessment of laparoscopic skills (GOALS).

DESIGN

Prospective interobserver study. In collaboration with an expert group, we developed the procedure-specific rating scale, competence assessment tool for laparoscopic supracervical hysterectomy (CAT-LSH). LSH was performed by gynecologists with different levels of surgical competence levels (13 procedures were performed by inexperienced trainees, 13 by intermediate experienced, and 15 by laparoscopic experts). All procedures were video-recorded. Surgical performance was evaluated in all procedures using both CAT-LSH and GOALS by the surgical assistant, as well as by 2 blinded observers evaluating the video recordings.

SETTING

University teaching hospital.

PARTICIPANTS

Laparoscopic experts, consultants and gynecological registrars from the Department of Gynecology.

RESULTS

There were significant differences between the 3 proficiency groups in both the rating scales. Mean GOALS score evaluated by the operating assistant and the 2 observers were for inexperienced surgeons 16.4 vs. 13.6 (p < 0.01), for surgeons with intermediate experienced 22.6 vs. 19.5 (p < 0.05) and for expert surgeons 26.1 vs. 22.4 (p < 0.01), respectively. Corresponding results for the CAT-LSH scores were 41.0/34.6 (p < 0.01), 49.2/43.1 (p < 0.01), and 58.7/51.1 (p < 0.01), respectively. The interrater reliability measured by the interclass correlation coefficient between the surgical assistant and the 2 blinded observers for GOALS and CAT-LSH were 0.71 and 0.75, respectively.

CONCLUSIONS

The GOALS and CAT-LSH appear to have construct validity and high interrater reliability. Assessment of surgical competence during LSH is feasible in daily practice with objective rating scales like CAT-LSH and GOALS.

Evaluating Intraoperative Laparoscopic Skill: Direct Observation Versus Blinded Videotaped Performances

The Global Operative Assessment of Laparoscopic Skill (GOALS) has been shown to meet high standards for direct observation. The purpose of this study was to investigate the reliability and validity of GOALS when applied to blinded, videotaped performances. Five novice surgeons and 5 experienced surgeons were each evaluated by 2 observers during a laparoscopic cholecystectomy. Subsequently, 4 laparoscopists (V1 to V4) evaluated the videotaped procedures using GOALS. Two of the raters (V1 and V3) had prior experience using GOALS. The interrater reliabilities between video raters (VRs) and between VRs and direct raters (DRs) were calculated using the intraclass correlation coefficient. Construct validity was assessed using 2-way analysis of variance. Interrater reliability between the 4 VRs and the 2 DRs was 0.72. The intraclass correlation coefficient for the 4 VRs was 0.68 and for each VR compared with the mean DR was 0.86, 0.39, 0.94, and 0.76, respectively. All raters, except V2, differentiated between novice and experienced groups ( P values ranged from .01 to .05). These data suggest that GOALS can be used to assess laparoscopic skill based on videotaped performances but that rater training may play an important role in ensuring the reliability and validity of the instrument. Experience with the tool in the operating room may improve the reliability of video rating and could be of value in training evaluators.

Training and Assessment in Pediatric Endoscopy

A key aspect of pediatric gastroenterology practice is the ability to perform endoscopy procedures safely, effectively, and efficiently. Similar to adult endoscopy, performance of pediatric endoscopy requires the acquisition of related technical, cognitive, and integrative competencies to effectively diagnose and manage gastrointestinal disorders in children. However, the distinctive requirements of pediatric patients and their families and the differential spectrum of disease highlight the need for a pediatric-specific training curriculum and assessment framework to ensure endoscopic procedures are performed safely and successfully in children. This review outlines the current state of evidence as it pertains to pediatric endoscopy training and assessment.

Continued Validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): Use in the Simulated Environment

CONSTRUCT: The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) is a 9-item surgical evaluation tool designed to assess technical competence in surgical trainees using behavioral anchors.

BACKGROUND

The initial development of the O-SCORE produced evidence for valid results. Further work is required to determine if the use of a single surgeon or an unblinded rater introduces bias. In addition, the relationship of the O-SCORE to other currently used technical assessment tools should be explored to provide validity evidence related to the relationship to other measures. We have designed this project to provide continued validity evidence for the O-SCORE related to these two issues.

APPROACH

Nineteen residents and 2 staff Orthopedic Surgeons from the University of Ottawa volunteered to participate in a 2-part OSCE style station. Participants completed a written questionnaire followed by a videotaped 10-minute simulated open reduction and internal fixation of a midshaft radius fracture. Videos were rated individually by 2 blinded staff orthopedic surgeons using an Objective Structured Assessment of Technical Skills (OSATS) global rating scale, an OSATS checklist, and the O-SCORE in random order.

RESULTS

O-SCORE results appeared sensitive to surgical training level even when raters were blinded. In addition, strong agreement between two independent observers using the O-SCORE suggests that the measure captures a performance easily recognized by surgical observers. Ratings on the O-SCORE also were strongly associated with global ratings on the currently most validated technical evaluation tool (OSATS). Collectively, these results suggest that the O-SCORE generates accurate, reproducible, and meaningful results when used in a randomized and blinded fashion, providing continued validity evidence for using this tool to evaluate surgical trainee competence.

CONCLUSIONS

The O-SCORE was able to differentiate surgical trainee level using blinded raters providing further evidence of validity for the O-SCORE. There was strong agreement between two independent observers using the O-SCORE. Ratings on the O-SCORE also demonstrated equivalence to scores on the most validated technical evaluation tool (OSATS). These results suggest that the O-SCORE demonstrates accurate and reproducible results when used in a randomized and blinded fashion providing continued validity evidence for this tool in the evaluation of surgical competence in the trainees.

Teaching Vaginal Surgery to Junior Residents: Initial Validation of 3 Novel Procedure-Specific Low-Fidelity Models

OBJECTIVE

Competency-based surgical education relies on operative models to teach surgical skills within a curriculum. Low fidelity simulation has been shown to improve surgical performance. Our objectives were: to develop procedure-specific models to teach anterior repair (AR), posterior repair (PR), and vaginal hysterectomy (VH) to junior residents; to establish model reliability and validity.

DESIGN

Residents were randomized to control (no training) and intervention (model training) groups. They were filmed while performing a series of tasks. Experts were also filmed. Each video was scored by 2 blinded raters.

SETTING

Multicenter collaboration within the Western Society of Pelvic Medicine (Vancouver, Calgary, and Edmonton). Face and content validity were evaluated. A standard scoring tool was developed for performance evaluation. Interrater reliability was assessed using intraclass correlation coefficient. Cronbach α was calculated for internal consistency. Jonckheere-Terpstra test verified whether the scores increased with operator skill level.

PARTICIPANTS

A total of 14 junior gynecology residents, 2 urogynecology fellows, and 3 staff urogynecologists were rated by a total of 6 gynecologic surgeons who scored 42 videos each.

RESULTS

Experienced pelvic surgeons from 3 participating sites agreed the models captured essential elements of real surgical skills (face validity) and of the true procedures (content validity). Intraclass correlation coefficient was adequate (AR = 0.86, PR = 0.90, and VH = 0.87). Cronbach α for the total scores was adequate (AR = 0.85, PR = 0.8, and VH = 0.71). Performance score increased with operator skill level for all 3 procedures (AR, p = <0.001; PR, p = 0.008; and VH, p = 0.007).

CONCLUSIONS

Our low fidelity procedure-specific vaginal surgery models had adequate initial validity. Future research will investigate transferability of acquired skills to the operating room.

The Surgical Clerkship and Medical Student Performance in a Standardized Patient Case of Acute Cholecystitis

BACKGROUND

Although an Observed Structured Clinical Examination (OSCE) has been used to evaluate patient interaction and general knowledge competencies of third-year students during their required surgical clerkships, whether surgical clerkship experience predicts satisfactory performance with a surgical patient in an OSCE environment has not been investigated.

OBJECTIVE

We hypothesized that completion of the third-year surgery clerkship would improve student ability to diagnose acute cholecystitis and recognize the further need for hospital admission and treatment.

DESIGN

An observational study design was used to determine student skills in evaluating a simulated surgical patient with abdominal pain from acute cholecystitis. The skills included key data gathering, physical examination, and information-sharing tasks.

SETTING

Tertiary care academic medical center.

PARTICIPANTS

Performance was compared between a cohort of 101 medical students who had completed the third-year surgical clerkship and 72 who had not. A secondary analysis compared performance for 91 students who had completed their third-year clerkship in sites near the University of California, San Francisco School of Medicine, and 10 who did so at a regional campus geographically distant from the medical school.

RESULTS

Of the 173 students who participated in the OSCE, only 42% correctly identified the diagnosis of acute cholecystitis, though 71% did suggest the possibility of a biliary process to the standardized patient. Most of the students who identified the condition as acute cholecystitis or gallbladder-related process had completed their third-year surgical clerkship (odds ratio [OR] = 3.26). Students who completed their surgical clerkship were also better able to recommend appropriate treatment for the patient (OR = 2.35), and recommend admission to the hospital or emergency department (OR = 2.00). Approximately one-third (35.3%) of all students documented a positive Murphy's sign, but only 6.4% identified the triad of leukocytosis, fever, and a Murphy's sign as diagnostic of acute cholecystitis and the need for surgical intervention. Student performance on the clinical examination did not differ depending on whether the students completed their clerkship at a Bay Area or regional hospital.

CONCLUSIONS

Student recognition of the key physical examination and laboratory findings diagnostic of acute cholecystitis was low, but students were better able to recommend further treatment for a patient with acute cholecystitis after completing the third-year surgical clerkship. Our study reveals areas where surgical educators can improve medical student ability to accurately diagnose acute cholecystitis and evaluate acute abdominal processes.

Development and testing for an operative competency assessment tool for nasal septoplasty surgery

BACKGROUND

Assessing surgical competency in otolaryngology is challenging, and residency programs are now responsible for ensuring the surgical competency of their graduates. Therefore, more objective assessment tools are being incorporated into the evaluation process. Objective structured assessment of technical skills (OSATSs) tools have been developed for multiple otolaryngology procedures. These include tonsillectomy, endoscopic sinus surgery, thyroidectomy, mastoidectomy, direct laryngoscopy, and rigid bronchoscopy. The purpose of this study was to develop and test a new assessment tool for septoplasty surgery and ensuring its feasibility, reliability, and construct validity. This study was designed to develop and test a valid, reliable, and feasible evaluation tool designed to measure the development of trainees' surgical skills in the operating room for septoplasty surgery.

METHODS

A new OSATSs-based instrument form for septoplasty was developed. During the study period of 2 years, 21 otolaryngology-head and neck surgery residents (ranging from postgraduate year 2 to 5) were evaluated intraoperatively by one faculty member obtaining a total of 175 evaluations. Surgical performance was rated using a seven-item task-specific checklist (TSC) and a global rating scale (GRS). The TSC assessed specific septoplasty technical skills, and the GRS assessed the overall surgical performance.

RESULTS

Our tool showed construct validity for both components of the assessment instrument, with increasing mean scores with advancing clinical levels. Cronbach's α, a measure of internal consistency, was 0.911 for TSC and 0.898 for GRS. Strong correlation between the TSC and GRS was established (r = 0.955; p < 0.01).

CONCLUSION

This study proved our educational tool to be a valid, reliable, and feasible method for assessing competency in septoplasty surgery. It can be integrated into surgical training programs to facilitate direct formative feedback. Assessing trainees' learning curves enables insight into their progression, ensuring their appropriate development.

Assessment of competence in pediatric gastrointestinal endoscopy

Gastrointestinal endoscopy is an invaluable tool for the diagnosis and treatment of upper and lower gastrointestinal diseases in children. Pediatric and adult endoscopy differ in several respects including differences in procedural indications, sedation practices, pre-procedure preparation, equipment, and the importance of routine tissue sampling and terminal ileum intubation. In the same way that performance of endoscopy in children requires pediatric-specific training, assessment of pediatric endoscopists requires an approach that is tailored to pediatric practice and the use of assessment methods and measures that have been developed and validated specifically within the pediatric context.

The introduction and the validation of a surgical encounter template to facilitate surgical coaching of gynaecologists at a metropolitan tertiary obstetrics and gynaecology hospital

INTRODUCTION

At a metropolitan tertiary obstetrics and gynaecology hospital some gynaecologists identified a need for surgical coaching. Full-time specialists in a teaching hospital are expected to teach surgery whilst having limited access to improving their own surgical skills. Over time, this resulted in some degree of technical deskilling. This in turn led to a loss of confidence in their technical ability to perform complex procedures. The trainee was potentially taught surgery by gynaecologists who were not confident in some aspects of their own surgical skill.

METHODS

A pilot program was introduced to validate a structured surgical encounter template to facilitate surgical coaching in a dedicated regular operating list.

RESULTS

There was a high degree of satisfaction amongst participants in the program. Learning objectives were predominantly related to acquisition of psychomotor skills, but cognitive and affective skills were also identified. Participants felt they gained confidence and improved their surgical skill as a result of the program. The descriptive feedback validated the structured encounter template in facilitating the learning and teaching environment in the operating room. Most participants expressed an intention to adopt a more structured approach in their surgical teaching.

DISCUSSION

A structured surgical coaching template used in a surgical coaching program facilitated short-term self-perceived improvement in surgical skill and confidence. Participants also expressed an intention to introduce a more structured approach in their teaching. The structured programme using the encounter template may have a potential role in remediating a surgeon identified as an outlier by a credentialing body.

Arthroscopic proficiency: methods in evaluating competency

BACKGROUND

The current paradigm of arthroscopic training lacks objective evaluation of technical ability and its adequacy is concerning given the accelerating complexity of the field. To combat insufficiencies, emphasis is shifting towards skill acquisition outside the operating room and sophisticated assessment tools. We reviewed (1) the validity of cadaver and surgical simulation in arthroscopic training, (2) the role of psychomotor analysis and arthroscopic technical ability, (3) what validated assessment tools are available to evaluate technical competency, and (4) the quantification of arthroscopic proficiency.

METHODS

The Medline and Embase databases were searched for published articles in the English literature pertaining to arthroscopic competence, arthroscopic assessment and evaluation and objective measures of arthroscopic technical skill. Abstracts were independently evaluated and exclusion criteria included articles outside the scope of knee and shoulder arthroscopy as well as original articles about specific therapies, outcomes and diagnoses leaving 52 articles cited in this review.

RESULTS

Simulated arthroscopic environments exhibit high levels of internal validity and consistency for simple arthroscopic tasks, however the ability to transfer complex skills to the operating room has not yet been established. Instrument and force trajectory data can discriminate between technical ability for basic arthroscopic parameters and may serve as useful adjuncts to more comprehensive techniques. There is a need for arthroscopic assessment tools for standardized evaluation and objective feedback of technical skills, yet few comprehensive instruments exist, especially for the shoulder. Opinion on the required arthroscopic experience to obtain proficiency remains guarded and few governing bodies specify absolute quantities.

CONCLUSIONS

Further validation is required to demonstrate the transfer of complex arthroscopic skills from simulated environments to the operating room and provide objective parameters to base evaluation. There is a deficiency of validated assessment tools for technical competencies and little consensus of what constitutes a sufficient case volume within the arthroscopy community.

Direct observed procedural skills assessment in the undergraduate setting

BACKGROUND

Medical students are required to undertake procedural clinical skills training before they qualify as doctors, and an assessment of these skills is a critical element of their fitness to practice.

CONTEXT

Challenges facing educators include the logistics of observing: i.e. who is best placed to assess their competence? Evidence appears to be inconclusive about the competence of students in the workplace to adhere to standards of practice, and the time required for an effective assessment.

INNOVATION

In this article the aim is to discuss who is best placed to assess final-year medical students in the workplace. We explore the use of direct observed procedural skills (DOPS) to assess students undertaking procedural skills in a simulated workplace setting by tutor-, peer- and self-assessment. The DOPS tool has been used to assess foundation doctors, but can it be used to effectively assess undergraduate medical students?

IMPLICATIONS

The main purpose of formative assessment in the simulated setting is to support student learning through the provision of feedback and debriefing. The use of the DOPS tool in this way can provide an insightful perspective of a students' assessment of procedural clinical skills. Tutors can use the DOPS tool to guide their teaching practice by tailoring their lessons towards areas in which students require more guidance. The DOPS assessment tool presents an opportunity to provide immediate and relevant feedback.

Remote evaluation of laparoscopic performance using the global operative assessment of laparoscopic skills

BACKGROUND

Although numerous assessment tools currently exist to evaluate laparoscopic surgical skills, no studies have demonstrated the reliability of such tools when used with telementoring technology. This study aimed to determine the reliability of the Global Operative Assessment of Laparoscopic Skills (GOALS) rating scale for assessing laparoscopic skills remotely and to identify how factors unique to remote assessment such as bandwidth and image quality influence its reliability.

METHODS

Four trained observers evaluated 19 participants for their technical performance during a laparoscopic cholecystectomy using the GOALS assessment tool. One observer assessed the study participants directly in the operating room, whereas the three remaining observers were randomly assigned and blinded to a high- (1.5 Mbps), medium- (256 kbps), or low- (64.4 kbps) bandwidth restriction and observed remotely via Skype. The Maryland Visual Comfort Scale was used to evaluate the video quality of the respective connections.

RESULTS

The intraclass correlation coefficient (ICC) calculated for the total GOALS score demonstrated a statistically significant correlation of high, medium, and low bandwidths respectively with ICC 0.693 (95 % confidence interval [CI], 0.226-0.883), 0.518 (95 % CI 0.089-0.783), and 0.499 (95 % CI 0.025-0.781). There was a statistically significant difference in the overall perceived visual quality between the high/low (Z = -3.222; P = 0.001) and the medium/low (Z = -3.567; P < 0.001) bandwidth comparison but no difference between the high/medium bandwidths (Z = -0.610; P = 0.542).

CONCLUSION

The data suggest that the GOALS assessment tool retains its reliability for intraoperative assessment of laparoscopic skills when used remotely. This is a key requirement in telesimulation programs allowing for structured feedback between the mentor and the mentee. This study quantifies the effect that bandwidth has on the reliability of remote assessment, demonstrating that higher bandwidths improve the utility of these tools.

Interprofessional non-technical skills for surgeons in disaster response: a qualitative study of the Australian perspective

Interprofessional non-technical skills for surgeons in disaster response have not yet been developed. The aims of this study were to identify the non-technical skills required of surgeons in disaster response and training for disaster response and to explore the barriers and facilitators to interprofessional practice in surgical teams responding to disasters. Twenty health professionals, with prior experience in natural disaster response or education, participated in semi-structured in-depth interviews. A qualitative matrix analysis design was used to thematically analyze the data. Non-technical skills for surgeons in disaster response identified in this study included skills for austere environments, cognitive strategies and interprofessional skills. Skills for austere environments were physical self-care including survival skills, psychological self-care, flexibility, adaptability, innovation and improvisation. Cognitive strategies identified in this study were "big picture" thinking, situational awareness, critical thinking, problem solving and creativity. Interprofessional attributes include communication, team-player, sense of humor, cultural competency and conflict resolution skills. "Interprofessionalism" in disaster teams also emerged as a key factor in this study and incorporated elements of effective teamwork, clear leadership, role adjustment and conflict resolution. The majority of participants held the belief that surgeons needed training in non-technical skills in order to achieve best practice in disaster response. Surgeons considerring becoming involved in disaster management should be trained in these skills, and these skills should be incorporated into disaster preparation courses with an interprofessional focus.

Master surgeons' operative teaching philosophies: a qualitative analysis of parallels to learning theory

OBJECTIVES

Practicing within the Halstedian model of surgical education, academic surgeons serve dual roles as physicians to their patients and educators of their trainees. Despite this significant responsibility, few surgeons receive formal training in educational theory to inform their practice. The goal of this work was to gain an understanding of how master surgeons approach teaching uncommon and highly complex operations and to determine the educational constructs that frame their teaching philosophies and approaches.

DESIGN

Individuals included in the study were queried using electronically distributed open-ended, structured surveys. Responses to the surveys were analyzed and grouped using grounded theory and were examined for parallels to concepts of learning theory.

SETTING

Academic teaching hospital.

PARTICIPANTS

Twenty-two individuals identified as master surgeons.

RESULTS

Twenty-one (95.5%) individuals responded to the survey. Two primary thematic clusters were identified: global approach to teaching (90.5% of respondents) and approach to intraoperative teaching (76.2%). Many of the emergent themes paralleled principles of transfer learning theory outlined in the psychology and education literature. Key elements included: conferring graduated responsibility (57.1%), encouraging development of a mental set (47.6%), fostering or expecting deliberate practice (42.9%), deconstructing complex tasks (38.1%), vertical transfer of information (33.3%), and identifying general principles to structure knowledge (9.5%).

CONCLUSIONS

Master surgeons employ many of the principles of learning theory when teaching uncommon and highly complex operations. The findings may hold significant implications for faculty development in surgical education.

Computer-aided feedback of surgical knot tying using optical tracking

BACKGROUND

Quantifying the information content of hand motion during surgical knot tying using information theory based entropy measures enables the comparison of different groups: novice and expert. We hypothesized that complexity would differ between the 2 groups and predicted based on motor learning models that complexity/information would reduce with increased expertise.

METHODS

Six degrees of freedom hand-motion data during surgical knot tying were acquired using an infrared optical hand tracking device. Multiple data samples were obtained from 2 groups: novice (third-year medical students) and expert (attending surgeons). After preprocessing each knot tying data sample into a binary symbolic time series, 3 nonlinear complexity measures were calculated: Lempel Ziv complexity, Shannon entropy, and Renyi entropy. The Shannon and Renyi entropies were calculated using a word length of 6. A Student t test was used to test whether the 2 groups were from the same population when using these entropy measures, applying a p value of 0.05 to reject the null hypothesis.

RESULTS

The expert surgeons were found to have less complex patterns of motion compared with the novice group. This finding was statistically significant using Lempel Ziv complexity (p = 0.004), Shannon entropy (p = 0.006), and Renyi entropy with q = 2 (p = 0.006). Using Renyi entropy with q = 0.5, the 2 groups were not significantly different (p = 0.26).

CONCLUSIONS

The ability to separate novice from expert populations during surgical knot tying using information theory entropy measures could form the basis of a low-cost educational tool to provide feedback and to assess skill acquisition using low-fidelity bench models.

Learning basic laparoscopic skills: a randomized controlled study comparing box trainer, virtual reality simulator, and mental training

OBJECTIVES

The objectives of this study were (1) to compare different methods of learning basic laparoscopic skills using box trainer (BT), virtual reality simulator (VRS) and mental training (MT); and (2) to determine the most effective method of learning laparoscopic skills.

DESIGN

Randomized controlled trial.

SETTING

King's College, London.

METHODS

41 medical students were included in the study. After randomization, they were divided into 5 groups. Group 1 was the control group without training; group 2 was box trained; group 3 was also box trained with an additional practice session; group 4 was VRS trained; and group 5 was solely mentally trained. The task was to cut out a circle marked on a stretchable material. All groups were assessed after 1 week on both BT and VRS. Four main parameters were assessed, namely time, precision, accuracy, and performance.

RESULTS

Time: On BT assessment, the box-trained group with additional practice group 3 was the fastest, and the mental-trained group 5 was the slowest. On VRS assessment, the time difference between group 3 and the control group 1 was statistically significant. Precision: On BT assessment, the box-trained groups 2 and 3 scored high, and mental trained were low on precision. On VRS assessment, the VRS-trained group ranked at the top, and the MT group was at the bottom on precision. Accuracy: On BT assessment, the box-trained group 3 was best and the mental-trained group was last. On VRS assessment, the VRS-trained group 4 scored high closely followed by box-trained groups 2 and 3. Performance: On BT assessment, the box-trained group 3 ranked above all the other groups, and the mental-trained group ranked last. On VRS assessment, the VRS group 4 scored best, followed closely by box-trained groups 2 and 3.

CONCLUSIONS

The skills learned on box training were reproducible on both VRS and BT. However, not all the skills learned on VRS were transferable to BT. Furthermore, VRS was found to be a reliable and the most convenient method of assessment. MT alone cannot replace conventional training.

Training and simulation in otolaryngology

This article focuses on key issues surrounding the needs and application of simulation technologies for technical skills training in otolaryngology. The discussion includes an overview of key topics in training and learning, the application of these issues in simulation environments, and the subsequent applications of these simulation environments to otolaryngology. Examples of past applications are presented, with discussion of how the interplay of cultural changes in surgical training in general along with the rapid advancements in technology have shaped and influenced their adoption and adaptation. The authors conclude with emerging trends and potential influences advanced simulation and training will have on technical skills training in otolaryngology.

Using elements from an acute abdominal pain Objective Structured Clinical Examination (OSCE) leads to more standardized grading in the surgical clerkship for third-year medical students

BACKGROUND

There is poor reliability in the Likert-based assessments of patient interaction and general knowledge base for medical students in the surgical clerkship. The Objective Structured Clinical Examination (OSCE) can be used to assess these competencies.

OBJECTIVE

We hypothesize that using OSCE performance to replace the current Likert-based patient interaction and general knowledge base assessments will not affect the pass/fail rate for third-year medical students in the surgical clerkship.

METHODS

In this retrospective study, third-year medical student clerkship data from a three-station acute abdominal pain OSCE were collected from the 2009-2010 academic year. New patient interaction and general knowledge base assessments were derived from the performance data and substituted for original assessments to generate new clerkship scores and ordinal grades. Two-sided nonparametric statistics were used for comparative analyses, using an α = 0.05.

RESULTS

Seventy third-year medical students (50.0% female) were evaluated. A sign test showed a difference in the original (4.45/5) and the new (4.20/5) median patient interaction scores (p < 0.01). A sign test did not show a difference in the original (4.00/5) and the new (4.11/5) median general knowledge base scores (p = 0.28). Nine clerkship grades changed between these different grading schemes (p = 0.045), with an overall agreement of 87.1% and a kappa statistic of 0.81. There were no differences in the pass/fail rate (p > 0.99).

CONCLUSIONS

We conclude that there are no differences in pass/fail rate, but there is a more standardized distribution of patient interaction assessments and utilization of the full spectrum of possible passing grades. We recommend that the current patient interaction assessment for third-year medical students in the surgical clerkship be replaced with that found through trained standardized patients in this three-station acute abdominal pain OSCE.