Crowd-Sourced Assessment of Technical Skills: a novel method to evaluate surgical performance

Association of Crowd-Sourced Assessment of Technical Skills (CSATSTM) and Outcomes of Robotic Assisted Radical Prostatectomy

OBJECTIVE

To investigate if use of the Crowd-Sourced Assessment of Technical Skills (CSATSTM) platform and video peer review with constructive feedback is associated with improvement in technical skill and patient outcomes for robotic-assisted laparoscopic prostatectomy (RALP).

METHODS

Five fellowship-trained urologists voluntarily submitted RALP cases for CSATSTM Global Evaluative Assessment of Robotic Skills (GEARS) scoring and expert narrative review between April 15, 2022 - April 30, 2023. Surgeon-selected and randomly-selected cases were reviewed. Surgeons underwent local peer review of videos with constructive feedback. Change in GEARS scores and frequency of post-operative outcomes over the 12-month periods before and during the study were analyzed in logistic regression models. Bias was assessed with sensitivity analysis comparing surgeon-selected to randomly-selected cases.

RESULTS

GEARS scores for randomly-selected versus surgeon-selected cases did not differ significantly. Overall GEARS score correlated positively with annual surgical RALP volume (r=0.39, p=0.003) and negatively with years in practice (r=-0.34, p=0.01). After adjusting for confounders, there was no significant improvement in overall GEARS Score (0.01±0.02/month, p=0.48); but likelihood of sepsis (Odds Ratio 0.07, 95% CI 0.01-1.00, p=0.05) and pelvic fluid collection (Odds Ratio 0.09, 95% CI 0.01-0.99, p=0.049) were significantly decreased during the intervention period (n=165) compared to the prior 12 months (n=144). No outcome increased in likelihood (p>0.05).

CONCLUSIONS

Integration of CSATSTM and local video peer review is associated with significant improvement in patient outcomes after RALP, despite no significant change in surgeons' GEARS scores. This is the first study demonstrating improvement in patient RALP outcomes after implementation of such a paradigm in practicing surgeons.

Head-to-Head Analysis of Vertical vs Horizontal Incision Patterns in Breast Reconstruction: Surgical Outcomes and Aesthetic Implications

BACKGROUND

Modern aesthetically optimized breast reconstruction requires collaboration between the patient, breast surgeon, and plastic surgeon. To optimize both surgical outcome and aesthetic results, incision patterns must be carefully planned.

OBJECTIVES

We aimed to determine whether vertical or horizontal orientation of mastectomy incision was preferred in the general population and to analyze corresponding complication profiles.

METHODS

A retrospective review was performed of all patients undergoing bilateral mastectomy followed by autologous breast reconstruction utilizing either vertical or horizontal incision from January 2011 to November 2022. Postoperative complications of the 2 incision patterns were analyzed. Additionally, crowdsourcing was utilized to assess aesthetic implications of horizontal and vertical incision patterns on postoperative pictures of completed breast reconstruction. Survey rater demographics were also analyzed to assess differences in scoring based on voter characteristics.

RESULTS

There were no significant differences in postoperative breast complications between patients with horizontal or vertical incisions when considering wound, infection, seroma, hematoma, fat necrosis, or overall complications (P > .05). Crowdsourcing showed that, regardless of voter demographics, vertical incisions were preferred over horizontal incisions (P < .001). Additionally, voters who knew someone who had undergone breast reconstruction were more likely to rate all incision patterns higher than other voters (P < .001).

CONCLUSIONS

Although there are no significant differences in complication profiles between vertical and horizontal incisions in autologous breast reconstruction patients, vertical incision patterns are preferred aesthetically by the general population.

LEVEL OF EVIDENCE: 4

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Does Surgeon Experience Correlate with Crowd-Sourced Skill Assessment in Robotic Bariatric Surgery?

BACKGROUND

The Global Evaluative Assessment of Robotic Skills (GEARS) rubric provides a measure of skill in robotic surgery. We hypothesize surgery performed by more experienced operators will be associated with higher GEARS scores.

METHOD

Patients undergoing sleeve gastrectomy from 2016 to 2020 were analyzed. Three groups were defined by time in practice: less than 5, between 5 and 15, and more than 15 years. Continuous variables were compared with ANOVA and multivariable regression was performed.

RESULTS

Fourteen operators performing 154 cases were included. More experienced surgeons had higher GEARS scores and shorter operative times. On multivariable regression, operative time (P = 0.027), efficiency (P = .022), depth perception (P = 0.033), and bimanual dexterity (P = 0.047) were associated with experience.

CONCLUSIONS

In our video-based assessment (VBA) model, operative time and several GEARS subcomponent scores were associated with surgical experience. Further studies should determine the association between these metrics and surgical outcomes.

Laypersons versus experienced surgeons in assessing simulated robot-assisted radical prostatectomy

BACKGROUND

Feedback is important for surgical trainees but it can be biased and time-consuming. We examined crowd-sourced assessment as an alternative to experienced surgeons' assessment of robot-assisted radical prostatectomy (RARP).

METHODS

We used video recordings (n = 45) of three RARP modules on the RobotiX, Simbionix simulator from a previous study in a blinded comparative assessment study. A group of crowd workers (CWs) and two experienced RARP surgeons (ESs) evaluated all videos with the modified Global Evaluative Assessment of Robotic Surgery (mGEARS).

RESULTS

One hundred forty-nine CWs performed 1490 video ratings. Internal consistency reliability was high (0.94). Inter-rater reliability and test-retest reliability were low for CWs (0.29 and 0.39) and moderate for ESs (0.61 and 0.68). In an Analysis of Variance (ANOVA) test, CWs could not discriminate between the skill level of the surgeons (p = 0.03-0.89), whereas ES could (p = 0.034).

CONCLUSION

We found very low agreement between the assessments of CWs and ESs when they assessed robot-assisted radical prostatectomies. As opposed to ESs, CWs could not discriminate between surgical experience using the mGEARS ratings or when asked if they wanted the surgeons to perform their robotic surgery.

Crowd-sourced and expert video assessment in minimally invasive esophagectomy

BACKGROUND

Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review.

METHODS

Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT).

RESULTS

Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42).

CONCLUSION

GOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.

New generation evaluations: video-based surgical assessments : A technology update

BACKGROUND

Surgical skill training, assessment, and feedback are the backbone of surgical training. High-quality skills require expert supervision and evaluation throughout a resource-intensive multi-year training process. As technological barriers to internet access and the ability to save and upload surgical videos continue to improve, video-based assessment technology is emerging as a tool that could reshape surgical training for the next generation of surgeons. Video-based assessment platforms have the potential to allow surgeons from across the globe to upload their surgical videos online and receive high-quality, standardized, and unbiased feedback. They combine visual recordings of a surgeon's operative technique, with standardized grading tools that have the potential to significantly impact surgical training and technical skill acquisition across the world.

METHOD

The platforms included in this review are in various stages of development after a thorough discussion with national experts on the SAGES TAVAC (Technology and Value Assessments) Committee. For each VBA program, a description of its platform was given and a literature review was obtained using a PubMed search performed from inception until December 2021.

RESULTS

The study reviewed all video-based assessment programs currently available in the market, identified their strengths and weaknesses, and how they can be optimized in future.

CONCLUSION

The technological platforms will play a key role in the training and technical skill acquisition of the next generation of surgeons and can have an immense impact on patient care across the world. There is immense potential for all these platforms to grow and become incorporated within the framework of an effective surgical training program.

Intentional enterotomies: validation of a novel robotic surgery training exercise

While laparoscopic simulation-based training is a well-established component of general surgery training, no such requirement or standardized curriculum exists for robotic surgery. Furthermore, there is a lack of high-fidelity electrocautery simulation training exercises in the literature. Using Messick's validity framework, we sought to determine the content, response process, internal content and construct validity of a novel inanimate tissue model that utilizes electrocautery for potential incorporation in such curricula. A multi-institutional, prospective study involving medical students (MS) and general surgery residents (PGY1-3) was conducted. Participants performed an exercise using a biotissue bowel model on the da Vinci Xi robotic console during which they created an enterotomy using electrocautery, followed by approximation with interrupted sutures. Participant performance was recorded and then scored by crowd-sourced assessors of technical skill, along with three of the authors. Construct validity was determined via difference in Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and total number of errors between the two cohorts. Upon completion of the exercise, participants were surveyed on their perception of the exercise and its impact on their robotic training to determine content validity. 31 participants were enrolled and separated into two cohorts: MS + PGY1 vs. PGY2-3. Time spent on the robotic trainer (0.8 vs. 8.13 h, p = 0.002), number of bedside robotic assists (5.7 vs. 14.8, p < 0.001), and number of robotic cases as primary surgeon (0.3 vs. 13.1, p < 0.001) were statistically significant between the two groups. Differences in GEARS scores (18.5 vs. 19.9, p = 0.001), time to completion (26.1 vs. 14.4 min, p < 0.001), and total errors (21.5 vs. 11.9, p = 0.018) between the groups were statistically significant as well. Of the 23 participants that completed the post-exercise survey, 87% and 91.3% reported improvement in robotic surgical ability and confidence, respectively. On a 10-point Likert scale, respondents rated the realism of the exercise 7.5, educational benefit 9.1, and effectiveness in teaching robotic skills 8.7. Controlling for the upfront investment of certain training materials, each exercise iteration cost ~ $30. This study confirmed the content, response process, internal structure and construct validity of a novel, high-fidelity and cost-effective inanimate tissue exercise which successfully incorporates electrocautery. Consideration should be given to its addition to robotic surgery training programs.

Utilization of a 3D Printed Simulation Training Model to Improve Microsurgical Training

Simulation is integral to the development and maintenance of micro- surgical skills. Several simulation models have been described ranging from bench- top to live animal models. High fidelity models are often burdened by cost and ethical issues limiting widespread implementation. This study aims to determine the feasibility of a microsurgical training platform using the Konjac noodle model.

Effects of automated skill assessment on robotic surgery training

BACKGROUND

Several automated skill-assessment approaches have been proposed for robotic surgery, but their utility is not well understood. This article investigates the effects of one machine-learning-based skill-assessment approach on psychomotor skill development in robotic surgery training.

METHODS

N = 29 trainees (medical students and residents) with no robotic surgery experience performed five trials of inanimate peg transfer with an Intuitive Surgical da Vinci Standard robot. Half of the participants received no post-trial feedback. The other half received automatically calculated scores from five Global Evaluative Assessment of Robotic Skill domains post-trial.

RESULTS

There were no significant differences between the groups regarding overall improvement or skill improvement rate. However, participants who received post-trial feedback rated their overall performance improvement significantly lower than participants who did not receive feedback.

CONCLUSIONS

These findings indicate that automated skill evaluation systems might improve trainee self-awareness but not accelerate early stage psychomotor skill development in robotic surgery training.

Toward Correcting Anxious Movements Using Haptic Cues on the Da Vinci Surgical Robot

Surgical movements have an important stylistic quality that individuals without formal surgical training can use to identify expertise. In our prior work, we sought to characterize quantitative metrics associated with surgical style and developed a near-real-time detection framework for stylistic deficiencies using a commercial haptic device. In this paper, we implement bimanual stylistic detection on the da Vinci Research Kit (dVRK) and focus on one stylistic deficiency, "Anxious", which may describe movements under stressful conditions. Our goal is to potentially correct these "Anxious" movements by exploring the effects of three different types of haptic cues (time-variant spring, damper, and spring-damper feedback) on performance during a basic surgical training task using the da Vinci Research Kit (dVRK). Eight subjects were recruited to complete peg transfer tasks using a randomized order of haptic cues and with baseline trials between each task. Overall, all cues lead to a significant improvement over baseline economy of volume and time-variant spring haptic cues lead to significant improvements in reducing the classified "Anxious" movements and also corresponded with significantly lower path length and economy of volume for the non-dominant hand. This work is the first step in evaluating our stylistic detection model on a surgical robot and could lay the groundwork for future methods to actively and adaptively reduce the negative effect of stress in the operating room.

Crowd-Sourced Assessment of Surgical Skills of Urology Resident Applicants: Four-Year Experience

OBJECTIVE

To determine a) if surgical skills among urology resident applicants could be reliably assessed via crowdsourcing and b) to what extent surgical skills testing impacts resident selection.

DESIGN

Interviewees completed the following surgical skills tasks during their interview day: open knot tying (OKT), laparoscopic peg transfer (LPT), and robotic suturing (RS). Urology faculty and crowd-workers evaluated each applicant's video-recorded performance using validated scoring and were assessed for agreement using Cronbach's alpha. Applicants' USMLE scores, interview scores, and Jefferson Scale of Physician Empathy (JSPE-S) scores were assessed for correlation with skills testing scores and match rank. Additionally, a survey was distributed to interviewees assessing match outcomes.

SETTING

University of California Irvine Department of Urology, Surgical Skills Laboratory PARTICIPANTS: All 94 urology residency interviewees at the University of California Irvine Department of Urology from 2015-2018 were invited to complete the three surgical skills tasks on their interview day.

RESULTS

Survey responses were received from all 94 interviewees (100%). Crowd and expert agreement was good (α=0.88), fair (α=0.67), and poor (α=0.32) for LPT, RS, and OKT scores, respectively. The skills testing scores did not correlate with match rank, USMLE score, or JSPE-S score. On multivariate analysis, only interview score (r= -0.723; p<0.001) and faculty LPT score (r=-0.262; p=0.001) were significant predictors of match rank. Interviewees who reported matching into a top 3 residency choice had significantly higher faculty LPT scores than those who did not (11.9 vs. 9.7, p=0.03).

CONCLUSIONS

Surgical skills overall did not significantly impact match rank. Expert assessment of laparoscopic peg transfer skills and interview performance among urology resident applicants correlated with match rank.

Is Technical Expertise Necessary to Assess Technical Expertise? Let's Ask the Kids

Background: Simulation has an increasing role in surgical training; however, using validated tools such as the Objective Structured Assessment of Technical Skills (OSATS) is time-consuming, which may be a potential barrier to simulation-based training. This study tests the hypothesis that assessors with technical expertise are necessary to objectively score a technical task. Methods: Three tasks, ring transfer, needle pass, and atresia cut, were performed using a synthetic thoracoscopic simulator for esophageal atresia/tracheoesophageal fistula. Three pediatric surgeons, 6 novice adults, and 3 children aged 9-13 years scored each attempt using the overall global OSATS rating from 1 to 5 (1 "repeated or awkward movements" to 5 "fluid movement. No awkwardness"). Results: For the ring transfer, all assessors scored novice participants significantly less than expert and intermediate scores (surgeon P = .0004, nonsurgeon adults P = .0009 and children P = .0003). For the needle pass, all assessors gave significantly different scores between novices and experts (surgeon P = .0007, nonsurgeon adult P = .0008, and children P = .0040). For the atresia cut, surgeon assessors gave significantly higher scores for experts and intermediate and novice (P = .0004). Nonsurgeon assessors gave experts and intermediates significantly higher scores than novices (P = .0001). Surgeon assessors achieved good reliability for ring transfer (0.8252) and needle pass (0.7769) compared with nonsurgeon assessors who showed poor reliability for the ring transfer (0.3959) and moderate for the needle pass (0.6551). Conclusions: Expertise in performing these procedures is not a prerequisite for an assessor to evaluate the technical skill, hence assessors of skill acquisition can be nonexpert, a nonsurgeon, or even a child. The variability in all groups suggests that reliability overall is increased with multiple assessors. Although nonsurgeon assessors may be appropriate for formative assessments, they lack the reliability to provide assessment of competence for high stakes complex tasks. Summative assessment will likely require at least 1 surgeon/expert assessor to provide reliability.

Development of a laparoscopic surgical skills simulation curriculum: Enhancing resident training through directed coaching and closed-loop feedback

BACKGROUND

Performance feedback through peer coaching and rigorous self-assessment is a critical part of technical skills improvement. However, formal collaborative programs using operative video-based skills assessments to generate peer coaching feedback have only been validated among attending surgeons. In this study, we developed a unique longitudinal, simulation video-based laparoscopic skills resident curriculum using video-based peer coaching and evaluated its association with skills acquisition among surgical trainees.

METHODS

The laparoscopic simulation curriculum consists of a pre-practice laparoscopic skill video recording, followed by receipt of directed coaching and feedback on performance from a faculty coach, a peer coach, and self-coaching. Residents then completed 6 weeks of feedback-directed practice and submitted a second post-practice laparoscopic skill video recording of the same skill, which was evaluated by a minimally invasive surgery expert grader. All general surgery residents in a single institution were enrolled, with 107 residents completing the curriculum in its initial 2 years.

RESULTS

Overall, more than two-thirds of residents achieved skills proficiency on their expert assessments, with similar rates of residents achieving skills proficiency at all postgraduate year levels. Significant improvements between the pre-practice assessments and post-practice assessments were most frequently seen in the instrument handling, precision, and motion & flow categories (P < .05 each). Faculty provided the highest number and proportion of closed-loop comments; residents' self-coaching feedback had the lowest number of closed-loop comments, with 83% of self-assessments containing none.

CONCLUSION

In this study, we describe the successful implementation of a longitudinal laparoscopic skills video-based coaching curriculum designed to improve residents' laparoscopic technical abilities through iterative directed practice supplemented by formative closed-loop feedback. This feasible, reproducible, and low-cost simulation curriculum can be adapted to other training programs and skills acquisition endeavors. This program also prepares trainees for ongoing performance feedback after completion of residency through rigorous self-assessment and peer-to-peer coaching.

Video-Based Coaching: Current Status and Role in Surgical Practice (Part 1) From the Society for Surgery of the Alimentary Tract, Health Care Quality and Outcomes Committee

Patient safety and outcomes are directly related to surgical performance. Surgical training emphasizes the importance of the surgeon in determining these outcomes. After training is complete, there is a lack of structured programs for surgeons to audit their skills and continue their individual development. There is a significant linear relationship between surgeon technical skill and surgical outcomes; however, measuring technical performance is difficult. Video-based coaching matches an individual surgeon in practice with a surgical colleague who has been trained in the core principles of coaching for individualizing instruction. It can provide objective assessment for teaching higher-level concepts, such as technical skills, cognitive skills, and decision-making. There are many benefits to video-based coaching. While the concept is gaining acceptance as a method of surgical education, it is still novel in clinical practice. As more surgeons look towards video-based coaching for quality improvement, a consistent definition of the program, goals, and metrics for assessment will be critical. This paper is a review on the status of the video-based coaching as it applies to practicing surgeons.

Patient Safety in the Operating Room During Urologic Surgery: The OR Black Box Experience

PURPOSE

To mitigate intraoperative adverse events, it is important to understand the context in which these errors occur. The purpose of this study is to characterize the IAEs and potential distractions that occur in minimally invasive urologic procedures.

METHODS

We conducted a prospective cohort study in patients undergoing laparoscopic urologic surgery at an academic health center. The OR Black Box, a unique technology system which captures video and audio recordings of the operating room as well as the operative field, was used to collect data regarding procedure type, critical step, IAEs, and distractions.

RESULTS

Of a total of 80 cases analyzed, the majority of these cases were partial nephrectomy (n = 36; 45%), radical nephrectomy (n = 20; 25%), and adrenalectomy (n = 4; 5%). Across all cases, there were a total of 138 clinically significant IAEs, 10 of which (14%) were of the highest severity (five on the SEVerity of intraoperative Events and Rectification Tool (SEVERE) matrix). Of these, 70 (51%) occurred during an a priori defined critical step of the operation. Distractions were common across all cases. The median rate of external communication per case was 16 events (IQR 11-22); and per critical step was 4 (IQR 2.75-8), while median room traffic per case was 65 entries/exits (IQR 42-76); and per critical step was 17 (IQR 10-65).

CONCLUSION

Our data demonstrate that IAEs occur frequently during all phases of the operation at hand. Future study will be required to examine the role of distractions and IAE as well as IAE and their relationship to post-operative clinical outcomes.

Virtual Education in Pediatric Surgery during the COVID-19 Era: Facing and Overcoming Current Challenges

The novel coronavirus disease 2019 (COVID-19) pandemic has impacted our way of living in an unprecedented manner. Medical professionals at all levels have been forced to adapt to the novel virus. The delivery of surgical services and the subsequent learning opportunities for surgical residents have especially been disrupted and the pediatric surgical community has not been exempted by this. This article highlights the challenges imposed by the pandemic and outlines the various learning modalities that can be implemented to ensure continued learning opportunities throughout the pandemic and beyond. Furthermore, it aims to show how the utilization and expansion of technologies maintain and further increase the communication, as well as the exchange of and access to knowledge among peers. Virtual education-, application-, and simulation-based learning and social media, as well as telemedicine and online conferences, will play a considerable role in the future of surgical specialties and surgical education.

Crowdsourced Assessment of Surgical Skill Proficiency in Cataract Surgery

OBJECTIVE

To test whether crowdsourced lay raters can accurately assess cataract surgical skills.

DESIGN

Two-armed study: independent cross-sectional and longitudinal cohorts.

SETTING

Washington University Department of Ophthalmology.

PARTICIPANTS AND METHODS

Sixteen cataract surgeons with varying experience levels submitted cataract surgery videos to be graded by 5 experts and 300+ crowdworkers masked to surgeon experience. Cross-sectional study: 50 videos from surgeons ranging from first-year resident to attending physician, pooled by years of training. Longitudinal study: 28 videos obtained at regular intervals as residents progressed through 180 cases. Surgical skill was graded using the modified Objective Structured Assessment of Technical Skill (mOSATS). Main outcome measures were overall technical performance, reliability indices, and correlation between expert and crowd mean scores.

RESULTS

Experts demonstrated high interrater reliability and accurately predicted training level, establishing construct validity for the modified OSATS. Crowd scores were correlated with (r = 0.865, p < 0.0001) but consistently higher than expert scores for first, second, and third-year residents (p < 0.0001, paired t-test). Longer surgery duration negatively correlated with training level (r = -0.855, p < 0.0001) and expert score (r = -0.927, p < 0.0001). The longitudinal dataset reproduced cross-sectional study findings for crowd and expert comparisons. A regression equation transforming crowd score plus video length into expert score was derived from the cross-sectional dataset (r² = 0.92) and demonstrated excellent predictive modeling when applied to the independent longitudinal dataset (r² = 0.80). A group of student raters who had edited the cataract videos also graded them, producing scores that more closely approximated experts than the crowd.

CONCLUSIONS

Crowdsourced rankings correlated with expert scores, but were not equivalent; crowd scores overestimated technical competency, especially for novice surgeons. A novel approach of adjusting crowd scores with surgery duration generated a more accurate predictive model for surgical skill. More studies are needed before crowdsourcing can be reliably used for assessing surgical proficiency.

The influence of reviewer's occupation on the skill assessment of urethrovesical anastomosis in robot-assisted radical prostatectomy

INTRODUCTION

In recent years, crowd-sourced assessments have been reported as a timesaving, cost-efficient, and practicable method of surgical skill evaluation. However, the differences in the assessment of surgical skills by the individual reviewers cannot be further examined in terms of characteristics of the reviewers because they are usually anonymously and randomly selected. This study aimed to reveal the effects of reviewers' occupations on their assessment of a surgeon's skill.

METHODS

In total, 42 urologists, 19 paramedics, 73 medical students, and 28 non-medical personnel used the Global Evaluative Assessment of Robotic Skills (GEARS) validated robotic surgery rating tool to assess the surgical skill of surgeons in nine edited video clips of complete urethrovesical anastomosis during a robot-assisted radical prostatectomy. The total GEARS scores of the four groups of reviewers were compared, and the similarities and the differences between the ratings of the urologists group and those of the other three groups were subsequently investigated.

RESULTS

The rankings of video clips in the order of GEARS scores were very similar in each group, and a strong positive correlation (R² values >0.8) was observed between the scores assigned by the urologists group and those assigned by the other three groups.

CONCLUSION

Our findings indicate that the crude evaluation of robot-assisted urethrovesical anastomosis is not affected by the reviewers' occupations. Non-medical personnel may be able to provide a rudimentary screening evaluation of surgical skill.

The Use of Crowdsourcing Technology to Evaluate Preoperative Severity in Patients With Unilateral Cleft Lip in a Multiethnic Population

ABSTRACT

Crowd sourcing has been used in multiple disciplines to quickly generate large amounts of diverse data. The objective of this study was to use crowdsourcing to grade preoperative severity of unilateral cleft lip phenotype in a multiethnic cohort with the hypothesis that crowdsourcing could efficiently achieve similar rankings compared to expert surgeons. Deidentified preoperative photos were collected for patients with primary, unilateral cleft lip with or without cleft palate (CL ± P). A platform was developed with C-SATS for pairwise comparisons utilizing Elo rankings by crowdsource workers through Amazon Mechanical Turk. Images were independently ranked by 2 senior surgeons for comparison. Seventy-six patients with varying severity of unilateral (CL ± P) phenotype were chosen from Operation Smile missions in Bolivia, Madagascar, Vietnam, and Morocco. Patients were an average of 1.2 years' old, ranging from 3 months to 3.3 years. Each image was compared with 10 others, for a total of 380 unique pairwise comparisons. A total of 4627 total raters participated with a median of 12 raters per pair. Data collection was completed in <20 hours. The crowdsourcing ranking and expert surgeon rankings were highly correlated with Pearson correlation coefficient of R = 0.77 (P = 0.0001). Crowdsourcing provides a rapid and convenient method of obtaining preoperative severity ratings, comparable to expert surgeon assessment, across multiple ethnicities. The method serves as a potential solution to the current lack of rating systems for preoperative severity and overcomes the difficulty of acquiring large-scale assessment from expert surgeons.

Challenges in surgical video annotation

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

Mindful Medical Education Online

The COVID-19 pandemic disrupted medical education. In-person classes and clinical rotations were urgently canceled, followed by a historic and unprecedented migration to online teaching. Most medical school courses were not designed to be fully online, and faculty and students are novices in the process. The purpose of this article is to provide recommendations for educators to optimize their approach to online curricular transformation. Mindful teaching online creates presences that set climate and support discourse, establish routines that build practice, model professional expectations, and challenge but support learners.

Crowdsourced Assessment of Aesthetic Outcomes after Breast Reconstruction

BACKGROUND

Evaluating the aesthetic success of breast reconstruction can be difficult. Patients, surgeons, and the general population may differ in what constitutes a successful outcome. Recently, crowdsourcing has emerged to accumulate and analyze data on a massive scale. The authors propose that crowdsourcing can be a useful tool to reliably rate aesthetic outcomes of breast reconstruction.

METHODS

One hundred one deidentified photographs of patients at various stages of breast reconstruction were gathered. Assessment tools included a five-point Likert scale and the transverse rectus abdominis myocutaneous (TRAM) visual assessment scale. Anonymous crowd workers and a group of expert reconstructive surgeons rated an identical set of photographs on the Likert scale. Crowd workers also rated the set of photographs on the TRAM scale.

RESULTS

The authors obtained 901 anonymous, layperson evaluations on both Likert and TRAM scales. Crowdsourced assessment data collection took 28.6 hours. Expert assessment took 15 months. Expert and crowdsourced scores were equivalent on the Likert scale (overall interrater reliability, κ = 0.99; 95 percent CI, 0.98 to 0.99). Intrarater reliability among each subcomponent was highly reproducible for the crowd (r = 0.98; 95 percent CI, 0.97 to 0.99) and experts (r = 0.82; 95 percent CI, 0.77 to 0.87). Breast contour and positioning were most predictive of overall aesthetic result. Skin patch and scar were least predictive of overall aesthetic appearance.

CONCLUSIONS

Aesthetic outcomes rated by crowds were reliable and correlated closely with those by expert surgeons. Crowdsourcing can be a rapid, reliable, and valid way to assess aesthetic outcomes in the breast reconstruction patient.

Current methods for assessing technical skill in cataract surgery

Surgery is a major source of errors in patient care. Preventing complications from surgical errors in the operating room is estimated to lead to reduction of up to 41 846 readmissions and save $620.3 million per year. It is now established that poor technical skill is associated with an increased risk of severe adverse events postoperatively and traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. This review discusses the current methods available for evaluating technical skills in cataract surgery and the recent technological advancements that have enabled capture and analysis of large amounts of complex surgical data for more automated objective skills assessment.

Implementation of the ACS/ APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory

BACKGROUND

The American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) Resident Skills Curriculum includes validated task-specific checklists and global rating scales (GRS) for Objective Structured Assessment of Technical Skills (OSATS). However, it does not include instructions on use of these assessment tools. Since consistency of ratings is a key feature of assessment, we explored rater reliability for two skills.

METHODS

Surgical faculty assessed hand-sewn bowel and vascular anastomoses in real-time using the OSATS GRS. OSATS were video-taped and independently evaluated by a research resident and surgical attending. Rating consistency was estimated using intraclass correlation coefficients (ICC) and generalizability analysis.

RESULTS

Three-rater ICC coefficients across 24 videos ranged from 0.12 to 0.75. Generalizability reliability coefficients ranged from 0.55 to 0.8. Percent variance attributable to raters ranged from 2.7% to 32.1%. Pairwise agreement showed considerable inconsistency for both tasks.

CONCLUSIONS

Variability of ratings for these two skills indicate the need for rater training to increase scoring agreement and decrease rater variability for technical skill assessments.

Robotic Surgery: The Impact of Simulation and Other Innovative Platforms on Performance and Training

OBJECTIVE

To review the current status of robotic training and the impact of various training platforms on the performance of robotic surgical trainees.

DATA SOURCES

Literature review of Google Scholar and PubMed. The search terms included a combination of the following: "robotic training," "simulation," "robotic curriculum," "obgyn residency robotic training," "virtual reality robotic training," "DaVinci training," "surgical simulation," "gyn surgical training." The sources considered for inclusion included peer-reviewed articles, literature reviews, textbook chapters, and statements from various institutions involved in resident training.

METHODS OF STUDY SELECTION

A literature search of Google Scholar and PubMed using terms related to robotic surgery and robotics training, as mentioned in the "Data Sources" section.

RESULTS

Multiple novel platforms that use machine learning and real-time video feedback to teach and evaluate robotic surgical skills have been developed over recent years. Various training curricula, virtual reality simulators, and other robotic training tools have been shown to enhance robotic surgical education and improve surgical skills. The integration of didactic learning, simulation, and intraoperative teaching into more comprehensive training curricula shows positive effects on robotic skills proficiency. Few robotic surgery training curricula have been validated through peer-reviewed study, and there is more work to be completed in this area. In addition, there is a lack of information about how the skills obtained through robotics curricula and simulation translate into operating room performance and patient outcomes.

CONCLUSION

Data collected to date show promising advances in the training of robotic surgeons. A diverse array of curricula for training robotic surgeons continue to emerge, and existing teaching modalities are evolving to keep up with the rapidly growing demand for proficient robotic surgeons. Futures areas of growth include establishing competency benchmarks for existing training tools, validating existing curricula, and determining how to translate the acquired skills in simulation into performance in the operating room and patient outcomes. Many surgical training platforms are beginning to expand beyond discrete robotic skills training to procedure-specific and team training. There is still a wealth of research to be done to understand how to create an effective training experience for gynecologic surgical trainees and robotics teams.

Surgical Training 2.0: A systematic approach reviewing the literature focusing on oral maxillofacial surgery - Part II

PURPOSE

Many technologies are emerging in the medical field. Having an overview of the technological arsenal available to train new surgeons seems very interesting to guide subsequent surgical training protocols.

METHODS

This article is a systematic approach reviewing new technologies in surgical training, in particular in oral and maxillofacial surgery. This review explores what new technologies can do compared to traditional methods in the field of surgical education. A structured literature search of PubMed was performed in adherence to PRISMA guidelines. The articles were selected when they fell within predefined inclusion criteria while respecting the key objectives of this systematic review. We looked at medical students and more specifically in surgery and analysed whether exposure to new technologies improved their surgical skills compared to traditional methods. Each technology is reviewed by highlighting its advantages and disadvantages and studying the feasibility of integration into current practice.

RESULTS

The results are encouraging. Indeed, all of these technologies make it possible to reduce the learning time, the operating times, the operating complications and increase the enthusiasm of the students compared to more conventional methods. The start-up cost, the complexity to develop new models, and the openness of mind necessary for the integration of these technologies are all obstacles to immediate development. The main limitations of this review are that many of the studies have been carried out on small numbers, they are not interested in acquiring knowledge or skills over the long term and obviously there is a publication bias.

CONCLUSION

Surgical education methods will probably change in the years to come, integrating these new technologies into the curriculum seems essential so as not to remain on the side. This second part therefore reviews, social networks, serious games and virtual reality. This Systematic review is registered on PROSPERO (CRD42020181376).

Robotic Surgery Training: Current Trends and Future Directions

The use of robotic surgery in urology has grown exponentially in the past 2 decades, but robotic surgery training has lagged behind. Most graduating residents report a lack of comfort independently performing common robotic urologic surgeries, despite an abundance of available resources. There is a general consensus on the key components of a comprehensive robotics curriculum, and well-validated tools have been developed to assess trainee competency. However, no single curriculum has emerged as the gold standard on which individual programs can build their own robotics curricula.

Crowd-Sourced and Attending Assessment of General Surgery Resident Operative Performance Using Global Ratings Scales

OBJECTIVE

We sought to assess the extent to which both crowd and intraoperative attending ratings using objective structured assessment of technical skill (OSATS) or global objective assessment of laparoscopic skills (GOALS) would correlate with the system for improving procedural learning (SIMPL) Zwisch and Performance scales.

DESIGN

Comparison of directly observed versus crowd sourced review of operative video.

SETTING

Operative video captured at 2 institutions.

PARTICIPANTS

Six (6) core general surgery procedures, 3 open and 3 laparoscopic, were selected from the American Board of Surgery's Resident Assessments list. Thirty-two cases performed by General Surgery residents across all training levels at 2 institutions were filmed. Videos were condensed using a standardized protocol to include the critical portion of the procedure.  Condensed videos were then submitted to crowd-sourced assessment of technical skills (C-SATS), an online crowd source-driven assessment service, for assessment using the appropriate resident assessment form (GOALS or OSATS) as well as with the SIMPL Zwisch and Performance scales. Crowd workers watched an educational tutorial on how to use the Zwisch and SIMPL Performance rating scales prior to participating. Attendings scored residents using the same tools immediately after the shared operative experience. Statistical analysis was performed using Pearson's correlation coefficient.

RESULTS

Crowd raters evaluated 32 procedures using GOALS/OSATS, Zwisch and Performance (35-50 ratings per video). Attendings also evaluated all 32 procedures using GOALS/OSATS and 26 of the procedures using SIMPL Zwisch and Performance. Pearson correlation coefficients with 95% confidence intervals for crowd ratings were: GOALS and Zwisch -0.40 [-0.73 to 0.10], OSATS and Zwisch 0.11 [-0.41 to 0.57], GOALS and Performance -0.06 [-0.44 to 0.35], and OSATS and Performance 0.22 [-0.46 to 0.20]. Pearson correlation coefficients for attendings were: GOALS and Zwisch (0.77), OSATS and Zwisch (0.65), GOALS and Performance (0.93), and OSATS and Performance (0.59).

CONCLUSIONS

Overall, correlations between crowd-sourced ratings using GOALS/OSATS and SIMPL global operative performance ratings tools were weak, yet for attendings, they were strong. Direct attending assessment may be required for evaluation of global performance while crowd sourcing may be more suitable for technical assessment.  Further studies are needed to see if more extensive crowd training would result in improved ability for global performance evaluation.

Bidirectional long short-term memory for surgical skill classification of temporally segmented tasks

PURPOSE

The majority of historical surgical skill research typically analyzes holistic summary task-level metrics to create a skill classification for a performance. Recent advances in machine learning allow time series classification at the sub-task level, allowing predictions on segments of tasks, which could improve task-level technical skill assessment.

METHODS

A bidirectional long short-term memory (LSTM) network was used with 8-s windows of multidimensional time-series data from the Basic Laparoscopic Urologic Skills dataset. The network was trained on experts and novices from four common surgical tasks. Stratified cross-validation with regularization was used to avoid overfitting. The misclassified cases were re-submitted for surgical technical skill assessment to crowds using Amazon Mechanical Turk to re-evaluate and to analyze the level of agreement with previous scores.

RESULTS

Performance was best for the suturing task, with 96.88% accuracy at predicting whether a performance was an expert or novice, with 1 misclassification, when compared to previously obtained crowd evaluations. When compared with expert surgeon ratings, the LSTM predictions resulted in a Spearman coefficient of 0.89 for suturing tasks. When crowds re-evaluated misclassified performances, it was found that for all 5 misclassified cases from peg transfer and suturing tasks, the crowds agreed more with our LSTM model than with the previously obtained crowd scores.

CONCLUSION

The technique presented shows results not incomparable with labels which would be obtained from crowd-sourced labels of surgical tasks. However, these results bring about questions of the reliability of crowd sourced labels in videos of surgical tasks. We, as a research community, should take a closer look at crowd labeling with higher scrutiny, systematically look at biases, and quantify label noise.

Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery

INTRODUCTION

Operative time has been traditionally used as a proxy for surgical skill and is commonly utilized to measure the learning curve, assuming that faster operations indicate a more skilled surgeon. The Global Evaluative Assessment of Robotic Skills (GEARS) rubric is a validated Likert scale for evaluating technical skill. We hypothesize that operative time will not correlate with the GEARS score.

METHODS

Patients undergoing elective robotic sleeve gastrectomy at a single bariatric center of excellence hospital from January 2019 to March 2020 were captured in a prospectively maintained database. For step-specific scoring, videos were broken down into three steps: ligation of short gastric vessels, gastric transection, and oversewing the staple line. Overall and step-specific GEARS scores were assigned by crowd-sourced evaluators. Correlation between operative time and GEARS score was assessed with linear regression and calculation of the R² statistic.

RESULTS

Sixty-eight patients were included in the study, with a mean operative time of 112 ± 27.4 min. The mean GEARS score was 20.1 ± 0.81. Mean scores for the GEARS subcomponents were: bimanual dexterity 4.06 ± 0.17; depth perception 3.96 ± 0.24; efficiency 3.82 ± 0.19; force sensitivity 4.06 ± 0.20; robotic control 4.16 ± 0.21. Operative time and overall score showed no correlation (R² = 0.0146, p = 0.326). Step-specific times and scores showed weak correlation for gastric transection (R² = 0.0737, p = 0.028) and no correlation for ligation of short gastric vessels (R² = 0.0262, p = 0.209) or oversewing the staple line (R² = 0.0142, p = 0.344).

CONCLUSIONS

Operative time and crowd-sourced GEARS score were not correlated. Operative time and GEARS scores measure different performance characteristics, and future studies should consider using both a validated skills assessment tool and operative time for a more complete evaluation of skill.

Forensic Footwear Reliability: Part II-Range of Conclusions, Accuracy, and Consensus

Between February 2017 and August 2018, West Virginia University conducted a reliability study to determine expert performance among forensic footwear examiners in the United States. Throughout the study's duration, 70 examiners each performed 12 comparisons and reported a total of 840 conclusions. In order to assess the accuracy of conclusions, the similarities and differences between mated and nonmated pairs were evaluated according to three criteria: (i) inherent agreement/disagreement in class, wear, and randomly acquired features, (ii) limitations as a function of questioned impression quality, clarity, and totality, and (iii) adherence to the Scientific Working Group for Shoeprint and Tire Tread Evidence (SWGTREAD) 2013 conclusion standard. Using these criteria, acceptable/expected categorical conclusions were defined. Preliminary results from this study are divided into a series of three summaries. This manuscript (Part II) reports accuracy and reproducibility. For mated pairs, accuracy equals 76.3% ± 13.0% (median of 78.6% and a 90% confidence interval between 72.2% and 80.0%). For nonmated pairs, accuracy equals 87.4% ± 9.24% (median of 91.4% and a 90% confidence interval between 84.7% and 89.8%). In addition, the community assessed agreement (denoted by IQR) of reported results equals the research team's accepted/expected conclusions for 10 out of 12 comparisons. In terms of reproducibility, the 90% confidence interval for consensus was computed and found to equal 0.71-0.86 (median of 0.77) for the combined dataset. Although based on a limited sample size, these results provide a baseline estimate of accuracy and consensus/reproducibility as a function of the existing seven-point SWGTREAD 2013 conclusion standard.

Innovation in Orthopaedic Surgery Education: Novel Tools for Modern Times

Orthopaedic residency training has and will continue to evolve with a wide variety of changes. Hands-on surgical simulation "boot camps," computerized simulation of surgical process, and even virtual reality simulators, all can help trainees acquire surgical experience without compromising patient care. Low-cost training modules help remedy the rising costs associated with teaching complex orthopaedic surgery skills. Motion tracking and checklists help refine standardization of assessment. As technology and healthcare systems continue to grow, we encourage training programs to keep pace by considering engagement of these tools.

Temporal variability of surgical technical skill perception in real robotic surgery

PURPOSE

Summary score metrics, either from crowds of non-experts, faculty surgeons or from automated performance metrics, have been trusted as the prevailing method of reporting surgeon technical skill. The aim of this paper is to learn whether there exist significant fluctuations in the technical skill assessments of a surgeon throughout long durations of surgical footage.

METHODS

A set of 12 videos of robotic surgery cases from common human patient robotic surgeries were used to evaluate the perceived technical skill at each individual minute of the surgical videos, which were originally 12-15 min in length. A linear mixed-effects model for each video was used to compare the ratings of each minute to those from every other minute in order to learn whether a change in scores over time can be detected and reliably measured apart from inter- and intrarater variation.

RESULTS

Modeling the change over time of the global evaluative assessment of robotic skills scores significantly contributed to the prediction models for 11 of the 12 surgeons. This demonstrates that measurable changes in technical skill occur over time during robotic surgery.

CONCLUSION

The findings from this research raise questions about the optimal duration of footage needed to be evaluated to arrive at an accurate rating of surgical technical skill for longer procedures. This may imply non-negligible label noise for supervised machine learning approaches. In the future, it may be necessary to report a surgeon's skill variability in addition to their mean score to have proper knowledge of a surgeon's overall skill level.

Crowdsourcing: A novel tool to elicit the student voice in the curriculum design process for an undergraduate diagnostic radiography degree programme

INTRODUCTION

Stakeholder participation in healthcare curriculum design is an important aspect of higher education with stakeholders including students, staff members, clinical partners, healthcare organisations, patients and members of the public. Significantly, student co-creation, of the curriculum, has become increasingly important. Yet there is limited research which addresses how to engage this group in design processes.

METHODS

This paper represents the first phase of a three stage action research spiral whereby the authors evaluated the use of a novel tool for curriculum design processes, anonymised crowdsourcing. This initial phase was open to all students enrolled on an undergraduate diagnostic radiography programme in the UK. To confirm the reliability of the crowdsource design an established eight point crowdsourcing verification tool was applied.

RESULTS

Twenty-three unique ideas were generated by participants, 40 comments made and 173 votes cast. Inductive analysis of the comments generated five themes. These included: the role of technology enhanced learning; simulation activities; patient focused curriculum; mental wealth (resilience) authentic assessment approaches. An evaluation of those who had and had not engaged highlighted areas of improvement for the administration of the second and third iterations which will include a wider pool of participants.

CONCLUSION

This study from a single programme offers lessons for others wishing to adopt and develop this approach elsewhere.

IMPLICATIONS FOR PRACTICE

Several ideas elicited by the crowdsource have been considered by the curriculum design team and will be implemented in the 2020 curriculum thus demonstrating the impact on local education practice of this research approach.

Development and validation of a transanal endoscopic rectal purse string simulator

BACKGROUND

Rectal purse string placement for transanal mesorectal excision is challenging, and practice is difficult. The objective of this study is to build an endoscopic rectal purse string simulator and provide evidence for the validity of its use.

METHODS

A low-cost transanal endoscopic rectal purse string simulator was created and used to measure the performance of participants. Participants included general surgery residents, fellows, and staff surgeons from several Canadian university-affiliated institutions across Canada. The performance of the rectal purse string placement was measured by the time to completion of the task as well scoring with a modified objective structured assessment of technical skills (OSATS) score and a modified advanced laparoscopic suturing (ALS) score.

RESULTS

Thirty-nine participants were recruited into the study. Participants were split into three groups, based on prior experience with laparoscopic suturing, for the analysis of suturing a rectal purse string on the simulator based on three performance measures. There was a significant difference found in all three measures of performance in the three groups (time to completion p = 0.014, mean blinded OSATS score p = 0.007, mean blinded ALS score p = 0.020). Participants with previous laparoscopic suturing had significantly faster times to completion and higher skills scores when compared to residents (time: 5.1 ± 1.4 min vs 9.0 ± 4.8 min, p = 0.005; OSATS: 19.7 ± 2.8 vs 13.0 ± 5.8, p = 0.00398, ALS: 27.4 ± 4.0 vs 18.9 ± 8.5, p = 0.0151).

CONCLUSIONS

A transanal endoscopic purse string simulator was constructed and preliminary testing has shown variable performance based on prior laparoscopic suturing experience.

The effect of video playback speed on surgeon technical skill perception

PURPOSE

 Finding effective methods of discriminating surgeon technical skill has proved a complex problem to solve computationally. Previous research has shown that obtaining non-expert crowd evaluations of surgical performances is as accurate as the gold standard, expert surgeon review. The aim of this research is: (1) to learn whether crowdsourced evaluators give higher ratings of technical skill to video of performances with increased playback speed, (2) its effect in discriminating skill levels, and (3) whether this increase is related to the evaluator consciously being aware that the video is manually manipulated.

METHODS

 A set of ten peg transfer videos (five novices, five experts) were used to evaluate the perceived technical skill of the performers at each video playback speed used ([Formula: see text]). Objective metrics used for measuring technical skill were also computed for comparison by manipulating the corresponding kinematic data of each performance. Two videos of an expert and novice performing dry laboratory laparoscopic trials of peg transfer tasks were used to obtain evaluations at each playback speed ([Formula: see text]) of perception of whether a video is played at real-time playback speed or not.

RESULTS

 We found that while both novices and experts had increased perceived technical skill as the video playback was increased, the amount of increase was significantly greater for experts. Each increase in the playback speed by [Formula: see text] was associated with, on average, a 0.72-point increase in the GOALS score (95% CI 0.60-0.84 point increase; [Formula: see text]) for expert videos and only a 0.24-point increase in the GOALS score (95% CI 0.13-0.36 point increase; [Formula: see text]) for novice videos.

CONCLUSION

 Due to the differential increase in perceived technical skill due to increased playback speed for experts, the difference between novice and expert skill levels of surgical performances may be more easily discerned by manually increasing the video playback speed.

Performance assessment - The knowledge, skills and attitudes of surgical performance

Pediatric surgical education has traditionally focused on the hard cognitive and psychomotor (technical) skills. While more and more attention is being paid to softer skills such as communication, collaboration, leadership, health advocacy, professionalism and scholarship, the bulk of curricula remain focused on the hard skills. An intricate part of education is the assessment of performance. This article reviews the current literature on the assessment of cognitive and psychomotor skills, focusing particularly on the assessment of technical skills in the realm of simulation.

A video anchored rating scale leads to high inter-rater reliability of inexperienced and expert raters in the absence of rater training

BACKGROUND

Our objective was to assess the impact of incorporating videos in a behaviorally anchored performance rating scale on the inter-rater reliability (IRR) of expert, intermediate and novice raters.

METHODS

The Intra-corporeal Suturing Assessment Tool (ISAT) was modified to include short video clips demonstrating poor, average, and expert performances. Blinded raters used this tool to assess videos of trainees performing suturing on a porcine model. Three attending surgeons, 4 residents, and 4 novice raters participated; no rater training was provided. The IRR was then compared among rater groups.

RESULTS

The IRR using the modified ISAT was high at 0.80 (p < 0.001). Ratings were significantly correlated with trainee objective suturing scores for all rater groups (experts: R = 0.84, residents: R = 0.81, and novices: R = 0.69; p < 0.001).

CONCLUSIONS

Incorporating video anchors (to define performance) in the ISAT led to high IRR and enabled novices to achieve similar consistency in their ratings as experts.

Video review for measuring and improving skill in urological surgery

Interest is growing within the urological surgery community for objective assessments of technical skill. Surgical video review relies on the use of objective assessment tools to evaluate both global and procedure-specific skill. These evaluations provide structured feedback to surgeons with the aim of improving technique, which has been associated with patient outcomes. Currently, skill assessments can be performed by using expert peer-review, crowdsourcing or computer-based methods. Given the relationship between skill and patient outcomes, surgeons might be required in the future to provide empirical evidence of their technical skill for certification, employment, credentialing and quality improvement. Interventions such as coaching and skills workshops incorporating video review might help surgeons improve their skill, with the ultimate goal of improving patient outcomes.

Crowdsourcing in health and medical research: a systematic review

Background

Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health.

Methods

We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach.

Results

We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardio-pulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-of-hospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and layperson-initiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled.

Conclusions

Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine.

Trial registration

PROSPERO: CRD42017052835. December 27, 2016.

Crowdsourced Assessment of Inanimate Biotissue Drills: A Valid and Cost-Effective Way to Evaluate Surgical Trainees

OBJECTIVE

Providing feedback to surgical trainees is a critical component for assessment of technical skills, yet remains costly and time consuming. We hypothesize that statistical selection can identify a homogenous group of nonexpert crowdworkers capable of accurately grading inanimate surgical video.

DESIGN

Applicants auditioned by grading 9 training videos using the Objective Structured Assessment of Technical Skills (OSATS) tool and an error-based checklist. The summed OSATS, summed errors, and OSATS summary score were tested for outliers using Cronbach's Alpha and single measure intraclass correlation. Accepted crowdworkers then submitted grades for videos in 3 different compositions: full video 1× speed, full video 2× speed, and critical section segmented video. Graders were blinded to this study and a similar statistical analysis was performed.

SETTING

The study was conducted at the University of Pittsburgh Medical Center (Pittsburgh, PA), a tertiary care academic teaching hospital.

PARTICIPANTS

Thirty-six premedical students participated as crowdworker applicants and 2 surgery experts were compared as the gold-standard.

RESULTS

The selected hire intraclass correlation was 0.717 for Total Errors and 0.794 for Total OSATS for the first hire group and 0.800 for Total OSATS and 0.654 for Total Errors for the second hire group. There was very good correlation between full videos at 1× and 2× speed with an interitem statistic of 0.817 for errors and 0.86 for OSATS. Only moderate correlation was found with critical section segments. In 1 year 275hours of inanimate video was graded costing $22.27/video or $1.03/minute.

CONCLUSIONS

Statistical selection can be used to identify a homogenous cohort of crowdworkers used for grading trainees' inanimate drills. Crowdworkers can distinguish OSATS metrics and errors in full videos at 2× speed but were less consistent with segmented videos. The program is a comparatively cost-effective way to provide feedback to surgical trainees.

Crowdsourcing in health professions education: What radiography educators can learn from other disciplines

OBJECTIVES

Crowdsourcing works through an institution outsourcing a function normally performed by an employee or group of individuals. Within a crowdsource users, known as the crowd, form a community who voluntarily undertake a task which involves the pooling of knowledge resources. A literature review was undertaken to identify how the tool is being used in health professions education, and potential for use in radiography education.

KEY FINDINGS

17 papers were returned. Literature identified was assessed against an established crowdsourcing definition. Reviewing these yielded four themes for discussion: student selection procedures, lesson planning, teaching materials and assessment.

CONCLUSION

Crowdsourcing is associated with innovative activities through collective solution seeking via a large network of users. It is increasingly being adopted in healthcare training and maybe transferable to educational activities within the field of radiography education.