C-SATS: Assessing Surgical Skills Among Urology Residency Applicants

Medical Students Cannot Assess Robotic Surgeons Performing Radical Prostatectomy

INTRODUCTION

Medical students have previously been shown to be just as effective for video rating as experts. We want to compare medical students to experienced surgeons as video assessors of simulated robot-assisted radical prostatectomy (RARP) performance.

MATERIALS AND METHODS

Video recordings of three RARP modules on the RobotiX (formerly Simbionix) simulator from a previous study were used. Five novice surgeons, five experienced robotic surgeons, and five experienced robotic surgeons in RARP performed a total of 45 video-recorded procedures. The videos were assessed with the modified Global Evaluative Assessment of Robotic Skills tool as both full-length and an edited edition that only included the first 5 minutes of the procedure.

RESULTS

Fifty medical students and two experienced RARP surgeons (ES) performed a total of 680 video ratings of full-length videos and 5-minute videos (2-9 ratings per video). Medical students and ES showed poor agreement for both full-length videos and 5-minute videos (0.29 and -0.13, respectively). Medical students could not discriminate between the skill level of the surgeons in either full-length videos or 5-minute videos ( P = 0.053-0.36 and P = 0.21-0.82), whereas ES could discriminate between novice surgeons and experienced surgeons (full-length, P < 0.001, and 5 minutes, P = 0.007) and intermediate and experienced surgeons (full-length, P = 0.001, and 5 minutes, P = 0.01) in both full-length videos and 5-minute videos.

CONCLUSION

We found that medical students cannot be used to assess RARP because they showed poor agreement with the ES rating for both full-length videos and 5-minute videos. Medical students could not discriminate between surgical skill levels.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Does Surgeon Experience Correlate with Crowd-Sourced Skill Assessment in Robotic Bariatric Surgery?

BACKGROUND

The Global Evaluative Assessment of Robotic Skills (GEARS) rubric provides a measure of skill in robotic surgery. We hypothesize surgery performed by more experienced operators will be associated with higher GEARS scores.

METHOD

Patients undergoing sleeve gastrectomy from 2016 to 2020 were analyzed. Three groups were defined by time in practice: less than 5, between 5 and 15, and more than 15 years. Continuous variables were compared with ANOVA and multivariable regression was performed.

RESULTS

Fourteen operators performing 154 cases were included. More experienced surgeons had higher GEARS scores and shorter operative times. On multivariable regression, operative time (P = 0.027), efficiency (P = .022), depth perception (P = 0.033), and bimanual dexterity (P = 0.047) were associated with experience.

CONCLUSIONS

In our video-based assessment (VBA) model, operative time and several GEARS subcomponent scores were associated with surgical experience. Further studies should determine the association between these metrics and surgical outcomes.

Laypersons versus experienced surgeons in assessing simulated robot-assisted radical prostatectomy

BACKGROUND

Feedback is important for surgical trainees but it can be biased and time-consuming. We examined crowd-sourced assessment as an alternative to experienced surgeons' assessment of robot-assisted radical prostatectomy (RARP).

METHODS

We used video recordings (n = 45) of three RARP modules on the RobotiX, Simbionix simulator from a previous study in a blinded comparative assessment study. A group of crowd workers (CWs) and two experienced RARP surgeons (ESs) evaluated all videos with the modified Global Evaluative Assessment of Robotic Surgery (mGEARS).

RESULTS

One hundred forty-nine CWs performed 1490 video ratings. Internal consistency reliability was high (0.94). Inter-rater reliability and test-retest reliability were low for CWs (0.29 and 0.39) and moderate for ESs (0.61 and 0.68). In an Analysis of Variance (ANOVA) test, CWs could not discriminate between the skill level of the surgeons (p = 0.03-0.89), whereas ES could (p = 0.034).

CONCLUSION

We found very low agreement between the assessments of CWs and ESs when they assessed robot-assisted radical prostatectomies. As opposed to ESs, CWs could not discriminate between surgical experience using the mGEARS ratings or when asked if they wanted the surgeons to perform their robotic surgery.

Modeling Technical Skills Learning Curves of Incoming Surgical Trainees

OBJECTIVE

Not all trainees reach technical competency even after completing surgical training. While assessment of technical skill is not part of the residency interview process, identifying under-performers early on may help identify opportunities for individualized, targeted training. The objectives of this study were to (1) create predictive learning curve (LC) models for each of 3 basic laparoscopic tasks to identify performers versus underperformers and (2) evaluate the use of LCs to identify underperformers during selection into surgical training.

DESIGN

Predictive LC models were created for laparoscopic pattern cutting (PC), peg transfer (PT) and intra-corporeal knots (IC) over 40 repetitions by 65 novice trainees in 2014. Trainees were categorized as performers and underperformers. Receiver operator characteristic analysis determined the minimum number of repetitions required to predict individual LCs, which were then used to determine the proportion of underperformers.

SETTING

Technical performance was assessed onsite at the Canadian Residence Matching Service (CaRMS) interviews, after interview completion (January 2015).

PARTICIPANTS

Applicants to general surgery (GS) and gynecology (OBGYN) participated in a skills assessment during.

RESULTS

The PC, PT and IC tasks required a minimum of 8, 10, and 5 repetitions respectively, to predict overall performance. Predictive values for each task had excellent sensitivity and specificity: 1.00, 1.00 (PC); 1.00, 1.00 (PT); and 0.94, 1.00 (IC). Eighty applicants completed 8 PC repetitions; 16% were identified as underperformers.

CONCLUSIONS

Individual LCs for three different laparoscopic tasks can be predicted with excellent sensitivity and specificity based on 10 repetitions or less. This information can be used to identify trainees who may have difficulty with laparoscopic technical skills early on.

Innovations in Urologic Surgical Training

PURPOSE OF REVIEW

This review aims to summarize innovations in urologic surgical training in the past 5 years.

RECENT FINDINGS

Many assessment tools have been developed to objectively evaluate surgical skills and provide structured feedback to urologic trainees. A variety of simulation modalities (i.e., virtual/augmented reality, dry-lab, animal, and cadaver) have been utilized to facilitate the acquisition of surgical skills outside the high-stakes operating room environment. Three-dimensional printing has been used to create high-fidelity, immersive dry-lab models at a reasonable cost. Non-technical skills such as teamwork and decision-making have gained more attention. Structured surgical video review has been shown to improve surgical skills not only for trainees but also for qualified surgeons. Research and development in urologic surgical training has been active in the past 5 years. Despite these advances, there is still an unfulfilled need for a standardized surgical training program covering both technical and non-technical skills.

Quantifying the "Assistant Effect" in Robotic-Assisted Radical Prostatectomy (RARP): Measures of Technical Performance

PURPOSE

Surgeons are reliant on the bedside assistant during robotic surgeries. Using a modified global rating scale (GRS), we aim to assess the association between an assistant's technical skill on surgeon performance in Robotic-Assisted Radical Prostatectomy (RARP).

METHODS

Prospective, intraoperative video from RARP cases at three centers were collected. Baseline demographic and RARP-experience data were collected from participating surgeons and trainees. The dissection of the prostatic pedicle and neurovascular bundle step (NVB) was analyzed. Expert analysts scored the console surgeon performance using the Global Evaluative Assessment of Robotic Skills (GEARS), and the bedside assistant performance using a modified Objective Structured Assessment of Technical Skills (aOSATS). The primary outcome is the association between console surgeon performance, as measured by GEARS, and assistant skill, as measured by aOSATS. Spearman's rho correlations were used to test the relationship between assistant and surgeon technical performance, and a multivariable linear regression model was created to test this association while controlling for patient factors.

RESULTS

92 RARP cases were available for the analysis, comprising 14 console surgeons and 22 different bedside assistants. In only 5 (5.4%) cases, the neurovascular bundle step was completed by a trainee, and in 13 (14.1%) of cases, a staff-level surgeon acted as the bedside assistant. aOSATS score was significantly associated with robotic console experience (P = 0.011), and prior laparoscopic experience (P < 0.001). Assistant aOSATS score showed a weak but significant correlation with surgeon GEARS score during the neurovascular bundle step (spearman's rho = 0.248, P = 0.028). On linear regression, aOSATS remained a significant predictor of console surgeon performance (P = 0.016), after controlling for patient age and BMI, prostate volume, tumor stage, and presence of nerve-sparing.

CONCLUSIONS

This is the first study to assess the association between assistant technical skill and surgeon performance in RARP. Additionally, we have provided validity evidence for a modified OSATS global rating scale for training and assessing bedside assistant performance.

Expert Consensus Recommendations for Robotic Surgery Credentialing

OBJECTIVE

To define criteria for robotic credentialing using expert consensus.

BACKGROUND

A recent review of institutional robotic credentialing policies identified significant variability and determined current policies are largely inadequate to ensure surgeon proficiency and may threaten patient safety.

METHODS

28 national robotic surgery experts were invited to participate in a consensus conference. After review of available institutional policies and discussion, the group developed a 91 proposed criteria. Using a modified Delphi process the experts were asked to indicate their agreement with the proposed criteria in three electronic survey rounds after the conference. Criteria that achieved 80% or more in agreement (consensus) in all rounds were included in the final list.

RESULTS

All experts agreed that there is a need for standardized robotic surgery credentialing criteria across institutions that promote surgeon proficiency. 49 items reached consensus in the first round, 19 in the second, and 8 in the third for a total of 76 final items. Experts agreed that privileges should be granted based on video review of surgical performance and attainment of clearly defined objective proficiency benchmarks. Parameters for ongoing outcome monitoring were determined and recommendations for technical skills training, proctoring, and performance assessment were defined.

CONCLUSIONS

Using a systematic approach, detailed credentialing criteria for robotic surgery were defined. Implementation of these criteria uniformly across institutions will promote proficiency of robotic surgeons and has the potential to positively impact patient outcomes.

Resident Selection in the Wake of United States Medical Licensing Examination Step 1 Transition to Pass/Fail Scoring

INTRODUCTION

The numeric score for the United States Medical Licensing Examination Step 1 is one of the only universal, objective, scaled criteria for comparing the many students who apply to orthopaedic surgery residency. However, on February 12, 2020, it was announced that Step 1 would be transitioning to pass/fail scoring. The purpose of this study was to (1) determine the most important factors used for interview and resident selection after this change and (2) to assess how these factors have changed compared with a previous report on resident selection.

METHODS

A survey was distributed to the program directors (PDs) of all 179 orthopaedic surgery programs accredited by the Accreditation Council for Graduate Medical Education. Questions focused on current resident selection practices and the impact of the Step 1 score transition on expected future practices.

RESULTS

A total of 78 PDs (44%) responded to the survey. Over half of PDs (59%) responded that United States Medical Licensing Examination Step 2 clinical knowledge (CK) score is the factor that will increase most in importance after Step 1 transitions to pass/fail, and 90% will encourage applicants to include their Step 2 CK score on their applications. The factors rated most important in resident selection from zero to 10 were subinternship performance (9.05), various aspects of interview performance (7.49 to 9.01), rank in medical school (7.95), letters of recommendation (7.90), and Step 2 CK score (7.27). Compared with a 2002 report, performance on manual skills testing, subinternship performance, published research, letters of recommendations, and telephone call on applicants' behalf showed notable increases in importance.

DISCUSSION

As Step 2 CK is expected to become more important in the residency application process, current applicant stress on Step 1 scores may simply move to Step 2 CK scores. Performance on subinternships will remain a critical aspect of residency application, as it was viewed as the most important resident selection factor and has grown in importance compared with a previous report.

Best Approaches to Evaluation and Feedback in Post-Graduate Medical Education

PURPOSE OF REVIEW

The objectives of this literature review are to appraise current approaches and assess new technologies that have been utilized for evaluation and feedback of residents, with focus on surgical trainees.

RECENT FINDINGS

In 1999, the Accreditation Council for Graduate Medical Education introduced the Milestone system as a tool for summative evaluation. The organization allows individual program autonomy on how evaluation and feedback are performed. In the past, questionnaire evaluations and informal verbal feedback were employed. However, with the advent of technology, they have taken a different shape in the form of crowdsourcing, mobile platforms, and simulation. Limited data is available on new methods but studies show promise citing low cost and positive impact on resident education. No one "best approach" exists for evaluation and feedback. However, it is apparent that a multimodal approach that is based on the ACGME Milestones can be effective and aid in guiding programs.

Evaluation of Surgical Skills during Robotic Surgery by Deep Learning-Based Multiple Surgical Instrument Tracking in Training and Actual Operations

As the number of robotic surgery procedures has increased, so has the importance of evaluating surgical skills in these techniques. It is difficult, however, to automatically and quantitatively evaluate surgical skills during robotic surgery, as these skills are primarily associated with the movement of surgical instruments. This study proposes a deep learning-based surgical instrument tracking algorithm to evaluate surgeons’ skills in performing procedures by robotic surgery. This method overcame two main drawbacks: occlusion and maintenance of the identity of the surgical instruments. In addition, surgical skill prediction models were developed using motion metrics calculated from the motion of the instruments. The tracking method was applied to 54 video segments and evaluated by root mean squared error (RMSE), area under the curve (AUC), and Pearson correlation analysis. The RMSE was 3.52 mm, the AUC of 1 mm, 2 mm, and 5 mm were 0.7, 0.78, and 0.86, respectively, and Pearson’s correlation coefficients were 0.9 on the x-axis and 0.87 on the y-axis. The surgical skill prediction models showed an accuracy of 83% with Objective Structured Assessment of Technical Skill (OSATS) and Global Evaluative Assessment of Robotic Surgery (GEARS). The proposed method was able to track instruments during robotic surgery, suggesting that the current method of surgical skill assessment by surgeons can be replaced by the proposed automatic and quantitative evaluation method.

Crowdsourcing in health and medical research: a systematic review

Background

Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health.

Methods

We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach.

Results

We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardio-pulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-of-hospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and layperson-initiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled.

Conclusions

Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine.

Trial registration

PROSPERO: CRD42017052835. December 27, 2016.

Simulation-based training in urology residency programmes in the USA: Results of a nationwide survey

Objective

To evaluate the current usage of simulation in urological education in the USA and the barriers to incorporating a simulation-based educational curriculum, as the shift towards competency-based medical education has necessitated the introduction of simulation for training and assessing both non-technical and technical skills.

Materials and methods

Residency programme directors at Accreditation Council for Graduate Medical Education (ACGME)-accredited urology training programmes in the USA were invited to respond to an anonymous electronic survey. The study evaluated the programme directors’ experiences and opinions for the current usage of existing urology simulators. The survey also elicited receptiveness and the barriers for incorporating simulation-based training curricula within urology training programmes.

Results

In all, 43 completed surveys were received (35% response rate). Amongst responders, 97% (42/43) reported having access to a simulation education centre, and 60% (25/42) have incorporated simulation into their curriculum. A total of 87% (37/43) agreed that there is a role for a standardised simulator training curriculum, and 75% (30/40) agreed that simulators would improve operating room performance. A total of 64% (27/42) agreed that cost was a limiting factor, 12% (5/42) agreed on the cost-effectiveness of simulators, 35% (17/41) agreed there was an increased need for simulator education within work-hour limitations, and 38% (16/42) agreed a simulation programme would reduce patient risks and complications.

Conclusions

The majority of urology programme directors consider that there is a role for incorporating a simulation-based curriculum into urology training. Barriers to implementation include cost burden, need for constant technology updates, need for advanced planning, and willingness of faculty to participate in administration.

Crowdsourced versus expert evaluations of the vesico-urethral anastomosis in the robotic radical prostatectomy: is one superior at discriminating differences in automated performance metrics?

Crowdsourcing from the general population is an efficient, inexpensive method of surgical performance evaluation. In this study, we compared the discriminatory ability of experts and crowdsourced evaluators (the Crowd) to detect differences in robotic automated performance metrics (APMs). APMs (instrument motion tracking and events data directly from the robot system) of anterior vesico-urethral anastomoses (VUAs) of robotic radical prostatectomies were captured by the dVLogger (Intuitive Surgical). Crowdsourced evaluators and four expert surgeons evaluated video footage using the Global Evaluative Assessment of Robotic Skills (GEARS) (individual domains and total score). Cases were then stratified into performance groups (high versus low quality) for each evaluator based on GEARS. APMs from each group were compared using the Mann-Whitney U test. 25 VUAs performed by 11 surgeons were evaluated. The Crowd displayed moderate correlation with averaged expert scores for all GEARS domains (r > 0.58, p < 0.01). Bland-Altman analysis showed a narrower total GEARS score distribution by the Crowd compared to experts. APMs compared amongst performance groups for each evaluator showed that through GEARS scoring, the most common differentiated metric by evaluators was the velocity of the dominant instrument arm. The Crowd outperformed two out of four expert evaluators by discriminating differences in three APMs using total GEARS scores. The Crowd assigns a narrower range of GEARS scores compared to experts but maintains overall agreement with experts. The discriminatory ability of the Crowd at discerning differences in robotic movements (via APMs) through GEARS scoring is quite refined, rivaling that of expert evaluators.

Implementing assessments of robot-assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

OBJECTIVES

To systematically review and synthesise the validity evidence supporting intraoperative and simulation-based assessments of technical skill in urological robot-assisted surgery (RAS), and make evidence-based recommendations for the implementation of these assessments in urological training.

MATERIALS AND METHODS

A literature search of the Medline, PsycINFO and Embase databases was performed. Articles using technical skill and simulation-based assessments in RAS were abstracted. Only studies involving urology trainees or faculty were included in the final analysis.

RESULTS

Multiple tools for the assessment of technical robotic skill have been published, with mixed sources of validity evidence to support their use. These evaluations have been used in both the ex vivo and in vivo settings. Performance evaluations range from global rating scales to psychometrics, and assessments are carried out through automation, expert analysts, and crowdsourcing.

CONCLUSION

There have been rapid expansions in approaches to RAS technical skills assessment, both in simulated and clinical settings. Alternative approaches to assessment in RAS, such as crowdsourcing and psychometrics, remain under investigation. Evidence to support the use of these metrics in high-stakes decisions is likely insufficient at present.

E-learning teaches attendings "how to" objectively assess pediatric urology trainees' surgery skills for orchiopexy

INTRODUCTION

Established methods to train pediatric urology surgery by residency training programs require updating in response to administrative changes such as new, reduced trainee duty hours. Therefore, new objective methods must be developed to teach trainees. We approached this need by creating e-learning to teach attendings objective assessment of trainee skills using the Zwisch scale, an established assessment tool. The aim of this study was to identify whether or not e-learning is an appropriate platform for effective teaching of this assessment tool, by assessing inter-rater correlation of assessments made by the attendings after participation in the e-learning. Pediatric orchiopexy was used as the index case.

METHODS

An e-learning tool was created to teach attending surgeons objective assessment of trainees' surgical skills. First, e-learning content was created which showed the assessment method videotape of resident surgery done in the operating room. Next, attendings were enrolled to e-learn this method. Finally, the ability of enrollees to assess resident surgery skill performance was tested. Namely, test video was made showing a trainee performing inguinal orchiopexy. All enrollees viewed the same online videos. Assessments of surgical skills (Zwisch scale) were entered into an online survey. Data were analyzed by intercorrelation coefficient kappa analysis (strong correlation was ICC ≥ 0.7).

RESULTS

A total of 11 attendings were enrolled. All accessed the online learning and then made assessments of surgical skills trainees showed on videotapes. The e-learning comprised three modules: 1. "Core concepts," in which users learned the assessment tool methods; 2. "Learn to assess," in which users learned how to assess by watching video clips, explaining the assessment method; and 3. "Test," in which users tested their skill at making assessments by watching video clips and then actively inputting their ratings of surgical and global skills as viewed in the video clips (Figure). A total of 89 surgical skill ratings were performed with 56 (65%) exact matches between raters and 89 (100%) matched within one rank. Interclass correlation coefficient (ANOVA) showed statistically significant correlation. (r = 0.725, 95% CI 0.571-0.837, F = 3.976, p ≤ 0.00001). Kappa analysis of inter-rater reliability showed strong consensus between attendings for average measures with ICC = 0.71, 95% CI 0.46-0.95 (p = 0.03).

CONCLUSION

We launched e-learning to teach pediatric urology attendings "how to" assess trainee surgical skills objectively (Zwisch scale). After e-learning, there was strong inter-rater correlation in assessments made. We plan to extend such e-learning to pediatric urology surgical training programs.

Crowdsourcing Evaluation of Ureteroscopic Videos Using the Post-Ureteroscopic Lesion Scale to Assess Ureteral Injury

INTRODUCTION AND OBJECTIVES

We hypothesized that crowdsourcing assessments could be applied to the Postureteroscopic Lesion Scale (PULS) for ureteral injury.

METHODS

At a single institution, we prospectively digitally recorded 14 ureters at the terminal portion of standard ureteroscopic procedures. Each recording was reviewed by 10 global experts to determine a mean PULS score. Following training, the Crowd-Sourced Assessment of Technical Skills, C-SATS^® (C-SATS, Inc., Seattle, WA) platform was used to obtain crowd-based reviews. The mean crowd PULS scores was determined using the linear mixed-effects (LME) model. The intraclass correlation coefficient (ICC) was calculated to measure the agreement among experts. Spearman's rank correlation (rho) was used to quantify the strength of the relationship between the crowd LME mean and the experts.

RESULTS

Ten expert's reviews and 2100 layman reviews were obtained in 21 days and 49 hours, respectively. The ICC for the 10 experts was 0.68 (95% confidence interval 0.49, 0.86). When the expert mean PULS was <1, the crowd scored those recordings at 1 or greater. The highest scored recording by the experts was a 3.2, which the crowd scored at 2.25. The correlation between the crowd LME means and expert means across all videos was 0.70 (p = 0.0056) indicative of moderately strong agreement.

CONCLUSION

In this initial application of crowd-sourced evaluation of ureteral injury, there was a moderately strong correlation between crowd and expert ratings. Refinement of the training, through exposure to the nuances of ureteral injuries, in particular for PULS <1 or ≥3, may lead to better crowd/expert correlation. Compared to expert review, crowd data can be collected with much greater efficiency.

Crowdsourcing Disease Biomarker Discovery Research: The IP4IC Study

PURPOSE

Biomarker discovery is limited by readily assessable, cost efficient human samples available in large numbers that represent the entire heterogeneity of the disease. We developed a novel, active participation crowdsourcing method to determine BP-RS (Bladder Permeability Defect Risk Score). It is based on noninvasive urinary cytokines to discriminate patients with interstitial cystitis/bladder pain syndrome who had Hunner lesions from controls and patients with interstitial cystitis/bladder pain syndrome but without Hunner lesions.

MATERIALS AND METHODS

We performed a national crowdsourcing study in cooperation with the Interstitial Cystitis Association. Patients answered demographic, symptom severity and urinary frequency questionnaires on a HIPAA (Health Insurance Portability and Accountability Act) compliant website. Urine samples were collected at home, stabilized with a preservative and sent to Beaumont Hospital for analysis. The expression of 3 urinary cytokines was used in a machine learning algorithm to develop BP-RS.

RESULTS

The IP4IC study collected a total of 448 urine samples, representing 153 patients (147 females and 6 males) with interstitial cystitis/bladder pain syndrome, of whom 54 (50 females and 4 males) had Hunner lesions. A total of 159 female and 136 male controls also participated, who were age matched. A defined BP-RS was calculated to predict interstitial cystitis/bladder pain syndrome with Hunner lesions or a bladder permeability defect etiology with 89% validity.

CONCLUSIONS

In this novel participation crowdsourcing study we obtained a large number of urine samples from 46 states, which were collected at home, shipped and stored at room temperature. Using a machine learning algorithm we developed BP-RS to quantify the risk of interstitial cystitis/bladder pain syndrome with Hunner lesions, which is indicative of a bladder permeability defect etiology. To our knowledge BP-RS is the first validated urine biomarker assay for interstitial cystitis/bladder pain syndrome and one of the first biomarker assays to be developed using crowdsourcing.

Crowdsourcing in Surgical Skills Acquisition: A Developing Technology in Surgical Education

BACKGROUND

The application of crowdsourcing to surgical education is a recent phenomenon and adds to increasing demands on surgical residency training. The efficacy, range, and scope of this technology for surgical education remains incompletely defined.

OBJECTIVE

A systematic review was performed using the PubMed database of English-language literature on crowdsourced evaluation of surgical technical tasks up to April 2017.

METHODS

Articles were reviewed, abstracted, and analyzed, and were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI). Articles were evaluated with eligibility criteria for inclusion. Study information, performance task, subjects, evaluative standards, crowdworker compensation, time to response, and correlation between crowd and expert or standard evaluations were abstracted and analyzed.

RESULTS

Of 63 unique publications initially identified, 13 with MERSQI scores ranging from 10 to 13 (mean = 11.85) were included in the review. Overall, crowd and expert evaluations demonstrated good to excellent correlation across a wide range of tasks (Pearson's coefficient 0.59-0.95, Cronbach's alpha 0.32-0.92), with 1 exception being a study involving medical students. There was a wide range of reported interrater variability among experts. Nonexpert evaluation was consistently quicker than expert evaluation (ranging from 4.8 to 150.9 times faster), and was more cost effective.

CONCLUSIONS

Crowdsourced feedback appears to be comparable to expert feedback and is cost effective and efficient. Further work is needed to increase consistency in expert evaluations, to explore sources of discrepant assessments between surgeons and crowds, and to identify optimal populations and novel applications for this technology.