Crowd-Sourced Assessment of Technical Skills for Validation of Basic Laparoscopic Urologic Skills Tasks

Intentional enterotomies: validation of a novel robotic surgery training exercise

While laparoscopic simulation-based training is a well-established component of general surgery training, no such requirement or standardized curriculum exists for robotic surgery. Furthermore, there is a lack of high-fidelity electrocautery simulation training exercises in the literature. Using Messick's validity framework, we sought to determine the content, response process, internal content and construct validity of a novel inanimate tissue model that utilizes electrocautery for potential incorporation in such curricula. A multi-institutional, prospective study involving medical students (MS) and general surgery residents (PGY1-3) was conducted. Participants performed an exercise using a biotissue bowel model on the da Vinci Xi robotic console during which they created an enterotomy using electrocautery, followed by approximation with interrupted sutures. Participant performance was recorded and then scored by crowd-sourced assessors of technical skill, along with three of the authors. Construct validity was determined via difference in Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and total number of errors between the two cohorts. Upon completion of the exercise, participants were surveyed on their perception of the exercise and its impact on their robotic training to determine content validity. 31 participants were enrolled and separated into two cohorts: MS + PGY1 vs. PGY2-3. Time spent on the robotic trainer (0.8 vs. 8.13 h, p = 0.002), number of bedside robotic assists (5.7 vs. 14.8, p < 0.001), and number of robotic cases as primary surgeon (0.3 vs. 13.1, p < 0.001) were statistically significant between the two groups. Differences in GEARS scores (18.5 vs. 19.9, p = 0.001), time to completion (26.1 vs. 14.4 min, p < 0.001), and total errors (21.5 vs. 11.9, p = 0.018) between the groups were statistically significant as well. Of the 23 participants that completed the post-exercise survey, 87% and 91.3% reported improvement in robotic surgical ability and confidence, respectively. On a 10-point Likert scale, respondents rated the realism of the exercise 7.5, educational benefit 9.1, and effectiveness in teaching robotic skills 8.7. Controlling for the upfront investment of certain training materials, each exercise iteration cost ~ $30. This study confirmed the content, response process, internal structure and construct validity of a novel, high-fidelity and cost-effective inanimate tissue exercise which successfully incorporates electrocautery. Consideration should be given to its addition to robotic surgery training programs.

Minimally Trained Analysts Can Perform Fast, Objective Assessment of Orthopedic Technical Skill from Fluoroscopic Images

Skill assessment in orthopedics has traditionally relied on subjective impressions from a supervising surgeon. The feedback derived from these tools may be limited by bias and other practical issues. Objective analysis of intraoperative fluoroscopic images offers an inexpensive, repeatable, and precise assessment strategy without bias. Assessors generally refrain from using the scores of images obtained throughout the operation to evaluate skill for practical reasons. A new system was designed to facilitate rapid analysis of this fluoroscopy via minimally trained analysts. Four expert and four novice analysts independently measured one objective metric for skill using both a custom analysis software and a commercial alternative. Analysts were able to measure the objective metric three times faster when using the custom software, and without a practical difference in accuracy in comparison to the expert analysts using the commercial software. These results suggest that a well-designed fluoroscopy analysis system can facilitate inexpensive, reliable, and objective assessment of surgical skills.

Crowd-Sourced Assessment of Surgical Skills of Urology Resident Applicants: Four-Year Experience

OBJECTIVE

To determine a) if surgical skills among urology resident applicants could be reliably assessed via crowdsourcing and b) to what extent surgical skills testing impacts resident selection.

DESIGN

Interviewees completed the following surgical skills tasks during their interview day: open knot tying (OKT), laparoscopic peg transfer (LPT), and robotic suturing (RS). Urology faculty and crowd-workers evaluated each applicant's video-recorded performance using validated scoring and were assessed for agreement using Cronbach's alpha. Applicants' USMLE scores, interview scores, and Jefferson Scale of Physician Empathy (JSPE-S) scores were assessed for correlation with skills testing scores and match rank. Additionally, a survey was distributed to interviewees assessing match outcomes.

SETTING

University of California Irvine Department of Urology, Surgical Skills Laboratory PARTICIPANTS: All 94 urology residency interviewees at the University of California Irvine Department of Urology from 2015-2018 were invited to complete the three surgical skills tasks on their interview day.

RESULTS

Survey responses were received from all 94 interviewees (100%). Crowd and expert agreement was good (α=0.88), fair (α=0.67), and poor (α=0.32) for LPT, RS, and OKT scores, respectively. The skills testing scores did not correlate with match rank, USMLE score, or JSPE-S score. On multivariate analysis, only interview score (r= -0.723; p<0.001) and faculty LPT score (r=-0.262; p=0.001) were significant predictors of match rank. Interviewees who reported matching into a top 3 residency choice had significantly higher faculty LPT scores than those who did not (11.9 vs. 9.7, p=0.03).

CONCLUSIONS

Surgical skills overall did not significantly impact match rank. Expert assessment of laparoscopic peg transfer skills and interview performance among urology resident applicants correlated with match rank.

Virtual Reality Warm-up Before Robot-assisted Surgery: A Randomized Controlled Trial

TRIAL DESIGN

This was a randomized controlled trial.

BACKGROUND

Intraoperative errors correlate with surgeon skill and skill declines with intervals of inactivity. The goals of this research were to identify the optimal virtual reality (VR) warm-up curriculum to prime a surgeon's technical skill and validate benefit in the operating room.

MATERIALS AND METHODS

Surgeons were randomized to receive six trial sessions of a designated set of VR modules on the da Vinci Skills Simulator to identify optimal VR warm-up curricula to prime technical skill. After performing their curricula, warm-up effect was assessed based on performance on a criterion task. The optimal warm-up curriculum was chosen from the group with the best task time and video review-based technical skill. Robot-assisted surgery-experienced surgeons were then recruited to either receive or not receive warm-up before surgery. Skill in the first 15 min of surgery was assessed by blinded surgeon and crowdworker review as well as tool motion metrics. The intervention was performing VR warm-up before human robot-assisted surgery. Warm-up effect was measured using objective performance metrics and video review using the Global Evaluative Assessment of Robotic Skills tool. Linear mixed effects models with a random intercept for each surgeon and nonparametric modified Friedman tests were used for analysis.

RESULTS

The group performing only a Running Suture task on the simulator was on average 31.3 s faster than groups performing other simulation tasks and had the highest Global Evaluative Assessment of Robotic Skills scores from 41 surgeons who participated. This was chosen as the optimal curriculum. Thereafter, 34 surgeons completed 347 surgeries with corresponding video and tool motion data. No statistically significant differences in skill were observed with the warm-up intervention.

CONCLUSIONS

We conclude that a robotic VR warm-up before performing the early stages of surgery does not impact the technical skill of the surgeon.

Innovations in Urologic Surgical Training

PURPOSE OF REVIEW

This review aims to summarize innovations in urologic surgical training in the past 5 years.

RECENT FINDINGS

Many assessment tools have been developed to objectively evaluate surgical skills and provide structured feedback to urologic trainees. A variety of simulation modalities (i.e., virtual/augmented reality, dry-lab, animal, and cadaver) have been utilized to facilitate the acquisition of surgical skills outside the high-stakes operating room environment. Three-dimensional printing has been used to create high-fidelity, immersive dry-lab models at a reasonable cost. Non-technical skills such as teamwork and decision-making have gained more attention. Structured surgical video review has been shown to improve surgical skills not only for trainees but also for qualified surgeons. Research and development in urologic surgical training has been active in the past 5 years. Despite these advances, there is still an unfulfilled need for a standardized surgical training program covering both technical and non-technical skills.

Crowdsourced Assessment of Aesthetic Outcomes after Breast Reconstruction

BACKGROUND

Evaluating the aesthetic success of breast reconstruction can be difficult. Patients, surgeons, and the general population may differ in what constitutes a successful outcome. Recently, crowdsourcing has emerged to accumulate and analyze data on a massive scale. The authors propose that crowdsourcing can be a useful tool to reliably rate aesthetic outcomes of breast reconstruction.

METHODS

One hundred one deidentified photographs of patients at various stages of breast reconstruction were gathered. Assessment tools included a five-point Likert scale and the transverse rectus abdominis myocutaneous (TRAM) visual assessment scale. Anonymous crowd workers and a group of expert reconstructive surgeons rated an identical set of photographs on the Likert scale. Crowd workers also rated the set of photographs on the TRAM scale.

RESULTS

The authors obtained 901 anonymous, layperson evaluations on both Likert and TRAM scales. Crowdsourced assessment data collection took 28.6 hours. Expert assessment took 15 months. Expert and crowdsourced scores were equivalent on the Likert scale (overall interrater reliability, κ = 0.99; 95 percent CI, 0.98 to 0.99). Intrarater reliability among each subcomponent was highly reproducible for the crowd (r = 0.98; 95 percent CI, 0.97 to 0.99) and experts (r = 0.82; 95 percent CI, 0.77 to 0.87). Breast contour and positioning were most predictive of overall aesthetic result. Skin patch and scar were least predictive of overall aesthetic appearance.

CONCLUSIONS

Aesthetic outcomes rated by crowds were reliable and correlated closely with those by expert surgeons. Crowdsourcing can be a rapid, reliable, and valid way to assess aesthetic outcomes in the breast reconstruction patient.

Current methods for assessing technical skill in cataract surgery

Surgery is a major source of errors in patient care. Preventing complications from surgical errors in the operating room is estimated to lead to reduction of up to 41 846 readmissions and save $620.3 million per year. It is now established that poor technical skill is associated with an increased risk of severe adverse events postoperatively and traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. This review discusses the current methods available for evaluating technical skills in cataract surgery and the recent technological advancements that have enabled capture and analysis of large amounts of complex surgical data for more automated objective skills assessment.

Expert Consensus Recommendations for Robotic Surgery Credentialing

OBJECTIVE

To define criteria for robotic credentialing using expert consensus.

BACKGROUND

A recent review of institutional robotic credentialing policies identified significant variability and determined current policies are largely inadequate to ensure surgeon proficiency and may threaten patient safety.

METHODS

28 national robotic surgery experts were invited to participate in a consensus conference. After review of available institutional policies and discussion, the group developed a 91 proposed criteria. Using a modified Delphi process the experts were asked to indicate their agreement with the proposed criteria in three electronic survey rounds after the conference. Criteria that achieved 80% or more in agreement (consensus) in all rounds were included in the final list.

RESULTS

All experts agreed that there is a need for standardized robotic surgery credentialing criteria across institutions that promote surgeon proficiency. 49 items reached consensus in the first round, 19 in the second, and 8 in the third for a total of 76 final items. Experts agreed that privileges should be granted based on video review of surgical performance and attainment of clearly defined objective proficiency benchmarks. Parameters for ongoing outcome monitoring were determined and recommendations for technical skills training, proctoring, and performance assessment were defined.

CONCLUSIONS

Using a systematic approach, detailed credentialing criteria for robotic surgery were defined. Implementation of these criteria uniformly across institutions will promote proficiency of robotic surgeons and has the potential to positively impact patient outcomes.

A Method for Segmenting the Process of Needle Insertion during Simulated Cannulation using Sensor Data

Cannulation is a routine yet challenging medical procedure resulting in a direct impact on patient outcomes. While current training programs provide guidelines to learn this complex procedure, the lack of objective and quantitative feedback impedes learning this skill more effectively. In this paper, we present a simulator for performing hemodialysis cannulation that captures the process using multiple sensing modalities that provide a multi-faceted assessment of cannulation. Further, we describe an algorithm towards segmenting the cannulation process using specific events in the sensor data for detailed analysis. Results from three participants with varying levels of clinical cannulation expertise are presented along with a metric that successfully differentiates the three participants. This work could lead to sensor-based cannulation skill assessment and training in the future potentially resulting in improved patient outcomes.

Bidirectional long short-term memory for surgical skill classification of temporally segmented tasks

PURPOSE

The majority of historical surgical skill research typically analyzes holistic summary task-level metrics to create a skill classification for a performance. Recent advances in machine learning allow time series classification at the sub-task level, allowing predictions on segments of tasks, which could improve task-level technical skill assessment.

METHODS

A bidirectional long short-term memory (LSTM) network was used with 8-s windows of multidimensional time-series data from the Basic Laparoscopic Urologic Skills dataset. The network was trained on experts and novices from four common surgical tasks. Stratified cross-validation with regularization was used to avoid overfitting. The misclassified cases were re-submitted for surgical technical skill assessment to crowds using Amazon Mechanical Turk to re-evaluate and to analyze the level of agreement with previous scores.

RESULTS

Performance was best for the suturing task, with 96.88% accuracy at predicting whether a performance was an expert or novice, with 1 misclassification, when compared to previously obtained crowd evaluations. When compared with expert surgeon ratings, the LSTM predictions resulted in a Spearman coefficient of 0.89 for suturing tasks. When crowds re-evaluated misclassified performances, it was found that for all 5 misclassified cases from peg transfer and suturing tasks, the crowds agreed more with our LSTM model than with the previously obtained crowd scores.

CONCLUSION

The technique presented shows results not incomparable with labels which would be obtained from crowd-sourced labels of surgical tasks. However, these results bring about questions of the reliability of crowd sourced labels in videos of surgical tasks. We, as a research community, should take a closer look at crowd labeling with higher scrutiny, systematically look at biases, and quantify label noise.

Temporal variability of surgical technical skill perception in real robotic surgery

PURPOSE

Summary score metrics, either from crowds of non-experts, faculty surgeons or from automated performance metrics, have been trusted as the prevailing method of reporting surgeon technical skill. The aim of this paper is to learn whether there exist significant fluctuations in the technical skill assessments of a surgeon throughout long durations of surgical footage.

METHODS

A set of 12 videos of robotic surgery cases from common human patient robotic surgeries were used to evaluate the perceived technical skill at each individual minute of the surgical videos, which were originally 12-15 min in length. A linear mixed-effects model for each video was used to compare the ratings of each minute to those from every other minute in order to learn whether a change in scores over time can be detected and reliably measured apart from inter- and intrarater variation.

RESULTS

Modeling the change over time of the global evaluative assessment of robotic skills scores significantly contributed to the prediction models for 11 of the 12 surgeons. This demonstrates that measurable changes in technical skill occur over time during robotic surgery.

CONCLUSION

The findings from this research raise questions about the optimal duration of footage needed to be evaluated to arrive at an accurate rating of surgical technical skill for longer procedures. This may imply non-negligible label noise for supervised machine learning approaches. In the future, it may be necessary to report a surgeon's skill variability in addition to their mean score to have proper knowledge of a surgeon's overall skill level.

The effect of video playback speed on surgeon technical skill perception

PURPOSE

 Finding effective methods of discriminating surgeon technical skill has proved a complex problem to solve computationally. Previous research has shown that obtaining non-expert crowd evaluations of surgical performances is as accurate as the gold standard, expert surgeon review. The aim of this research is: (1) to learn whether crowdsourced evaluators give higher ratings of technical skill to video of performances with increased playback speed, (2) its effect in discriminating skill levels, and (3) whether this increase is related to the evaluator consciously being aware that the video is manually manipulated.

METHODS

 A set of ten peg transfer videos (five novices, five experts) were used to evaluate the perceived technical skill of the performers at each video playback speed used ([Formula: see text]). Objective metrics used for measuring technical skill were also computed for comparison by manipulating the corresponding kinematic data of each performance. Two videos of an expert and novice performing dry laboratory laparoscopic trials of peg transfer tasks were used to obtain evaluations at each playback speed ([Formula: see text]) of perception of whether a video is played at real-time playback speed or not.

RESULTS

 We found that while both novices and experts had increased perceived technical skill as the video playback was increased, the amount of increase was significantly greater for experts. Each increase in the playback speed by [Formula: see text] was associated with, on average, a 0.72-point increase in the GOALS score (95% CI 0.60-0.84 point increase; [Formula: see text]) for expert videos and only a 0.24-point increase in the GOALS score (95% CI 0.13-0.36 point increase; [Formula: see text]) for novice videos.

CONCLUSION

 Due to the differential increase in perceived technical skill due to increased playback speed for experts, the difference between novice and expert skill levels of surgical performances may be more easily discerned by manually increasing the video playback speed.

Video review for measuring and improving skill in urological surgery

Interest is growing within the urological surgery community for objective assessments of technical skill. Surgical video review relies on the use of objective assessment tools to evaluate both global and procedure-specific skill. These evaluations provide structured feedback to surgeons with the aim of improving technique, which has been associated with patient outcomes. Currently, skill assessments can be performed by using expert peer-review, crowdsourcing or computer-based methods. Given the relationship between skill and patient outcomes, surgeons might be required in the future to provide empirical evidence of their technical skill for certification, employment, credentialing and quality improvement. Interventions such as coaching and skills workshops incorporating video review might help surgeons improve their skill, with the ultimate goal of improving patient outcomes.

Crowdsourcing in health and medical research: a systematic review

Background

Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health.

Methods

We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach.

Results

We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardio-pulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-of-hospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and layperson-initiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled.

Conclusions

Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine.

Trial registration

PROSPERO: CRD42017052835. December 27, 2016.

How Do Thresholds of Principle and Preference Influence Surgeon Assessments of Learner Performance?

OBJECTIVE

The present study asks whether intraoperative principles are shared among faculty in a single residency program and explores how surgeons' individual thresholds between principles and preferences might influence assessment.

BACKGROUND

Surgical education continues to face significant challenges in the implementation of intraoperative assessment. Competency-based medical education assumes the possibility of a shared standard of competence, but intersurgeon variation is prevalent and, at times, valued in surgical education. Such procedural variation may pose problems for assessment.

METHODS

An entire surgical division (n = 11) was recruited to participate in video-guided interviews. Each surgeon assessed intraoperative performance in 8 video clips from a single laparoscopic radical left nephrectomy performed by a senior learner (>PGY5). Interviews were audio recorded, transcribed, and analyzed using the constant comparative method of grounded theory.

RESULTS

Surgeons' responses revealed 5 shared generic principles: choosing the right plane, knowing what comes next, recognizing normal and abnormal, making safe progress, and handling tools and tissues appropriately. The surgeons, however, disagreed both on whether a particular performance upheld a principle and on how the performance could improve. This variation subsequently shaped their reported assessment of the learner's performance.

CONCLUSIONS

The findings of the present study provide the first empirical evidence to suggest that surgeons' attitudes toward their own procedural variations may be an important influence on the subjectivity of intraoperative assessment in surgical education. Assessment based on intraoperative entrustment may harness such subjectivity for the purpose of implementing competency-based surgical education.

Machine learning methods for automated technical skills assessment with instructional feedback in ultrasound-guided interventions

OBJECTIVE

Currently, there is a worldwide shift toward competency-based medical education. This necessitates the use of automated skills assessment methods during self-guided interventions training. Making assessment methods that are transparent and configurable will allow assessment to be interpreted into instructional feedback. The purpose of this work is to develop and validate skills assessment methods in ultrasound-guided interventions that are transparent and configurable.

METHODS

We implemented a method based upon decision trees and a method based upon fuzzy inference systems for technical skills assessment. Subsequently, we validated these methods for their ability to predict scores of operators on a 25-point global rating scale in ultrasound-guided needle insertions and their ability to provide useful feedback for training.

RESULTS

Decision tree and fuzzy rule-based assessment performed comparably to state-of-the-art assessment methods. They produced median errors (on a 25-point scale) of 1.7 and 1.8 for in-plane insertions and 1.5 and 3.0 for out-of-plane insertions, respectively. In addition, these methods provided feedback that was useful for trainee learning. Decision tree assessment produced feedback with median usefulness 7 out of 7; fuzzy rule-based assessment produced feedback with median usefulness 6 out of 7.

CONCLUSION

Transparent and configurable assessment methods are comparable to the state of the art and, in addition, can provide useful feedback. This demonstrates their value in self-guided interventions training curricula.

Automatic and near real-time stylistic behavior assessment in robotic surgery

PURPOSE

Automatic skill evaluation is of great importance in surgical robotic training. Extensive research has been done to evaluate surgical skill, and a variety of quantitative metrics have been proposed. However, these methods primarily use expert selected features which may not capture latent information in movement data. In addition, these features are calculated over the entire task time and are provided to the user after the completion of the task. Thus, these quantitative metrics do not provide users with information on how to modify their movements to improve performance in real time. This study focuses on automatic stylistic behavior recognition that has the potential to be implemented in near real time.

METHODS

We propose a sparse coding framework for automatic stylistic behavior recognition in short time intervals using only position data from the hands, wrist, elbow, and shoulder. A codebook is built for each stylistic adjective using the positive and negative labels provided for each trial through crowd sourcing. Sparse code coefficients are obtained for short time intervals (0.25 s) in a trial using this codebook. A support vector machine classifier is trained and validated through tenfold cross-validation using the sparse codes from the training set.

RESULTS

The results indicate that the proposed dictionary learning method is able to assess stylistic behavior performance in near real time using user joint position data with improved accuracy compared to using PCA features or raw data.

CONCLUSION

The possibility to automatically evaluate a trainee's style of movement in short time intervals could provide the user with online customized feedback and thus improve performance during surgical tasks.

Automatic Surgical Skill Rating Using Stylistic Behavior Components

A gold standard in surgical skill rating and evaluation is direct observation, which a group of experts rate trainees based on a likert scale, by observing their performance during a surgical task. This method is time and resource intensive. To alleviate this burden, many studies have focused on automatic surgical skill assessment; however, the metrics suggested by the literature for automatic evaluation do not capture the stylistic behavior of the user. In addition very few studies focus on automatic rating of surgical skills based on available likert scales. In a previous study we presented a stylistic behavior lexicon for surgical skill. In this study we evaluate the lexicon's ability to automatically rate robotic surgical skill, based on the 6 domains in the Global Evaluative Assessment of Robotic Skills (GEARS). 14 subjects of different skill levels performed two surgical tasks on da Vinci surgical simulator. Different measurements were acquired as subjects performed the tasks, including limb (hand and arm) kinematics and joint (shoulder, elbow, wrist) positions. Posture videos of the subjects performing the task, as well as videos of the task being performed were viewed and rated by faculty experts based on the 6 domains in GEARS. The paired videos were also rated via crowd-sourcing based on our stylistic behavior lexicon. Two separate regression learner models, one using the sensor measurements and the other using crowd ratings for our proposed lexicon, were trained for each domain in GEARS. The results indicate that the scores predicted from both prediction models are in agreement with the gold standard faculty ratings.

Mapping of Crowdsourcing in Health: Systematic Review

Background

Crowdsourcing involves obtaining ideas, needed services, or content by soliciting Web-based contributions from a crowd. The 4 types of crowdsourced tasks (problem solving, data processing, surveillance or monitoring, and surveying) can be applied in the 3 categories of health (promotion, research, and care).

Objective

This study aimed to map the different applications of crowdsourcing in health to assess the fields of health that are using crowdsourcing and the crowdsourced tasks used. We also describe the logistics of crowdsourcing and the characteristics of crowd workers.

Methods

MEDLINE, EMBASE, and ClinicalTrials.gov were searched for available reports from inception to March 30, 2016, with no restriction on language or publication status.

Results

We identified 202 relevant studies that used crowdsourcing, including 9 randomized controlled trials, of which only one had posted results at ClinicalTrials.gov. Crowdsourcing was used in health promotion (91/202, 45.0%), research (73/202, 36.1%), and care (38/202, 18.8%). The 4 most frequent areas of application were public health (67/202, 33.2%), psychiatry (32/202, 15.8%), surgery (22/202, 10.9%), and oncology (14/202, 6.9%). Half of the reports (99/202, 49.0%) referred to data processing, 34.6% (70/202) referred to surveying, 10.4% (21/202) referred to surveillance or monitoring, and 5.9% (12/202) referred to problem-solving. Labor market platforms (eg, Amazon Mechanical Turk) were used in most studies (190/202, 94%). The crowd workers’ characteristics were poorly reported, and crowdsourcing logistics were missing from two-thirds of the reports. When reported, the median size of the crowd was 424 (first and third quartiles: 167-802); crowd workers’ median age was 34 years (32-36). Crowd workers were mainly recruited nationally, particularly in the United States. For many studies (58.9%, 119/202), previous experience in crowdsourcing was required, and passing a qualification test or training was seldom needed (11.9% of studies; 24/202). For half of the studies, monetary incentives were mentioned, with mainly less than US $1 to perform the task. The time needed to perform the task was mostly less than 10 min (58.9% of studies; 119/202). Data quality validation was used in 54/202 studies (26.7%), mainly by attention check questions or by replicating the task with several crowd workers.

Conclusions

The use of crowdsourcing, which allows access to a large pool of participants as well as saving time in data collection, lowering costs, and speeding up innovations, is increasing in health promotion, research, and care. However, the description of crowdsourcing logistics and crowd workers’ characteristics is frequently missing in study reports and needs to be precisely reported to better interpret the study findings and replicate them.

A scoping review of assessment tools for laparoscopic suturing

BACKGROUND

A needs assessment identified a gap in teaching and assessment of laparoscopic suturing (LS) skills. The purpose of this review is to identify assessment tools that were used to assess LS skills, to evaluate validity evidence available, and to provide guidance for selecting the right assessment tool for specific assessment conditions.

METHODS

Bibliographic databases were searched till April 2017. Full-text articles were included if they reported on assessment tools used in the operating room/simulation to (1) assess procedures that require LS or (2) specifically assess LS skills.

RESULTS

Forty-two tools were identified, of which 26 were used for assessing LS skills specifically and 26 for procedures that require LS. Tools had the most evidence in internal structure and relationship to other variables, and least in consequences.

CONCLUSION

Through identification and evaluation of assessment tools, the results of this review could be used as a guideline when implementing assessment tools into training programs.

Vessel Ligation Fundamentals: A Comparison of Technical Evaluations by Crowdsourced Nonclinical Personnel and Surgical Faculty

BACKGROUND

Evaluation of fundamental surgical skills is invaluable to the training of medical students and junior residents. This study assessed the effectiveness of crowdsourcing nonmedical personnel to evaluate technical proficiency at simulated vessel ligation.

STUDY DESIGN

Fifteen videos were captured of participants performing vessel ligation using a low-fidelity model (5 attending surgeons and 5 medical students before and after training). These videos were evaluated by nonmedical personnel recruited through Amazon Mechanical Turk, as well as by 3 experienced surgical faculty. Evaluation criteria were based on Objective Structured Assessment of Technical Skills (scale: 5-25). Results were compared using Wilcoxon signed rank-sum and Cronbach's alpha (α).

RESULTS

Thirty-two crowd workers evaluated all 15 videos. Crowd workers scored attending surgeon videos significantly higher than pretraining medical student videos (20.5 vs 14.9, p < 0.001), demonstrating construct validity. Across all videos, crowd evaluations were more lenient than expert evaluations (19.1 vs 14.5, p < 0.001). However, average volunteer evaluations correlated more strongly with average expert evaluations (α = 0.95) than the strength of correlation between any 2 individual expert evaluators (α = 0.72-0.88). Combined reimbursement for all workers was $80.00.

CONCLUSION

After adjustments for score inflation, crowdsourced can evaluate surgical fundamentals with excellent validity. This resource is considerably less costly and potentially more reliable than individual expert evaluations.

Meaningful Assessment of Robotic Surgical Style using the Wisdom of Crowds

OBJECTIVE

Quantitative assessment of surgical skills is an important aspect of surgical training; however, the proposed metrics are sometimes difficult to interpret and may not capture the stylistic characteristics that define expertise. This study proposes a methodology for evaluating the surgical skill, based on metrics associated with stylistic adjectives, and evaluates the ability of this method to differentiate expertise levels.

METHODS

We recruited subjects from different expertise levels to perform training tasks on a surgical simulator. A lexicon of contrasting adjective pairs, based on important skills for robotic surgery, inspired by the global evaluative assessment of robotic skills tool, was developed. To validate the use of stylistic adjectives for surgical skill assessment, posture videos of the subjects performing the task, as well as videos of the task were rated by crowd-workers. Metrics associated with each adjective were found using kinematic and physiological measurements through correlation with the crowd-sourced adjective assignment ratings. To evaluate the chosen metrics' ability in distinguishing expertise levels, two classifiers were trained and tested using these metrics.

RESULTS

Crowd-assignment ratings for all adjectives were significantly correlated with expertise levels. The results indicate that naive Bayes classifier performs the best, with an accuracy of [Formula: see text], [Formula: see text], [Formula: see text], and [Formula: see text] when classifying into four, three, and two levels of expertise, respectively.

CONCLUSION

The proposed method is effective at mapping understandable adjectives of expertise to the stylistic movements and physiological response of trainees.

Crowdsourcing Disease Biomarker Discovery Research: The IP4IC Study

PURPOSE

Biomarker discovery is limited by readily assessable, cost efficient human samples available in large numbers that represent the entire heterogeneity of the disease. We developed a novel, active participation crowdsourcing method to determine BP-RS (Bladder Permeability Defect Risk Score). It is based on noninvasive urinary cytokines to discriminate patients with interstitial cystitis/bladder pain syndrome who had Hunner lesions from controls and patients with interstitial cystitis/bladder pain syndrome but without Hunner lesions.

MATERIALS AND METHODS

We performed a national crowdsourcing study in cooperation with the Interstitial Cystitis Association. Patients answered demographic, symptom severity and urinary frequency questionnaires on a HIPAA (Health Insurance Portability and Accountability Act) compliant website. Urine samples were collected at home, stabilized with a preservative and sent to Beaumont Hospital for analysis. The expression of 3 urinary cytokines was used in a machine learning algorithm to develop BP-RS.

RESULTS

The IP4IC study collected a total of 448 urine samples, representing 153 patients (147 females and 6 males) with interstitial cystitis/bladder pain syndrome, of whom 54 (50 females and 4 males) had Hunner lesions. A total of 159 female and 136 male controls also participated, who were age matched. A defined BP-RS was calculated to predict interstitial cystitis/bladder pain syndrome with Hunner lesions or a bladder permeability defect etiology with 89% validity.

CONCLUSIONS

In this novel participation crowdsourcing study we obtained a large number of urine samples from 46 states, which were collected at home, shipped and stored at room temperature. Using a machine learning algorithm we developed BP-RS to quantify the risk of interstitial cystitis/bladder pain syndrome with Hunner lesions, which is indicative of a bladder permeability defect etiology. To our knowledge BP-RS is the first validated urine biomarker assay for interstitial cystitis/bladder pain syndrome and one of the first biomarker assays to be developed using crowdsourcing.

Crowdsourcing in Surgical Skills Acquisition: A Developing Technology in Surgical Education

BACKGROUND

The application of crowdsourcing to surgical education is a recent phenomenon and adds to increasing demands on surgical residency training. The efficacy, range, and scope of this technology for surgical education remains incompletely defined.

OBJECTIVE

A systematic review was performed using the PubMed database of English-language literature on crowdsourced evaluation of surgical technical tasks up to April 2017.

METHODS

Articles were reviewed, abstracted, and analyzed, and were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI). Articles were evaluated with eligibility criteria for inclusion. Study information, performance task, subjects, evaluative standards, crowdworker compensation, time to response, and correlation between crowd and expert or standard evaluations were abstracted and analyzed.

RESULTS

Of 63 unique publications initially identified, 13 with MERSQI scores ranging from 10 to 13 (mean = 11.85) were included in the review. Overall, crowd and expert evaluations demonstrated good to excellent correlation across a wide range of tasks (Pearson's coefficient 0.59-0.95, Cronbach's alpha 0.32-0.92), with 1 exception being a study involving medical students. There was a wide range of reported interrater variability among experts. Nonexpert evaluation was consistently quicker than expert evaluation (ranging from 4.8 to 150.9 times faster), and was more cost effective.

CONCLUSIONS

Crowdsourced feedback appears to be comparable to expert feedback and is cost effective and efficient. Further work is needed to increase consistency in expert evaluations, to explore sources of discrepant assessments between surgeons and crowds, and to identify optimal populations and novel applications for this technology.

Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy

INTRODUCTION

Development of uretero-ileal stricture (UIS) after robotic-assisted radical cystectomy (RARC) may be dependent on surgical technique. Video review of intraoperative technique is an emerging paradigm for surgical quality improvement. We examined whether surgeon-perceived risk of UIS or crowd-sourced assessment of robotic skill are associated with the development of UIS.

METHODS

We conducted a case-control study comparing the operative technique of uretero-ileal anastomoses resulting in clinically significant UIS with the contralateral anastomosis for the same patient. De-identified videos were analyzed by 1) five high-volume surgeons; and 2) crowd workers (Crowd-Sourced Assessment of Technical Skill, C-SATS) to determine Global Evaluative Assessment of Robotic Skill (GEARS) score. Mantel-Haenszel common odds ratio (OR) estimates were calculated to assess the association between surgeon performance and the development of UIS. Logistic regression models were used to examine the association between GEARS scores and the development of UIS.

RESULTS

A total of 10 UIS videos were compared to eight control videos by five surgeons and 2142 crowd workers. Expert surgeons systematically evaluated intraoperative footage, however, no association between the expert mode response and UIS (OR 0.42; 95% confidence interval [CI] 0.05-3.45; p=0.91) was identified. Crowd-sourced assessment was not predictive of UIS (p=0.62).

CONCLUSIONS

We used video review to systematically analyze procedure-specific content and technique. The inability of surgeons to predict UIS may reflect the questionnaire, uncontrolled patient factors, or a lack of power. Crowd-sourced GEARS score was unsuccessful in predicting UIS after RARC.

Simple visual review of pre- to post-operative renal ultrasound images predicts pyeloplasty success equally as well as geometric measurements: A blinded comparison with a gold standard

BACKGROUND

MAG3 diuretic renal scan remains the gold standard for determination of improvement in renal drainage following pyeloplasty for ureteropelvic junction obstruction. We hypothesized that (i) a change in geometric measurements between pre-operative and post-operative renal ultrasound (RUS) images and (ii) blinded simple visual review of images both would predict pyeloplasty success.

OBJECTIVE

To determine if simple visual review and/or novel geometric measurement of renal ultrasounds can detect pyeloplasty failure.

STUDY DESIGN

This study was a retrospective, blinded comparison with a gold standard. Included were children aged ≤18 years undergoing pyeloplasty at our institution from 2009 to 2015. For each kidney, representative pre-operative and post-operative RUS images were chosen. Our standard for pyeloplasty success was improved drainage curve on MAG3 and lack of additional surgery. Measurements for collecting system circularity, roundness, and renal parenchymal to collecting system area ratio (RPCSR) were obtained by three raters (Figure), who were blinded to the outcome of the pyeloplasty. Changes in geometric measurements were analyzed as a diagnostic test for MAG3-defined pyeloplasty success using ROC curve analysis. In addition, six reviewers blinded to pyeloplasty success reviewed pre-operative and post-operative images visually for improved hydronephrosis and categorized pyeloplasty as success or failure based on simple visual review of RUS.

RESULTS

Fifty-three repaired renal units were identified (50 children). There were five pyeloplasty failures, four of which underwent revision or nephrectomy. While all geometric measurements could discriminate pyeloplasty failure and success, the geometric measurements that discriminated best between pyeloplasty failure and success were change in collecting system roundness and change in RPCSR. Consensus opinion among six blinded reviewers using simple visual review had a sensitivity of 94% and PPV of 100% with respect to identifying pyeloplasty success (AUC 0.97 (95% CI 0.93-1.0)). This was not significantly different from AUC for change in roundness (p = 0.09) or change in RPCSR (p = 0.1).

DISCUSSION

Change in collecting system roundness and change in RPCSR were the most accurate geometric measurements in predicting pyeloplasty success. Simple visual review of ultrasound images for pyeloplasty success performed as well or better than geometric measurements. However, geometric measurements remain useful as a research tool or to communicate findings between clinicians.

CONCLUSIONS

Complex geometric measurements of hydronephrosis or post-operative MAG3 scans are not needed if hydronephrosis is visually significantly improved, as simple visual review is highly sensitive for detecting pyeloplasty failure.

The CREST Simulation Development Process: Training the Next Generation

BACKGROUND

The challenges of training and assessing endourologic skill have driven the development of new training systems. The Center for Research in Education and Simulation Technologies (CREST) has developed a team and a methodology to facilitate this development process.

METHODS

Backwards design principles were applied. A panel of experts first defined desired clinical and educational outcomes. Outcomes were subsequently linked to learning objectives. Gross task deconstruction was performed, and the primary domain was classified as primarily involving decision-making, psychomotor skill, or communication. A more detailed cognitive task analysis was performed to elicit and prioritize relevant anatomy/tissues, metrics, and errors. Reference anatomy was created using a digital anatomist and clinician working off of a clinical data set. Three dimensional printing can facilitate this process. When possible, synthetic or virtual tissue behavior and textures were recreated using data derived from human tissue. Embedded sensors/markers and/or computer-based systems were used to facilitate the collection of objective metrics. A learning Verification and validation occurred throughout the engineering development process.

RESULTS

Nine endourology-relevant training systems were created by CREST with this approach. Systems include basic laparoscopic skills (BLUS), vesicourethral anastomosis, pyeloplasty, cystoscopic procedures, stent placement, rigid and flexible ureteroscopy, GreenLight PVP (GL Sim), Percutaneous access with C-arm (CAT), Nephrolithotomy (NLM), and a vascular injury model. Mixed modalities have been used, including "smart" physical models, virtual reality, augmented reality, and video. Substantial validity evidence for training and assessment has been collected on systems. An open source manikin-based modular platform is under development by CREST with the Department of Defense that will unify these and other commercial task trainers through the common physiology engine, learning management system, standard data connectors, and standards.

CONCLUSION

Using the CREST process has and will ensure that the systems we create meet the needs of training and assessing endourologic skills.

C-SATS: Assessing Surgical Skills Among Urology Residency Applicants

BACKGROUND

We hypothesized that surgical skills assessment could aid in the selection process of medical student applicants to a surgical program. Recently, crowdsourcing has been shown to provide an accurate assessment of surgical skills at all levels of training. We compared expert and crowd assessment of surgical tasks performed by resident applicants during their interview day at the urology program at the University of California, Irvine.

MATERIALS AND METHODS

Twenty-five resident interviewees performed four tasks: open square knot tying, laparoscopic peg transfer, robotic suturing, and skill task 8 on the LAP Mentor™ (Simbionix Ltd., Lod, Israel). Faculty experts and crowd workers (Crowd-Sourced Assessment of Technical Skills [C-SATS], Seattle, WA) assessed recorded performances using the Objective Structured Assessment of Technical Skills (OSATS), Global Evaluative Assessment of Robotic Skills (GEARS), and the Global Operative Assessment of Laparoscopic Skills (GOALS) validated assessment tools.

RESULTS

Overall, 3938 crowd assessments were obtained for the four tasks in less than 3.5 hours, whereas the average time to receive 150 expert assessments was 22 days. Inter-rater agreement between expert and crowd assessment scores was 0.62 for open knot tying, 0.92 for laparoscopic peg transfer, and 0.86 for robotic suturing. Agreement between applicant rank on skill task 8 on the LAP Mentor assessment and crowd assessment was 0.32. The crowd match rank based solely on skills performance did not compare well with the final faculty match rank list (0.46); however, none of the bottom five crowd-rated applicants appeared in the top five expert-rated applicants and none of the top five crowd-rated applicants appeared in the bottom five expert-rated applicants.

CONCLUSIONS

Crowd-source assessment of resident applicant surgical skills has good inter-rater agreement with expert physician raters but not with a computer-based objective motion metrics software assessment. Overall applicant rank was affected to some degree by the crowd performance rating.

Scientific Skill Assessment of Basic Surgical Dissection and Overall Laparoscopic Performance

To improve surgical education, objective and scientific skill assessments are required. There are two types of skill evaluations: assessments of basic surgical skills and assessments of overall surgical performance. To establish a reliable assessment method for surgical dissection, we measured the force applied on the tip of a surgical instrument during dissection of the renal vessels of pigs. The experiments revealed that, during surgical dissection, expert laparoscopic surgeons applied vertical force at the beginning of the stroke and then horizontal force, with minimum vertical force, at the end of the stroke. As an assessment of overall surgical performance, the Endoscopic Surgical Qualification system was developed and has been used for 12 years in Japan. More than 3700 surgeons, including urologists, were determined to have appropriate laparoscopic surgical skills after assessments of unedited videos by referees.