1
|
Ketel MHM, Klarenbeek BR, Eddahchouri Y, Cuesta MA, van Daele E, Gutschow CA, Hölscher AH, Hubka M, Luyer MDP, Merritt RE, Nieuwenhuijzen GAP, Shen Y, Abma IL, Rosman C, van Workum F. Crowd-sourced and expert video assessment in minimally invasive esophagectomy. Surg Endosc 2023; 37:7819-7828. [PMID: 37605010 PMCID: PMC10520122 DOI: 10.1007/s00464-023-10297-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 07/02/2023] [Indexed: 08/23/2023]
Abstract
BACKGROUND Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review. METHODS Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT). RESULTS Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42). CONCLUSION GOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.
Collapse
Affiliation(s)
- Mirte H M Ketel
- Department of Surgery, Radboud University Medical Center, Nijmegen, The Netherlands.
| | | | - Yassin Eddahchouri
- Department of Surgery, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Miguel A Cuesta
- Department of Surgery, Amsterdam University Medical Centers, Location VUmc, Amsterdam, The Netherlands
| | - Elke van Daele
- Department of Digestive Surgery, Ghent University Hospital, Ghent, Belgium
| | - Christian A Gutschow
- Department of Surgery and Transplantation, University Hospital Zurich, Zurich, Switzerland
| | - Arnulf H Hölscher
- Department for General, Visceral and Trauma Surgery, Elisabeth-Krankenhaus-Essen GmbH, Essen, Germany
| | - Michal Hubka
- Department of Thoracic Surgery, Virginia Mason Medical Center, Seattle, SE, USA
| | - Misha D P Luyer
- Department of Surgery, Catharina Hospital, Eindhoven, The Netherlands
| | - Robert E Merritt
- Department of Surgery, Ohio State University - Wexner Medical Center, Columbus, OH, USA
| | | | - Yaxing Shen
- Department of Thoracic Surgery, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Inger L Abma
- IQ Healthcare, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Camiel Rosman
- Department of Surgery, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Frans van Workum
- Department of Surgery, Radboud University Medical Center, Nijmegen, The Netherlands
- Department of Surgery, Canisius-Wilhelmina Hospital, Nijmegen, The Netherlands
| |
Collapse
|
2
|
The Moderating Role of Age in the Effect of Video Playback Speed on Urgency Perception in the Context of Climate Change. SUSTAINABILITY 2022. [DOI: 10.3390/su14148923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
Urgency perception plays a vital role in addressing the issue of climate change. However, little is known about how to promote the perceived urgency of climate change and its potential influence on proenvironmental intention and behavior. This research focuses on a potentially significant but less studied factor in video communication: video playback speed. The current research explores the effectiveness of video playback speed as a subtle behavioral nudge to influence urgency perception and proenvironmental response in the context of climate change. We conducted two survey-embedded experiments in which participants watched a climate change video playing at either normal or fast speed and then completed measurements. Data were collected first in an undergraduate sample (n = 75) and then in a general population sample (n = 300) and analyzed using Mann–Whitney U tests, chi-squared tests, and moderation analysis in SPSS. The results reveal that a fast playback speed of climate change video decreases the perceived urgency of climate change for younger consumers, not for older consumers. However, video playback speed does not influence proenvironmental intention and behavior. These findings enhance understanding of when video playback speed affects urgency perception and proenvironmental tendency, and provide valuable insights for climate change communication.
Collapse
|
3
|
Olsen RG, Genét MF, Konge L, Bjerrum F. Crowdsourced assessment of surgical skills: A systematic review. Am J Surg 2022; 224:1229-1237. [DOI: 10.1016/j.amjsurg.2022.07.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2022] [Revised: 05/30/2022] [Accepted: 07/14/2022] [Indexed: 11/25/2022]
|
4
|
Bidirectional long short-term memory for surgical skill classification of temporally segmented tasks. Int J Comput Assist Radiol Surg 2020; 15:2079-2088. [PMID: 33000365 DOI: 10.1007/s11548-020-02269-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2020] [Accepted: 09/23/2020] [Indexed: 10/23/2022]
Abstract
PURPOSE The majority of historical surgical skill research typically analyzes holistic summary task-level metrics to create a skill classification for a performance. Recent advances in machine learning allow time series classification at the sub-task level, allowing predictions on segments of tasks, which could improve task-level technical skill assessment. METHODS A bidirectional long short-term memory (LSTM) network was used with 8-s windows of multidimensional time-series data from the Basic Laparoscopic Urologic Skills dataset. The network was trained on experts and novices from four common surgical tasks. Stratified cross-validation with regularization was used to avoid overfitting. The misclassified cases were re-submitted for surgical technical skill assessment to crowds using Amazon Mechanical Turk to re-evaluate and to analyze the level of agreement with previous scores. RESULTS Performance was best for the suturing task, with 96.88% accuracy at predicting whether a performance was an expert or novice, with 1 misclassification, when compared to previously obtained crowd evaluations. When compared with expert surgeon ratings, the LSTM predictions resulted in a Spearman coefficient of 0.89 for suturing tasks. When crowds re-evaluated misclassified performances, it was found that for all 5 misclassified cases from peg transfer and suturing tasks, the crowds agreed more with our LSTM model than with the previously obtained crowd scores. CONCLUSION The technique presented shows results not incomparable with labels which would be obtained from crowd-sourced labels of surgical tasks. However, these results bring about questions of the reliability of crowd sourced labels in videos of surgical tasks. We, as a research community, should take a closer look at crowd labeling with higher scrutiny, systematically look at biases, and quantify label noise.
Collapse
|
5
|
Kelly JD, Nash M, Heller N, Lendvay TS, Kowalewski TM. Temporal variability of surgical technical skill perception in real robotic surgery. Int J Comput Assist Radiol Surg 2020; 15:2101-2107. [PMID: 32860549 DOI: 10.1007/s11548-020-02253-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2020] [Accepted: 08/19/2020] [Indexed: 11/26/2022]
Abstract
PURPOSE Summary score metrics, either from crowds of non-experts, faculty surgeons or from automated performance metrics, have been trusted as the prevailing method of reporting surgeon technical skill. The aim of this paper is to learn whether there exist significant fluctuations in the technical skill assessments of a surgeon throughout long durations of surgical footage. METHODS A set of 12 videos of robotic surgery cases from common human patient robotic surgeries were used to evaluate the perceived technical skill at each individual minute of the surgical videos, which were originally 12-15 min in length. A linear mixed-effects model for each video was used to compare the ratings of each minute to those from every other minute in order to learn whether a change in scores over time can be detected and reliably measured apart from inter- and intrarater variation. RESULTS Modeling the change over time of the global evaluative assessment of robotic skills scores significantly contributed to the prediction models for 11 of the 12 surgeons. This demonstrates that measurable changes in technical skill occur over time during robotic surgery. CONCLUSION The findings from this research raise questions about the optimal duration of footage needed to be evaluated to arrive at an accurate rating of surgical technical skill for longer procedures. This may imply non-negligible label noise for supervised machine learning approaches. In the future, it may be necessary to report a surgeon's skill variability in addition to their mean score to have proper knowledge of a surgeon's overall skill level.
Collapse
Affiliation(s)
- Jason D Kelly
- Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN, USA.
| | - Michael Nash
- Department of Biostatistics, University of Washington, Seattle, WA, USA
| | - Nicholas Heller
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN, USA
| | - Thomas S Lendvay
- Department of Urology, University of Washington, Seattle, WA, USA
| | - Timothy M Kowalewski
- Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN, USA
| |
Collapse
|