At-home medical student simulation: achieving knot-tying proficiency using video-based assessment

Taking advantage of asynchronous digital feedback: development of an at-home basic suture skills training program for undergraduate medical students that facilitates skills retention

Purpose

To date, there are no training programs for basic suturing that allow remote deliberate practice. This study seeks to evaluate the effectiveness of a basic suture skills training program and its 6-month skill retention applying unsupervised practice and remote digital feedback.

Methods

Fourth-year medical-student trainees reviewed instructional videos from a digital platform and performed unsupervised practice as needed at their homes. When they felt competent, trainees uploaded a video of themselves practicing the skill. In < 72 h, they received expert asynchronous digital feedback. The course had two theoretical stages and five video-based assessments, where trainees performed different suturing exercises. For the assessment, a global (GRS) and specific rating scale (SRS) were used, with a passing score of 20 points (max:25) and 15 (max:20), respectively. Results were compared to previously published work with in-person expert feedback (EF) and video-guided learning without feedback (VGL). A subgroup of trainees underwent a 6-month skills retention assessment.

Results

Two-hundred and forty-three trainees underwent the course between March and December 2021. A median GRS of 24 points was achieved in the final assessment, showing significantly higher scores (p < 0.001) than EF and VGL (20.5 and 15.5, respectively). Thirty-seven trainees underwent a 6-month skills retention assessment, improving in GRS (23.38 vs 24.03, p value = 0.06) and SRS (18.59 vs 19, p value = 0.07).

Conclusion

It is feasible to teach basic suture skills to undergraduate medical students using an unsupervised training course with remote and asynchronous feedback through a digital platform. This methodology allows continuous training with the repetition of quality practice, personalized feedback, and skills retention at 6 months.

Developing artificial intelligence models for medical student suturing and knot-tying video-based assessment and coaching

BACKGROUND

Early introduction and distributed learning have been shown to improve student comfort with basic requisite suturing skills. The need for more frequent and directed feedback, however, remains an enduring concern for both remote and in-person training. A previous in-person curriculum for our second-year medical students transitioning to clerkships was adapted to an at-home video-based assessment model due to the social distancing implications of COVID-19. We aimed to develop an Artificial Intelligence (AI) model to perform video-based assessment.

METHODS

Second-year medical students were asked to submit a video of a simple interrupted knot on a penrose drain with instrument tying technique after self-training to proficiency. Proficiency was defined as performing the task under two minutes with no critical errors. All the videos were first manually rated with a pass-fail rating and then subsequently underwent task segmentation. We developed and trained two AI models based on convolutional neural networks to identify errors (instrument holding and knot-tying) and provide automated ratings.

RESULTS

A total of 229 medical student videos were reviewed (150 pass, 79 fail). Of those who failed, the critical error distribution was 15 knot-tying, 47 instrument-holding, and 17 multiple. A total of 216 videos were used to train the models after excluding the low-quality videos. A k-fold cross-validation (k = 10) was used. The accuracy of the instrument holding model was 89% with an F-1 score of 74%. For the knot-tying model, the accuracy was 91% with an F-1 score of 54%.

CONCLUSIONS

Medical students require assessment and directed feedback to better acquire surgical skill, but this is often time-consuming and inadequately done. AI techniques can instead be employed to perform automated surgical video analysis. Future work will optimize the current model to identify discrete errors in order to supplement video-based rating with specific feedback.