Acquisition of Surgical Skills in Medical Students via Telementoring: A Randomized Controlled Trial

Expanded Access to Video-Based Laparoscopic Skills Assessments: Ease, Reliability, and Accuracy

OBJECTIVE

Video-based performance assessments provide essential feedback to surgical residents, but in-person and remote video-based assessment by trained proctors incurs significant cost. We aimed to determine the reliability, accuracy, and difficulty of untrained attending staff surgeon raters completing video-based assessments of a basic laparoscopic skill. Secondarily, we aimed to compare reliability and accuracy between 2 different types of assessment tools.

DESIGN

An anonymous survey was distributed electronically to surgical attendings via a national organizational listserv. Survey items included demographics, rating of video-based assessment experience (1 = have never completed video-based assessments, 5 = often complete video-based assessments), and rating of favorability toward video-based and in-person assessments (0 = not favorable, 100 = favorable). Participants watched 2 laparoscopic peg transfer performances, then rated each performance using an Objective Structured Assessment of Technical Skill (OSATS) form and the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS). Participants then rated assessment completion ease (1 = Very Easy, 5 = Very Difficult).

SETTING

National survey of practicing surgeons.

PARTICIPANTS

Sixty-one surgery attendings with experience in laparoscopic surgery from 10 institutions participated as untrained raters. Six experienced laparoscopic skills proctors participated as expert raters.

RESULTS

Inter-rater reliability was substantial for both OSATS (k = 0.75) and MISTELS (k = 0.85). MISTELS accuracy was significantly higher than that of OSATS (κ: MISTELS = 0.18, 95%CI = [0.06,0.29]; OSATS = 0.02, 95%CI = [-0.01,0.04]). While participants were inexperienced with completing video-based assessments (median = 1/5), they perceived video-based assessments favorably (mean = 73.4) and felt assessment completion was "Easy" on average.

CONCLUSIONS

We demonstrate that faculty raters untrained in simulation-based assessments can successfully complete video-based assessments of basic laparoscopic skills with substantial inter-rater reliability without marked difficulty. These findings suggest an opportunity to increase access to feedback for trainees using video-based assessment of fundamental skills in laparoscopic surgery.

The use of household items to support online surgical knot-tying skills training: a mixed methods study

BACKGROUND

This study investigated the perceptions and performance of medical students regarding their engagement and learning of a knot-tying skill presented in an online demonstration format due to the emergency remote measures that accompanied COVID-19 restrictions.

METHODS

Final-year undergraduate medical students were invited to view an online demonstration of a one-handed knot-tying skill and practice the skill using common household items. They recorded their attempts and uploaded them onto the Flipgrid application. Completed attempts were scored using an adapted Objective Structured Assessment of Technical Skill (OSATS) validated tool. We used a mixed-methods sequential explanatory design; data regarding students' engagement was gathered via a short questionnaire, and a Focus Group Discussion (FGD) was conducted to understand their learning experiences better. Descriptive statistics such as proportions and percentages were used to summarize categorical variables and median for continuous variables. Each video attempt was scored independently by two surgeons; reliability was determined using intraclass correlation; statistical tests were conducted at a 5% level of significance. Responses to open-ended survey questions and qualitative data from the FGDs were analysed using thematic analysis.

RESULTS

Seventy-one students participated in the exercise. Most students (91.5%) expressed confidence in their ability to perform the skill and were able to follow the steps in the video demonstration (83.1%). Median number of times needed to practice before video submission was 7.0 (Interquartile range [IQR] 5.0-10.0). Using the adapted OSATS tool; median scores on student attempts were 19.0 out of 21 (IQR: 17.0-20.0) for Assessor 1 and 18.0 out of 21 (IQR: 17.0-20.0) for Assessor 2, and overall scores showed good reliability between assessors based on intraclass correlation (0.86, 95% CI 0.79-0.90, p < 0.001). Qualitative insights from the students' experiences in learning the skill were generally positive; it was a practical, experiential learning process and they valued the social aspects of learning via Flipgrid. Challenges expressed related to the need for in-person training and formal feedback on how to improve their technique. Suggestions to improve their learning included a request for an interactive session with immediate feedback on attempts, and being able to practice with a friend who would assist with videoing.

CONCLUSION

Basic knot-tying can be taught with acceptable efficiency and student satisfaction using online methods with items available at home.

Building the competency of health professionals in the Kyrgyz Republic for the Baby-Friendly Hospital Initiative

Health professional competency building is one of nine national responsibilities (to achieve universal coverage and sustainability) described in the 2018 World Health Organization/UNICEF implementation guidance for the Baby-Friendly Hospital Initiative (BFHI). Skilled breastfeeding support as a standard of newborn care is critical to the establishment of lactation and exclusive breastfeeding. This qualitative case study describes the Kyrgyz Republic's experience with health professional competency building related to breastfeeding counselling and support. We interviewed 38 key informants and reviewed national policies and international guidelines related to BFHI. The study found that although the country has a new policy reflecting BFHI global standards and guidance, the policy has not been disseminated nationally. Additionally, the policy lacks guidance on competency monitoring and verification and does not mention preservice training, even though preservice training on breastfeeding support exists. To achieve universal coverage for health professional competencies, the Kyrgyz Republic uses preservice, in-service and refresher training. However, the main limitations to aligning with the new guidance are a lack of preservice BFHI- and breastfeeding-specific curricula, experienced trainers and sufficient time and funding to dedicate to practical skill development. Conducted during the COVID-19 pandemic, this study confirmed disruptions to BFHI training and service delivery but also documents the Kyrgyz Republic's resilient strides to mitigate impacts on breastfeeding support through facility-level individual champions and adjustments to training such as going online. Opportunities exist for strengthening the competencies of service providers through strengthened preservice training, comprehensive and consistent in-service training, solutions for overworked service providers and clear and sufficiently funded monitoring guidance.