Obstacles Affecting the Implementation of the O-SCORE for Assessment of Orthopedic Surgical Skills Competency

Incorporating Intraoperative Assessments of surgical Trainees Performing Open Surgery Into Summative Trainee Evaluations: A Scoping Review

OBJECTIVE

Real-world assessment of trainee surgical competence and autonomy must occur in the operative environment. It is unclear if or how workplace-based operative skills assessments are systematically incorporated into summative evaluations. In this study, intraoperative surgical skills assessment tools were reviewed for actual or proposed incorporation into summative evaluations.

DESIGN

A scoping review was conducted. The Cochrane Central Registrar of Clinical Trials, Embase, Medline, and Web of Science were searched for articles published from January 2012 to December 2021. Articles describing an educational assessment paradigm of a post-graduate surgical trainee performing an open surgical procedure were selected. Using the PRISMA Extension Guidelines for Scoping Reviews, two independent reviewers completed data extraction including literature quality, assessment paradigm, assessment quality, incorporation into summative evaluations, and facilitators and barriers to implementation. Finally, a narrative synthesis of extracted data was completed.

RESULTS

Twenty-two articles met inclusion criteria describing 11 different surgical assessments. All were cohort studies, of which 21 involved the evaluation of an assessment tool and one involved participant perceptions of the assessment tool. All studies discussed the possible use of their assessment to track progress over time, but no study explicitly outlined the incorporation of the tool into a summative review. Four types of assessments automatically integrated results, which reportedly were utilized at Clinical Competency Committee (CCC) meetings. Facilitators and barriers of surgical assessment integration into summative assessments was not described.

CONCLUSIONS

While proponents of formative intra-operative assessment have claimed that their potential use in supporting summative evaluation and tracking learning trajectories are major benefits, a systematic method of integrating workplace surgical assessments into summative resident evaluation has not been described or uniformly implemented.

Prediction of Basic Robotic Competence for Robotic Visceral Operations Using the O-Score within the "Robotic Curriculum for Young Surgeons" (RoCS)

INTRODUCTION

Surgical residency programs lack structured assessments of robotic surgery. The validated O-Score is an assessment tool for tracking robotic operative proficiency consisting of 9 items on a 5 point Likert scale. Surgical autonomy is one comprehensive binary item. This study aimed to establish a benchmark for the number of procedures and the O-Score sum score to achieve surgical autonomy in robotic visceral procedures.

MATERIAL AND METHODS

This single-center prospective pilot cohort study assessed robotic procedures between 2020 and 2023. Bedside and console assistance performances were analyzed separately based on the O-Score and the calculated total numerical sum of the individual item values of the O-Score. Bedside assistance was conducted for upper gastrointestinal, hepatopancreatobiliary, and colorectal procedures, whereas console assistance referred to either one of the three areas. The study participants included inexperienced robotic surgeons who were evaluated by 2 robotic experts.

RESULTS

In total, 273 procedures were included in this study. For 13 bedside assistants, 273 O-Score assessments were identified, and 62 O-Score assessments for six console assistants. Surgical autonomy was achieved in 50.9% for bedside assistance and in 11.3 % for assistance at the robotic console. Surgical autonomy was positively correlated with the O-Score sum for bedside (p = < 0.001) and console assistance (p = 0.004). The positive prediction of surgical autonomy for bedside (console) assistance ranged from 74% (60%) to 93% (100%), correlated with a range of the O-Score sum between 37 (37) and 40 (40) and a robotic caseload between 19 (17) and 33 (24) procedures.

CONCLUSIONS

A significant improvement in the basic robotic performance was observed. Benchmarks regarding number of cases and O-Score sum were established for bedside assistance regardless of the type of visceral robotic operation. Currently, data on console assistance are limited. Monitoring robotic operative skills and skill progression is feasible in daily routine using the O-Score and O-Score sum.

Assessment in Undergraduate Competency-Based Medical Education: A Systematic Review

BACKGROUND

Studies that have methodically compiled the body of research on the competency-based medical education (CBME) assessment procedure and pinpointed knowledge gaps about the structure of the assessment process are few. Thus, the goals of the study were to create a model assessment framework for competency-based medical education that would be applicable in the Indian setting as well as to thoroughly examine the competency-based medical education assessment framework.

METHODS

PubMed, MEDLINE (Ovid), EMBASE (Ovid), Scopus, Web of Science, and Google Scholar were the databases that were searched. The search parameters were restricted to English language publications about competency-based education and assessment methods, which were published between January 2006 and December 2020. A descriptive overview of the included research (in tabular form) served as the foundation for the data synthesis.

RESULTS

Databases provided 732 records; out of which 36 fulfilled the inclusion and exclusion criteria. Thirty-six studies comprised a mix of randomized controlled trials, focus group interviews, and questionnaire studies, including cross-sectional studies, qualitative studies (03), mixed-method studies, etc. The papers were published in 10 different journals. The greatest number was published in BMC Medical Education (18). The average quality score for included studies was 62.53% (range: 35.71-83.33%). Most authors are from the UK (07), followed by the USA (05). The included studies were grouped into seven categories based on their dominant focus: moving away from a behavioristic approach to a constructive approach of assessment (01 studies), formative assessment (FA) and feedback (10 studies), the hurdles in the implementation of feedback (04 studies), utilization of computer or online based formative test with automated feedback (05 studies), video feedback (02 studies), e-learning platforms for formative assessment (04 studies), studies related to workplace-based assessment (WBA)/mini-clinical evaluation exercise (mini-CEX)/direct observation of procedural skills (DOPS) (10 studies).

CONCLUSIONS

Various constructivist techniques, such as concept maps, portfolios, and rubrics, can be used for assessments. Self-regulated learning, peer feedback, online formative assessment, an online computer-based formative test with automated feedback, the use of a computerized web-based objective structured clinical examination (OSCE) evaluation system, and the use of narrative feedback instead of numerical scores in mini-CEX are all ways to increase student involvement in the design and implementation of the formative assessment.

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

Developing and Assessing the Feasibility of Implementing a Surgical Objective Structured Clinical Skills Examination (S-OSCE)

OBJECTIVE

To1 describe the development and evaluate the feasibility of a surgical objective structured clinical examination (OSCE) for the purpose of competency assessment based on the Royal College of Canada's CanMEDS framework.

DESIGN

A unique surgical OSCE was developed to evaluate the clinical and surgical management of common orthopaedic problems using simulated patients and cadaveric specimens. Cases were graded by degree of difficulty (less complex, complex, more complex) Developing an assessment tool with significant resource utilization and good correlation with traditional methods is challenging. The feasibility of an OSCE that evaluates independent clinical and surgical decision making was evaluated. In addition, as part of establishing construct validity, correlation of OSCE scores with previously validated O-scores was performed.

SETTING

A tertiary level academic teaching hospital.

PARTICIPANTS

Thirty-four Postgraduate year 3-5 trainees of a 5-year Canadian orthopedic residency program creating 96 operative case performances available for final review.

RESULTS

The development of the OSCE cases involved a multistep process with attending surgeons, residents and a surgical education consultant. There were 4 different OSCE days, over a 3 year period (2016-2018) encompassing a variety of less complex and more complex procedures. Performance on the OSCE correlated strongly with the (O-SCORE, 0.89) and a linear regression analysis correlated moderately with year of training (r² = 0.5737). The feasibility analysis demonstrated good financial practicality with solid programmatic integration.

CONCLUSIONS

The unique surgical OSCE scores correlate strongly with an established entrustability scale. Administering this OSCE to evaluate preoperative and intraoperative decision making to complement other forms of assessment is feasible. The financial burden to training programs is modest in comparison to the insight gained by both residents and faculty.

Development and Evaluation of a Surgical Direct Assessment Tool for Resident Training

In podiatric residency training, minimum activity volume numbers are used to assess surgical competency. The purpose of this study was to develop a standardized direct assessment form as a complement to minimum activity volume numbers. Sixteen attending physicians completed 121 direct assessment forms, evaluating six podiatric medicine and surgery residents. Evaluation scores were highly correlated with residency year. Resident feedback was positive, with the open-response portion identified as especially useful. Although further efforts may help refine this approach, the use of standardized, competency-based direct assessment has the potential to improve the training of podiatric medicine and surgery residents.