An Innovative, No-cost, Evidence-Based Smartphone Platform for Resident Evaluation

A Resident-Driven Mobile Evaluation System Can Be Used to Augment Traditional Surgery Rotation Evaluations

BACKGROUND

The Accreditation Council for Graduate Medical Education requires residents to receive milestone-based evaluations in key areas. Shortcomings of the traditional evaluation system (TES) are a low completion rate and delay in completion. We hypothesized that adoption of a mobile evaluation system (MES) would increase the number of evaluations completed and improve their timeliness.

METHODS

Traditional evaluations for a general surgery residency program were converted into a web-based form via a widely available, free, and secure application and implemented in August 2017. After 8 months, MES data were analyzed and compared to that of our TES.

RESULTS

122 mobile evaluations were completed; 20% were solicited by residents. Introduction of the MES resulted in an increased number of evaluations per resident (P = .0028) and proportion of faculty completing evaluations (P = .0220). Timeliness also improved, with 71% of evaluations being completed during one's clinical rotation.

CONCLUSIONS

A resident-driven MES is an inexpensive and effective method to augment traditional end-of-rotation evaluations.

Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review

PURPOSE

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions.

METHOD

The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders.

RESULTS

In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain.

CONCLUSIONS

Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

Best Approaches to Evaluation and Feedback in Post-Graduate Medical Education

PURPOSE OF REVIEW

The objectives of this literature review are to appraise current approaches and assess new technologies that have been utilized for evaluation and feedback of residents, with focus on surgical trainees.

RECENT FINDINGS

In 1999, the Accreditation Council for Graduate Medical Education introduced the Milestone system as a tool for summative evaluation. The organization allows individual program autonomy on how evaluation and feedback are performed. In the past, questionnaire evaluations and informal verbal feedback were employed. However, with the advent of technology, they have taken a different shape in the form of crowdsourcing, mobile platforms, and simulation. Limited data is available on new methods but studies show promise citing low cost and positive impact on resident education. No one "best approach" exists for evaluation and feedback. However, it is apparent that a multimodal approach that is based on the ACGME Milestones can be effective and aid in guiding programs.

Comparison of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) to a Single-Item Performance Score

Construct: We compared a single-item performance score with the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for their ability in assessing surgical competency.

BACKGROUND

Surgical programs are adopting competency-based frameworks. The adoption of these frameworks for assessment requires tools that produce accurate and valid assessments of knowledge and technical performance. An assessment tool that is quick to complete could improve feasibility, reduce delays, and result in a higher volume of assessments of learners. Previous work demonstrated that the 9-item O-SCORE can produce valid results; the goal of this study was to determine if a single-item performance rating (Is candidate competent to independently complete procedure: yes or no) completed at a separate viewing would correlate to the O-SCORE, thus increasing feasibility of procedural competence assessment.

APPROACH

Nineteen residents and 2 staff orthopedic surgeons from the University of Ottawa volunteered for a 2-part OSCE-style station including a written questionnaire and videotaped simulated open reduction and internal fixation midshaft radius fracture. Each performance was rated independently by 3 orthopedic surgeons using a single-item performance score (Time 1). The performances were assessed again 6 weeks later by the 3 raters using the O-SCORE (Time 2). Correlation between the single-item performance score and the O-SCORE were evaluated.

RESULTS

Three orthopedic surgeons completed 21 ratings each resulting in 63 orthopedic ratings. There was a high level of correlation and agreement between the single-item performance score at Time 1 and Time 2 (κ correlation =0.72-1.00; p < .001; percentage agreement =90%-100%). The reliability of the O-SCORE at Time 2 with three raters was 0.83 and the internal consistency was 0.89. There was a tendency for each rater to assign more yes responses to the more senior trainees.

CONCLUSIONS

A single-item performance score correlated highly with the O-SCORE in an orthopedic setting. A single-item score could be used to supplement a multi-item score with similar results in orthopedics. There is still benefit in completing multi-item scores such as the O-SCORE evaluations to guide specific areas of improvement and direct feedback.

Point-of-Encounter Assessment: Using Health Belief Model Constructs to Change Grading Behaviors

OBJECTIVE

Fourth-year medical students need feedback to improve. Even during 1-month rotations, there needs to be a formal mid-clerkship feedback session. Better feedback involves multiple surgical evaluators at multiple levels. Constructs from the Health Belief Model of behavior change were used to assess faculty and resident grading behaviors to create a more usable evaluation system. A point-of-encounter (POE) system was created. The objective of this study was to review the efficacy of a POE clinical evaluation card (CEC) system which was initiated to increase evaluator's participation in grading and formative feedback prior to mid-clerkship evaluation.

DESIGN

The study was a 1-year retrospective cohort study reviewing the CECs for level of evaluator, content, and student compliance. A Likert-type scale survey regarding the usage of the clinical cards was also completed by evaluators.

SETTING

Texas Tech University Health Sciences Center at El Paso, during 2 fourth-year medical student rotations, Subinternship (Sub-I), and Surgical Intensive Care Unit (SICU).

PARTICIPANTS

34 fourth-year medical students and 20 evaluators.

RESULTS

Students turned in a mean of 10 cards, 75% in SICU and 65% in Sub-I turned in all 10 cards. There were significantly greater advanced residents evaluating during Sub-I vs SICU: mean evals by PGY3 were 1.9 vs 0.75 (p = .01) and mean evals by PGY5 were 1.4 vs 0.1 (p < .0001). There were significantly more faculty completing evaluations during SICU vs Sub-I: 2.5 faculty evals/student vs 1.4 faculty evals/student (p = .023). Evaluator ratings were high on a 5-point Likert-type scale, with most responses near the "strongly agree" rating of 4.7 to 4.8.

CONCLUSIONS

Use of POE CECs met goals of having at least 7 CECs turned in by mid-clerkship and 10 at end-clerkships. Formative evaluations by mid-clerkship went from 0 to 7 evaluations. Evaluator surveys highlighted clarity and efficiency as reasons for using CECs.

Identifying Priorities for Faculty Development in General Surgery Using the Delphi Consensus Method

OBJECTIVE

Faculty teaching skills are critical for effective surgical education, however, which skills are most important to be taught in a faculty development program have not been well defined. The objective of this study was to identify priorities for faculty development as perceived by surgical educators.

DESIGN

We used a modified Delphi methodology to assess faculty perceptions of the value of faculty development activities, best learning modalities, as well as barriers and priorities for faculty development. An expert panel developed the initial survey and distributed it to the membership of the Association of Program Directors in Surgery. Responses were reviewed by the expert panel and condensed to 3 key questions that were redistributed to the survey participants for final ranking.

PARTICIPANTS

Seven experts reviewed responses to 8 questions by 110 participants. 35 participants determined the final ranking responses to 3 key questions.

RESULTS

The top three priorities for faculty development were: 1) Resident assessment/evaluation and feedback 2) Coaching for faculty teaching, and 3) Improving intraoperative teaching skills. The top 3 learning modalities were: 1) Coaching 2) Interactive small group sessions, and 3) Video-based education. Barriers to implementing faculty development included time limitations, clinical workload, faculty interest, and financial support.

CONCLUSIONS

Faculty development programs should focus on resident assessment methods, intraoperative and general faculty teaching skills using a combination of coaching, small group didactic and video-based education. Concerted efforts to recognize and financially reward the value of teaching and faculty development is required to support these endeavors and improve the learning environment for both residents and faculty.