Factors affecting perceived credibility of assessment in medical education: A scoping review

Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review

BACKGROUND

Feedback processes are crucial for learning, guiding improvement, and enhancing performance. In workplace-based learning settings, diverse teaching and assessment activities are advocated to be designed and implemented, generating feedback that students use, with proper guidance, to close the gap between current and desired performance levels. Since productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured feedback activities within undergraduate workplace-based learning settings. However, these settings are characterized by their unpredictable nature, which can either promote learning or present challenges in offering structured learning opportunities for students. This scoping review maps literature on how feedback processes are organised in undergraduate clinical workplace-based learning settings, providing insight into the design and use of feedback.

METHODS

A scoping review was conducted. Studies were identified from seven databases and ten relevant journals in medical education. The screening process was performed independently in duplicate with the support of the StArt program. Data were organized in a data chart and analyzed using thematic analysis. The feedback loop with a sociocultural perspective was used as a theoretical framework.

RESULTS

The search yielded 4,877 papers, and 61 were included in the review. Two themes were identified in the qualitative analysis: (1) The organization of the feedback processes in workplace-based learning settings, and (2) Sociocultural factors influencing the organization of feedback processes. The literature describes multiple teaching and assessment activities that generate feedback information. Most papers described experiences and perceptions of diverse teaching and assessment feedback activities. Few studies described how feedback processes improve performance. Sociocultural factors such as establishing a feedback culture, enabling stable and trustworthy relationships, and enhancing student feedback agency are crucial for productive feedback processes.

CONCLUSIONS

This review identified concrete ideas regarding how feedback could be organized within the clinical workplace to promote feedback processes. The feedback encounter should be organized to allow follow-up of the feedback, i.e., working on required learning and performance goals at the next occasion. The educational programs should design feedback processes by appropriately planning subsequent tasks and activities. More insight is needed in designing a full-loop feedback process, in which specific attention is needed in effective feedforward practices.

Feedback that Lands: Exploring How Residents Receive and Judge Feedback During Entrustable Professional Activities

Introduction

Receiving feedback from different types of assessors (e.g., senior residents, staff supervisors) may impact trainees' perceptions of the quantity and quality of data during entrustable professional activity (EPA) assessments. We evaluated the quality of EPA feedback provided by different assessors (senior residents, chief medical residents/subspecialty residents, and staff) and explored residents' judgements of the value of this feedback.

Methods

From a database of 2228 EPAs, we calculated the frequency of contribution from three assessor groups. We appraised the quality of 60 procedure-related EPAs completed between July 2019 and March 2020 using a modified Completed Clinical Evaluation Report Rating (CCERR) tool. Next, we asked 15 internal medicine residents to sort randomly selected EPAs according to their judgements of value, as an elicitation exercise before a semi-structured interview. Interviews explored participants' perceptions of quality of written feedback and helpful assessors.

Results

Residents completed over 60% of EPA assessments. We found no difference in modified-CCERR scores between the three groups. When judging EPA feedback value, residents described a process of weighted deliberation, considering perceived assessor characteristics (e.g., credibility, experience with EPA system), actionable written comments, and their own self-assessment.

Discussion

Like other recent studies, we found that residents contributed most to procedure-related EPA assessments. To the established list of factors influencing residents' judgements of feedback value, we add assessors' adherence to, and their shared experiences of being assessed within, EPA assessment systems. We focus on the implications for how assessors and leaders can build credibility in themselves and in the practices of EPA assessments.

Undergraduate Learners' Receptiveness to Feedback in Medical Schools: A Scoping Review

Feedback from educators to learners is considered an important element of effective learning in medical school. While early studies were focused on the processes of providing feedback, recent work has showed that factors related to how learners receive feedback seems to be equally important. Considering that the literature on this topic is new in medical education, and studies are diverse and methodologically variable, we sought to conduct a scoping review to map the articles on receptiveness to feedback, to provide an overview of its related factors, to identify the types of research conducted in this area, and to document knowledge gaps in the existing literature. Using the Joanna Briggs Institute scoping review methodology, we searched four databases (CINAHL, Ovid, PubMed, and Web of Science) and screened 9120 abstracts, resulting in 98 articles for our final analysis. In this sample, 80% of studies on the feedback receiver were published in the last 10 years, and there is a vast variation in the studies' methodologies. The main factors that affect medical students' receptiveness to feedback are students' characteristics, feedback content, educators' credibility, and the learning environment. Feedback literacy is a very recent and rarely used term in medical education; therefore, an important area for further investigation. Lastly, we identified some gaps in the literature that might guide future research, such as studying receptiveness to feedback based on academic seniority and feedback literacy's long-term impacts on learning.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01858-0.

Process evaluation of paediatric fellowship training programs at a University Hospital in Pakistan

BACKGROUND

Fellowship programs offer training in a subspecialty focusing on distinct and advanced clinical/academic skills. This advanced postgraduate training allows physicians, who desire a more specialized practice, to further develop clinical, academic, research, and leadership/administrative skills. The Aga Khan University (AKU) is one of the few institutes offering paediatric sub-specialty training in Pakistan. We aimed to evaluate the current Paediatric fellowship programs at AKU.

METHODS

Process evaluation of six paediatric fellowship programs (cardiology, neurology, endocrinology, critical care medicine, neonatology, and infectious disease) was conducted from September 2020 to April 2021 by senior clinicians and medical educationists. Evidence was collected through document review (using existing postgraduate medical education program information form), observation of teaching and learning support, and focused group discussions/interviews with program faculty and fellows were conducted. A review of the evaluation report was done as part of this study. This study received an exemption from the ethical review committee. The quantitative data were analyzed using SPSS (22.0) while the reports of discussion with fellows and friends underwent content analysis.

RESULTS

All fellowship programs met the criteria for having a robust competency-based fellowship curriculum as per the institutional and national guidelines. Formative assessment in the form of continuous evaluation was found to be integrated into all the fellowship programs, however, most of the programs were found to lack a summative assessment plan. Fellows in training and program faculty were satisfied with the opportunities for mentorship, teaching, and learning. Thematic analysis of the discussion reports with faculty and fellows revealed three key themes including, program aspects translating into strengthening the training, gaps in the training program in delivering the expectations, and making ways to reach par excellence.

CONCLUSIONS

The process evaluation of paediatric fellowship programs provided an opportunity to holistically review the current strengths and quality of the training in individual programs along with the unmet needs of the trainees. This will help the program stakeholders to prioritize, align and allocate the resources to further enhance the quality of training and outcome of individual fellowship programs to ensure wider impacts at a regional, national, and international health system level.

Assessing electrodiagnostic skills among residents and fellows: Relationships between workplace-based assessments using the Electromyography Direct Observation Tool and other measures of trainee performance

INTRODUCTION/AIMS

Graduate medical education programs must ensure residents and fellows acquire skills needed for independent practice. Workplace-based observational assessments are informative but can be time- and resource-intensive. In this study we sought to gather "relations-to-other-variables" validity evidence for scores generated by the Electromyography Direct Observation Tool (EMG-DOT) to inform its use as a measure of electrodiagnostic skill acquisition.

METHODS

Scores on multiple assessments were compiled by trainees during Clinical Neurophysiology and Electromyography rotations at a large US academic medical center. Relationships between workplace-based EMG-DOT scores (n = 298) and scores on a prerequisite simulated patient exercise, patient experience surveys (n = 199), end-of-rotation evaluations (n = 301), and an American Association of Neuromuscular & Electrodiagnostic Medicine (AANEM) self-assessment examination were assessed using Pearson correlations.

RESULTS

Among 23 trainees, EMG-DOT scores assigned by physician raters correlated positively with end-of-rotation evaluations (r = 0.63, P = .001), but EMG-DOT scores assigned by technician raters did not (r = 0.10, P = .663). When physician and technician ratings were combined, higher EMG-DOT scores correlated with better patient experience survey scores (r = 0.42, P = .047), but not with simulated patient or AANEM self-assessment examination scores.

DISCUSSION

End-of-rotation evaluations can provide valid assessments of trainee performance when completed by individuals with ample opportunities to directly observe trainees. Inclusion of observational assessments by technicians and patients provides a more comprehensive view of trainee performance. Workplace- and classroom-based assessments provide complementary information about trainee performance, reflecting underlying differences in types of skills measured.