Medical competence as a multilayered construct

How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study

BACKGROUND

Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs.

METHODS

The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability.

RESULTS

Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability.

CONCLUSION

The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

Embracing the future: Technological developments and sustainability in health professional education

Pusic, Crampton, and Eva announce the release of Medical Education's 15th annual State of the Science issue: Embracing the Future

Contemplating the future of competency assessment

Health education's competency models are evolving to balance universal skills with those that are job‐ and organization‐specific. These authors argue that a three‐layer framework emphasizing person‐organization fit will create better outcomes.

Mentorship in health professions education - an AMEE guide for mentors and mentees: AMEE Guide No. 167

This AMEE guide discusses theoretical principles and practical strategies for health professions educators to promote impactful mentoring relationships. Traditional definitions are challenged, distinctions are made between roles such as mentor, advisor, coach and sponsor. As educational environments change and options for professional development expand, we argue that the traditional dyadic format of mentoring alone will not help mentees to maximise their professional growth. Newer formats of mentoring are discussed in detail and their advantages and disadvantages compared. We use a variety of theoretical concepts to anchor the practice of mentorship: self-focussed and other-focussed motives; psychological safety; personal interpretive framework; Daloz model for balancing support and challenge; zone of proximal development; communities of practice; and development along multiple layers of competence. Recommended strategies for effective mentoring are based on extensive review of literature, as well as combined professional mentoring experiences of the authors. We use key principles from the theories described and phases of mentoring relationships as foundations for the suggested best practices of mentorship. Finally, we emphasise the role of mentees in their own professional development and provide tips for them on seeking mentors, expanding their mentoring network and taking the lead in setting the agenda during mentoring meetings and formulating action plans for their own advancement.