Shadow systems in assessment: how supervisors make progress decisions in practice

Implementing Competence Committees on a National Scale: Design and Lessons Learned

Competence committees (CCs) are a recent innovation to improve assessment decision-making in health professions education. CCs enable a group of trained, dedicated educators to review a portfolio of observations about a learner's progress toward competence and make systematic assessment decisions. CCs are aligned with competency based medical education (CBME) and programmatic assessment. While there is an emerging literature on CCs, little has been published on their system-wide implementation. National-scale implementation of CCs is complex, owing to the culture change that underlies this shift in assessment paradigm and the logistics and skills needed to enable it. We present the Royal College of Physicians and Surgeons of Canada's experience implementing a national CC model, the challenges the Royal College faced, and some strategies to address them. With large scale CC implementation, managing the tension between standardization and flexibility is a fundamental issue that needs to be anticipated and addressed, with careful consideration of individual program needs, resources, and engagement of invested groups. If implementation is to take place in a wide variety of contexts, an approach that uses multiple engagement and communication strategies to allow for local adaptations is needed. Large-scale implementation of CCs, like any transformative initiative, does not occur at a single point but is an evolutionary process requiring both upfront resources and ongoing support. As such, it is important to consider embedding a plan for program evaluation at the outset. We hope these shared lessons will be of value to other educators who are considering a large-scale CBME CC implementation.

"I'm on an island": A qualitative study of underperforming surgical trainee perspectives on remediation

BACKGROUND

There is a significant gap in the literature regarding trainees' perceptions of remediation. This study aims to explore surgical trainees' experiences and perspectives of remediation.

METHODS

This qualitative study used semi-structured interviews with 11 doctors who have experienced formal remediation as a surgical trainee. Reflexive thematic analysis was used for data analysis.

RESULTS

In this study, trainees perceived remediation as a harrowing and isolating experience, with long-lasting emotions. There was a perceived lack of clarity regarding explanations of underperformance and subjective goals. Remediation was viewed as a 'performance' and tick-box exercise with superficial plans, with challenging trainee/supervisor dynamics.

CONCLUSIONS

These findings about trainees' perspectives on remediation show a need for trainees to be better emotionally supported during remediation and that remediation plans must be improved to address deficits. Integrating the perspectives and experiences of surgical trainees who have undergone remediation should help improve remediation outcomes and patient care.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

The role of feedback in supporting trainees who underperform in clinical environments

Introduction

Underperformance in clinical environments can be costly and emotional for all stakeholders. Feedback is an important pedagogical strategy for working with underperformance - both formal and informal strategies can make a difference. Feedback is a typical feature of remediation programs, and yet there is little consensus on how feedback should unfold in the context of underperformance.

Methods

This narrative review synthesises literature at the intersections of feedback and underperformance in clinical environments where service, learning and safety need to be considered. We do so with a critical eye towards generating insights for working with underperformance in the clinical environment.

Synthesis and discussion

There are compounding and multi-level factors that contribute to underperformance and subsequent failure. This complexity overwrites simplistic notions of 'earned' failure through individual traits and deficit. Working with such complexity requires feedback that goes beyond educator input or 'telling'. When we shift beyond feedback as input to process, we recognise that these processes are fundamentally relational, where trust and safety are necessary for trainees to share their weaknesses and doubts. Emotions are always present and they signal action. Feedback literacy might help us consider how to engage trainees with feedback so that they take an active (autonomous) role in developing their evaluative judgements. Finally, feedback cultures can be influential and take effort to shift if at all. A key mechanism running through all these considerations of feedback is enabling internal motivation, and creating conditions for trainees to feel relatedness, competence and autonomy. Broadening our perceptions of feedback, beyond telling, might help create environments for learning to flourish.

Learning, judgement and workforce tensions impact failure to fail in a medical postgraduate setting: A supervisor perspective

PURPOSE

Medical training organisations have a duty to prepare medical graduates for future safe, competent practice. Decisions about underperformance are high stakes at the postgraduate level and failure to fail can occur. We aimed to explore this concept from a systems and supervisor perspective.

METHOD

Supervisors of specialist physician trainees were invited to provide written feedback on failure to fail as part of a broader anonymous supervisor survey. They were provided with a trigger statement and responded in free-text format. A deductive content analysis was undertaken through the lenses of supervisor and institution.

RESULTS

Of 663 supervisors who responded to the broader survey, 373 (56%) provided feedback on the failure to fail trigger statement. Analyses indicated an interplay between trainee and supervisor characteristics, and broader system elements. System elements that contributed to failure to fail trainees included lack of longitudinal monitoring and quality of assessment information. Supervisor characteristics included confident, conflicted and avoidant behaviours towards underperforming trainees.

CONCLUSIONS

Individual and system challenges that contributed to failure to fail were identified in this study, and we propose a three-way tension among learning, judgement and workforce. Three potential mitigation strategies have been identified to reduce failure to fail, namely a stage-based approach to remediation, faculty development in supervisory skills and improved assessment-for-learning processes.

Validity evidence for summative performance evaluations in postgraduate community pharmacy education

INTRODUCTION

Workplace-based assessment of competencies is complex. In this study, the validity of summative performance evaluations (SPEs) made by supervisors in a two-year longitudinal supervisor-trainee relationship was investigated in a postgraduate community pharmacy specialization program in the Netherlands. The construct of competence was based on an adapted version of the 2005 Canadian Medical Education Directive for Specialists (CanMEDS) framework.

METHODS

The study had a case study design. Both quantitative and qualitative data were collected. The year 1 and year 2 SPE scores of 342 trainees were analyzed using confirmatory factor analysis and generalizability theory. Semi-structured interviews were held with 15 supervisors and the program director to analyze the inferences they made and the impact of SPE scores on the decision-making process.

RESULTS

A good model fit was found for the adapted CanMEDS based seven-factor construct. The reliability/precision of the SPE measurements could not be completely isolated, as every trainee was trained in one pharmacy and evaluated by one supervisor. Qualitative analysis revealed that supervisors varied in their standards for scoring competencies. Some supervisors were reluctant to fail trainees. The competency scores had little impact on the high-stakes decision made by the program director.

CONCLUSIONS

The adapted CanMEDS competency framework provided a valid structure to measure competence. The reliability/precision of SPE measurements could not be established and the SPE measurements provided limited input for the decision-making process. Indications of a shadow assessment system in the pharmacies need further investigation.

Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making

BACKGROUND

Programmatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student's performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making.

METHODS

In this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner's interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data.

RESULTS

The examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student's performance and quality of narrative feedback.

CONCLUSION

The perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these "difficult" portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.

How Trainees Come to Trust Supervisors in Workplace-Based Assessment: A Grounded Theory Study

PURPOSE

In competency-based medical education, workplace-based assessment provides trainees with an opportunity for guidance and supervisors the opportunity to judge the trainees' clinical practice. Learning from assessment is enhanced when trainees reveal their thinking and are open to critique, which requires trust in the assessor. If supervisors knew more about how trainees come to trust them in workplace-based assessment, they could better engender trainee trust and improve trainees' learning experience.

METHOD

From August 2018 to September 2019, semistructured interviews were conducted with 17 postgraduate anesthesia trainees across Australia and New Zealand. The transcripts were analyzed using constructivist grounded theory methods sensitized by a sociocultural view of learning informed by Wenger's communities of practice theory.

RESULTS

Participants described a continuum from a necessary initial trust to an experience-informed dynamic trust. Trainees assumed initial trust in supervisors based on accreditation, reputation, and a perceived obligation of trustworthiness inherent in the supervisor's role. With experience and time, trainees' trust evolved based on supervisor actions. Deeper levels of trainee trust arose in response to perceived supervisor investment and allowed trainees to devote more emotional and cognitive resources to patient care and learning rather than impression management. Across the continuum from initial trust to experience-informed trust, trainees made rapid trust judgments that were not preceded by conscious deliberation; instead, they represented a learned "feel for the game."

CONCLUSIONS

While other factors are involved, our results indicate that the trainee behavior observed in workplace-based assessment is a product of supervisor invitation. Supervisor trustworthiness and investment in trainee development invite trainees to work and present in authentic ways in workplace-based assessment. This authentic engagement, where learners "show themselves" to supervisors and take risks, creates assessment for learning.

The shift from disbelieving underperformance to recognising failure: A tipping point model

CONTEXT

Coming face to face with a trainee who needs to be failed is a stern test for many supervisors. In response, supervisors have been encouraged to report evidence of failure through numerous assessment redesigns. And yet, there are lingering signs that some remain reluctant to engage in assessment processes that could alter a trainee's progression in the programme. Failure is highly consequential for all involved and, although rare, requires explicit study. Recent work identified a phase of disbelief that preceded identification of underperformance. What remains unknown is how supervisors come to recognise that a trainee needs to be failed.

METHODS

Following constructivist grounded theory methodology, 42 physicians and surgeons in British Columbia, Canada shared their experiences supervising trainees who profoundly underperformed, required extensive remediation or were dismissed from the programme. We identified recurring themes using an iterative, constant comparative process.

RESULTS

The shift from disbelieving underperformance to recognising failure involves three patterns: accumulation of significant incidents, discovery of an egregious error after negligible deficits or illumination of an overlooked deficit when pointed out by someone else. Recognising failure was accompanied by anger, certainty and a sense of duty to prevent harm.

CONCLUSION

Coming to the point of recognising that a trainee needs to fail is akin to the psychological process of a tipping point where people first realise that noise is signal and cross a threshold where the pattern is no longer an anomaly. The co-occurrence of anger raises the possibility for emotions to be a driver of, and not only a barrier to, recognising failure. This warrants caution because tipping points, and anger, can impede detection of improvement. Our findings point towards possibilities for supporting earlier identification of underperformance and overcoming reluctance to report failure along with countermeasures to compensate for difficulties in detecting improvement once failure has been verified.

Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review

PURPOSE

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions.

METHOD

The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders.

RESULTS

In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain.

CONCLUSIONS

Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

Exploring patient experiences in the student outpatient clinic - A contribution to learning

OBJECTIVE

To understand how patients experience participation in student encounters.

METHODS

Qualitative semi-structured telephone interviews with ten cardiac surgery patients who had attended voluntary postoperative consultations in a student outpatient clinic. The interview guide included questions about reasons for and experiences of being part of a teaching situation. Interviews were analysed through inductive thematic analysis where pieces of text in each interview were assigned different codes and condensed into themes.

RESULTS

The patients expressed a duality in their reasons for participating in student consultations: (1) a personal need for assurance (safety) and (2) a wish to help students (altruism). Students were perceived as professional and sometimes insecure. Being part of an educational situation was meaningful to the patients because they did not feel objectified. Knowing that there was a backup supervisor made the patients feel safe even though the supervisor was not present during all parts of the consultation.

CONCLUSIONS

Patients experienced safety, understood their role in all parts of the consultation, and shared a wish to help students learn.

PRACTICE IMPLICATIONS

A sequential consultation model alternating between student- and supervisor-driven supervision can balance student autonomy and patient safety. This knowledge could guide future patient-centred medical education in student clinics.

Ottawa 2020 consensus statement for programmatic assessment - 1. Agreement on the principles

INTRODUCTION

In the Ottawa 2018 Consensus framework for good assessment, a set of criteria was presented for systems of assessment. Currently, programmatic assessment is being established in an increasing number of programmes. In this Ottawa 2020 consensus statement for programmatic assessment insights from practice and research are used to define the principles of programmatic assessment.

METHODS

For fifteen programmes in health professions education affiliated with members of an expert group (n = 20), an inventory was completed for the perceived components, rationale, and importance of a programmatic assessment design. Input from attendees of a programmatic assessment workshop and symposium at the 2020 Ottawa conference was included. The outcome is discussed in concurrence with current theory and research.

RESULTS AND DISCUSSION

Twelve principles are presented that are considered as important and recognisable facets of programmatic assessment. Overall these principles were used in the curriculum and assessment design, albeit with a range of approaches and rigor, suggesting that programmatic assessment is an achievable education and assessment model, embedded both in practice and research. Knowledge on and sharing how programmatic assessment is being operationalized may help support educators charting their own implementation journey of programmatic assessment in their respective programmes.

Key dimensions of innovations in workplace-based assessment for postgraduate medical education: a scoping review

BACKGROUND

Specialist training bodies continue to devise innovative methods of gathering information on trainee workplace performance to meet the requirements of competency-based medical education. We reviewed recent innovations in workplace-based assessment (WBA) tools to identify strengths, weaknesses, and trade-offs inherent in their design and use.

METHODS

In this scoping review, using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we systematically searched databases between 2009 and 2019 for WBA tools with novel characteristics not typically seen in traditional WBAs. These included innovations in rating scales, ways of collecting information, technological innovations, ways of triggering WBAs, and approaches to compiling and using information.

RESULTS

We identified 30 innovative WBA tools whose characteristics could be categorised into seven dimensions: frequency of assessment, granularity (unit of performance assessed), coverage of the curriculum, rating method, initiation of the WBA, information use, and incentives. These dimensions had multiple interdependencies and trade-offs, often balancing generating assessment data with available resources. Philosophical stance on assessment also influenced WBA choice, for example prioritising trainee-centred learning (i.e. initiation of WBA and transparency of assessment data), perceptions of assessment and feedback as burdensome or beneficial, and holistic vs reductionist views on assessment of performance.

CONCLUSIONS

Our synthesis of the literature on innovative WBAs provides a framework for categorising tool characteristics across seven dimensions, systematically teasing apart the considerations in design and use of workplace assessments. It also draws attention to the trade-offs inherent in tool design and selection, and enables a more deliberate consideration of the tool characteristics most appropriate to the local context.

Implementing programmatic assessment transforms supervisor attitudes: An explanatory sequential mixed methods study

INTRODUCTION

Programmatic assessment (PA) is an increasingly popular approach to competency-based assessment (CBA), yet evaluation evidence is limited. This study aimed to identify and explore supervisor attitudes before and after implementing a novel PA using a sequential explanatory mixed methods design. In phase one, a survey was used to identify supervisor perspectives on work-based placements, PA and CBA. Survey results were then applied to develop focus group questions to further explore supervisor attitudes.

RESULTS

PA was found to improve supervisor-student relationships by removing high-stakes assessment decisions and creating greater capacity for feedback and teaching, leading to a productive learning environment. Assessment was perceived as an important role and supervisors wanted to feel valued and heard within PA. Trust was conceptualised as a triad between supervisor, student and university, and enabled supervisors to engage with PA which was important for success. Supervisor learning of PA was experiential and often supported by students, highlighting the need for hands-on training.

CONCLUSION

Participants reported a high level of agreement with PA and CBA principles which may have made them amenable to educational change. Further research is needed to explore the experience of all stakeholders and to understand how worldviews and culture influence assessment initiatives.

Team member expectations of trainee communicator and collaborator competencies - so shines a good deed in a weary world?

BACKGROUND

Workplace-based assessment may be further optimized by drawing upon the perspectives of multiple assessors, including those outside the trainee's discipline. Interprofessional competencies like communication and collaboration are often considered suitable for team input.

AIM

We sought to characterize multidisciplinary expectations of communicator and collaborator competency roles.

METHODS

We adopted a constructivist grounded theory approach to explore perspectives of multidisciplinary team members on a clinical teaching unit. In semi-structured interviews, participants described expectations for competent collaboration and communication of trainees outside their own discipline. Data were analyzed to identify recurring themes, underlying concepts and their interactions using constant comparison.

RESULTS

Three main underlying perspectives influenced interprofessional characterization of competent communication and collaboration: (1) general expectations of best practice; (2) specific expectations of supportive practice; and (3) perceived commitment to teaching practice. However, participants seemingly judged trainees outside their discipline according to how competencies were exercised to advance their own professional patient care decision-making, with minimal attention to the trainee's specific skillset demonstrated.

CONCLUSION

While team members expressed commitment to supporting interprofessional competency development of trainees outside their discipline, service-oriented judgement of performance loomed large. The potential impact on the credibility of multidisciplinary sources for workplace-based assessment requires consideration.

Where the rubber meets the road - An integrative review of programmatic assessment in health care professions education

INTRODUCTION

Programmatic assessment was introduced as an approach to design assessment programmes with the aim to simultaneously optimize the decision-making and learning function of assessment. An integrative review was conducted to review and synthesize results from studies investigating programmatic assessment in health care professions education in practice.

METHODS

The authors systematically searched PubMed, Web of Science, and ERIC to identify studies published since 2005 that reported empirical data on programmatic assessment. Characteristics of the included studies were extracted and synthesized, using descriptive statistics and thematic analysis.

RESULTS

Twenty-seven studies were included, which used quantitative methods (n = 10), qualitative methods (n = 12) or mixed methods (n = 5). Most studies were conducted in clinical settings (77.8%). Programmatic assessment was found to enable meaningful triangulation for robust decision-making and used as a catalyst for learning. However, several problems were identified, including overload in assessment information and the associated workload, counterproductive impact of using strict requirements and summative signals, lack of a shared understanding of the nature and purpose of programmatic assessment, and lack of supportive interpersonal relationships. Thematic analysis revealed that the success and challenges of programmatic assessment were best understood by the interplay between quantity and quality of assessment information, and the influence of social and personal aspects on assessment perceptions.

CONCLUSION

Although some of the evidence may seem compelling to support the effectiveness of programmatic assessment in practice, tensions will emerge when simultaneously stimulating the development of competencies and assessing its result. The identified factors and inferred strategies provide guidance for navigating these tensions.

Sociocultural learning in emergency medicine: a holistic examination of competence

Objectives

In the medical community of practice, the resident is situated in systems of professional and cultural activities. How diagnostic competence manifests in their sociocultural context is examined through situativity and systems theories. A holistic model is proposed that could examine diagnostic competence across micro (individual), meso (activity), and macro (cultural) systems.

Methods

Two short scenarios are presented resulting from observations of emergency medicine residents and their supervising physicians. These scenarios are analyzed using a trans-theoretical model of situativity and systems theories to understand how diagnostic competence manifests in practice (activity system).

Results

Assessment of diagnostic competence in a sociocultural context may require assessment of responses to contextual factors that seem immaterial to clinical reasoning. This information may signal that the resident also has the skills to identify appropriate information channels within an activity system and can accurately collect and prioritize clinical information within those channels. Therefore, the formal assessment of competent clinical reasoning performance, as a situated practice, may benefit from delineating how much of the context of an activity system is required to be competently managed and synthesized across the competency milestones.

Conclusions

The examination of diagnostic competence as a sociocultural practice can provide a unique and holistic examination of clinical reasoning performance and assessment.