Workplace-based assessments in postgraduate medical education: A hermeneutic review

Does authentic assessment undermine authentic learning?

In this editorial the editors consider the ideals and realities of high and low stakes assessments in clinical workplaces, the impact of these assessments on clinical workplace learning, and the clash between authenticity in assessment and authenticity in learning.

A phenomenological study of resident and faculty experiences with learner engagement in the normalization of workplace-based assessment

Background

Workplace-based assessments (WPBA) have become integral to learner-centred medical education. As previous research has linked learner engagement to WPBA implementation, this study explores residents' and faculty members' experiences with learner engagement in the normalisation of WPBA practice.

Methods

Transcendental phenomenology was used as the qualitative approach, focusing on the participants' lived experiences. A semi-structured interview guide was used to interview five faculty members and five residents who had conducted WPBA. The interviews were transcribed and analysed using phenomenological data analysis.

Results

Three themes were identified between learner engagement and WPBA conduct: (a) work environment, (b) roles and relationships, and (c) mutually beneficial teaching and learning. WPBA learner engagement occurred when participants interacted with each other and with the clinical setting to facilitate teaching and learning. Both participant groups reported a desire to participate in WPBA, but time constraints at times hindered participation. The residents indicated that WPBA improved their knowledge and admitted to experiencing negative emotions during the assessment. Overall, participants recognised the reciprocal benefits of WPBA participation for their professional development.

Conclusion

The findings of the study suggest that learner engagement influences the use of WPBA. Consequently, it may be beneficial to consider the role of learner engagement to normalise WPBA application for teaching and learning in the clinical context.

The Ottawa resident observation form for nurses (O-RON): evaluation of an assessment tool's psychometric properties in different specialties

BACKGROUND

Workplace-based assessment (WBA) used in post-graduate medical education relies on physician supervisors' feedback. However, in a training environment where supervisors are unavailable to assess certain aspects of a resident's performance, nurses are well-positioned to do so. The Ottawa Resident Observation Form for Nurses (O-RON) was developed to capture nurses' assessment of trainee performance and results have demonstrated strong evidence for validity in Orthopedic Surgery. However, different clinical settings may impact a tool's performance. This project studied the use of the O-RON in three different specialties at the University of Ottawa.

METHODS

O-RON forms were distributed on Internal Medicine, General Surgery, and Obstetrical wards at the University of Ottawa over nine months. Validity evidence related to quantitative data was collected. Exit interviews with nurse managers were performed and content was thematically analyzed.

RESULTS

179 O-RONs were completed on 30 residents. With four forms per resident, the ORON's reliability was 0.82. Global judgement response and frequency of concerns was correlated (r = 0.627, P < 0.001).

CONCLUSIONS

Consistent with the original study, the findings demonstrated strong evidence for validity. However, the number of forms collected was less than expected. Exit interviews identified factors impacting form completion, which included clinical workloads and interprofessional dynamics.

Croix R, van Melle E. Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Barriers and Facilitators in Continuous Medical Education Related to Allogeneic Stem Cell Transplantation: A Qualitative Study of Physicians

INTRODUCTION

This study explored qualitatively, in a sample of German hematologists working in clinical allogeneic hematopoietic stem cell transplantation (alloHSCT), perceptions of barriers and facilitators to participate in continuous medical education (CME), to provide detailed information on how to improve participation in CME activities related to alloHSCT, which may also be applicable to other areas of medicine.

METHODS

Based on a recruitment campaign of the German Association for Hematopoietic Stem Cell Transplantation (DAG-HSZT), 21 semi-structured telephone interviews were conducted, transcribed, and analyzed using framework analysis.

RESULTS

Three clusters of barriers were identified that explain why alloHSCT physicians may or may not participate in CME: individual constraints (e.g., better networking, young physicians being overwhelmed by the complexity of alloHSCT), structural constraints (e.g., time and financial issues, tailoring CME courses according to the targeted audience), and content-related constraints (e.g., requirement of CME sessions, provision of an overview of CME courses, more flexible offers). We discuss the ten most frequently raised issues, including the use of incentives and the need for support at the start of residency, staff shortages, and requirements for learning sessions.

CONCLUSION

There is a need for a paradigm shift in CME related to alloHSCT toward a more individualized and needs-based approach. Close monitoring of residents' needs and learning progress, as well as feedback systems, could help identify appropriate CME courses that should be integrated into a tiered learning system. CME should be more targeted to specific audiences (i.e., residents, fellows, and attendees) to provide training that is tailored to individual CME needs. On-demand courses can help balance work and family obligations. Finally, peer-reviewed, up-to-date information platforms should be expanded.

Specialist training: workplace-based assessments impact on teaching, learning and feedback to support competency-based postgraduate programs

BACKGROUND

Workplace-based assessments (WBAs) are part of a competency-based curriculum where training progression is dependent on the achievement of defined competencies in a real-world clinical environment. There is a significant literature gap on the impact of WBAs implemented in resource constrained countries and their contextual challenges. This study aimed to examine the use, impact, and educational context of WBAs in South African medical specialist training programs drawing on perspectives from both trainees and trainers to identify educational challenges and propose effective solutions.

METHODS

A mixed methods national electronic survey was conducted with specialist medical trainees and supervising trainers from all eight specialist training institutions in South Africa involving 16 specialities. The survey responses were voluntary and anonymous. The survey was closed after seven months when data saturation was achieved. Descriptive statistical analysis was performed using SPSS Version 27 (SPSS Inc, 2012, Chicago, IL) for the quantitative analysis. The thematic coding framework for the qualitative analysis was facilitated by NVivo Version 12 software.

RESULTS

There were 108 ethnically diverse supervising trainers and 248 specialist trainees' survey respondents. Across the 16 medical specialities, 45% of the respondents were using WBAs. Despite contextual resource and staff challenges, this study found that WBAs had a positive impact to Kirkpatrick level 2 in providing actionable feedback to improve competency. WBA users had a significantly higher rating for trainee supervision (p < 0.01), general quality of feedback on trainee competence (< 0.01) and the specialist training program (p = 0.03) compared to WBA non-users. They also had a higher rating for the assessment of the trainee as a professional (p < 0.01); scholar (p < 0.01); communicator (p < 0.01); collaborator (p = 0.001) and leader/manager (p < 0.001) based on the AfriMEDS competency framework. Racism, sexism and favouritism were challenges that negatively affected the training programs.

CONCLUSION

Overall, this study reports that the use of WBAs had a substantially favourable impact on teaching, learning, feedback and supports a competency-based approach to specialist training programs. Addressing the contextual concerns that negatively impact training; training the trainees and trainers about their relationship, roles and responsibilities; and focusing on a trainee-centred, inclusive and empowering teaching approach will help further enhance its effectiveness.

Modelling the rate of trainees transitioning to Fellowship before achieving competence under the RANZCP's Alternative Assessment Pathway to the Objective Structured Clinical Examination

OBJECTIVE

The Objective Structured Clinical Examination (OSCE) has been removed from the Royal Australian and New Zealand College of Psychiatrists' (RANZCP) training pathway. This decision occurred in the context of an Alternative Assessment Pathway (AAP) necessitated by Covid-19, justified by logistical, methodological and equity concerns. The false positive rate of trainees progressing to Fellowship before achieving competence is a key indicator for evaluating any assessment leading to psychiatric Fellowship. Variations in the statistical properties of the AAP and OSCE were analysed for their impact on pre-competent trainees progressing to Fellowship.

METHOD

Starting with the false positive scenario presented to justify discontinuing the OSCE, false positive rates associated with the AAP and OSCE were calculated based on different assumptions about reliability and accuracy.

RESULTS

The analyses suggest that less reliable and less accurate alternatives to the OSCE, such as the AAP, increase the number of pre-competent trainees progressing to Fellowship.

CONCLUSIONS

Given possible increases in pre-competent trainees progressing to Fellowship while alternatives to the OSCE are finalised, confidence in the RANZCP's training program demands robust public analyses of those alternatives.

Workplace-based assessments-Articulating the playbook

INTRODUCTION

A workplace-based assessment (WBA) is a learning recording device that is widely used in medical education globally. Although entrenched in medical curricula, and despite a substantial body of literature exploring them, it is not yet fully understood how WBAs play out in practice. Adopting a constructivist standpoint, we examine these assessments, in the workplace, using principles based upon naturalist inquiry, drawing from a theoretical framework based on Goffman's dramaturgical analogy for the presentation of self, and using qualitative research methods to articulate what is happening as learners complete them.

METHODS

Learners were voluntarily recruited to participate in the study from a single teaching hospital. Data were generated, in-situ, through observations with field notes and audiovisual recording of WBAs, along with accompanying interviews with learners.

RESULTS

Data from six learners was analysed to reveal a set of general principles-the WBA playbook. These four principles were tacit, unwritten, unofficial and learners applied them to complete their WBA proformas: (1) maintain the impression of progression, (2) manage the authenticity of the individual proforma, (3) avoid losing face with the assessor and (4) complete the proforma in an effort-efficient way. By adhering to these principles, learners expressed their understanding of their social position in their world at that time the documents were created.

DISCUSSION

This paper recognises the value of the WBA as a lived experience, and of the WBA document as a social space, where learners engage in a social performance before the readers of the proforma. Such an interpretation better represents what happens as learners undergo and record WBAs in the real-world, recognising WBAs as learner-centred, learner-driven, meaning-making phenomena. In this way, as a record of interpretation and meanings, the subjective nature of the WBA process is a strength to be harnessed, rather than a weakness to be glossed over.

Exam prediction and the general Practice Registrar Competency Assessment Grid (GPR-CAG)

BACKGROUND

In GP training, identifying early predictors of poor summative examination performance can be challenging. We aimed to establish whether external clinical teaching visit (ECTV) performance, measured using a validated instrument (GP Registrar Competency Assessment Grid, GPR-CAG) is predictive of Royal Australian College of General Practitioners (RACGP) Fellowship examination performance.

METHODS

A retrospective cohort study including GP registrars in New South Wales/Australian Capital Territory with ECTV data recorded during their first training term (GPT1), between 2014 and 2018, who attempted at least one Fellowship examination. Independent variables of interest included the four GPR-CAG factors assessed in GPT1 ('patient-centredness/caring', 'formulating hypotheses/management plans', 'professional responsibilities', 'physical examination skills'). Outcomes of interest included individual scores of the three summative examinations (Applied Knowledge Test (AKT); Key Feature Problem (KFP); and the Objective Structured Clinical Examination (OSCE)) and overall Pass/Fail status. Univariable and multivariable regression analyses were performed.

RESULTS

Univariably, there were statistically significant associations (p < 0.01) between all four GPR-CAG factors and all four summative examination outcomes, except for 'formulating hypotheses/management plans' and OSCE score (p = 0.07). On multivariable analysis, each factor was significantly associated (p < 0.05) with at least one exam outcome, and 'physical examination skills' was significantly associated (p < 0.05) with all four exam outcomes.

DISCUSSION

ECTV performance, via GPR-CAG scores, is predictive of RACGP Fellowship exam performance. The univariable findings highlight the pragmatic utility of ECTVs in flagging registrars who are at-risk of poor exam performance, facilitating early intervention. The multivariable associations of GPR-CAG scores and examination performance suggest that these scores provide predictive ability beyond that of other known predictors.

The Use, Effectiveness, and Impact of Workplace-Based Assessments on Teaching, Supervision and Feedback Across Surgical Specialties

OBJECTIVE

To investigate the use and effectiveness of Workplace-based assessments (WBAs) and their impact on training, feedback, and perioperative teaching in surgical training programs.

DESIGN

A mixed methods cross-sectional, national electronic survey was conducted with surgical trainees and consultant trainers.

SETTINGS

The trainees and supervising faculty were from all 8 major surgical training universities across 11 surgical disciplines in South Africa.

PARTICIPANTS

A total of 108 surgical trainees and 41 supervising consultant trainers from 11 surgical disciplines across 8 surgical training universities responded to the survey.

RESULTS

The most significant educational gap identified by both the surgical trainees and trainers across all surgical disciplines was inadequate perioperative feedback. A third of the respondents were currently using workplace-based assessments. The WBA users (both trainees and trainers) had a higher rating for the general quality of surgical feedback than WBA nonusers (p = 0.02). WBA users also had a higher rating for the general quality of feedback given to trainees on their skills and competence (p = 0.04) and a higher rating for trainee supervision (p = 0.01) and the specialist training program overall (p = 0.01). The WBA users also had a higher rating for the assessment of competencies such as the trainee as an effective communicator (p < 0.01) and collaborator (p = 0.04).

CONCLUSION

This study found that the use of WBAs enhances the quality and effectiveness of feedback in surgical training programs. We also found that the use of WBAs enhance perioperative teaching and learning and improves the assessment of relational competencies. This was also associated with high ratings for the quality of trainee supervision. Faculty and trainee development, strengthening the trainee-trainer relationship, and integrating iterative stakeholder feedback could help realize the full potential of WBAs to augment surgical training across disciplines.

Assessment-Seeking Strategies: Navigating the Decision to Initiate Workplace-Based Assessment

Phenomenon: Competency-based medical education (CBME) relies on workplace-based assessment (WBA) to generate formative feedback (assessment for learning-AfL) and make inferences about competence (assessment of learning-AoL). When approaches to CBME rely on residents to initiate WBA, learners experience tension between seeking WBA for learning and for establishing competence. How learners resolve this tension may lead to unintended consequences for both AfL and AoL. We sought to explore the factors that impact both decisions to seek and not to seek WBA and use the findings to build a model of assessment-seeking strategy used by residents. In building this model we consider how the link between WBA and promotion or progression within a program impacts an individual's assessment-seeking strategy. Approach: We conducted 20 semi-structured interviews with internal medicine residents at Queen's University about the factors that influence their decision to seek or avoid WBA. Using grounded theory methodology, we applied a constant comparative analysis to collect data iteratively and identify themes. A conceptual model was developed to describe the interaction of factors impacting the decision to seek and initiate WBA. Findings: Participants identified two main motivations when deciding to seek assessments: the need to fulfill program requirements and the desire to receive feedback for learning. Analysis suggested that these motivations are often at odds with each other. Participants also described several moderating factors that impact the decision to initiate assessments, irrespective of the primary underlying motivation. These included resident performance, assessor factors, training program expectations, and clinical context. A conceptual framework was developed to describe the factors that lead to strategic assessment-seeking behaviors. Insights: Faced with the dual purpose of WBA in CBME, resident behavior in initiating assessment is guided by specific assessment-seeking strategies. Strategies reflect individual underlying motivations, influenced by four moderating factors. These findings have broad implications for programmatic assessment in a CBME context including validity considerations for assessment data used in summative decision-making including readiness for unsupervised practice.

Grading practice as a strategy to improve proficiencies in undergraduate nurse education: Modelling key areas of competence

BACKGROUND

In undergraduate nursing grading practice is generally avoided as it is considered educationally flawed.

OBJECTIVES

To test an innovative online grading practice tool (GPT) in undergraduate nurse education. To model the determinants of the final practice grade in four areas of clinical competence and in one cohort analysis the relationship between final practice grade and each area of clinical competence and an OSCE grade.

DESIGN

A cross-sectional study.

PARTICIPANTS

A convenience sample of 782 nursing students from one Higher Education Institution in the North-East of England were included. The sample involved two sequential cohorts of final-year students with 391 students in each cohort.

METHODS

A specifically designed online grading practice tool (GPT) composed of thirty-six objectives equally divided across four areas of clinical competence. The GPT was applied to two consecutive student cohorts on completion of their final practice learning placement.

RESULTS

There was a statistically significant difference in the mean final practice grade between the two cohorts. In the overall sample, regression modelling showed that all four areas of student assessment contributed equally to the final grade. Analysis by cohort showed that in Cohort 1 clinical thinking and professionalism had the most influence on the final grade with person-centered care and patient safety most strongly impacting on the final grades of Cohort 2. In Cohort 2 there is no statistically significant correlation between final practice grade, each area of clinical competence and an OSCE grade.

CONCLUSIONS

Practice learning is fundamental to how students develop professional awareness and learn to nurse. Findings from a novel grading practice tool applied in undergraduate nursing reveal how effectively the tool works. Nurse educators must be responsive to the realities of learning in practice and explore new ways of assessing clinical competence.

How workplace-based assessments guide learning in postgraduate education: A scoping review

INTRODUCTION

Competency-based medical education (CBME) led to the widespread adoption of workplace-based assessment (WBA) with the promise of achieving assessment for learning. Despite this, studies have illustrated tensions between the summative and formative role of WBA which undermine learning goals. Models of workplace-based learning (WBL) provide insight, however, these models excluded WBA. This scoping review synthesizes the primary literature addressing the role of WBA to guide learning in postgraduate medical education, with the goal of identifying gaps to address in future studies.

METHODS

The search was applied to OVID Medline, Web of Science, ERIC and CINAHL databases, articles up to September 2020 were included. Titles and abstracts were screened by two reviewers, followed by a full text review. Two members independently extracted and analysed quantitative and qualitative data using a descriptive-analytic technique rooted in Billett's four premises of WBL. Themes were synthesized and discussed until consensus.

RESULTS

All 33 papers focused on the perception of learning through WBA. The majority applied qualitative methodology (70%), and 12 studies (36%) made explicit reference to theory. Aligning with Billett's first premise, results reinforce that learning always occurs in the workplace. WBA helped guide learning goals and enhanced feedback frequency and specificity. Billett's remaining premises provided an important lens to understand how tensions that existed in WBL have been exacerbated with frequent WBA. As individuals engage in both work and WBA, they are slowly transforming the workplace. Culture and context frame individual experiences and the perceived authenticity of WBA. Finally, individuals will have different goals, and learn different things, from the same experience.

CONCLUSION

Analysing WBA literature through the lens of WBL theory allows us to reframe previously described tensions. We propose that future studies attend to learning theory, and demonstrate alignment with philosophical position, to advance our understanding of assessment-for-learning in the workplace.

'Ticked off'? Can a new outcomes-based postgraduate curriculum utilising programmatic assessment reduce assessment burden in Intensive Care Medicine?

Context

Increasing dissatisfaction with existing methods of assessment in the workplace alongside a national drive towards outcomes-based postgraduate curricula led to a recent overhaul of the way Intensive Care Medicine (ICM) trainees are assessed in the United Kingdom. Programmatic assessment methodology was utilised; the existing 'tick-box' approach using workplace-based assessment to demonstrate competencies was de-emphasised and the expertise of trainers used to assess capability relating to fewer, high-level outcomes related to distinct areas of specialist practice.

Methods

A thematic analysis was undertaken investigating attitudes from 125 key stakeholders, including trainees and trainers, towards the new assessment strategy in relation to impact on assessment burden and acceptability.

Results

This qualitative study suggests increased satisfaction with the transition to an outcomes-based model with capability judged by educational supervisors. However, reflecting frustration relating to current assessment in the workplace, participants felt assessment burden has been significantly reduced. The approach taken was felt to be an improved method for assessing professional practice; there was enthusiasm for this change. However, this research highlights trainee and trainer anxiety regarding how to 'pass' these expert judgement decisions of capability in the real world. Additionally, concerns relating to the impact on subgroups of trainees due to the potential influence of implicit biases on the resultant fewer but 'higher stakes' interrogative judgements became apparent.

Conclusion

The move further towards a constructivist paradigm in workplace assessment in ICM reduces assessment burden yet can provoke anxiety amongst trainees and trainers requiring considered implementation. Furthermore, the perception of potential for bias in global judgements of performance requires further exploration.

Measuring changes in nursing students' workplace performance following feedback encounters: A quasi-experimental study

BACKGROUND

Work-integrated learning and performance is intimately linked with assessment and feedback; however, empirical evidence concerning the impact of feedback on student performance in workplace settings is limited. A novel practice-based feedback intervention-the REMARK (nuRse fEedback iMplementAtion frameworRK) program-was developed and implemented. Sustainable feedback processes that involve both learners and learning partners can ultimately maximise learning outcomes, optimise self-regulation, and impact performance.

OBJECTIVES

To determine the impact of implementing an educational intervention based on best practice principles of feedback on undergraduate nursing student performance during clinical placements in acute healthcare settings.

METHODS

A non-equivalent, quasi-experimental design was used. A pre- and post-assessment strategy for both control and intervention groups was employed. The REMARK program was based on known attributes of effective feedback critiqued in the literature that foster interactions between a learner and learning partner and, hence, support dialogic feedback.

RESULTS

Clinical assessment data of 214 final-year nursing students collected at three time points were analysed. Using multivariate modelling, the results indicated that students participating in the REMARK programme (intervention group) had statistically higher performance scores than students in the control group when time and placement setting were controlled.

CONCLUSIONS

This research provides evidence that engagement with an intervention based on attributes of effective feedback improves nursing students' workplace performance during clinical placement. To achieve the greatest advantage, incorporating strategies that encourage students to adopt an active role in feedback conversations that relate to the student's learning goals are recommended.

Beyond Checking Boxes: Exploring Tensions With Use of a Workplace-Based Assessment Tool for Formative Assessment in Clerkships

PURPOSE

To understand the role of a workplace-based assessment (WBA) tool in facilitating feedback for medical students, this study explored changes and tensions in a clerkship feedback activity system through the lens of cultural historical activity theory (CHAT) over 2 years of tool implementation.

METHOD

This qualitative study uses CHAT to explore WBA use in core clerkships by identifying feedback activity system elements (e.g., community, tools, rules, objects) and tensions among these elements. University of California, San Francisco core clerkship students were invited to participate in semistructured interviews eliciting experience with a WBA tool intended to enhance direct observation and feedback in year 1 (2019) and year 2 (2020) of implementation. In year 1, the WBA tool required supervisor completion in the school's evaluation system on a computer. In year 2, both students and supervisors had WBA completion abilities and could access the form via a smartphone separate from the school's evaluation system.

RESULTS

Thirty-five students participated in interviews. The authors identified tensions that shifted with time and tool iterations. Year 1 students described tensions related to cumbersome tool design, fear of burdening supervisors, confusion over WBA purpose, WBA as checking boxes, and WBA usefulness depending on clerkship context and culture. Students perceived dissatisfaction with the year 1 tool version among peers and supervisors. The year 2 mobile-based tool and student completion capabilities helped to reduce many of the tensions noted in year 1. Students expressed wider WBA acceptance among peers and supervisors in year 2 and reported understanding WBA to be for low-stakes feedback, thereby supporting formative assessment for learning.

CONCLUSIONS

Using CHAT to explore changes in a feedback activity system with WBA tool iterations revealed elements important to WBA implementation, including designing technology for tool efficiency and affording students autonomy to document feedback with WBAs.

The WFME global standards for quality improvement of postgraduate medical education: Which standards are also applicable in Germany? Recommendations for physicians with a license for postgraduate training and training agents

Background: In Germany, the (model) regulation for postgraduate medical education 2018, the professional codes of conduct of the regional medical councils and the health professions chamber laws of the federal states are the formal basis of postgraduate medical education, but say little about its structure, processes and results. The World Federation for Medical Education (WFME) has developed global standards for improving the quality of postgraduate medical education and published them in a revised edition in 2015. A German version which takes the specifics of medical training in Germany into account has not been published to date. Objective: The Committee for Postgraduate Medical Education (PGME) of the Society for Medical Education (GMA) has set itself the goal of firstly translating the WFME standards into German and secondly making recommendations for physicians with a license for post-graduate training (PLT) and training agents (TA) in clinics and practices which have been adapted to the German context. Methods: The WFME standards were translated into German by a working group of the GMA Committee for PGME, the terminology adapted to PGME in Germany and checked by an interdisciplinary panel of experts made up of 9 members of the committee. In a second step, the WFME basic standards and quality standards for PGME relevant to PLTs and TAs in Germany were iteratively determined by this panel of experts using the Nominal Group Technique (NGT) and compiled in the form of recommendations. Results: The translation of the WFME guidelines was approved by the expert group without any changes to the content, taking into account the terminological system of PGME in Germany. In a second step, 90 standards were identified which were considered helpful for PGME in Germany, especially for PLTs and TAs (such as development of a professional identity, a more patient-centered approach or support of self-directed learning). Care was taken to only give recommendations which can be influenced by PLTs and TAs. These standards have been summarized as recommendations to PLTs and TAs and take into account all chapters of the WFME standards. Conclusion: The WFME standards selected here are recommended to PLTs and TAs in clinics and practices to achieve high-quality PGME. Empirical longitudinal studies will be required to examine both the implementation and the results of applying the modified WFME criteria in Germany.

“Could You Work in My Team?”: Exploring How Professional Clinical Role Expectations Influence Decision-Making of Assessors During Exit-Level Medical School OSCEs

Decision-making in clinical assessment, such as exit-level medical school Objective Structured Clinical Examinations (OSCEs), is complex. This study utilized an empirical phenomenological qualitative approach with thematic analysis to explore OSCE assessors' perceptions of the concept of a “prototypical intern” expressed during focus group discussions. Topics discussed included the concept of a prototypical intern, qualities to be assessed, and approaches to clinical assessment decision-making. The thematic analysis was then applied to a theoretical framework (Cultural Historical Activity Theory—CHAT) that explored the complexity of making assessment decisions amidst potentially contradicting pressures from academic and clinical perspectives. Ten Australasian medical schools were involved with 15 experienced and five less experienced assessors participating. Thematic analysis of the data revealed four major themes in relation to how the prototypical intern concept influences clinical assessors' judgements: (a) Suitability of marking rubric based on assessor characteristics and expectations; (b) Competence as final year student vs. performance as a prototypical intern; (c) Safety, trustworthiness and reliability as constructs requiring assessment and (d) Contradictions in decision making process due to assessor differences. These themes mapped well within the interaction between two proposed activity systems in the CHAT model: academic and clinical. More clinically engaged and more experienced assessors tend to fall back on a heuristic, mental construct of a “prototypical intern,” to calibrate judgements, particularly, in difficult situations. Further research is needed to explore whether consensus on desirable intern qualities and their inclusion into OSCE marksheets decreases the cognitive load and increases the validity of assessor decision making.

The shift from disbelieving underperformance to recognising failure: A tipping point model

CONTEXT

Coming face to face with a trainee who needs to be failed is a stern test for many supervisors. In response, supervisors have been encouraged to report evidence of failure through numerous assessment redesigns. And yet, there are lingering signs that some remain reluctant to engage in assessment processes that could alter a trainee's progression in the programme. Failure is highly consequential for all involved and, although rare, requires explicit study. Recent work identified a phase of disbelief that preceded identification of underperformance. What remains unknown is how supervisors come to recognise that a trainee needs to be failed.

METHODS

Following constructivist grounded theory methodology, 42 physicians and surgeons in British Columbia, Canada shared their experiences supervising trainees who profoundly underperformed, required extensive remediation or were dismissed from the programme. We identified recurring themes using an iterative, constant comparative process.

RESULTS

The shift from disbelieving underperformance to recognising failure involves three patterns: accumulation of significant incidents, discovery of an egregious error after negligible deficits or illumination of an overlooked deficit when pointed out by someone else. Recognising failure was accompanied by anger, certainty and a sense of duty to prevent harm.

CONCLUSION

Coming to the point of recognising that a trainee needs to fail is akin to the psychological process of a tipping point where people first realise that noise is signal and cross a threshold where the pattern is no longer an anomaly. The co-occurrence of anger raises the possibility for emotions to be a driver of, and not only a barrier to, recognising failure. This warrants caution because tipping points, and anger, can impede detection of improvement. Our findings point towards possibilities for supporting earlier identification of underperformance and overcoming reluctance to report failure along with countermeasures to compensate for difficulties in detecting improvement once failure has been verified.

Perceptions of postgraduate family medicine supervision at decentralised training sites, South Africa

Background

Specialist training in family medicine (FM) is growing rapidly in sub-Saharan Africa. The strong emphasis on workplace-based learning for speciality training makes it vital to gain in-depth insights into registrar supervision. Previous studies have explored aspects of supervision at decentralised sites in high-income countries, however, little is known about the benefits and constraints of decentralised postgraduate supervision in low- to middle-income countries, especially in Africa.

Aim

This study aimed to explore family physicians’ and registrars’ perceptions of the strengths and challenges of clinical and educational supervision across decentralised training sites.

Setting

The study was conducted across two provinces at five decentralised training sites affiliated with the University of the Witwatersrand, Johannesburg.

Methods

This qualitative study involved semi-structured interviews with a purposive sample of 11 FPs and 11 registrars. The data were thematically analysed.

Results

Two of the four themes identified, ‘supervision is context-specific and supervisor-dependent’, and ‘the nature of engagement matters’, involved strengths and challenges. The other two, ‘supervision is not ideal’ and ‘the training environment is challenging’, focussed on challenges.

Conclusion

Supervisors and registrars described the postgraduate FM supervision as context-specific and supervisor-dependent. Supervisors displayed good clinical-teacher characteristics and supervisory relationships. However, several challenges, including registrars’ workload, resource shortages and a lack of standardisation across training sites, need to be addressed. Regular faculty development is essential for supervisors to be aware of relevant aspects of, and current trends in, postgraduate training.

Wellbeing Conceptualizations in Family Medicine Trainees: A Hermeneutic Review

PHENOMENON

High levels of burnout have been widely reported among postgraduate medical trainees, however relatively little literature has examined what 'wellbeing' means for this group. Moreover, the literature that does exist has generally overlooked the potential role of specialty factors in influencing such conceptualizations. This is particularly true for family medicine and general practice trainees - a specialty considered to be unique due, in part, to its focus on community-based care. The present review sought to explore conceptualizations of wellbeing specifically within the context of family medicine and general practice training.

APPROACH

The Embase, Ovid Medline, and PsycINFO databases were searched from inception to November 2019 for literature examining wellbeing in family medicine and general practice trainees. Literature was iteratively thematically analyzed through the process of a hermeneutic cycle. In total, 36 articles were reviewed over seven rounds, at which point saturation was reached.

FINDINGS

The findings confirm the complex and multifaceted nature of wellbeing as experienced by family medicine and general practice trainees. An emphasis on psychological factors - including emotional intelligence, positive mental health, self-confidence and resilience - alongside positive interpersonal relationships, rewards, and balanced interactions between trainees' personal and professional demands were deemed critical elements.

INSIGHTS

A model of wellbeing that emphasizes rich connections between trainees' personal and professional life domains is proposed. Further qualitative research will help to extend current understanding of wellbeing among medical trainees, including the individuality of each specialty's experiences, with the potential to enhance interventional efforts.

A Document Analysis of Nationally Available Faculty Assessment Forms of Resident Performance

BACKGROUND

Written feedback by faculty of resident performance is valuable when it includes components based on assessment for learning. However, it is not clear how often assessment forms include these components for summative and formative feedback.

OBJECTIVE

To analyze prompts used in forms for faculty assessment of resident performance, guided by best practices in survey research methodology, self-regulation theory, and competency-based assessment.

METHODS

A document analysis, which is a qualitative approach used to analyze content and structure of texts, was completed on assessment forms nationally available in MedHub. Due to the number of forms available, only internal medicine and surgery specialties were included. A document summary form was created to analyze the assessments. The summary form guided researchers through the analysis.

RESULTS

Forty-eight forms were reviewed, each from a unique residency program. All forms provided a textbox for comments, and 54% made this textbox required for assessment completion. Eighty-three percent of assessments placed the open textbox at the end of the form. One-third of forms contained a simple prompt, "Comments," for the narrative section. Fifteen percent of forms included a box to check if the information on the form had been discussed with the resident. Fifty percent of the assessments were unclear if they were meant to be formative or summative in nature.

CONCLUSIONS

Our document analysis of assessment forms revealed they do not always follow best practices in survey design for narrative sections, nor do they universally address elements deemed important for promotion of self-regulation and competency-based assessment.

Ottawa 2020 consensus statement for programmatic assessment - 1. Agreement on the principles

INTRODUCTION

In the Ottawa 2018 Consensus framework for good assessment, a set of criteria was presented for systems of assessment. Currently, programmatic assessment is being established in an increasing number of programmes. In this Ottawa 2020 consensus statement for programmatic assessment insights from practice and research are used to define the principles of programmatic assessment.

METHODS

For fifteen programmes in health professions education affiliated with members of an expert group (n = 20), an inventory was completed for the perceived components, rationale, and importance of a programmatic assessment design. Input from attendees of a programmatic assessment workshop and symposium at the 2020 Ottawa conference was included. The outcome is discussed in concurrence with current theory and research.

RESULTS AND DISCUSSION

Twelve principles are presented that are considered as important and recognisable facets of programmatic assessment. Overall these principles were used in the curriculum and assessment design, albeit with a range of approaches and rigor, suggesting that programmatic assessment is an achievable education and assessment model, embedded both in practice and research. Knowledge on and sharing how programmatic assessment is being operationalized may help support educators charting their own implementation journey of programmatic assessment in their respective programmes.

Identifying the at-risk General Practice trainee: a retrospective cohort meta-analysis of General Practice registrar flagging

A central principle of programmatic assessment is that the final decision is not a surprise to the learner. To achieve this, assessments must demonstrate predictive and consequential validity, however, to date, research has only focussed on the former. The present study attempts to address this gap by examining the predictive and consequential validity of flagging systems used by Australian General Practice regional training organisations (RTOs) in relation to Fellowship examinations. Informed by unstructured interviews with Senior Medical Educators to understand the flagging system of each RTO, meta-analyses of routinely-collected flagging data were used to examine the predictive validity of flagging at various points in training and exam performance. Additionally, flagging system features identified from the interviews were used to inform exploratory subgroup analyses and meta-regressions to further assess the predictive and consequential validity of these systems. Registrars flagged near the end of their training were two to four times more likely to fail Fellowship exams than their non-flagged counterparts. Regarding flagging system features, having graded (i.e. ordinal) flagging systems was associated with higher accuracy, whilst involving the assigned medical educator in remediation and initiating a formal diagnostic procedure following a flag improved registrars' chances of passing exams. These results demonstrate both predictive and consequential validity of flagging systems. We argue that flagging is most effective when initiated early in training in conjunction with mechanisms to maximise diagnostic accuracy and the quality of remediation programs.

Numbers Encapsulate, Words Elaborate: Toward the Best Use of Comments for Assessment and Feedback on Entrustment Ratings

The adoption of entrustment ratings in medical education is based on a seemingly simple premise: to align workplace-based supervision with resident assessment. Yet it has been difficult to operationalize this concept. Entrustment rating forms combine numeric scales with comments and are embedded in a programmatic assessment framework, which encourages the collection of a large quantity of data. The implicit assumption that more is better has led to an untamable volume of data that competency committees must grapple with. In this article, the authors explore the roles of numbers and words on entrustment rating forms, focusing on the intended and optimal use(s) of each, with a focus on the words. They also unpack the problematic issue of dual-purposing words for both assessment and feedback. Words have enormous potential to elaborate, to contextualize, and to instruct; to realize this potential, educators must be crystal clear about their use. The authors set forth a number of possible ways to reconcile these tensions by more explicitly aligning words to purpose. For example, educators could focus written comments solely on assessment; create assessment encounters distinct from feedback encounters; or use different words collected from the same encounter to serve distinct feedback and assessment purposes. Finally, the authors address the tyranny of documentation created by programmatic assessment and urge caution in yielding to the temptation to reduce words to numbers to make them manageable. Instead, they encourage educators to preserve some educational encounters purely for feedback, and to consider that not all words need to become data.

[Competence-oriented specialist training in dermatology]

The new M WBO (Musterweiterbildungsordnung) has been developed for 6 years and will be put into force by most LÄK (Landesärztekammern) on 01.07.2020. Future training to become a dermatologist is competency-based. This is to enable observable, successful problem solving in practice. Thus, the acquisition of competence is only dependent on whether the goal has been achieved. New procedures are required to determine achievement of the respective competencies, including annual continuing education interviews, workplace-based examinations, and an eLogbuch (electronic logbook). Minimum inpatient periods will be eliminated in the future. In principle, continuing education can take place entirely on an outpatient basis. The M WBO Dermatology comprises 14 thematic blocks with differentiated description of cognitive and methodological or action competencies. The guideline numbers of dermatological additional training courses have been shifted considerably into the period of basic training. Additional training courses can be acquired while working. Fachlich empfohlene Weiterbildungspläne (FEWP) are the concrete implementation regulations of the M WBO. They are not part of the WBO and can be adapted. In connection with this new approach, numerous questions are currently still open, such as documentation in practice or financing.

Workplace-based assessments of entrustable professional activities in a psychiatry core clerkship: an observational study

BACKGROUND

Entrustable professional activities (EPAs) in competency-based, undergraduate medical education (UME) have led to new formative workplace-based assessments (WBA) using entrustment-supervision scales in clerkships. We conducted an observational, prospective cohort study to explore the usefulness of a WBA designed to assess core EPAs in a psychiatry clerkship.

METHODS

We analyzed changes in self-entrustment ratings of students and the supervisors' ratings per EPA. Timing and frequencies of learner-initiated WBAs based on a prospective entrustment-supervision scale and resultant narrative feedback were analyzed quantitatively and qualitatively. Predictors for indirect supervision levels were explored via regression analysis, and narrative feedback was coded using thematic content analysis. Students evaluated the WBA after each clerkship rotation.

RESULTS

EPA 1 ("Take a patient's history"), EPA 2 ("Assess physical & mental status") and EPA 8 ("Document & present a clinical encounter") were most frequently used for learner-initiated WBAs throughout the clerkship rotations in a sample of 83 students. Clinical residents signed off on the majority of the WBAs (71%). EPAs 1, 2, and 8 showed the largest increases in self-entrustment and received most of the indirect supervision level ratings. We found a moderate, positive correlation between self-entrusted supervision levels at the end of the clerkship and the number of documented entrustment-supervision ratings per EPA (p < 0.0001). The number of entrustment ratings explained 6.5% of the variance in the supervisors' ratings for EPA 1. Narrative feedback was documented for 79% (n = 214) of the WBAs. Most narratives addressed the Medical Expert role (77%, n = 208) and used reinforcement (59%, n = 161) as a feedback strategy. Students perceived the feedback as beneficial.

CONCLUSIONS

Using formative WBAs with an entrustment-supervision scale and prompts for written feedback facilitated targeted, high-quality feedback and effectively supported students' development toward self-entrusted, indirect supervision levels.