Perceived effects of innovations in postgraduate medical education: a Q study focusing on workplace-based assessment

Patterns of Medical Residents' Preferences for Organizational Socialization Strategies to Facilitate Their Transitions: A Q-study

Introduction

To facilitate various transitions of medical residents, healthcare team members and departments may employ various organizational socialization strategies, including formal and informal onboarding methods. However, residents' preferences for these organizational socialization strategies to ease their transition can vary. This study identifies patterns (viewpoints) in these preferences.

Methods

Using Q-methodology, we asked a purposeful sample of early-career residents to rank a set of statements into a quasi-normal distributed grid. Statements were based on previous qualitative interviews and organizational socialization theory. Participants responded to the question, 'What are your preferences regarding strategies other health care professionals, departments, or hospitals should use to optimize your next transition?' Participants then explained their sorting choices in a post-sort questionnaire. We identified different viewpoints based on by-person (inverted) factor analysis and Varimax rotation. We interpreted the viewpoints using distinguishing and consensus statements, enriched by residents' comments.

Results

Fifty-one residents ranked 42 statements, among whom 36 residents displayed four distinct viewpoints: Dependent residents (n = 10) favored a task-oriented approach, clear guidance, and formal colleague relationships; Social Capitalizing residents (n = 9) preferred structure in the onboarding period and informal workplace social interactions; Autonomous residents (n = 12) prioritized a loosely structured onboarding period, independence, responsibility, and informal social interactions; and Development-oriented residents (n = 5) desired a balanced onboarding period that allowed independence, exploration, and development.

Discussion

This identification of four viewpoints highlights the inadequacy of one-size-fits-all approaches to resident transition. Healthcare professionals and departments should tailor their socialization strategies to residents' preferences for support, structure, and formal/informal social interaction.

Evaluating the perceptions of workplace-based assessments in surgical training: a systematic review

INTRODUCTION

Workplace-based assessments (WBAs) are intended to maximise learning opportunities in surgical training. There is speculation as to whether mandatory assessments in this form contribute to a tick-box culture. The objective of this review was to investigate surgical trainees' attitudes towards WBAs.

METHODS

This systematic review of qualitative studies was performed in accordance with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement. The literature was searched on the Medline®, Embase™, PubMed and Web of Science™ databases on 22 March 2022.

RESULTS

Sixteen studies were included in the review, mostly carried out on users of the Intercollegiate Surgical Curriculum Programme portfolio in the UK. Trainees felt that WBAs were educationally useful, providing opportunity for feedback, but this was overshadowed by a pressure to reach a set annual quota for WBAs and achieve high scores. Other themes included inaccurate recording of WBAs, the role of WBAs as formative or summative assessments, engagement and accessibility of trainers, and lack of time to complete WBAs.

CONCLUSIONS

Negative perceptions about WBAs were widespread among surgical trainees despite a recognition of their capacity to facilitate learning. This review supports the recent removal of the annual quota for WBAs in UK surgical training programmes.

Assessing electrodiagnostic skills among residents and fellows: Relationships between workplace-based assessments using the Electromyography Direct Observation Tool and other measures of trainee performance

INTRODUCTION/AIMS

Graduate medical education programs must ensure residents and fellows acquire skills needed for independent practice. Workplace-based observational assessments are informative but can be time- and resource-intensive. In this study we sought to gather "relations-to-other-variables" validity evidence for scores generated by the Electromyography Direct Observation Tool (EMG-DOT) to inform its use as a measure of electrodiagnostic skill acquisition.

METHODS

Scores on multiple assessments were compiled by trainees during Clinical Neurophysiology and Electromyography rotations at a large US academic medical center. Relationships between workplace-based EMG-DOT scores (n = 298) and scores on a prerequisite simulated patient exercise, patient experience surveys (n = 199), end-of-rotation evaluations (n = 301), and an American Association of Neuromuscular & Electrodiagnostic Medicine (AANEM) self-assessment examination were assessed using Pearson correlations.

RESULTS

Among 23 trainees, EMG-DOT scores assigned by physician raters correlated positively with end-of-rotation evaluations (r = 0.63, P = .001), but EMG-DOT scores assigned by technician raters did not (r = 0.10, P = .663). When physician and technician ratings were combined, higher EMG-DOT scores correlated with better patient experience survey scores (r = 0.42, P = .047), but not with simulated patient or AANEM self-assessment examination scores.

DISCUSSION

End-of-rotation evaluations can provide valid assessments of trainee performance when completed by individuals with ample opportunities to directly observe trainees. Inclusion of observational assessments by technicians and patients provides a more comprehensive view of trainee performance. Workplace- and classroom-based assessments provide complementary information about trainee performance, reflecting underlying differences in types of skills measured.

Q-Methodology: A Technique to Assess Perceptions in Simulation Research

SUMMARY STATEMENT

Learner perceptions of simulation can be challenging to assess and are usually limited to qualitative strategies. Q-methodology is an exploratory, systematic research technique that combines quantitative and qualitative procedures to measure differences in subjectivity. This article briefly reviews Q-methodology, provides an example of how it could be applied to assess the affective domain of simulation-based learning, and discusses implications for the use of Q-methodology in future simulation scholarship.

'One size does not fit all': The value of person-centred analysis in health professions education research

Health professions education (HPE) research is dominated by variable-centred analysis, which enables the exploration of relationships between different independent and dependent variables in a study. Although the results of such analysis are interesting, an effort to conduct a more person-centred analysis in HPE research can help us in generating a more nuanced interpretation of the data on the variables involved in teaching and learning. The added value of using person-centred analysis, next to variable-centred analysis, lies in what it can bring to the applications of the research findings in educational practice. Research findings of person-centred analysis can facilitate the development of more personalized learning or remediation pathways and customization of teaching and supervision efforts. Making the research findings more recognizable in practice can make it easier for teachers and supervisors to understand and deal with students. The aim of this article is to compare and contrast different methods that can be used for person-centred analysis and show the incremental value of such analysis in HPE research. We describe three methods for conducting person-centred analysis: cluster, latent class and Q‑sort analyses, along with their advantages and disadvantage with three concrete examples for each method from HPE research studies.

A scoping review of Q-methodology in healthcare research

BACKGROUND

Q-methodology is an approach to studying complex issues of human 'subjectivity'. Although this approach was developed in the early twentieth century, the value of Q-methodology in healthcare was not recognised until relatively recently. The aim of this review was to scope the empirical healthcare literature to examine the extent to which Q-methodology has been utilised in healthcare over time, including how it has been used and for what purposes.

METHODS

A search of three electronic databases (Scopus, EBSCO-CINAHL Complete, Medline) was conducted. No date restriction was applied. A title and abstract review, followed by a full-text review, was conducted by a team of five reviewers. Included articles were English-language, peer-reviewed journal articles that used Q-methodology (both Q-sorting and inverted factor analysis) in healthcare settings. The following data items were extracted into a purpose-designed Excel spreadsheet: study details (e.g., setting, country, year), reasons for using Q-methodology, healthcare topic area, participants (type and number), materials (e.g., ranking anchors and Q-set), methods (e.g., development of the Q-set, analysis), study results, and study implications. Data synthesis was descriptive in nature and involved frequency counting, open coding and the organisation by data items.

RESULTS

Of the 2,302 articles identified by the search, 289 studies were included in this review. We found evidence of increased use of Q-methodology in healthcare, particularly over the last 5 years. However, this research remains diffuse, spread across a large number of journals and topic areas. In a number of studies, we identified limitations in the reporting of methods, such as insufficient information on how authors derived their Q-set, what types of analyses they performed, and the amount of variance explained.

CONCLUSIONS

Although Q-methodology is increasingly being adopted in healthcare research, it still appears to be relatively novel. This review highlight commonalities in how the method has been used, areas of application, and the potential value of the approach. To facilitate reporting of Q-methodological studies, we present a checklist of details that should be included for publication.

Appreciating small-group active learning: What do medical students want, and why? A Q-methodology study

INTRODUCTION

For Small-Group Active Learning (SMAL) to be effective, students need to engage meaningfully in learning activities to construct their knowledge. Teachers can have difficulty engaging their students in this process. To improve engagement, we aimed to identify the diversity in medical students' appreciation of SMAL, using the concepts of epistemic beliefs and approaches to learning.

METHOD

Q-methodology is a mixed-method research design used for the systematic study of subjectivity. We developed a set of 54 statements on active learning methods. In individual interviews, first-year medical students rank ordered their agreement with these statements and explained their reasons. Data were analyzed using a by-person factor analysis to group participants with shared viewpoints.

RESULTS

A four-factor solution (i.e. profiles) fit the data collected from 52 students best and explained 52% of the variance. Each profile describes a shared viewpoint on SMAL. We characterized the profiles as 'understanding-oriented', 'assessment-oriented', 'group-oriented', and 'practice-oriented'.

DISCUSSION

The four profiles describe how and why students differ in their appreciation of SMAL. Teachers can use the profiles to make better-informed decisions when designing and teaching their SMAL classes, by relating to students' epistemic beliefs, and approaches to learning. This may improve student motivation and engagement for SMAL.

Workplace-based assessments in postgraduate medical education: A hermeneutic review

OBJECTIVES

Since their introduction, workplace-based assessments (WBAs) have proliferated throughout postgraduate medical education. Previous reviews have identified mixed findings regarding WBAs' effectiveness, but have not considered the importance of user-tool-context interactions. The present review was conducted to address this gap by generating a thematic overview of factors important to the acceptability, effectiveness and utility of WBAs in postgraduate medical education.

METHOD

This review utilised a hermeneutic cycle for analysis of the literature. Four databases were searched to identify articles pertaining to WBAs in postgraduate medical education from the United Kingdom, Canada, Australia, New Zealand, the Netherlands and Scandinavian countries. Over the course of three rounds, 30 published articles were thematically analysed in an iterative fashion to deeply engage with the literature in order to answer three scoping questions concerning acceptability, effectiveness and assessment training. As each round was coded, themes were refined and questions added until saturation was reached.

RESULTS

Stakeholders value WBAs for permitting assessment of trainees' performance in an authentic context. Negative perceptions of WBAs stem from misuse due to low assessment literacy, disagreement with definitions and frameworks, and inadequate summative use of WBAs. Effectiveness is influenced by user (eg, engagement and assessment literacy) and tool attributes (eg, definitions and scales), but most fundamentally by user-tool-context interactions, particularly trainee-assessor relationships. Assessors' assessment literacy must be combined with cultural and administrative factors in organisations and the broader medical discipline.

CONCLUSIONS

The pivotal determinants of WBAs' effectiveness and utility are the user-tool-context interactions. From the identified themes, we present 12 lessons learned regarding users, tools and contexts to maximise WBA utility, including the separation of formative and summative WBA assessors, use of maximally useful scales, and instituting measures to reduce competitive demands.

Q sample construction: a novel approach incorporating a Delphi technique to explore opinions about codeine dependence

BACKGROUND

Q methodology is an evidenced approach to researching subjectivity, involving a combination of qualitative and quantitative techniques. The methodology has been used successfully in healthcare research to explore the opinions of patients and healthcare providers about topics such as the illness experience, healthcare services, clinical practice and professional training. Q methodology studies require the generation of a Q sample, a set of opinion statements representing the phenomenon of interest. This paper describes a novel and rigorous approach to develop a Q sample for a study exploring misusers' opinions about over-the-counter (OTC) codeine dependence and critically examines the associated methodological issues.

METHODS

Development of the Q sample in this study involved three steps; (1) identification of opinion statements via a comprehensive literature search, (2) application of a theoretical framework, the Capability, Opportunity, Motivation - Behaviour (COM-B) model of behaviour, to group and then reduce the number of statements and (3) use of a Delphi technique to achieve expert consensus on the final selection of statements. The Delphi component involved a multidisciplinary panel of 15 addiction experts comprised of doctors, nurses, pharmacists, psychologists and researchers, who were recruited purposively. Experts rated each statement using a 5-point scale of perceived importance. Two Delphi rounds were undertaken and consensus for inclusion of a statement was set at a median score of ≥4 and an interquartile range of ≤1.

RESULTS

A total of 842 statements representing codeine misusers' opinions about OTC codeine dependence were identified from the literature. Statements were grouped thematically using the COM-B framework and representative statements were selected, reducing the number to 111. After two Delphi rounds, addiction experts achieved consensus on 46 statements which formed the final Q sample.

CONCLUSIONS

This paper describes a new and systematic approach to Q sample construction and explores associated methodological issues that could be useful for those considering Q methodology and for furthering the rigour of this research technique.

Managing tensions in assessment: moving beyond either-or thinking

CONTEXT

In health professions education, assessment systems are bound to be rife with tensions as they must fulfil formative and summative assessment purposes, be efficient and effective, and meet the needs of learners and education institutes, as well as those of patients and health care organisations. The way we respond to these tensions determines the fate of assessment practices and reform. In this study, we argue that traditional 'fix-the-problem' approaches (i.e. either-or solutions) are generally inadequate and that we need alternative strategies to help us further understand, accept and actually engage with the multiple recurring tensions in assessment programmes.

METHODS

Drawing from research in organisation science and health care, we outline how the Polarity Thinking™ model and its 'both-and' approach offer ways to systematically leverage assessment tensions as opportunities to drive improvement, rather than as intractable problems. In reviewing the assessment literature, we highlight and discuss exemplars of specific assessment polarities and tensions in educational settings. Using key concepts and principles of the Polarity Thinking™ model, and two examples of common tensions in assessment design, we describe how the model can be applied in a stepwise approach to the management of key polarities in assessment.

DISCUSSION

Assessment polarities and tensions are likely to surface with the continued rise of complexity and change in education and health care organisations. With increasing pressures of accountability in times of stretched resources, assessment tensions and dilemmas will become more pronounced. We propose to add to our repertoire of strategies for managing key dilemmas in education and assessment design through the adoption of the polarity framework. Its 'both-and' approach may advance our efforts to transform assessment systems to meet complex 21st century education, health and health care needs.

Connected, attracted, and concerned: A Q study on medical crossborder curriculum partnerships

INTRODUCTION

A new form of internationalization has been trending upward in the medical education realm: crossborder medical curriculum partnerships established to deliver the same, or adapted, curriculum to groups of geographically separated students. This study aims to investigate crossborder medical curriculum partnerships by exploring the experiences of teachers at the recipient institution who have a key role in delivering the program.

METHODS

From four pioneering recipient medical schools, 24 teachers participated in a Q-sort study. Each participant rank-ordered 42 statements about teaching in a crossborder medical curriculum on a scale from -5 (indicating strong disagreement) to +5 (indicating strong agreement). The authors conducted a "by-person" factor analysis to uncover distinct patterns in the ranking of statements, using the statistical results and participants' comments about their Q sorts to interpret these patterns and translate them into distinct viewpoints.

RESULTS

Three viewpoints emerged, reflecting: (1) a feeling of connectedness with the partner institution, trust in the quality of the curriculum, and appreciation of interinstitutional relationships; (2) the partnership's attractiveness because of the career opportunities it offers; and (3) concerns over the quality of graduates because of doubts about the appropriateness of the didactic model and insufficient attention to local healthcare needs, and over the practical feasibility of such partnerships.

CONCLUSIONS

The three viewpoints identified revealed a pallet of views on how host teachers might experience their work. It shows the heterogeneous features of this group and seems to counterbalance reports that they are feeling "deprived" from their role as teacher. Two viewpoints featured an appreciation of interinstitutional relationships and of the partnership, especially when perceiving a degree of autonomy. Partners can capitalize on all different viewpoints by deploying procedure and policies to raise the quality of education delivery.

Changes in Perceived Supervision Quality After Introduction of Competency-Based Orthopedic Residency Training: A National 6-Year Follow-Up Study

OBJECTIVE

To evaluate the perceived quality of the learning environment, before and after introduction of competency-based postgraduate orthopedic education.

DESIGN

From 2009 to 2014, we conducted annual surveys among Dutch orthopedic residents. The validated Dutch Residency Educational Climate Test (D-RECT, 50 items on 11 subscales) was used to assess the quality of the learning environment. Scores range from 1 (poor) to 5 (excellent).

SETTING

Dynamic cohort follow-up study.

PARTICIPANTS

All Dutch orthopedic residents were surveyed during annual compulsory courses.

RESULTS

Over the 6-year period, 641 responses were obtained (response rate 92%). Scores for "supervision" (95% CI for difference 0.06-0.28, p = 0.002) and "coaching and assessment" (95% CI 0.11-0.35, p < 0.001) improved significantly after introduction of competency-based training. There was no significant change in score on the other subscales of the D-RECT.

CONCLUSIONS

After the introduction of some of the core components of competency-based postgraduate orthopedic education the perceived quality of "supervision" and "coaching and assessment" improved significantly.

Flexibility in Postgraduate Medical Training in the Netherlands

Postgraduate medical training in the Netherlands has become increasingly individualized. In this article, the authors describe current practices for three residency programs at the University Medical Center Utrecht: anesthesiology, pediatrics, and ophthalmology. These programs are diverse yet share characteristics allowing for individualized residency training. New residents enter each program throughout the year, avoiding a large simultaneous influx of inexperienced doctors. The usual duration of each is five years. However, the actual duration of rotations or of the program as a whole can be reduced because of residents' previous medical experience or demonstration of early mastery of relevant competencies. If necessary, the duration of training can also increase.Although working hours are already restricted by the European Working Time Directive, most residents choose to train on a part-time basis. The length of their program then is extended proportionally. The extension period added for those residents training part-time can be used to develop specific competencies, complete an elective rotation or research, or explore a focus area. If the resident meets all training objectives before the extension period is completed, the program director can choose to shorten the program length. Recently, entrustable professional activities have been introduced to strengthen workplace-based assessment. The effects on program duration have yet to be demonstrated.Flexible postgraduate training is feasible. Although improving work-life balance for residents is a necessity, attention must be paid to ensuring that they gain the necessary experience and competencies and maintain continuity of care to ensure that high-quality patient care is provided.

Stakeholder perspectives on workplace-based performance assessment: towards a better understanding of assessor behaviour

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

Staging a performance: learners' perceptions about direct observation during residency

CONTEXT

Evidence strongly supports that direct observation is a valid and reliable assessment tool; support for its impact on learning is less compelling, and we know that some learners are ambivalent about being observed. However, learners' perceptions about the impact of direct observation on their learning and professional development remain underexplored. To promote learning, we need to understand what makes direct observation valuable for learners.

METHODS

Informed by constructivist grounded theory, we interviewed 22 learners about their observation experiences. Data collection and analysis occurred iteratively; themes were identified using constant comparative analysis.

RESULTS

Direct observation was widely endorsed as an important educational strategy, albeit one that created significant anxiety. Opaque expectations exacerbated participants' discomfort, and participants described that being observed felt like being assessed. Consequently, participants exchanged their 'usual' practice for a 'textbook' approach; alterations to performance generated uncertainty about their role, and raised questions about whether observers saw an authentic portrayal of their knowledge and skill.

CONCLUSION

An 'observer effect' may partly explain learners' ambivalence about direct observation; being observed seemed to magnify learners' role ambiguity, intensify their tensions around professional development and raise questions about the credibility of feedback. In turn, an observer effect may impact learners' receptivity to feedback and may explain, in part, learners' perceptions that useful feedback is scant. For direct observation to be valuable, educators must be explicit about expectations, and they must be aware that how learners perform in the presence of an observer may not reflect what they do as independent practitioners. To nurture learners' professional development, educators must create a culture of observation-based coaching that is divorced from assessment and is tailored to developing learners' identities as practitioners of both the art and the science of medicine.

Patterns in clinical students' self-regulated learning behavior: a Q-methodology study

Students feel insufficiently supported in clinical environments to engage in active learning and achieve a high level of self-regulation. As a result clinical learning is highly demanding for students. Because of large differences between students, supervisors may not know how to support them in their learning process. We explored patterns in undergraduate students' self-regulated learning behavior in the clinical environment, to improve tailored supervision, using Q-methodology. Q-methodology uses features of both qualitative and quantitative methods for the systematic investigation of subjective issues by having participants sort statements along a continuum to represent their opinion. We enrolled 74 students between December 2014 and April 2015 and had them characterize their learning behavior by sorting 52 statements about self-regulated learning behavior and explaining their response. The statements used for the sorting were extracted from a previous study. The data was analyzed using by-person factor analysis to identify clusters of individuals with similar sorts of the statements. The resulting factors and qualitative data were used to interpret and describe the patterns that emerged. Five resulting patterns were identified in students' self-regulated learning behavior in the clinical environment, which we labelled: Engaged, Critically opportunistic, Uncertain, Restrained and Effortful. The five patterns varied mostly regarding goals, metacognition, communication, effort, and dependence on external regulation for learning. These discrete patterns in students' self-regulated learning behavior in the clinical environment are part of a complex interaction between student and learning context. The results suggest that developing self-regulated learning behavior might best be supported regarding individual students' needs.

Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings

Workplace based assessments (WBAs) are now commonplace in postgraduate medical training. User acceptability and engagement is essential to the success of any medical education innovation. To this end, possessing an insight into trainee and trainer perceptions towards WBAs will help identify the major problems, permitting strategies to be introduced to improve WBA implementation. A review of literature was performed to identify studies examining trainee and trainer perceptions towards WBAs. Studies were excluded if non-English or sampling a non-medical/dental population. The identified literature was synthesised for the purpose of this critical narrative review. It is clear that there is widespread negativity towards WBAs in the workplace. This has negatively impacted on the effectiveness of WBA tools as learning aids. This negativity exists in trainees but also to an extent in their trainers. Insight gained from the literature reveals three dominant problems with WBA implementation: poor understanding as to the purpose of WBAs; insufficient time available for undertaking these assessments; and inadequate training of trainers. Approaches to addressing these three problems with WBA implementation are discussed. It is likely that a variety of solutions will be required. The prevalence of negativity towards WBAs is substantial in both trainees and trainers, eroding the effectiveness of learning that is consequent upon them. The educational community must now listen to the concerns being raised by the users and consider the range of strategies being proposed to improve the experiences of trainees, and their trainers.

Innovating the practice of medical speciality training

Educational innovations are being introduced into medical speciality training. But how do people who participate in medical speciality training (residents, consultants, programme directors) deal with these innovations? And what effects do educational innovations have according to these people?By addressing these questions, this thesis contributes to the knowledge about the challenging process of innovating medical speciality training.

Guidelines: the do's, don'ts and don't knows of feedback for clinical education

INTRODUCTION

The guidelines offered in this paper aim to amalgamate the literature on formative feedback into practical Do's, Don'ts and Don't Knows for individual clinical supervisors and for the institutions that support clinical learning.

METHODS

The authors built consensus by an iterative process. Do's and Don'ts were proposed based on authors' individual teaching experience and awareness of the literature, and the amalgamated set of guidelines were then refined by all authors and the evidence was summarized for each guideline. Don't Knows were identified as being important questions to this international group of educators which if answered would change practice. The criteria for inclusion of evidence for these guidelines were not those of a systematic review, so indicators of strength of these recommendations were developed which combine the evidence with the authors' consensus.

RESULTS

A set of 32 Do and Don't guidelines with the important Don't Knows was compiled along with a summary of the evidence for each. These are divided into guidelines for the individual clinical supervisor giving feedback to their trainee (recommendations about both the process and the content of feedback) and guidelines for the learning culture (what elements of learning culture support the exchange of meaningful feedback, and what elements constrain it?)

CONCLUSION

Feedback is not easy to get right, but it is essential to learning in medicine, and there is a wealth of evidence supporting the Do's and warning against the Don'ts. Further research into the critical Don't Knows of feedback is required. A new definition is offered: Helpful feedback is a supportive conversation that clarifies the trainee's awareness of their developing competencies, enhances their self-efficacy for making progress, challenges them to set objectives for improvement, and facilitates their development of strategies to enable that improvement to occur.

How faculty members experience workplace-based assessment rater training: a qualitative study

CONTEXT

Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members' reactions to rater training.

OBJECTIVES

This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches - performance dimension training (PDT) and a modified approach to frame of reference training (FoRT) - to elucidate how such faculty development can be optimally designed.

METHODS

In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data.

RESULTS

Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one's approach to assessment, and (iv) concerns about institutional culture and buy-in.

CONCLUSIONS

Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.