The enablers and barriers to the use of direct observation of trainee clinical skills by supervising faculty in a psychiatry residency program

Assessing electrodiagnostic skills among residents and fellows: Relationships between workplace-based assessments using the Electromyography Direct Observation Tool and other measures of trainee performance

INTRODUCTION/AIMS

Graduate medical education programs must ensure residents and fellows acquire skills needed for independent practice. Workplace-based observational assessments are informative but can be time- and resource-intensive. In this study we sought to gather "relations-to-other-variables" validity evidence for scores generated by the Electromyography Direct Observation Tool (EMG-DOT) to inform its use as a measure of electrodiagnostic skill acquisition.

METHODS

Scores on multiple assessments were compiled by trainees during Clinical Neurophysiology and Electromyography rotations at a large US academic medical center. Relationships between workplace-based EMG-DOT scores (n = 298) and scores on a prerequisite simulated patient exercise, patient experience surveys (n = 199), end-of-rotation evaluations (n = 301), and an American Association of Neuromuscular & Electrodiagnostic Medicine (AANEM) self-assessment examination were assessed using Pearson correlations.

RESULTS

Among 23 trainees, EMG-DOT scores assigned by physician raters correlated positively with end-of-rotation evaluations (r = 0.63, P = .001), but EMG-DOT scores assigned by technician raters did not (r = 0.10, P = .663). When physician and technician ratings were combined, higher EMG-DOT scores correlated with better patient experience survey scores (r = 0.42, P = .047), but not with simulated patient or AANEM self-assessment examination scores.

DISCUSSION

End-of-rotation evaluations can provide valid assessments of trainee performance when completed by individuals with ample opportunities to directly observe trainees. Inclusion of observational assessments by technicians and patients provides a more comprehensive view of trainee performance. Workplace- and classroom-based assessments provide complementary information about trainee performance, reflecting underlying differences in types of skills measured.

Does direct observation influence the quality of workplace-based assessment documentation?

BACKGROUND

A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect).

METHODS

The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis.

RESULTS

A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84.

CONCLUSIONS

Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

Supervisory dyads' communication and alignment regarding the use of workplace-based observations: a qualitative study in general practice residency

BACKGROUND

In medical residency, performance observations are considered an important strategy to monitor competence development, provide feedback and warrant patient safety. The aim of this study was to gain insight into whether and how supervisor-resident dyads build a working repertoire regarding the use of observations, and how they discuss and align goals and approaches to observation in particular.

METHODS

We used a qualitative, social constructivist approach to explore if and how supervisory dyads work towards alignment of goals and preferred approaches to performance observations. We conducted semi-structured interviews with supervisor-resident dyads, performing a template analysis of the data thus obtained.

RESULTS

The supervisory dyads did not frequently communicate about the use of observations, except at the start of training and unless they were triggered by internal or external factors. Their working repertoire regarding the use of observations seemed to be primarily driven by patient safety goals and institutional assessment requirements rather than by providing developmental feedback. Although intended as formative, the institutional test was perceived as summative by supervisors and residents, and led to teaching to the test rather than educating for purposes of competence development.

CONCLUSIONS

To unlock the full educational potential of performance observations, and to foster the development of an educational alliance, it is essential that supervisory dyads and the training institute communicate clearly about these observations and the role of assessment practices of- and for learning, in order to align their goals and respective approaches.

What happens under the flag of direct observation, and how that matters: A qualitative study in general practice residency

INTRODUCTION

In competency-based medical education, direct observation (DO) of residents' skills is scarce, notwithstanding its undisputed importance for credible feedback and assessment. A growing body of research is investigating this discrepancy. Strikingly, in this research, DO as a concrete educational activity tends to remain vague. In this study, we concretised DO of technical skills in postgraduate longitudinal training relationships.

METHODS

Informed by constructivist grounded theory, we performed a focus group study among general practice residents. We asked residents about their experiences with different manifestations of DO of technical skills. A framework describing different DO patterns with their varied impact on learning and the training relationship was constructed and refined until theoretical sufficiency was reached.

RESULTS

The dominant DO pattern was ad hoc, one-way DO. Importantly, in this pattern, various unpredictable, and sometimes unwanted, scenarios could occur. Residents hesitated to discuss unwanted scenarios with their supervisors, sometimes instead refraining from future requests for DO or even for help. Planned bi-directional DO sessions, though seldom practiced, contributed much to collaborative learning in a psychologically safe training relationship.

DISCUSSION AND CONCLUSION

Patterns matter in DO. Residents and supervisors should be made aware of this and educated in maintaining an open dialogue on how to use DO for the benefit of learning and the training relationship.

Teaching residents how to break bad news: piloting a resident-led curriculum and feedback task force as a proof-of-concept study

Background

Breaking bad news (BBN) is a critically important skill set for residents. Limited formal supervision and unpredictable timing of bad news delivery serve as barriers to the exchange of meaningful feedback.

Purpose of study

The goal of this educational innovation was to improve internal medicine residents’ communication skills during challenging BBN encounters. A formal BBN training programme and innovative on-demand task force were part of this two-phase project.

Study design

Internal medicine residents at a large academic medical centre participated in an interactive workshop focused on BBN. Workshop survey results served as a needs assessment for the development of a novel resident-led BBN task force. The task force was created to provide observations at the bedside and feedback after BBN encounters. Training of task force members incorporated video triggers and a feedback checklist. Inter-rater reliability was analysed prior to field testing, which provided data on real-world implementation challenges.

Results

148 residents were trained during the 2-hour communications skills workshop. Based on survey results, 73% (108 of 148) of the residents indicated enhanced confidence in BBN after participation. Field testing of the task force on a hospital ward revealed potential workflow barriers for residents requesting observations and prompted troubleshooting. Solutions were implemented based on field testing results.

Conclusions

A trainee-led BBN task force and communication skills workshop is offered as an innovative model for improving residents’ interpersonal and communication skills in BBN. We believe the model is both sustainable and reproducible. Lessons learnt are offered to aid in implementation in other settings.

Dedicated Assessors: description of an innovative education intervention to facilitate direct observation in the clinical setting

BACKGROUND

The Department of Pediatrics at Queen's University undertook a pilot project in July 2017 to increase the frequency of direct observations (DO) its residents received without affecting the patient flow in a busy hospital-based pediatric ambulatory care clinic. Facilitating DO for authentic workplace-based assessments is essential for assessing resident's core competencies. The purpose of this study was to pilot an innovative education intervention to address the challenge of implementing DO in the clinical setting.

METHODS

The project allowed for staff physicians to act as "dedicated assessors" (DA), a faculty member who was scheduled to conduct direct observations of trainees' clinical skills, while not acting as the attending physician on duty. At the end of the project, focus group interviews were conducted with faculty and residents, and thematic analysis was completed.

RESULTS

Participants reported an increase in the overall quality of feedback received during the observations performed by a DA, with more specific feedback and a broader focus of assessment. There seemed to be little disruption to patient care. Some residents described the observations as anxiety-provoking.

CONCLUSIONS

Overall, this project provides insight into an educational approach that medical residency programs can apply to increase the frequency of workplace-based DO and boost the quality of feedback residents receive while maintaining the flow of already busy ambulatory care clinics.

Longitudinal resident coaching in the outpatient setting: A novel intervention to improve ambulatory consultation skills

BACKGROUND

Direct observation with feedback to learners should be a mainstay in resident education, yet it is infrequently done and its impact on consultation skills has rarely been assessed.

APPROACH

This project presents the framework and implementation of a longitudinal low-frequency, high-intensity direct observation and coaching intervention, and elaborates on insights learned. Internal medicine interns at one residency training program were randomized to an ambulatory coaching intervention or usual precepting. Over one year, coached interns had three complete primary care visits directly observed by a faculty clinician-coach who provided feedback informed by a behavior checklist. Immediately after each of the coached patient encounters, interns completed a structured self-assessment and coaches led a 30-minute feedback session informed by intern self-reflection and checklist items. Interns with usual precepting had two mini-CEX observations over the course of the year without other formal direct observation in the ambulatory setting.

EVALUATION

As part of the post-intervention assessment, senior faculty members blinded to intervention and control group assignments evaluated videotaped encounters. Coached interns completed an average of 21/23 behaviors from the checklist, while interns from the control group completed 18 (p < 0.05). The median overall grade for coached interns was B+, compared to B-/C+ for controls (p < 0.05).

REFLECTION

Coaching interns longitudinally using a behavior checklist is feasible and associated with improved consultation performance. Direct observation of complete clinical encounters followed by systematic coaching is educationally valuable, but time and resource intensive.

Student perspectives on competency-based portfolios: Does a portfolio reflect their competence development?

INTRODUCTION

Portfolio-based assessments require that learners' competence development is adequately reflected in portfolio documentation. This study explored how students select and document performance data in their portfolios and how they perceive these data to be representative for their competence development.

METHODS

Students uploaded performance data in a competency-based portfolio. During one clerkship period, twelve students also recorded an audio diary in which they reflected on experiences and feedback that they perceived to be indicants of their competence development. Afterwards, these students were interviewed to explore the extent to which the performance documentation in the portfolio corresponded with what they considered illustrative evidence of their development. The interviews were analyzed using thematic analysis.

RESULTS

Portfolios provide an accurate but fragmented picture of student development. Portfolio documentation was influenced by tensions between learning and assessment, student beliefs about the goal of portfolios, student performance evaluation strategies, the learning environment and portfolio structure.

DISCUSSION

This study confirms the importance of taking student perceptions into account when implementing a competency-based portfolio. Students would benefit from coaching on how to select meaningful experiences and performance data for documentation in their portfolios. Flexibility in portfolio structure and requirements is essential to ensure optimal fit between students' experienced competence development and portfolio content.

Supervisors' intention to observe clinical task performance: an exploratory study using the theory of planned behaviour during postgraduate medical training

BACKGROUND

Direct observation of clinical task performance plays a pivotal role in competency-based medical education. Although formal guidelines require supervisors to engage in direct observations, research demonstrates that trainees are infrequently observed. Supervisors may not only experience practical and socio-cultural barriers to direct observations in healthcare settings, they may also question usefulness or have low perceived self-efficacy in performing direct observations. A better understanding of how these multiple factors interact to influence supervisors' intention to perform direct observations may help us to more effectively implement the aforementioned guidelines and increase the frequency of direct observations.

METHODS

We conducted an exploratory quantitative study, using the Theory of Planned Behaviour (TPB) as our theoretical framework. In applying the TPB, we transfer a psychological theory to medical education to get insight in the influence of cognitive and emotional processes on intentions to use direct observations in workplace based learning and assessment. We developed an instrument to investigate supervisors intention to perform direct observations. The relationships between the TPB measures of our questionnaire were explored by computing bivariate correlations using Pearson's R tests. Hierarchical regression analysis was performed in order to assess the impact of the respective TPB measures as predictors on the intention to perform direct observations.

RESULTS

In our study 82 GP supervisors completed our TPB questionnaire. We found that supervisors had a positive attitude towards direct observations. Our TPB model explained 45% of the variance in supervisors' intentions to perform them. Normative beliefs and past behaviour were significant determinants of this intention.

CONCLUSION

Our study suggests that supervisors use their past experiences to form intentions to perform direct observations in a careful, thoughtful manner and, in doing so, also take the preferences of the learner and other stakeholders potentially engaged in direct observations into consideration. These findings have potential implications for research into work-based assessments and the development of training interventions to foster a shared mental model on the use of direct observations.

Perceptions of scheduled vs. unscheduled directly observed visits in an internal medicine residency outpatient clinic

INTRODUCTION

Learners may subconsciously change their behavior once they know they are being observed, and this Hawthorne effect should be considered when designing assessments of learner behavior. While there is a growing body of literature to suggest direct observation is the ideal standard for formative assessment, the best method to directly observe learners is unknown. We explored scheduled and unscheduled methods of direct observation among internal medicine residents in the outpatient continuity clinic to advance the understanding of both observation methods.

METHODS

We conducted a thematic analysis of faculty and internal medicine residents in an outpatient clinic setting. A semi-structured interview guide for focus group sessions was created. Focus groups were used to explore the internal medicine resident and core teaching faculty perceptions of the scheduled and unscheduled direct observation methods in the outpatient clinc. An experienced qualitative research interviewer external to the internal medicine residency was moderating the sessions. Eight peer focus groups were held. Abstraction of themes from focus group transcripts identified resident and faculty perceptions of the different observation methods.

RESULTS

Focus groups had 14 resident participants and 14 faculty participants. Unscheduled observations were felt to be more authentic than scheduled observations since residents perceived their behavior to be unmodified. Unscheduled observations allowed for increased numbers of observations per resident, which permitted more frequent formative assessments. Residents and faculty preferred remote video observation compared to in-room observation. Participants found direct observation a useful learning tool for high-yield, specific feedback.

CONCLUSIONS

Unscheduled remote direct observation captures authentic clinical encounters while minimizing learner behavior modification. An unscheduled observation approach results in more frequent formative assessment and therefore in more instances of valuable feedback compared to scheduled observations. These findings can help guide the best practice approaches to direct clinical observation in order to enhance residents learning and experience.

Physical Examination Skills Among Chief Residents in Psychiatry: Practices, Attitudes, and Self-Perceived Knowledge

OBJECTIVES

The authors investigated the attitudes, self-perceived competence, and the need for a dedicated curriculum on physical examination skills among chief residents in psychiatry.

METHODS

A voluntary 28-item web-based questionnaire was distributed to psychiatry chief residents in the USA between January 2019 and February 2019.

RESULTS

Of 181 chief residents, 79 (response rate, 44%) completed the online survey. The majority of chief residents want to improve their physical exam skills (64%) and believe that there should be a targeted curriculum aimed at incorporating these skills into everyday psychiatric practice (63%). However, most (57%) chief residents reported that they only conduct physical exams on a few selected patients (< 25% of the time) and almost half (48%) last used a stethoscope a year ago, if not longer. Self-perceived competence and comfort level with neurology-related exam findings was especially low: only 35% could identify discrepant neurological findings and 33% elicit Hoover's sign of leg paresis. A significant majority (86%) believed that performing a physical exam would not interfere with the therapeutic relationship.

CONCLUSIONS

Although chief residents in psychiatry believe that developing competence in physical examinations is important to their education, the current educational landscape does not support the development of these skills. Future educational strategies should focus on addressing this need.

Where philosophy meets culture: exploring how coaches conceptualise their roles

CONTEXT

Although conceptually attractive, coaching in medicine remains ill-defined, with little examination of the transferability of coaching principles from other fields. Here we explore how coaching is enacted both within and outside of medicine; we aim to understand both the elements required for coaching to be useful and the factors that may influence its translation to the medical education context.

METHODS

In this constructivist grounded theory study, we interviewed 24 individuals across three groups: physicians who consider themselves coaches in clinical learning settings (n = 8), physicians with experience as sports, arts or business coaches (n = 10), and sports coaches without medical backgrounds (n = 6). Data collection and analysis were conducted iteratively using constant comparison to identify themes and explore their relationships.

RESULTS

We identified a shared philosophy of coaching, comprising three core elements that our participants endorsed regardless of the coaching context: (i) mutual engagement, with a shared orientation towards growth and development; (ii) ongoing reflection involving both learners and coaches, and (iii) an embrace of failure as a catalyst for learning. Enacting these features appeared to be influenced by culture, which affected how coaching was defined and developed, how the coaching role was positioned within the learning context, and how comfortably vulnerability could be expressed. Participants struggled to clearly define the coaching role in medicine, instead acknowledging that the lines between educational roles were often blurred. Further, the embrace of failure appeared challenging in medicine, where showing vulnerability was perceived as difficult for both learners and teachers.

CONCLUSIONS

Medical education's embrace of coaching should be informed by an understanding of both coach and learner behaviours that need to be encouraged and trained, and the cultural and organisational supports that are required to foster success.

Assessment, feedback and the alchemy of learning

CONTEXT

Models of sound assessment practices increasingly emphasise assessment's formative role. As a result, assessment must not only support sound judgements about learner competence, but also generate meaningful feedback to guide learning. Reconciling the tension between assessment's focus on judgement and decision making and feedback's focus on growth and development represents a critical challenge for researchers and educators.

METHODS

We synthesise the literature related to this tension, framed around four trends in education research: (i) shifting perspectives on assessment; (ii) shifting perspectives on feedback; (iii) increasing attention on learners' perceptions of assessment and feedback, and (iv) increasing attention on the influence of culture on assessment and feedback. We describe factors that produce and sustain this tension.

RESULTS

The lines between assessment and feedback frequently blur in medical education. Models of programmatic assessment deliberately use the same data for both purposes: low-stakes individual data points are used formatively, but then are added together to support summative judgements. However, the translation of theory to practice is not straightforward. Efforts to embed meaningful feedback in programmes of learning face a multitude of threats. Learners may perceive assessment with formative intent as summative, restricting their engagement with it as feedback, and thus diminishing its learning value. A learning culture focused on assessment may limit learners' sense of safety to explore, to experiment, and sometimes to fail.

CONCLUSIONS

Successfully blending assessment and feedback demands clarity of purpose, support for learners, and a system and organisational commitment to a culture of improvement rather than a culture of performance.

Resident and attending perceptions of direct observation in internal medicine: a qualitative study

OBJECTIVES

Direct observation is the foundation of assessment and learning in competency-based medical education (CBME). Despite its importance, there is significant uncertainty about how to effectively implement frequent and high-quality direct observation. This is particularly true in specialties where observation of non-procedural skills is highly valued and presents unique challenges. It is therefore important to understand perceptions of direct observation to ensure successful acceptance and implementation. In this study, we explored perceptions of direct observation in internal medicine.

METHODS

We interviewed internal medicine attending physicians (n = 9) and residents (n = 8) at the University of Toronto, purposively sampled for diversity. Using a constructivist grounded theory approach, constant comparative analysis was performed to develop a framework to understand perceptions of direct observation on the clinical teaching units.

RESULTS

Participants articulated a narrow perception of what constitutes direct observation, in contrast to their own descriptions of skills that were observed. This resulted in the perception that certain valuable skills that participants felt were routinely observed were nonetheless not 'directly observable', such as clinical reasoning, observed through case presentations and patient care discussions. Differentiating direct observation from informal observation led to overestimation of the time and resource requirements needed to enhance direct observation, which contributed to scepticism and lack of engagement related to CBME implementation.

CONCLUSIONS

In an internal medicine training programme, perceptions of what constitutes direct observation can lead to under-recognition and hinder acceptance in workplace-based assessment and learning. Our results suggest a reframing of 'direct observation' for residents and attending physicians, by explicitly identifying desired skills in non-procedurally-based specialties. These findings may help CBME-based training programmes improve the process of direct observation, leading to enhanced assessment and learning.

Coaching Residents in the Ambulatory Setting: Faculty Direct Observation and Resident Reflection

BACKGROUND

Direct observation can be valuable for learners' skill development in graduate medical education, but it is done infrequently. Information on how to optimize trainee learning from, and best practices of, direct observation interventions in the ambulatory setting is limited.

OBJECTIVE

We explored the impact of a focused outpatient direct observation and coaching intervention on internal medicine residents.

METHODS

Using a behavior checklist based on tenets of clinical excellence, 2 faculty preceptors observed outpatient primary care visits with 96% (46 of 48) of the internal medicine residents in 2017. Residents self-assessed their performance after the visit using the same checklist. Next, a focused coaching feedback session, emphasizing reflection, was structured to highlight areas of discrepancy between resident self-assessment and coach observation (blind spots), and residents were asked to identify goals for practice improvement.

RESULTS

Common blind spots in resident self-assessment related to collaborating with patients while using the electronic health record (48%, 21 of 44), hand washing (43%, 20 of 46), and asking thoughtful questions (40%, 18 of 45). At 1-month follow-up, 93% (43 of 46) of responding residents reported change in practice toward goals often or sometimes. All residents reported that the intervention felt comfortable, and 98% (45 of 46) noted that it helped them identify new behaviors to incorporate into clinical practice.

CONCLUSIONS

Structured episodes of direct observation and coaching in the outpatient setting, with a behavior checklist, appear acceptable and useful for internal medicine residents' learning and development.

Guidelines: The do's, don'ts and don't knows of direct observation of clinical skills in medical education

INTRODUCTION

Direct observation of clinical skills is a key assessment strategy in competency-based medical education. The guidelines presented in this paper synthesize the literature on direct observation of clinical skills. The goal is to provide a practical list of Do's, Don'ts and Don't Knows about direct observation for supervisors who teach learners in the clinical setting and for educational leaders who are responsible for clinical training programs.

METHODS

We built consensus through an iterative approach in which each author, based on their medical education and research knowledge and expertise, independently developed a list of Do's, Don'ts, and Don't Knows about direct observation of clinical skills. Lists were compiled, discussed and revised. We then sought and compiled evidence to support each guideline and determine the strength of each guideline.

RESULTS

A final set of 33 Do's, Don'ts and Don't Knows is presented along with a summary of evidence for each guideline. Guidelines focus on two groups: individual supervisors and the educational leaders responsible for clinical training programs. Guidelines address recommendations for how to focus direct observation, select an assessment tool, promote high quality assessments, conduct rater training, and create a learning culture conducive to direct observation.

CONCLUSIONS

High frequency, high quality direct observation of clinical skills can be challenging. These guidelines offer important evidence-based Do's and Don'ts that can help improve the frequency and quality of direct observation. Improving direct observation requires focus not just on individual supervisors and their learners, but also on the organizations and cultures in which they work and train. Additional research to address the Don't Knows can help educators realize the full potential of direct observation in competency-based education.

'Sometimes the work just needs to be done': socio-cultural influences on direct observation in medical training

CONTEXT

Direct observation promises to strengthen both coaching and assessment, and calls for its increased use in medical training abound. Despite its apparent potential, the uptake of direct observation in medical training remains surprisingly limited outside the formal assessment setting. The limited uptake of observation raises questions about cultural barriers to its use. In this study, we explore the influence of professional culture on the use of direct observation within medical training.

METHODS

Using a constructivist grounded theory approach, we interviewed 22 residents or fellows (10 male, 12 female) about their experiences of being observed during training. Participants represented a range of specialties and training levels. Data collection and analysis were conducted iteratively. Themes were identified using constant comparative analysis.

RESULTS

Observation was used selectively; specialties tended to observe the clinical acts that they valued most. Despite these differences, we found two cultural values that consistently challenged the ready implementation of direct observation across specialties: (i) autonomy in learning and (ii) efficiency in health care provision. Furthermore, we found that direct observation was a primarily learner-driven activity, which left learners caught in the middle, wanting observation but also wanting to appear independent and efficient.

CONCLUSIONS

The cultural values of autonomy in learning and practice and efficiency in health care provision challenge the integration of direct observation into clinical training. Medical learners are often expected to ask for observation, but such requests are socially and culturally fraught, and likely to constrain the wider uptake of direct observation.