"I'll never play professional football" and other fallacies of self-assessment

Clinical educator self-efficacy, self-evaluation and its relationship with student evaluations of clinical teaching

BACKGROUND

In a whole-of-system approach to evaluation of teaching across any degree, multiple sources of information can help develop an educators' understanding of their teaching quality. In the health professions, student evaluations of clinical teaching are commonplace. However, self-evaluation of teaching is less common, and exploration of clinical educators' self-efficacy even less so. The aim of the study was to evaluate how a clinical educator's self-evaluation of teaching intersects with their self-efficacy, to ascertain if that matches student evaluation of their teaching. This information may assist in facilitating targeted professional development to improve teaching quality.

METHODS

Clinical educators in the osteopathy program at Victoria University (VU) were invited to complete: a) self-evaluation version of the Osteopathy Clinical Teaching Questionnaire (OCTQ); and b) the Self-Efficacy in Clinical Teaching (SECT) questionnaire. Students in the VU program completed the OCTQ for each of the clinical educators they worked with during semester 2, 2017.

RESULTS

Completed OCTQ and SECT were received from 37 clinical educators. These were matched with 308 student evaluations (mean of 6 student ratings per educator). Three possible educator cohorts were identified: a) high clinical eductor self-OCTQ with low student evaluation; b) low clinical educator self-evaluation and high student evaluations; and, c) no difference between self- and student evaulations. Clinical educators in the first cohort demonstrated significantly higher SECT subscale scores (effect size > 0.42) than their colleagues. Age, gender, teaching qualification, and years practicing or years as a clinical educator were not associated with clinical educator OCTQ scores or the SECT subscales.

CONCLUSIONS

Targeted professional development directed towards fostering self-efficacy may provide an avenue for engaging those clinical educators whose self-efficacy is low and/or those who did not receive high student evaluations. Given there is no gold standard measure of clinical teaching quality, educators should engage with multiple sources of feedback to benchmark their current performance level, and identify opportunities to improve. Student and self-evaluations using the OCTQ and evaluation of self-efficacy using the SECT, are useful tools for inclusion in a whole-of-system approach to evaluation of the clinical learning environment.

Empowering Clinician Education With Patient-outcome Feedback

Emergency physicians (EPs) often lack the information they need about their patients' outcomes so that they can both optimally adjust and refine their diagnostic and treatment processes and recognize their clinical errors. Patient-outcome feedback (POF) provides that information by informing clinicians about a patient's clinical course after that clinician's evaluation and treatment. This feedback may encompass the period after the EP has transferred a patient's care to another EP or after the patient has left the ED or hospital. EPs obtain POF through various active and passive methods, depending on their institutional and medical record systems. Active methods require that clinicians or others spend time and effort acquiring the information; passive methods deliver it automatically. POF is an excellent performance-based measurement that helps clinicians to stimulate their learning and to build their own validated mental library of outcomes with which to make clinical decisions, i.e., heuristics and System 1 thinking. POF offers especially useful feedback about patients who have been admitted, were referred to specialists, had major interventions, had potentially significant tests pending on discharge, or were handed off to another EP. The current health care system makes it difficult for EPs to discover their patients' outcomes, squandering significant educational opportunities. Three stimuli to improve this situation would be to require EPs to receive passive POF as part of hospital accreditation, for reviewing POF to be classified as a Category 1 Continuing Medical Education activity, and to reimburse clinicians for learning activities related to POF. Research indicates that our health care institutions and systems would be well served to provide clinicians with ongoing automatic information about their patients' outcomes.

START - evaluating a novel assessment of consultant readiness in paediatrics: The entry not the exit

The Royal College of Paediatrics and Child Health (RCPCH) incepted a new end-of-training assessment in 2012, known as START, the Speciality Trainee Assessment of Readiness for Tenure [as a Consultant]. It is a novel, formative, multi-scenario, OSCE-style, out-of-workplace assessment using unseen scenarios with generic, external assessors undertaken in the trainees' penultimate training year. This study considers whether this assessment assists in preparing senior paediatric trainees for consultant working. A mixed qualitative and quantitative study in the post-positivist paradigm was designed. Subjects were paediatricians who have taken START and completed their paediatric training. Methods were an on-line questionnaire survey and a key informant interview. The assessment is viewed positively, but some trainees report negative experiences. They find value in the formative feedback which generally helps direct trainees towards focussing their training in their final year before ending their training and consultant appointment. For many respondents, the assessment highlighted areas for further development, was relevant for consultant working and useful for consultant interview preparation. Of least value was travelling, cost, assessor performance, feedback quality, feeling like a summative exam and sub-speciality involvement. Many respondents felt the assessment highlighted areas to develop in their subsequent training. Overall START supports transition to consultant working.

Learning in the workplace: Use of informal feedback cues in doctor-patient communication

OBJECTIVES

We expect physicians to be lifelong learners. Participation in clinical practice is an important potential source of that learning. To support physicians in this process, a better understanding of how they learn in clinical practice is necessary. This study investigates how physicians recognise and use informal feedback from interactions with patients in outpatient settings as learning cues to adjust their communication behaviours in daily practice.

METHODS

To understand physicians' use of informal feedback, we combined non-participant observations with semi-structured interviews. We enrolled 10 respiratory physicians and observed 100 physician-patient interactions at two teaching hospitals in the Netherlands. Data collection and analysis were performed iteratively according to the principles of constructivist grounded theory.

RESULTS

Following stages of open, axial and selective coding, we were able to conceptualise how physicians use cues to reflect on and adjust their communication. In addition to vast variations within and across patient encounters, we observed recurring adjustments in physicians' communication behaviours in response to specific informal feedback cues. Physicians recognised and used these cues to self-monitor communication performance. They had established 'communication repertoires' based on multiple patient interactions, which many saw as learning opportunities contributing to the development of expertise. Our findings, however, show differences in physicians' individual levels of sensitivity in recognising and using learning opportunities in daily practice, which were further influenced by contextual, personal and interpersonal factors. Whereas some described themselves as having little inclination to change, others used critical incidents to fine-tune their communication repertoires, and yet others constantly reshaped them, seeking learning opportunities in their daily work.

CONCLUSIONS

There is large variation in how physicians use learning cues from daily practice. To enhance learning in and from daily practice, we propose turning workplace learning into a collaborative effort with the aim of increasing awareness and the use of informal performance-relevant feedback.

Exploration of constructive feedback practices in dental education chairside teaching: A case study

BACKGROUND

Chairside teaching is one of the teaching-learning methods in clinical dental education in which direct care is provided to patients. When students have been deemed competent in carrying out procedures on a dental phantom, they need clinical experience on patients, with guidance and constructive feedback from clinical teachers. Constructive feedback is an important learning platform in helping students analyse the strong and the weak aspects of their performance in order to identify required improvements. This study aimed to explore the practice of giving constructive feedback in chairside teaching.

METHOD

A qualitative method with a case study design. Data were collected through in-depth interviews and focus group discussions (FGDs) with clinical teachers and students in clinical rotation. Data triangulation was carried out by observing the practice of giving constructive feedback in chairside teaching and document analysis from January to April 2019. Results of the in-depth interviews and FGDs were transcribed verbatim and analysed using a thematic analysis approach.

RESULT

In-depth interviews with five programme coordinators and FGDs with two groups of clinical teachers (N = 8 and N = 6) and two clinical student groups (N = 8 each). Three main themes emerged in this study: ways to provide feedback, challenges on feedback provision and challenges on feedback follow-up.

CONCLUSION

Differences in perception between clinical teachers and students were identified, influenced by students' interactions with the learning environment. Interventions are to be encouraged that involve teachers, students and institutions, and the interactions amongst the three.

An observational study of self-monitoring in ad hoc health care teams

BACKGROUND

Working in ad hoc teams in a health care environment is frequent but a challenging and complex undertaking. One way for teams to refine their teamwork could be through post-resuscitation reflection and debriefing. However, this would require that teams have insight into the quality of their teamwork. This study investigates (1) the accuracy of the self-monitoring of ad hoc resuscitation teams and their leaders relative to external observations of their teamwork and (2) the relationship of team self-monitoring and external observations to objective performance measures.

METHODS

We conducted a quantitative observational study of real-world ad hoc interprofessional teams responding to a simulated cardiac arrest in an emergency room. Teams consisting of residents, consultants, and nurses were confronted with an unexpected, simulated, standardized cardiac arrest situation. Their teamwork was videotaped to allow for subsequent external evaluation on the team emergency assessment measure (TEAM) checklist. In addition, objective performance measures such as time to defibrillation were collected. All participants completed a demographic questionnaire prior to the simulation and a questionnaire tapping their perceptions of teamwork directly after it.

RESULTS

22 teams consisting of 115 health care professionals showed highly variable performance. All performance measures intercorrelated significantly, with the exception of team leaders' evaluations of teamwork, which were not related to any other measures. Neither team size nor cumulative experience were correlated with any measures, but teams led by younger leaders performed better than those led by older ones.

CONCLUSION

Team members seem to have better insight into their team's teamwork than team leaders. As a practical consequence, the decision to debrief and the debriefing itself after a resuscitation should be informed by team members, not just leaders.

Perceptions of the feedback process: a case study of veterinary residents and supervisors

BACKGROUND

Appraisal of resident clinical performance is critical during training. The most common method for this is feedback to residents based on impressions of supervising clinicians. How effective these practices are for veterinary residents remains unknown. This study focused on establishing perceptions of veterinary residents and supervisors regarding the feedback process.

METHODS

A qualitative case study format was chosen to investigate the perceptions and experiences within a well-developed residency programme. The study cohort consisted of veterinary residents and supervisors from the same specialties. Qualitative data were collected through individual semistructured interviews continuing iteratively until theoretical saturation was reached (14 in total).

RESULTS

Mismatches in resident and supervisor perceptions were evident regarding positive feedback delivery and the importance of dialogue. The nature of the resident/supervisor relationship and the efficacy of feedback were closely interlinked. The development of a feedback-friendly culture would be beneficial. Residents perceived that feedback on teaching was lacking. Milestones were perceived to be lacking.

CONCLUSION

The results highlight a need for change away from the 'no news is good news' culture. Development of training workshops, formation of closer relationships between supervisors and residents, and a transition to competency-based education may be necessary.

Self-monitoring accuracy does not increase throughout undergraduate medical education

CONTEXT

Accurate self-assessment of one's performace on a moment-by-moment basis (ie, accurate self-monitoring) is vital for the self-regulation of practising physicians and indeed for the effective regulation of self-directed learning during medical education. However, little is currently known about the functioning of self-monitoring and its co-development with medical knowledge across medical education. This study is the first to simultaneously investigate a number of relevant aspects and measures that have so far been studied separately: different measures of self-monitoring for a broad area of medical knowledge across 10 different performance levels.

METHODS

This study assessed the self-monitoring accuracy of medical students (n = 3145) across 10 semesters. Data collected during the administration of the formative Berlin Progress Test Medicine (PTM) were analysed. The PTM comprises 200 multiple-choice questions covering all major medical disciplines and organ systems. A self-report indicator (ie, confidence) and two behavioural indicators of self-monitoring accuracy (ie, response time and the likelihood of changing an initial answer to a correct rather than an incorrect item) were examined for their development over semesters.

RESULTS

Analyses of more than 390 000 observations (of approximately 250 students per semester) showed that confidence was higher for correctly than for incorrectly answered items and that 86% of items answered with high confidence were indeed correct. Response time and the likelihood of the initial answer being changed were higher when the initial answer was incorrect than when it was correct. Contrary to expectations, no differences in self-monitoring accuracy were observed across semesters.

CONCLUSIONS

Convergent evidence from different measures of self-monitoring suggests that medical students self-monitor their knowledge on a question-by-question basis well, although not perfectly, and to the same degree as has been found in studies outside medicine. Despite large differences in performance, no variations in self-monitoring across semesters (with the exception of the first semester) were observed.

With a grain of salt? Supervisor credibility and other factors influencing trainee decisions to seek in-consultation assistance: a focus group study of Australian general practice trainees

Background

‘Ad hoc’ help-seeking by trainees from their supervisors during trainee consultations is important for patient safety, and trainee professional development. We explored trainee objectives and activities in seeking supervisor assistance, and trainee perceptions of the outcomes of this help-seeking (including the utility of supervisor responses).

Methods

Focus groups with Australian general practice trainees were undertaken. All data was audio-recorded and transcribed, coded using in-vivo and descriptive codes, and analysed by the constant comparison of provisional interpretations and themes with the data. Findings are reported under the over-arching categories of help-seeking objectives, activities and outcomes.

Results

Early in their general practice placements trainees needed information about practice facilities, and the “complex maze” of local patient resources and referral preferences: some clinical presentations were also unfamiliar, and many trainees were unaccustomed to making patient management decisions. Subsequent help-seeking was often characterised informally as “having a chat” or “getting a second opinion” so as not to “miss anything” when trainees were “not 100% sure”. Trainees emphasised the importance of being (and demonstrating that they were) clinically safe. Workflow constraints, and supervisory and doctor-patient relationships, had a powerful influence on trainee help-seeking activities. An etiquette for providing help in front of patients was described. Trainees assessed the credibility of supervisors based on their approach to risk and their clinical expertise in the relevant area. Several trainees reported reservations about their supervisor’s advice on occasions.

Conclusion

A trainee’s subsequent help-seeking is strongly influenced by how their supervisor responds when their help is sought. Trainees prefer to seek help from credible supervisors who respond promptly and maintain trainee ‘face’ in front of patients. Trainees learn through help-seeking to make their own clinical decisions but may remain uncertain about professional and societal expectations, and curious about how other general practitioners practise. Trainees value opportunities throughout their training to observe expert general practice.

In-the-Moment Feedback and Coaching: Improving R2C2 for a New Context

BACKGROUND

The R2C2, a 4-phase feedback and coaching model, builds relationships, explores reactions, determines content and coaches for change, and facilitates formal feedback conversations between clinical supervisors/preceptors and residents. Formal discussions about performance are typically based on collated information from daily encounter sheets, objective structured clinical examinations, multisource feedback, and other data. This model has not been studied in settings where brief feedback and coaching conversations occur immediately after a specific clinical experience.

OBJECTIVE

We explored how supervisors adapt the R2C2 model for in-the-moment feedback and coaching and developed a guide for its use in this context.

METHODS

Eleven purposefully selected supervisors were interviewed in 2018 to explore where they used the R2C2 model, how they adapted it for in-the-moment conversations, and phrases used corresponding to each phase that could guide design of a new R2C2 in-the-moment model.

RESULTS

Participants readily adapted the model to varied feedback situations; each of the 4 phases were relevant for conversations. Phase-specific phrases that could enable effective coaching conversations in a limited amount of time were identified. Data facilitated a revision of the original R2C2 model for in-the-moment feedback and coaching conversations and design of an accompanying trifold brochure to enable its effective use.

CONCLUSIONS

The R2C2 in-the-moment model offers a systematic approach to feedback and coaching that builds on the original model, yet addresses time constraints and the need for an iterative conversation between the reaction and content phases. The model enables supervisors to coach and co-create an action plan with residents to improve performance.

Peak Performance: A Communications-Based Leadership and Teamwork Simulation for Fourth-Year Medical Students

BACKGROUND

Medical education has traditionally been rooted in the teaching of health and disease processes, with little attention to the development of teamwork and leadership competencies.

OBJECTIVE

In an era of value-based health care provided by high-functioning teams, new approaches are needed to develop communication, leadership, and teamwork skills for medical students.

DESIGN

We designed and piloted a simulation-based educational activity called Peak Performance that linked a workbook, which focused on self-reflection on communication and leadership skills, with professional coaching. The simulation scenario placed students in the role of an upper-level resident on an inpatient service, followed by a small group debrief with students, a clinical faculty member, and a professional executive coach. After the debriefing session, students were invited to complete a self-reflection workbook within 1 week of the initial simulation. The final element of the curriculum was an individualized session with an executive coach. Peak Performance was offered to all fourth-year medical students enrolled in the Social and Health Systems Science required course at the University of North Carolina School of Medicine.

RESULTS

Pre-/post-self-assessments of leadership competencies were completed by students. Pre-simulation self-assessment scores ranged from 3.72 to 4.33 on a 5-point scale. The lowest scores were in "Managing Conflict" and "Managing Others." The highest score was in "Self-Awareness." The post-simulation scores decreased in every competency, with "Managing Others" dropping significantly from 3.72 pre-simulation to 3.36 post-simulation (0.31, P < .05). Satisfaction with the curriculum was high, as reflected by a Net Promoter Score of 91% ("excellent" > 50%).

CONCLUSIONS

A novel simulation-based educational activity linked to professional coaching is a feasible and impactful strategy to develop leadership, communication, and teamwork skills in medical students. Student insight and self-awareness increased as evidenced by a decrease in competency self-assessment after guided reflection and individualized coaching.

Toward Practice-Based Continuing Education Protocols: Using Testing to Help Physicians Update Their Knowledge

INTRODUCTION

Using assessment to facilitate learning is a well-established priority in education but has been associated with variable effectiveness for continuing professional development. What factors modulate the impact of testing in practitioners are unclear. We aimed to improve capacity to support maintenance of competence by exploring variables that influence the value of web-based pretesting.

METHODS

Family physicians belonging to a practice-based learning program studied two educational modules independently or in small groups. Before learning sessions they completed a needs assessment and were assigned to either sit a pretest intervention or read a relevant review article. After the learning session, they completed an outcome test, indicated plans to change practice, and subsequently documented changes made.

RESULTS

One hundred twelve physicians completed the study, 92 in small groups. The average lag between tests was 6.3 weeks. Relative to those given a review article, physicians given a pretest intervention: (1) reported spending less time completing the assigned task (16.7 versus 25.7 minutes); (2) performed better on outcome test questions that were repeated from the pretest (65.9% versus 58.7%); and (3) when the learning module was completed independently, reported making a greater proportion of practice changes to which they committed (80.0% versus 45.0%). Knowledge gain was unrelated to physicians' stated needs.

DISCUSSION

Low-stakes formative quizzes, delivered with feedback, can influence the amount of material practicing physicians remember from an educational intervention independent of perceptions regarding the need to engage in continuing professional development on the particular topic.

A Delphi study to establish consensus on an educational package of musculoskeletal clinical prediction rules for physiotherapy clinical educators

BACKGROUND

Clinical prediction rules (CPRs) are evidence-based tools to aid clinical decision-making, and there are many that are relevant for physiotherapists, especially in the musculoskeletal field. However, a lack of awareness and understanding by physiotherapy clinical educators could limit students' exposure to these potentially valuable tools. An educational package tailored for clinical educators could help them recognise the value of CPRs and implement them in clinical practice with students.

OBJECTIVES

To determine consensus on the essential content and optimal delivery of an educational package on musculoskeletal CPRs for physiotherapy clinical educators.

DESIGN

An online survey of physiotherapy experts who have published on CPRs, using a Delphi approach.

METHOD

Sixteen experts were recruited for a two-round reactive Delphi study in which they rated previously identified elements, as well as suggesting new items for an educational package.

FINDINGS

A pre-defined consensus of ≥70% identified that the content of an educational package should cover fundamental aspects of CPRs including why, when and how to use them clinically, and their limitations. Information on the evidence-base of different types of CPRs, with specific examples, was also identified as important. Online delivery was recommended via self-directed learning and webinars, along with electronic versions of actual CPRs. A self-assessment component was also supported.

CONCLUSIONS

An educational package on musculoskeletal CPRs for clinical educators was supported with key elements outlined by an international panel of experts.

IMPLICATIONS

Improving clinical educators' knowledge of CPRs may lead to physiotherapy students having a greater understanding and ability to use CPRs.

Meaningful feedback through a sociocultural lens

This AMEE guide provides a framework and practical strategies for teachers, learners and institutions to promote meaningful feedback conversations that emphasise performance improvement and professional growth. Recommended strategies are based on recent feedback research and literature, which emphasise the sociocultural nature of these complex interactions. We use key concepts from three theories as the underpinnings of the recommended strategies: sociocultural, politeness and self-determination theories. We view the content and impact of feedback conversations through the perspective of learners, teachers and institutions, always focussing on learner growth. The guide emphasises the role of teachers in forming educational alliances with their learners, setting a safe learning climate, fostering self-awareness about their performance, engaging with learners in informed self-assessment and reflection, and co-creating the learning environment and learning opportunities with their learners. We highlight the role of institutions in enhancing the feedback culture by encouraging a growth mind-set and a learning goal-orientation. Practical advice is provided on techniques and strategies that can be used and applied by learners, teachers and institutions to effectively foster all these elements. Finally, we highlight throughout the critical importance of congruence between the three levels of culture: unwritten values, espoused values and day to day behaviours.

Benefits and Limitations of Transurethral Resection of the Prostate Training With a Novel Virtual Reality Simulator

PURPOSE

Profound endourological skills are required for optimal postoperative outcome parameters after transurethral resection of the prostate (TURP). We investigated the Karl Storz (Tuttlingen, Germany) UroTrainer for virtual simulation training of the TURP.

MATERIALS AND METHODS

Twenty urologists underwent a virtual reality (VR) TURP training. After a needs analysis, performance scores and self-rated surgical skills were compared before and after the curriculum, the realism of the simulator was assessed, and the optimal level of experience for VR training was evaluated. Statistical testing was done with SPSS 25.

RESULTS

Forty percent of participants indicated frequent intraoperative overload during real-life TURP and 80% indicated that VR training might be beneficial for endourological skills development, underlining the need to advance classical endourological training. For the complete cohort, overall VR performance scores (P = 0.022) and completeness of resection (P < 0.001) significantly improved. Self-rated parameters including identification of anatomical structures (P = 0.046), sparing the sphincter (P = 0.002), and handling of the resectoscope (P = 0.033) became significantly better during the VR curriculum. Participants indicated progress regarding handling of the resectoscope (70%), bleeding control (55%), and finding the correct resection depth (50%). Although overall realism and handling of the resectoscope was good, virtual bleeding control and correct tissue feedback should be improved in future VR simulators. Seventy percent of participants indicated 10 to 50 virtual TURP cases to be optimal and 80% junior residents to be the key target group for VR TURP training.

CONCLUSIONS

There is a need to improve training the TURP and VR simulators might be a valuable supplement, especially for urologists beginning with the endourological desobstruction of the prostate.

Situational cues surrounding family physicians seeking external resources while self-monitoring in practice

Many models of safe and effective clinical decision making in medical practice emphasize the importance of recognizing moments of uncertainty and seeking help accordingly. This is not always done effectively, but we know little about what cues prompt health professionals to call on resources beyond their own knowledge or skill set. Such information would offer guidance regarding how systems might be designed to offer better individual support. In this study, the authors explored the situational factors that are present during moments of uncertainty that lead primary care physicians to access external resources. To do so, a generic qualitative exploratory analysis was conducted on 72 narratives collected through audio recorder-based, self-observational, journaling completed by 12 purposively selected family physicians. Participants were asked to provide a detailed descriptive account of the circumstances surrounding their consultation of external resources immediately after 6 sequential patient encounters in which they felt compelled to seek such support. Thematic analysis of the transcripts was performed to better understand participants' experiences of the social, contextual, and personal features surrounding decisions to seek support. When doing so we observed that specific features of patient encounters were routinely present when physicians decided to access external sources for help. These included medical aspects of the case (e.g., complex presentations), social aspects (e.g., the presence of another individual), and personal factors (e.g., feeling a need for reassurance). External resources were seen as an opportunity for verification, a mechanism to increase patient satisfaction, and a means through which to defend decision-making. Accessing such resources appeared to influence the physician-patient relationship for various reasons. Recognition and further study of the cues that prompt use of external information will further our understanding of physicians' behavioural responses to challenging/uncertain situations, highlight mechanisms through which a culture of self-directed assessment seeking might be encouraged, and offer guidance regarding ways in which physicians can be encouraged to practice mindfully. Our results make it clear that reasons for which primary care physicians seek the support of external resources may be multifactorial and, therefore, one should be cautious when inferring reasons for the pursuit of such support.

Clinical Improvement Interventions for Residents and Practicing Physicians: A Scoping Review of Coaching and Mentoring for Practice Improvement

BACKGROUND

Graduate medical education (GME) bodies are beginning to mandate coaching as an integral part of the learning process, in addition to current requirements for mentorship. Once an emergency medicine physician transitions beyond graduate training, there is no requirement and little focus on coaching as a method of improving or maintaining clinical practice. Our objective was to understand and describe the current state of the published literature with regard to the use of coaching and mentorship for both GME and practicing physicians.

METHODS

We conducted a structured review of the literature through PubMed and Google Scholar and included all articles applying coaching or mentorship modalities to GME trainees or practicing physicians. A Google Form was used for standardized data abstraction. Data were collected pertaining to the settings of intervention, the nature of the intervention, its effect, and its resource requirements.

RESULTS

A total of 3,546 papers were isolated during the literature review. After exclusion, 186 underwent full-text review by the authors of which 126 articles were included in the final data analysis. Eighty-two articles (65%) pertained to mentorship and 14 (11%) to coaching; the remainder of the articles discussed a combination or variation of these two concepts. Fifty-three (42%) articles were descriptive studies and 35 (28%) were narrative reviews or commentaries. Forty-seven (37%) articles originated from within surgical specialties and coaching was most commonly applied to procedural or manual skills with 22 (17%) instances among all studies.

CONCLUSIONS

Most literature on coaching and mentorship is descriptive or narrative, and few papers are in the specialty of emergency medicine. Most interventions are limited to single instances of coaching or mentorship without longitudinal application of the intervention. There is an important need to study and publish further evidence on coaching interventions.

Toward effective feedback: From concept to practice

Feedback is the building block of assessment and an essential component of effective teaching. In this article, we will provide a comprehensive definition of feedback and a relevant conceptual framework for learning to explain how feedback can be used to improve performance. We will discuss the process of providing feedback, and the role teacher and learner can play to make it more effective. We will also examine the misunderstandings and pitfalls around feedback, as well as generational differences that may influence its impact.

Ability of Ophthalmology Residents to Self-Assess Their Performance Through Established Milestones

OBJECTIVES

Accurate self-assessment is an important aspect of practice-based learning and improvement and a critical skill for resident growth. The Accreditation Council for Graduate Medical Education mandates semiannual milestones assessments by a clinical competency committee (CCC) for all ophthalmology residents. There are six core competencies: patient care (PC), medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication skills. These competencies are assessed by the milestones rubric, which has detailed behavioral anchors and are also used for trainee self-assessments. This study compares resident self-assessed (SA) and faculty CCC milestones scores.

DESIGN

Residents completed milestones self-assessments prior to receiving individual score reports from the CCC. Correlation coefficients were calculated comparing the SA and CCC scores. In addition, statistical models were used to determine predictors of disparities and differences between the SA and CCC scores.

SETTING

Wilmer Eye Institute, Johns Hopkins Hospital.

PARTICIPANTS

Twenty-one residents in the Wilmer Ophthalmology Residency program from July 2014 to June 2016.

RESULTS

Fifty-seven self-assessments were available for the analysis. For each resident's first assessment, SA and CCC scores were strongly correlated (r ≥ 0.6 and p < 0.05) for four milestones, and not correlated for the remaining 20 milestones. In multivariable models, the SA and CCC scores are less disparate for medical knowledge and systems-based practice competencies compared to practice-based learning and improvement. Higher year of training, PC and professionalism competencies were predictive of statistically significant resident overestimation of scores relative to the CCC. In addition, higher CCC scores predicted statistically significant lower SA-CCC disparities and differences. SA-CCC differences did not lower to a significant extent with repeated assessments or modification to the end-of-rotation evaluation forms.

CONCLUSIONS

Self-assessments by ophthalmology residents are not well-correlated with faculty assessments, emphasizing the need for improved and frequent timely feedback. Residents have the greatest difficulty self-assessing their professionalism and PC competency. In general, senior residents and underperforming residents have more inaccurate self-assessments.

Accuracy of self-monitoring: does experience, ability or case difficulty matter?

CONTEXT

The ability to self-monitor one's performance in clinical settings is a critical determinant of safe and effective practice. Various studies have shown this form of self-regulation to be more trustworthy than aggregate judgements (i.e. self-assessments) of one's capacity in a given domain. However, little is known regarding what cues inform learners' self-monitoring, which limits an informed exploration of interventions that might facilitate improvements in self-monitoring capacity. The purpose of this study is to understand the influence of characteristics of the individual (e.g. ability) and characteristics of the problem (e.g. case difficulty) on the accuracy of self-monitoring by medical students.

METHODS

In a cross-sectional study, 283 medical students from 5 years of study completed a computer-based clinical reasoning exercise. Confidence ratings were collected after completing each of six cases and the accuracy of self-monitoring was considered to be a function of confidence when the eventual answer was correct relative to when the eventual answer was incorrect. The magnitude of that difference was then explored as a function of year of seniority, gender, case difficulty and overall aptitude.

RESULTS

Students demonstrated accurate self-monitoring by virtue of giving higher confidence ratings (57.3%) and taking a shorter time to work through cases (25.6 seconds) when their answers were correct relative to when they were wrong (41.8% and 52.0 seconds, respectively; p< 0.001 and d > 0.5 in both instances). Self-monitoring indices were related to student seniority and case difficulty, but not to overall ability or student gender.

CONCLUSIONS

This study suggests that the accuracy of self-monitoring is context specific, being heavily influenced by the struggles students experience with a particular case rather than reflecting a generic ability to know when one is right or wrong. That said, the apparent capacity to self-monitor increases developmentally because increasing experience provides a greater likelihood of success with presented problems.

Twelve tips to promote a feedback culture with a growth mind-set: Swinging the feedback pendulum from recipes to relationships

Feedback in medical education has traditionally showcased techniques and skills of giving feedback, and models used in staff development have focused on feedback providers (teachers) not receivers (learners). More recent definitions have questioned this approach, arguing that the impact of feedback lies in learner acceptance and assimilation of feedback with improvement in practice and professional growth. Over the last decade, research findings have emphasized that feedback conversations are complex interpersonal interactions influenced by a multitude of sociocultural factors. However, feedback culture is a concept that is challenging to define, thus strategies to enhance culture are difficult to pin down. In this twelve tips paper, we have attempted to define elements that constitute a feedback culture from four different perspectives and describe distinct strategies that can be used to foster a learning culture with a growth mind-set.

Strategies to effectively implement self-assessment techniques using multiple methods

BACKGROUND

This commentary accompanies a companion article that used post-test predictions or postdictions as a self-assessment strategy to help third-year pharmacy students evaluate and understand their test performance.

IMPACT

In addition to providing constructive critiques to the companion article, two educational studies using different self-assessment techniques are highlighted.

TEACHABLE MOMENT

To help improve the design and implementation of self-assessment strategies in the future, the goal of this teachable moments matter is to unpack the relationship between self-assessment and metacognition. Specifically, this commentary will address the consequences of using singular self-assessments as well as provide strategies on how to appropriately utilize self-assessments through the triangulation of multiple methods.

Challenges & Issues: Evidence-Based Clinical Skills Teaching and Learning: What Do We Really Know?

The recent programmatic focus on skills development in veterinary medicine means that many programs are devoting increased time to formal clinical skills teaching. This expansion makes it essential that we use the time as effectively as possible. This review examines current practices and veterinary training principles using the broader field of evidence-based motor skills learning as a lens. In many areas, current practices may be hindering learning. Proposed practices include using videos and discussions for pre-laboratory training, focusing on a single complex skill at a time, using more near-peer instructors rather than faculty, including assessments in each teaching or practice session, and encouraging supervised distributed practice by incorporating practice sessions into the formal curriculum. Ensuring mastery of a few core skills rather than exposure to many may be the new goal. Further research is urgently needed on block versus spiral curricula, optimum instructor-to-student ratios, learning and practice schedules, hours required for proficiency, and the benefits of exercise on motor skills learning.

Window to the Unknown: Using Storytelling to Identify Learning Needs for the Intrinsic Competencies Within an Online Needs Assessment

INTRODUCTION

Needs assessments are important for developing online educational resources, but they frequently do not capture learning needs in the intrinsic physician competencies. Storytelling exercises, by promoting critical reflection and emphasizing values and context, may assist curriculum developers in identifying emergent knowledge gaps in these areas that are initially unknown to learners.

METHODS

We developed an online curriculum for thrombosis and hemostasis based on an open-access online needs assessment comprised of a topic list, case scenarios, and storytelling exercise. In the storytelling exercise, learners described 1) a difficult clinical case and 2) why that case was difficult. In this qualitative descriptive study, we performed a secondary thematic analysis of this storytelling data, coded for the CanMEDS 2015 intrinsic roles. Two investigators independently coded transcripts with iterative comparison.

RESULTS

A total of 143 respondents completed the storytelling exercise. All responses yielded a gap in medical expertise, while 25 (17.5%) described an additional intrinsic role. Learning needs in all six intrinsic roles were identified. The most commonly cited learning needs were in the leader (recognizing how resource allocation impacts health care), communicator (communicating knowledge with patients), and collaborator (unclear communication between providers) roles. These excerpts were notable for how they expressed the complexity and affective components of medicine.

CONCLUSIONS

Storytelling exercises can highlight context, attitudes, and relationships that provide depth to needs assessments. These narratives are a novel method of identifying gaps in intrinsic physician competencies that are initially unknown by learners (Johari window). These emergent intrinsic learning needs may be used to enrich learner-centered curricula.

Notes from the Field: Evaluating a Budget-Based Approach to Peer Assessment for Measuring Collaboration Among Learners on Interprofessional Teams

Assessing interprofessional skills poses challenges for health professions educators. While competency frameworks define the skills graduating students should possess, they do not provide guidance for assessment. This brief report explores validity evidence for use of peer assessment to assess learners and provide feedback for improvement. The context was an online learning experience for 477 fourth-year students from medicine, nursing, and pharmacy who worked together on small interprofessional teams to care for a virtual geriatric patient. At the end of each case unit, students were given a budget of points to allocate among teammates to assess their communication and interprofessional collaboration. Ratings were averaged to provide learners with feedback about their performance. Scores were normally distributed, did not demonstrate a leniency effect, were moderately correlated with ratings that preceptors assigned to students, and had smaller correlations with knowledge scores and other case activity measures. Findings support budget-based peer assessment as a valid and feasible approach for differentiating between students with high interprofessional competency and those who may be deficient. Further exploration should focus on the longitudinal effect of peer assessment, how it may influence individual learning and team dynamics, and whether it could be used for other assessment purposes.

The Evaluation of Physicians' Communication Skills From Multiple Perspectives

PURPOSE

To examine how family physicians', patients', and trained clinical raters' assessments of physician-patient communication compare by analysis of individual appointments.

METHODS

Analysis of survey data from patients attending face-to-face appointments with 45 family physicians at 13 practices in England. Immediately post-appointment, patients and physicians independently completed a questionnaire including 7 items assessing communication quality. A sample of videotaped appointments was assessed by trained clinical raters, using the same 7 communication items. Patient, physician, and rater communication scores were compared using correlation coefficients.

RESULTS

Included were 503 physician-patient pairs; of those, 55 appointments were also evaluated by trained clinical raters. Physicians scored themselves, on average, lower than patients (mean physician score 74.5; mean patient score 94.4); 63.4% (319) of patient-reported scores were the maximum of 100. The mean of rater scores from 55 appointments was 57.3. There was a near-zero correlation coefficient between physician-reported and patient-reported communication scores (0.009, P = .854), and between physician-reported and trained rater-reported communication scores (-0.006, P = .69). There was a moderate and statistically significant association, however, between patient and trained-rater scores (0.35, P = .042).

CONCLUSIONS

The lack of correlation between physician scores and those of others indicates that physicians' perceptions of good communication during their appointments may differ from those of external peer raters and patients. Physicians may not be aware of how patients experience their communication practices; peer assessment of communication skills is an important approach in identifying areas for improvement.

Motivation for feedback-seeking among pediatric residents: a mixed methods study

BACKGROUND

For effective self-directed life-long learning physicians need to engage in feedback-seeking, which means fostering such behavior during training. Self-determination theory (SDT) posits that intrinsic motivation is fostered when the environment optimizes the individual's experience of autonomy, relatedness, and competence. Educational settings meeting these psychological needs may foster intrinsic motivation in trainees, enhance their desire for feedback, and promote feedback-seeking. We sought to examine residents' feedback-seeking behaviors through the lens of SDT and explore the association with intrinsic motivation and career choice.

METHODS

We used a mixed-methods approach with an explanatory sequential design. Residents participated in simulation training, completed an inventory of intrinsic motivation (IMI) and responded to sequential opportunities for performance feedback requiring different levels of effort. We compared IMI scores and career choice between groups with different effort. We interviewed high-effort feedback-seekers and conducted thematic analysis of interview data.

RESULTS

Thirty-four of 35 residents completed the survey (97%). Of those completing the study, 12 engaged in low-effort feedback-seeking only, 10 indicated intent for high-effort feedback-seeking and 10 actually engaged in higher effort to get feedback. Groups did not differ in mean IMI scores. Among high-effort feedback-seekers more residents were interested in critical care-related fields compared to the other groups. We identified 5 themes around autonomy, relatedness, and competence clarifying residents' reasons for feedback-seeking.

CONCLUSIONS

Our findings suggest that among residents, the relationship between motivation and feedback-seeking is complex and cannot be predicted by IMI score. Career plans and relationships with feedback providers impact feedback-seeking, which can inform educational interventions.

Do State Continuing Medical Education Requirements for Physicians Improve Clinical Knowledge?

OBJECTIVE

To evaluate the effect of state continuing medical education (CME) requirements on physician clinical knowledge.

DATA SOURCES

Secondary data for 19,563 general internists who took the Internal Medicine Maintenance of Certification (MOC) examination between 2006 and 2013.

STUDY DESIGN

We took advantage of a natural experiment resulting from variations in CME requirements across states over time and applied a difference-in-differences methodology to measure associations between changes in CME requirements and physician clinical knowledge. We measured changes in clinical knowledge by comparing initial and MOC examination performance 10 years apart. We constructed difference-in-differences estimates by regressing examination performance changes against physician demographics, county and year fixed effects, trend-state indicators, and state CME change indicators.

DATA COLLECTION

Physician data were compiled by the American Board of Internal Medicine. State CME policies were compiled from American Medical Association reports.

PRINCIPAL FINDINGS

More rigorous CME credit-hour requirements (mostly implementing a new requirement) were associated with an increase in examination performance equivalent to a shift in examination score from the 50th to 54th percentile.

CONCLUSIONS

Among physicians required to engage in a summative assessment of their clinical knowledge, CME requirements were associated with an improvement in physician clinical knowledge.

Effect of reflection on medical students' situational interest: an experimental study

CONTEXT

Reflection has been considered crucial to learning. Engaging in reflection while solving problems is expected to foster identification of knowledge gaps and interest in learning more about them, the latter being a major motivational force in learning. Although theoretically sound, this assumption still lacks empirical evidence. This experiment investigated whether reflection while diagnosing clinical cases of different levels of difficulty influences medical students' awareness of knowledge gaps and situational interest.

METHODS

Forty-two fourth-year students from a Brazilian medical school were randomly allocated to diagnose six clinical cases (three difficult; three easy), either by following a structured reflection procedure (reflection group) or by giving alternative diagnoses (control group). Subsequently, for each case, all students rated their situational interest and awareness of knowledge gaps.

RESULTS

Situational interest was significantly higher in the reflection group than in the control group (mean = 4.10, standard deviation = 0.50 versus mean = 3.65, standard deviation = 0.48, respectively; p = 0.003; range, 1-5). The effect size was large (d = 0.92). Awareness of knowledge gaps was higher in the reflection group than in the control group, but the difference was not significant. Case difficulty influenced both situational interest, which was significantly higher on easy than on difficult cases (mean = 3.96, standard deviation = 0.56 versus mean = 3.80, standard deviation = 0.55, respectively; p = 0.004), and awareness of knowledge gaps, with higher scores observed on difficult compared with easy cases (mean = 3.99, standard deviation = 0.46 versus mean = 3.66, standard deviation = 0.53, respectively; p < .001). No interaction between experimental condition and case difficulty emerged.

CONCLUSION

Relative to providing alternative diagnoses while solving cases, structured reflection increased medical students' interest and may therefore be a useful tool for teachers concerned with enhancing students' motivation for learning. Surprisingly, easy cases promoted higher situational interest despite the higher awareness of knowledge gaps on difficult cases. This suggests the potential for case difficulty to inhibit students' interest in learning, a possibility that demands further investigation.

Randomized Controlled Pilot Study of Video Self-assessment for Resident Mastoidectomy Training

A prospective randomized controlled pilot study was performed to determine if video self-assessment improves competency in mastoidectomy and to assess interrater agreement between expert and resident evaluations of recorded mastoidectomy. Sixteen otolaryngology residents were recorded while performing cadaveric mastoidectomy and randomized into video self-assessment and control groups. All residents performed a second recorded mastoidectomy. Performance was evaluated by blinded experts with a validated assessment scale. Video self-assessment did not lead to greater skill improvement between the first and second mastoidectomy. Interrater agreement was fair to substantial between the expert evaluators and between resident self-evaluations by recall and video review. Agreement between experts and residents was only slight to fair; residents consistently rated their performance higher than experts (P < .05). In conclusion, 1 session of video self-review did not lead to improved competence in mastoidectomy over standard practice. While experts agree on assessments, residents may overestimate their competency in performing cadaveric mastoidectomy.

Collaboration and Teamwork in the Health Professions: Rethinking the Role of Conflict

Whereas the business professions have long recognized that conflict can be a source of learning and innovation, the health professions still tend to view conflict negatively as being disruptive, inefficient, and unprofessional. As a consequence, the health professions tend to avoid conflict or resolve it quickly. This neglect to appreciate conflict's positive attributes appears to be driven in part by (1) individuals' fears about being negatively perceived and the potential negative consequences in an organization of being implicated in conflict, (2) constrained views and approaches to professionalism and to evaluation and assessment, and (3) lingering autocracies and hierarchies of power that view conflict as a disruptive threat.The author describes changing perspectives on collaboration and teamwork in the health professions, discusses how the health professions have neglected to appreciate the positive attributes of conflict, and presents three alternative approaches to more effectively integrating conflict into collaboration and teamwork in the health professions. These three approaches are (1) cultivating psychological safety on teams to make space for safe interpersonal risk taking, (2) viewing conflict as a source of expansive learning and innovation (via models such as activity theory), and (3) democratizing hierarchies of power through health humanities education ideally by advancing the health humanities to the core of the curriculum.The author suggests that understanding conflict's inevitability and its innovative potential, and integrating it into collaboration and teamwork, may have a reassuring and emancipating impact on individuals and teams. This may ultimately improve performance in health care organizations.

Narrative Derived From Medical Student Reflection in Action: Lessons Learned and Implications for Assessment

A process and tool that prompts learners to think about and reflect on their clinical performance was implemented. Learner narrative reflections about their work and faculty feedback, both captured in the moment, provided data for decisions about level of performance in a competency-based assessment system.

The impact of informed self-assessment on the development of medical students' learning goals

PURPOSE

This study investigates the contributions of self-assessment (SA) and external feedback on the development of learning goals (LG) and the influence on LG recall and implementation in medical students.

METHODS

Following a standardized patient (SP) assessment, 168 pre-clinical medical students completed a SA, received SP feedback and created a LG. LG were categorized by source. Two weeks later, students recalled LG and described implementation. Chi-squared analyses were used to test the associations.

RESULTS

SA influenced LG for 82.8% of students whereas SP feedback influenced LG for 45.9%. Students rarely generated LG based on SA when they received discordant feedback (5.4%), but sometimes incorporated feedback discordant from their SA into LG (14.9%). Students who created LG based on SP feedback were more likely to recall LG than those who created LG based on SA, 89.7 versus 67.6%, p < 0.05 and implement their LG, 72.4 versus 48.9%, χ²(1) = 5.3, p = 0.017. Students who reported receiving effective feedback were more likely to implement their LG than those reporting adequate feedback, 60.9 versus 37.9%, χ²(1) = 8.0, p = 0.01.

CONCLUSIONS

SA is an essential part of goal setting and subsequent action. Perception of feedback plays a crucial role in LG implementation.

What should we teach the teachers? Identifying the learning priorities of clinical supervisors

Clinicians who teach are essential for the health workforce but require faculty development to improve their educational skills. Curricula for faculty development programs are often based on expert frameworks without consideration of the learning priorities as defined by clinical supervisors themselves. We sought to inform these curricula by highlighting clinical supervisors own requirements through answering the research question: what do clinical supervisors identify as relative strengths and areas for improvement in their teaching practice? This mixed methods study employed a modified version of the Maastricht Clinical Teaching Questionnaire (mMCTQ) which included free-text reflections. Descriptive statistics were calculated and content analysis was conducted on textual comments. 481 (49%) of 978 clinical supervisors submitted their mMCTQs and associated reflections for the research study. Clinical supervisors self-identified relatively strong capability with interpersonal skills or attributes and indicated least capability with assisting learners to explore strengths, weaknesses and learning goals. The qualitative category 'establishing relationships' was the most reported strength with 224 responses. The qualitative category 'feedback' was the most reported area for improvement, with 151 responses. Key areas for curricular focus include: improving feedback practices; stimulating reflective and agentic learning; and managing the logistics of a clinical education environment. Clinical supervisors' self-identified needs provide a foundation for designing engaging and relevant faculty development programs.

Peer group reflection on student ratings stimulates clinical teachers to generate plans to improve their teaching

BACKGROUND

Previous studies have demonstrated that student ratings of a teachers' performance do not incentivize clinical teachers to reflect critically and generate plans to improve their teaching. Peer group reflection might offer a solution in mediating this change.

AIM

To investigate: (a) to which extent clinical teachers perceive self-evaluation, student ratings and peer group reflection effective; and (b) whether additional peer group reflection fosters critical reflection and the translation of feedback into concrete plans of action.

METHOD

We conducted a quasi-experiment, inviting two groups of 10 clinical teachers each (1) to complete a self-evaluation and (2) subsequently examine their student ratings. One group participated in (3) an additional peer group reflection meeting. All participants were finally requested to define plans for improvement and evaluate each activity's effectiveness.

RESULTS

Participants perceived all three activities to be effective. Levels of reflection did not differ across the two groups. However, participation in peer group reflection did result in generating more concrete plans to change clinical teaching.

CONCLUSIONS

Peer group reflection on student ratings shows promise as tool to assist clinical teachers in generating plans for improvement. Future research should focus on whether teaching indeed improves with the introduction of peer group reflection.

Coaching Experts: Applications to Surgeons and Continuing Professional Development

Surgery is a science and an art, which is mastered through years of training and refined by the accumulation of individual experience and preference. Continuing professional development (CPD) is a concept that emphasizes a self-directed approach to education. Coaching is a process that leads to increased utilization of a person's current skills and resources without counselling or advising. Coaching in surgery could be used to facilitate and optimize feedback and reflection, thus enhancing performance and outcomes through elite performance of an operative procedure. Therefore, it can be applied under the umbrella of CPD. Ultimately also emphasizing that better quality surgery is not necessarily purely based on technical outcomes, it is a combination of both technical and nontechnical practice. Coaching of surgeons is a conceptually formidable tool in the successful implementation of effective CPD programs. CPD currently provides an opportunity for surgeons to gain access to constantly evolving medical knowledge and technique; however, there is no accountability to its understanding or implementation. Coaches have the potential to provide confidential appraisal and feedback in a constructive approach with the aim to eliminate any barriers to the transfer of technique and knowledge.

Educational Technologies for Physician Continuous Professional Development: A National Survey

PURPOSE

To determine the past experiences with, current use of, and anticipated use of online learning and simulation-based education among practicing U.S. physicians, and how findings vary by age.

METHOD

The authors surveyed 4,648 randomly sampled board-certified U.S. physicians, September 2015 to April 2016, using Internet-based and paper questionnaires. Survey items (some optional) addressed past and current technology usage, perceived technology effectiveness, and anticipated future use of specific technology innovations.

RESULTS

Of 988 respondents, 444 completed optional items. Of these, 429/442 (97.1%) had used online learning and 372/442 (84.2%) had used simulation-based education in the past five years. Desire for more online learning was modest (mean [standard deviation], 4.6 [1.5]; 1 = strongly disagree, 7 = strongly agree), as was desire for more simulation-based education (4.2 [1.7]). Both online learning and simulation-based education were perceived as effective (5.2 [1.4]; 5.0 [1.4]). Physicians believed they possess adequate skills for online learning (5.8 [1.2]) and that point-of-care learning is vital to effective patient care (5.3 [1.3]). Only 39.0% used objective performance data to guide their learning choices, although 64.6% agreed that such information would be useful. The highest-rated innovations included a central repository for listing educational opportunities and tracking continuing education credits, an app to award credit for answering patient-focused questions, 5-minute and 20-minute clinical updates, and an e-mailed "question of the week." Responses to most survey items were similar across age groups.

CONCLUSIONS

Practicing physicians generally seem receptive and prepared to use a variety of educational technologies, regardless of age.

Assessment and Change: An Exploration of Documented Assessment Activities and Outcomes by Canadian Psychiatrists

INTRODUCTION

Fellows of the Royal College of Physicians and Surgeons of Canada are required to participate in assessment activities for all new 5-year cycles beginning on or after January 2014 to meet the maintenance of certification program requirements. This study examined the assessment activities which psychiatrists reported in their maintenance of certification e-portfolios to determine the types and frequency of activities reported; the resultant learning, planned learning, and/or changes to the practice they planned or implemented; and the interrelationship between the types of assessment activities, learning that was affirmed or planned, and changes planned or implemented.

METHODS

A total of 5000 entries from 2195 psychiatrists were examined. A thematic analysis drawing on the framework analysis was undertaken of the 2016 entries.

RESULTS

There were 3841 entries for analysis; 1159 entries did not meet the criteria for assessment. The most commonly reported activities were self-assessment programs, feedback on teaching, regular performance reviews, and chart reviews. Less frequent were direct observation, peer supervision, and reviews by provincial medical regulatory authorities. In response to the data, psychiatrists affirmed that their practices were appropriate, identified gaps they intended to address, planned future learning, and/or planned or implemented changes. The assessment activities were internally or externally initiated and resulted in no or small changes (accommodations and adjustments) or redirections.

DISCUSSION

Psychiatrists reported participating in a variety of assessment activities that resulted in variable impact on learning and change. The study underscores the need to ensure that assessments being undertaken are purposeful, relevant, and designed to enable identification of outcomes that impact practice.

CPD of the future: a partnership between quality improvement and competency-based education

CONTEXT

Many of those involved in continuing professional development (CPD) over the past 10 years have engaged in discussions about its goals and activities. Whereas in the past CPD was viewed as an education intervention directed towards the medical expert role, recent research highlights the need to expand the scope of CPD and to promote its more explicit role in improving patient care and health outcomes. Recent developments in quality improvement (QI) and competency-based medical education (CBME), guided by appropriate theories of learning and change, can shed light on how the field might best advance. This paper describes principles of QI and CBME and how they might contribute to CPD, explores theoretical perspectives that inform such an integration and suggests a future model of CPD.

DISCUSSION

Continuing professional development seeks to improve patient outcomes by increasing physician knowledge and skills and changing behaviours, whereas QI takes the approach of system and process change. Combining the strengths of a CPD approach with strategies known to be effective from the field of QI has the potential to harmonise the contributions of each, and thereby to lead to better patient outcomes. Similarly, competency-based CPD is envisioned to place health needs and patient outcomes at the centre of a CPD system that will be guided by a set of competencies to enhance the quality of practice and the safety of the health system.

CONCLUSIONS

We propose that the future CPD system should adhere to the following principles: it should be grounded in the everyday workplace, integrated into the health care system, oriented to patient outcomes, guided by multiple sources of performance and outcome data, and team-based; it should employ the principles and strategies of QI, and should be taken on as a collective responsibility by physicians, CPD provider organisations, regulators and the health system.

A Randomized Trial of SMART Goal Enhanced Debriefing after Simulation to Promote Educational Actions

Introduction

Goal setting is used in education to promote learning and performance. Debriefing after clinical scenario-based simulation is a well-established practice that provides learners a defined structure to review and improve performance. Our objective was to integrate formal learning goal generation, using the SMART framework (Specific, Measurable, Attainable, Realistic, and Time-bound), into standard debriefing processes (i.e., “SMART Goal Enhanced Debriefing”) and subsequently measure the impact on the development of learning goals and execution of educational actions.

Methods

This was a prospective multicenter randomized controlled study of 80 emergency medicine residents at three academic hospitals comparing the effectiveness of SMART Goal Enhanced Debriefing to a standard debriefing. Residents were block randomized on a rolling basis following a simulation case. SMART Goal Enhanced Debriefing included five minutes of formal instruction on the development of SMART learning goals during the summary/application phase of the debrief. Outcome measures included the number of recalled learning goals, self-reported executed educational actions, and quality of each learning goal and educational action after a two-week follow-up period.

Results

The mean number of reported learning goals was similar in the standard debriefing group (mean 2.05 goals, SD 1.13, n=37 residents), and in the SMART Goal Enhanced Debriefing group (mean 1.93, SD 0.96, n=43), with no difference in learning goal quality. Residents receiving SMART Goal Enhanced Debriefing completed more educational actions on average (Control group actions completed 0.97 (SD 0.87), SMART debrief group 1.44 (SD 1.03) p=0.03).

Conclusion

The number and quality of learning goals reported by residents was not improved as a result of SMART Goal Enhanced Debriefing. Residents did, however, execute more educational actions, which is consistent with the overarching intent of any educational intervention.

Efficacy of an asynchronous electronic curriculum in emergency medicine education in the United States

PURPOSE

The aim of this study was to measure the effect of an iPad-based asynchronous curriculum on emergency medicine resident performance on the in-training exam (ITE). We hypothesized that the implementation of an asynchronous curriculum (replacing 1 hour of weekly didactic time) would result in non-inferior ITE scores compared to the historical scores of residents who had participated in the traditional 5-hour weekly didactic curriculum.

METHODS

The study was a retrospective, non-inferiority study. conducted at the University of California, Irvine Emergency Medicine Residency Program. We compared ITE scores from 2012 and 2013, when there were 5 weekly hours of didactic content, with scores from 2014 and 2015, when 1 hour of conference was replaced with asynchro-nous content. Examination results were compared using a non-inferiority data analysis with a 10% margin of difference.

RESULTS

Using a non-inferiority test with a 95% confidence interval, there was no difference between the 2 groups (before and after implementation of asynchronous learning), as the confidence interval for the change of the ITE was -3.5 to 2.3 points, whereas the 10% non-inferiority margin was 7.8 points.

CONCLUSION

Replacing 1 hour of didactic conference with asynchronous learning showed no negative impact on resident ITE scores.