The effects of self-assessment and supervisor feedback on residents' patient-education competency using videoed outpatient consultations

Radiologist preferences for faculty development initiatives to improve resident feedback in the era of competency-based medical education

INTRODUCTION

Since 2022, all Canadian post-graduate medical programs have transitioned to a Competence by Design (CBD) model within a Competency-Based Medical Education (CBME) framework. The CBME model emphasized more frequent, formative assessment of residents to evaluate their progress towards predefined competencies in comparison to traditional medical education models. Faculty members therefore have increased responsibility for providing assessments to residents on a more regular basis, which has associated challenges. Our study explores faculty assessment behaviours within the CBD framework and assesses their openness to opportunities aimed at improving the quality of written feedback. Specifically, we explore faculty's receptiveness to routine metric performance reports that offer comprehensive feedback on their assessment patterns.

METHODS

Online surveys were distributed to all 28 radiology faculty at Queen's University. Data were collected on demographics, feedback practices, motivations for improving the teacher-learner feedback exchange, and openness to metric performance reports and quality improvement measures. Following descriptive statistics, unpaired t-tests and one-way analysis of variance were conducted to compare groups based on experience and subspecialty.

RESULTS

The response rate was 89% (25/28 faculty). 56% of faculty were likely to complete evaluations after working with a resident. Regarding the degree to which faculty felt written feedback is important, 62% found it at least moderately important. A majority (67%) believed that performance reports could influence their evaluation approach, with volume of written feedback being the most likely to change. Faculty expressed interest in feedback-focused development opportunities (67%), favouring Grand Rounds and workshops.

CONCLUSION

Assessment of preceptor perceptions reveals that faculty recognize the importance of offering high-quality written feedback to learners. Faculty openness to quality improvement interventions for curricular reform relies on having sufficient time, knowledge, and skills for effective assessments. This suggests that integrating routine performance metrics into faculty assessments could serve as a catalyst for enhancing future feedback quality.

"Bring your worst": Residents' perspectives on video review of challenging patient communication as a learning tool

Objective

To investigate residents' experiences recording and receiving feedback on a challenging video of a patient encounter. Methods: We used a qualitative design with first year residents who took part in a mandatory communication skills course in which all participants were asked to bring a challenging video of a patient encounter. The methods consisted of brief reflection texts and focus groups related to their perspectives on the use of challenging videos. Results: 106 residents wrote brief reflection texts, and 13 residents participated in four focus groups. Residents mainly expressed positive experiences with the challenging video exercise. Residents reported that the pressure to perform was felt to be less than on previous teaching sessions because the focus was on choosing an encounter which was less than perfect. They also reported that they appreciated the opportunity to see that other doctors were not performing optimally. Conclusion: The use of challenging videos as a teaching method for communication skills was experienced as encouraging by residents and facilitated enhanced learning.

Innovation

We recommend adding more focus on challenging situations in video review. This could support learning by providing what our participants found to be a less daunting learning environment.

Can written prompts help medical residents to accurately monitor their own communication skills and those of others?

In healthcare, effective communication in complex situations such as end of life conversations is critical for delivering high quality care. Whether residents learn from communication training with actors depends on whether they are able to select appropriate information or 'predictive cues' from that learning situation that accurately reflect their or their peers' performance and whether they use those cues for ensuing judgement. This study aimed to explore whether prompts can help medical residents improving use of predictive cues and judgement of communication skills. First and third year Kenyan residents (N = 41) from 8 different specialties were randomly assigned to one of two experimental groups during a mock OSCE assessing advanced communication skills. Residents in the intervention arm received paper predictive cue prompts while residents in the control arm received paper regular prompts for self-judgement. In a pre- and post- test, residents' use of predictive cues and the appropriateness of peer-judgements were evaluated against a pre-rated video of another resident. The intervention improved both the use of predictive cues in self-judgement and peer-judgement. Ensuing accuracy of peer-judgements in the pre- to post-test only partly improved: no effect from the intervention was found on overall appropriateness of judgements. However, when analyzing participants' completeness of judgements over the various themes within the consultation, a reduction in inappropriate judgments scores was seen in the intervention group. In conclusion, predictive cue prompts can help learners to concentrate on relevant cues when evaluating communication skills and partly improve monitoring accuracy. Future research should focus on offering prompts more frequently to evaluate whether this increases the effect on monitoring accuracy in communication skills.

Exploring the use of metacognitive monitoring cues following a diagram completion intervention

Studying texts constitutes a significant part of student learning in health professions education. Key to learning from text is the ability to effectively monitor one's own cognitive performance and take appropriate regulatory steps for improvement. Inferential cues generated during a learning experience typically guide this monitoring process. It has been shown that interventions to assist learners in using comprehension cues improve their monitoring accuracy. One such intervention is having learners to complete a diagram. Little is known, however, about how learners use cues to shape their monitoring judgments. In addition, previous research has not examined the difference in cue use between categories of learners, such as good and poor monitors. This study explored the types and patterns of cues used by participants after being subjected to a diagram completion task prior to their prediction of performance (PoP). Participants' thought processes were studied by means of a think-aloud method during diagram completion and the subsequent PoP. Results suggest that relying on comprehension-specific cues may lead to a better PoP. Poor monitors relied on multiple cue types and failed to use available cues appropriately. They gave more incorrect responses and made commission errors in the diagram, which likely led to their overconfidence. Good monitors, on the other hand, utilized cues that are predictive of learning from the diagram completion task and seemed to have relied on comprehension cues for their PoP. However, they tended to be cautious in their judgement, which probably made them underestimate themselves. These observations contribute to the current understanding of the use and effectiveness of diagram completion as a cue-prompt intervention and provide direction for future research in enhancing monitoring accuracy.

A scoping review of self-monitoring in graduate medical education

BACKGROUND

Physicians and physicians-in-training have repeatedly demonstrated poor accuracy of global self-assessments, which are assessments removed from the context of a specific task, regardless of any intervention. Self-monitoring, an in-the-moment self-awareness of one's performance, offers a promising alternative to global self-assessment. The purpose of this scoping review is to better understand the state of self-monitoring in graduate medical education.

METHODS

We performed a scoping review following Arksey and O'Malley's six steps: identifying a research question, identifying relevant studies, selecting included studies, charting the data, collating and summarising the results and consulting experts. Our search queried Ovid Medline, Web of Science, PsychINFO, Eric and EMBASE databases from 1 January 1999 to 12 October 2022.

RESULTS

The literature search yielded 5363 unique articles. The authors identified 77 articles for inclusion. The search process helped create a framework to identify self-monitoring based on time and context dependence. More than 20 different terms were used to describe self-monitoring, and only 13 studies (17%) provided a definition for the equivalent term. Most research focused on post-performance self-judgements of a procedural skill (n = 31, 42%). Regardless of task, studies focused on self-judgement (n = 66, 86%) and measured the accuracy or impact on performance of self-monitoring (n = 41, 71%). Most self-monitoring was conducted post-task (n = 65, 84%).

CONCLUSION

Self-monitoring is a time- and context-dependent phenomenon that seems promising as a research focus to improve clinical performance of trainees in graduate medical education and beyond. The landscape of current literature on self-monitoring is sparse and heterogeneous, suffering from a lack of theoretical underpinning, inconsistent terminology and insufficiently clear definitions.

Teaching and assessing communication skills in the postgraduate medical setting: a systematic scoping review

BACKGROUND

Poor communication skills can potentially compromise patient care. However, as communication skills training (CST) programs are not seen as a priority to many clinical departments, there is a discernible absence of a standardised, recommended framework for these programs to be built upon. This systematic scoping review (SSR) aims to gather prevailing data on existing CSTs to identify key factors in teaching and assessing communication skills in the postgraduate medical setting.

METHODS

Independent searches across seven bibliographic databases (PubMed, PsycINFO, EMBASE, ERIC, CINAHL, Scopus and Google Scholar) were carried out. Krishna's Systematic Evidence-Based Approach (SEBA) was used to guide concurrent thematic and content analysis of the data. The themes and categories identified were compared and combined where possible in keeping with this approach and then compared with the tabulated summaries of the included articles.

RESULTS

Twenty-five thousand eight hundred ninety-four abstracts were identified, and 151 articles were included and analysed. The Split Approach revealed similar categories and themes: curriculum design, teaching methods, curriculum content, assessment methods, integration into curriculum, and facilitators and barriers to CST. Amidst a wide variety of curricula designs, efforts to develop the requisite knowledge, skills and attitudes set out by the ACGME current teaching and assessment methods in CST maybe categorised into didactic and interactive methods and assessed along Kirkpatrick's Four Levels of Learning Evaluation.

CONCLUSIONS

A major flaw in existing CSTs is a lack of curriculum structure, focus and standardisation. Based upon the findings and current design principles identified in this SSR in SEBA, we forward a stepwise approach to designing CST programs. These involve 1) defining goals and learning objectives, 2) identifying target population and ideal characteristics, 3) determining curriculum structure, 4) ensuring adequate resources and mitigating barriers, 5) determining curriculum content, and 6) assessing learners and adopting quality improvement processes.

Effectiveness of online communication skills training for cancer and palliative care health professionals: A systematic review

OBJECTIVE

To determine the reported effect of online communication skills training (CST) on health professional (HP) communication skills and patient care outcomes in cancer and palliative care.

METHODS

Primary research published in English between January 2003 and April 2019 was identified in bibliographic databases including Medline, Embase and Proquest (Prospero: CRD42018088681). An integrated mixed-method approach included studies describing a CST intervention and its effect, for cancer or palliative care HPs, delivered online or blended with an online component. Included studies' outcomes were categorised then findings were stratified by an evaluation framework and synthesised in an effect direction plot. Risk of bias was assessed using Joanna Briggs Institute's tools.

RESULTS

Nineteen included studies (five randomised controlled trials, 11 pre-post, two post-test and one qualitative study) evaluated a CST intervention (median duration = 3.75 h; range 0.66-96 h) involving 1116 HPs, 422 students and 732 patients. Most interventions taught communication skills for specific scenarios and approximately half were delivered solely online and did not involve role plays. Online CST improved HPs' self-assessed communication skills (three studies, 215 participants), confidence (four studies, 533 participants), and objective knowledge (five studies, 753 participants). While few studies evaluated patient outcomes, CST may benefit observed communication skills in care settings (two studies, 595 participants).

CONCLUSIONS

Online CST benefits oncology HPs' subjectively-reported communication skills and confidence, and objective knowledge. Translation to patient outcomes requires further investigation. The quality of research varied and few studies had a control group. We recommend improvements to study design, evaluation and implementation.

Pilot project to assess and improve neurosurgery resident and staff perception of feedback to residents for self-improvement goal formation

OBJECTIVE

The Accreditation Council for Graduate Medical Education (ACGME) has pushed for more frequent and comprehensive feedback for residents during their training, but there is scant evidence for how neurosurgery residents view the current feedback system as it applies to providing information for self-improvement and goal formation. The authors sought to assess neurosurgery resident and staff perceptions of the current resident feedback system in providing specific, meaningful, achievable, realistic, and timely (SMART) goals. The authors then created a pilot project to improve the most unfavorably viewed aspect of the feedback system.

METHODS

The authors conducted an anonymous survey of neurosurgery residents and staff at an academic medical institution to assess SMART goals for resident feedback and used the results to create a pilot intervention to address the most unfavorably viewed aspect of the feedback system. The authors then conducted a postintervention survey to see if perceptions had improved for the target of the intervention.

RESULTS

Neurosurgery residents and staff completed an anonymous online survey, for which the results indicated that resident feedback was not occurring in a timely manner. The authors created a simple anonymous feedback form. The form was distributed monthly to neurosurgery residents, neurosurgical staff, and nurses, and the results were reported monthly to each resident for 6 months. A postintervention survey was then administered, and the results indicated that the opinions of the neurosurgery residents and staff on the timeliness of resident feedback had changed from a negative to a nonnegative opinion (p = 0.01).

CONCLUSIONS

The required ACGME feedback methods may not be providing adequate feedback for goal formation for self-improvement for neurosurgery residents. Simple interventions, such as anonymous feedback questionnaires, can improve neurosurgery resident and staff perception of feedback to residents for self-improvement and goal formation.

[Accomplishment of preventive activities proposed by resident doctors of Family Medicine in Primary Care and its relationship with communication skills]

OBJECTIVE

To determine the offer of preventive activities by resident physicians of family medicine in the Primary Care consultations and the relation with their communication habilities.

DESIGN

A descriptive multicentre study assessing medical consultations video recording.

LOCATION

Eight Primary Healthcare centres in Jaen (Andalucia).

PARTICIPANTS

Seventy-three resident physicians (4th year) filmed and observed with patients.

PRINCIPAL MEASUREMENTS

Offer of preventive activities (according to the Spanish Program of Preventive Activities and Health Promotion -PAPPS-). Doctor, patient and consultation characteristics. Peer-review of the communication between physicians and patients, using a CICAA scale. A descriptive, bivariate, logistic regression analysis was performed.

RESULTS

Two hundred and sixty interviews were evaluated (duration 8.5±4.0min) of 73 residents (50.7% women, mean age 32.9±7.7 years, 79% urban environment). The patient is more frequently a woman (60%) who comes alone (72%) due to acute processes (80%) and with 2.1±1.0 demands. Preventive activities are offered in 47% (duration less than one minute) of primary (70%) and secondary (59%) prevention, offered through advice (72%) or screening (52%) and focused on the cardiovascular area (52%) and lifestyles (53%). Eighty percent related to the patient's reason for consultation. Communication skills 41% improvable, 26% adequate, 23% excellent. The offer of preventive activities is related to the duration of the consultation (OR=1.1, 95% CI 1.01; 1.16) and communication skills (OR=1.03, 95% CI 1.01; 1.10).

CONCLUSIONS

Preventive activities are carried out in almost half of the consultations, although focused on advice and screening and linked to the patient's demand. Consultation time and communication skills favor a greater preventive offer.

Interdisciplinary video review: Assessing milestones and providing feedback

Literature within residency education states that directly observing resident–patient visits with the goal of providing formative and summative feedback to learners is helpful for resident skill development. However, limited literature exists regarding what specifically is most effective to observe and evaluate. Furthermore, the perspectives of learners are not always taken into consideration in the development and implementation of direct observation or video review of resident–patient encounters. This article overviews some of the current literature relevant to family medicine training and provides a description of some of the changes in one residency’s use of recorded encounters. Suggestions are provided for future steps for family medicine residencies to effectively utilize video review.

Assessment Tools for Feedback and Entrustment Decisions in the Clinical Workplace: A Systematic Review

BACKGROUND: Entrustable Professional Activities (EPAs) combine feedback and evaluation with a permission to act under a specified level of supervision and the possibility to schedule learners for clinical service. This literature review aims to identify workplace-based assessment tools that indicate progression toward unsupervised practice, suitable for entrustment decisions and feedback to learners. METHODS: A systematic search was performed in the PubMed, Embase, ERIC, and PsycINFO databases. Based on title/abstract and full text, articles were selected using predetermined inclusion and exclusion criteria. Information on workplace-based assessment tools was extracted using data coding sheets. The methodological quality of studies was assessed using the medical education research study quality instrument (MERSQI). RESULTS: The search yielded 6,371 articles (180 were evaluated in full text). In total, 80 articles were included, identifying 67 assessment tools. Only a few studies explicitly mentioned assessment tools used as a resource for entrustment decisions. Validity evidence was frequently reported, and the MERSQI score was 10.0 on average. CONCLUSIONS: Many workplace-based assessment tools were identified that potentially support learners with feedback on their development and support supervisors with providing feedback. As expected, only few articles referred to entrustment decisions. Nevertheless, the existing tools or the principals could be used for entrustment decisions, supervision level, or autonomy.

Video Review Process: Enhancing Impact and Resident/Faculty Satisfaction

Introduction

Although video review (VR) is a common component of family medicine (FM) resident education, levels of faculty and resident satisfaction and the perceived impact of the VR process are unknown. Our FM residency program made several changes to our VR process, including a new feedback form highlighting strengths and areas for growth, a post-VR email to the resident including a tip sheet on an identified skill, and follow-up shadowing by a behavioral health clinician to increase VR impact and reinforce skill development.

Methods

FM residency faculty (n=11) and residents (n=34) completed an anonymous 20-item online survey. The survey was administered twice, first at baseline and then 9 months after the changes were instituted. Survey questions targeted the preceding 9 months and assessed satisfaction with VRs and perceived value and impact of the VR process.

Results

At baseline, both faculty and residents reported low levels of satisfaction and perceived impact and value of the VR process. Among both residents and faculty, significant improvements emerged in ratings of satisfaction and perceived value of the VR process. At postsurvey, significantly more residents reported feeling equipped with specific tips on how to improve their communication skills after VR than at baseline.

Conclusion

Our evaluation documented significant improvements in satisfaction and perceived value in our VR process following the changes in our procedures. Future research could develop measures to assess communication skills that are more sensitive to change, thereby strengthening this line of research by supplementing self-report data with more objective observational data.

Benefits and Barriers of Cancer Practitioners Discussing Physical Activity with their Cancer Patients

Our aim was to synthesise the existing empirical literature and theoretical perspectives on the physical activity (PA) promotion practices and determinants of cancer clinicians and health professionals. We conducted a narrative review of theory and evidence to develop practice recommendations for improving the promotion of PA to cancer patients. Surveyed health professionals were aware of many benefits of PA for their cancer patients, although only ~40 % promoted PA to selected cancer patients. Walking was the most commonly promoted form of PA, with this promoted to assist patients control their weight and cardiovascular health risk. Barriers to promotion of PA included lack of time and knowledge of PA and behaviour change skills. Health professionals appear interested in promoting PA to their cancer patients, yet encounter several barriers. Further research is warranted to assist health professionals improve their PA promotion. An adapted reflective-impulsive model of social behaviour shows promise for assisting health professionals overcome barriers and provides an evidence-based theoretical framework for improving communication with patients. Universities, hospitals and/or health-care accreditation organisations also have important roles to play in assisting health professionals improve their promotion of PA to patients.

Interprofessional simulation to improve patient participation in transitional care

BACKGROUND

Educating and training healthcare professionals is known to improve the quality of transitional care for older adults. Arranging interprofessional meetings for healthcare professionals might be useful to improve patient participation skills in transitional care.

AIM

To describe the learning activities used in The Meeting Point programme, focusing on patient participation in transitional care, and assess whether they increase healthcare professionals' awareness of and competencies relating to patient participation in the transitional care of older patients.

DESIGN

Data were collected as part of an educational intervention programme, The Meeting Point, including three seminars on 'Patient participation in the transitional care of older patients' and four follow-up meetings. Participants were nurses, care assistants, doctors, physiotherapists, patient coordinators and administrative personnel from hospital, nursing homes and home-based care services.

METHOD

The Meeting Point was organised around four pillars: introduction, teaching session, group work activity and plenary discussion. Qualitative data included log reports, summaries of meetings, notes from group work activities, and reports from participants and from follow-up meetings.

RESULTS

Feedback from participants shows that they were satisfied with meeting healthcare professionals from other units of care. A film scenario was perceived relevant for group work activity and useful in focusing participants' attention to patient participation. Follow-up meetings show that some nursing home wards, the emergency department and one medical ward at the hospital continued with ongoing work to improve quality of care. Efforts included implementation of an observational waiting room with comfortable chairs, planning for discharge in hospital admission, a daily patient flow registration system and motivational interviewing during admission to nursing home.

CONCLUSIONS

The description of the learning activities used at The Meeting Point seminars shows that they were useful to increase awareness of and competencies on patient participation in transitional care.

Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice

As a part of a special collection in this issue of Academic Medicine, which is focused on mastery learning in medical education, this Perspective describes how the expert-performance approach with deliberate practice is consistent with many characteristics of mastery learning. Importantly, this Perspective also explains how the expert-performance approach provides a very different perspective on the acquisition of skill. Whereas traditional education with mastery learning focuses on having students attain an adequate level of performance that is based on goals set by the existing curricula, the expert-performance approach takes an empirical approach and first identifies the final goal of training-namely, reproducibly superior objective performance (superior patient outcomes) for individuals in particular medical specialties. Analyzing this superior complex performance reveals three types of mental representations that permit expert performers to plan, execute, and monitor their own performance. By reviewing research on medical performance and education, the author describes evidence for these representations and their development within the expert-performance framework. He uses the research to generate suggestions for improved training of medical students and professionals. Two strategies-designing learning environments with libraries of cases and creating opportunities for individualized teacher-guided training-should enable motivated individuals to acquire a full set of refined mental representations. Providing the right resources to support the expert-performance approach will allow such individuals to become self-regulated learners-that is, members of the medical community who have the tools to improve their own and their team members' performances throughout their entire professional careers.

Supervisors' and residents' patient-education competency in challenging outpatient consultations

OBJECTIVES

We compared supervisors' and residents' patient-education competency in challenging consultations in order to establish whether supervisors demonstrate sufficient patient-education competency to act credibly as role models and coaches for residents.

METHODS

All consultations conducted at one, two, or three of the outpatient clinics of each of the participating physicians were videoed. Each participant selected two challenging consultations from each clinic for assessment. We assessed their patient-education competency using the CELI instrument, we calculated net consultation length for all videoed consultations and we measured patient opinion about the patient education received using a questionnaire.

RESULTS

Forty-four residents and fourteen supervisors participated in the study. They selected 230 consultations for assessment. On average, supervisors and residents demonstrated similar patient-education competency. Net consultation length was longer for supervisors. Patient opinion did not differ between supervisors and residents.

CONCLUSIONS

Supervising consultants generally do not possess sufficient patient-education competency to fulfill their teaching roles in workplace-based learning that is aimed at improving residents' patient-education competency.

PRACTICE IMPLICATIONS

Not only residents but also supervising consultants should improve their patient-education competency. Workplace-based learning consisting of self-assessment of and feedback on videoed consultations could be useful in attaining this goal.