Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review

Exploring Interprofessional Development of Entrustable Professional Activities For Pediatric Intensive Care Fellows: A Proof-of-Concept Study

Phenomenon: Entrustable professional activities (EPAs) delineate major professional activities that an individual in a given specialty must be "entrusted" to perform, ultimately without supervision, to provide quality patient care. Until now, most EPA frameworks have been developed by professionals within the same specialty. As safe, effective, and sustainable health care ultimately depends on interprofessional collaboration, we hypothesized that members of interprofessional teams might have clear and possibly additional insight into which activities are essential to the professional work of a medical specialist. Approach: We recently employed a national modified Delphi study to develop and validate a set of EPAs for Dutch pediatric intensive care fellows. In this proof-of-concept study, we explored what pediatric intensive care physicians' non-physician team members (physician assistants, nurse practitioners, and nurses) constitute as essential professional activities for PICU physicians and how they regarded the newly developed set of nine EPAs. We compared their judgments with the PICU physicians' opinions. Findings: This study shows that non-physician team members share a mental model with physicians about which EPAs are indispensable for pediatric intensive care physicians. Despite this agreement however, descriptions of EPAs are not always clear for non-physician team members who have to work with them on a daily basis. Insights: Ambiguity as to what an EPA entails when qualifying a trainee can have implications for patient safety and trainees themselves. Input from non-physician team members may add to the clarity of EPA descriptions. This finding supports the involvement of non-physician team members in the developmental process of EPAs for (sub)specialty training programs.

Reducing ' probably benign ' assessments in normal mammograms: The role of radiologist experience

Rationale and objectives

to investigate the relationship between radiologists' experience in reporting mammograms, their caseloads, and the classification of category '3' or 'Probably Benign' on normal mammograms.

Materials and Methods

A total of 92 board-certified radiologists participated. Self-reported parameters related to experience, including age, years since qualifying as a radiologist, years of experience reading mammograms, number of mammograms read per year, and hours spent reading mammograms per week, were documented. To assess the radiologists' accuracy, "Probably Benign fractions" was calculated by dividing the number of "Probably Benign findings" given by each radiologist in the normal cases by the total number of normal cases Probably Benign fractions were correlated with various factors, such as the radiologists' experience.

Results

The results of the statistical analysis revealed a significant negative correlation between radiologist experience and 'Probably Benign' fractions for normal images. Specifically, for normal cases, the number of mammograms read per year (r = -0.29, P = 0.006) and the number of mammograms read over the radiologist's lifetime (r = -0.21, P = 0.049) were both negatively correlated with 'Probably Benign' fractions.

Conclusion

The results indicate that a relationship exists between increased reading volumes and reduced assessments of 'Probably Benign' in normal mammograms. The implications of these findings extend to the effectiveness of screening programs and the recall rates.

Catalytic effect of multisource feedback for trauma team captains: a mixed-methods prospective study

OBJECTIVES

To evaluate the impact and feasibility of multisource feedback compared with traditional feedback for trauma team captains (TTCs).

DESIGN

A mixed-methods, non-randomised prospective study.

SETTING

A level one trauma centre in Ontario, Canada.

PARTICIPANTS

Postgraduate medical residents in emergency medicine and general surgery participating as TTCs. Selection was based on a convenience sampling method.

INTERVENTION

Postgraduate medical residents participating as TTCs received either multisource feedback or standard feedback following trauma cases.

MAIN OUTCOME MEASURES

TTCs completed questionnaires designed to measure the self-reported intention to change practice (catalytic effect), immediately following a trauma case and 3 weeks later. Secondary outcomes included measures of perceived benefit, acceptability, and feasibility from TTCs and other trauma team members.

RESULTS

Data were collected following 24 trauma team activations: TTCs from 12 activations received multisource feedback and 12 received standard feedback. The self-reported intention for practice change was not significantly different between groups initially (4.0 vs 4.0, p=0.57) and at 3 weeks (4.0 vs 3.0, p=0.25). Multisource feedback was perceived to be helpful and superior to the existing feedback process. Feasibility was identified as a challenge.

CONCLUSIONS

The self-reported intention for practice change was no different for TTCs who received multisource feedback and those who received standard feedback. Multisource feedback was favourably received by trauma team members, and TTCs perceived multisource feedback as useful for their development.

Patients as Feedback Providers: Exploring Medical Students' Credibility Judgments

Introduction

Patient feedback is becoming ever more important in medical education. Whether students engage with feedback is partly determined by how credible they think the feedback provider is. Despite its importance for feedback engagement, little is known about how medical students judge the credibility of patients. The purpose of this study was therefore to explore how medical students make credibility judgments regarding patients as feedback providers.

Methods

This qualitative study builds upon McCroskey's conceptualization of credibility as a three-dimensional construct comprising: competence, trustworthiness, and goodwill. Since credibility judgments are shaped by the context, we studied students' credibility judgments in both a clinical and non-clinical context. Medical students were interviewed after receiving feedback from patients. Interviews were analyzed through template and causal network analysis.

Results

Students based their credibility judgments of patients on multiple interacting arguments comprising all three dimensions of credibility. In estimating a patient's credibility, students reasoned about aspects of the patient's competence, trustworthiness, and goodwill. In both contexts students perceived elements of an educational alliance between themselves and patients, which could increase credibility. Yet, in the clinical context students reasoned that therapeutic goals of the relationship with patients might impede educational goals of the feedback interaction, which lowered credibility.

Discussion

Students' credibility judgments of patients were a weighing of multiple sometimes conflicting factors, within the context of relationships and their associated goals. Future research should explore how goals and roles can be discussed between students and patients to set the stage for open feedback conversations.

The Impact of Multisource Feedback on Continuing Medical Education, Clinical Performance and Patient Experience: Innovation in a Child and Adolescent Mental Health Service

This paper reiterates the importance of the role of multisource feedback (MSF) in continuing medical education/continuing professional development (CME/CPD) and its impact on doctors' performance and patient experience globally. It summarises a unique initiative of robust utilisation of internationally recognised multisource feedback tools in an outpatient child and adolescent mental health service (CAMHS) in Qatar. The process involved the effective adoption and administering of the General Medical Council's (GMC) self-assessment questionnaire (SQ), patient questionnaire (PQ), and colleague questionnaire (CQ) followed by the successful incorporation of these tools in CME/CPD. The original version of the PQ questionnaire and the instructions to the patient document were translated into Arabic through the blind back-translation technique. This initiative of introducing gold-standard MSF tools and processes into clinical practice, among other quality-improvement projects, has contributed to the improvement of service standards and doctors' clinical practice. Patient satisfaction was measured through the annual patient experience analysis using the Experience of Service Questionnaire (ESQ) whereas changes in doctors' performance were evaluated by comparing annual appraisal scores before and after implementation of this initiative. We have demonstrated that when MSF is obtained impartially and transparently using recognised and valid tools, it can improve patient experience and enhance doctors' performance.

UK palliative medicine trainees and multisource communication skills feedback: an educational tool?

Background

Multisource feedback provides ratings of a trainee doctor’s performance from a range of assessors and enables 360 degree feedback on communication skills and team working behaviours. It is a tool used throughout palliative medicine training in the UK. There are limited data on the value of multisource feedback from a palliative medicine trainee perspective.

Aim

To study the views of palliative medicine trainees regarding multisource feedback as an educational tool to develop communication skills.

Design

A multimodal study encompassing a focus group and questionnaire mailed to all deanery palliative doctors.

Setting/participants

All palliative medicine trainees within a UK training deanery.

Results

Over half of responding trainees thought multisource feedback had little or no impact on their clinical practice. Improvements in delivery of multisource feedback to maximise learning were identified, including skilled feedback and facilitation by educational supervisors.

Conclusions

Despite multisource feedback currently having limited benefits, a number of recommendations are suggested to improve this.

Family Physician Quality Improvement Plans: A Realist Inquiry Into What Works, for Whom, Under What Circumstances

INTRODUCTION

Evaluation of quality improvement programs shows variable impact on physician performance often neglecting to examine how implementation varies across contexts and mechanisms that affect uptake. Realist evaluation enables the generation, refinement, and testing theories of change by unpacking what works for whom under what circumstances and why. This study used realist methods to explore relationships between outcomes, mechanisms (resources and reasoning), and context factors of a national multisource feedback (MSF) program.

METHODS

Linked data for 50 physicians were examined to determine relationships between action plan completion status (outcomes), MSF ratings, MSF comments and prescribing data (resource mechanisms), a report summarizing the conversation between a facilitator and physician (reasoning mechanism), and practice risk factors (context). Working backward from outcomes enabled exploration of similarities and differences in mechanisms and context.

RESULTS

The derived model showed that the completion status of plans was influenced by interaction of resource and reasoning mechanisms with context mediating the relationships. Two patterns were emerged. Physicians who implemented all their plans within six months received feedback with consistent messaging, reviewed data ahead of facilitation, coconstructed plan(s) with the facilitator, and had fewer risks to competence (dyscompetence). Physicians who were unable to implement any plans had data with fewer repeated messages and did not incorporate these into plans, had difficult plans, or needed to involve others and were physician-led, and were at higher risk for dyscompetence.

DISCUSSION

Evaluation of quality improvement initiatives should examine program outcomes taking into consideration the interplay of resources, reasoning, and risk factors for dyscompetence.

How does multisource feedback influence residency training? A qualitative case study

INTRODUCTION

Multisource feedback (MSF), also called 360-degree assessment, is one form of assessment used in postgraduate training. However, there is an ongoing discussion on its value, because the factors that influence the impact of MSF and the main impact of MSF are not fully understood. In this study, we investigated both the influencing factors and the impact of MSF on residency training.

METHODS

We conducted a qualitative case study within the boundaries of the residency training for paediatricians and paediatric surgeons at a University Hospital. We collected data from seven focus group interviews with stakeholders of MSF (residents, raters and supervisors). By performing a reflexive thematic analysis, we extracted the influencing factors and the impact of MSF.

RESULTS

We found seven influencing factors: MSF is facilitated by the announcement of a clear goal of MSF, the training of raters on the MSF instrument, a longitudinal approach of observation, timing not too early and not too late during the rotation, narrative comments as part of the ratings, the residents' self-assessment and a supervisor from the same department. We found three themes on the impact of MSF: MSF supports the professional development of residents, enhances interprofessional teamwork and increases the raters' commitment to the training of residents.

CONCLUSION

This study illuminates the influencing factors and impact of MSF on residency training. We offer novel recommendations on the continuity of observation, the timing during rotations and the role of the supervisor. Moreover, by discussing our results through the lens of identity formation theory, this work advances our conceptual understanding of MSF. We propose identity formation theory as a framework for future research on MSF to leverage the potential of MSF in residency training.

Unobserved Observers: Nurses' Perspectives About Sharing Feedback on the Performance of Resident Physicians

PURPOSE

Postgraduate training programs are incorporating feedback from registered nurses (RNs) to facilitate holistic assessments of resident performance. RNs are a potentially rich source of feedback because they often observe trainees during clinical encounters when physician supervisors are not present. However, RN perspectives about sharing feedback have not been deeply explored. This study investigated RN perspectives about providing feedback and explored the facilitators and barriers influencing their engagement.

METHOD

Constructivist grounded theory methodology was used in interviewing 11 emergency medicine and 8 internal medicine RNs at 2 campuses of a tertiary care academic medical center in Ontario, Canada, between July 2019 and March 2020. Interviews explored RN experiences working with and observing residents in clinical practice. Data collection and analysis were conducted iteratively. Themes were identified using constant comparative analysis.

RESULTS

RNs felt they could observe authentic day-to-day behaviors of residents often unwitnessed by supervising physicians and offer unique feedback related to patient advocacy, communication, leadership, collaboration, and professionalism. Despite a strong desire to contribute to resident education, RNs were apprehensive about sharing feedback and reported barriers related to hierarchy, power differentials, and a fear of overstepping professional boundaries. Although infrequent, a key stimulus that enabled RNs to feel safe in sharing feedback was an invitation from the supervising physician to provide input.

CONCLUSIONS

Perceived hierarchy in academic medicine is a critical barrier to engaging RNs in feedback for residents. Accessing RN feedback on authentic resident behaviors requires dismantling the negative effects of hierarchy and fostering a collaborative interprofessional working environment. A critical step toward this goal may require supervising physicians to model feedback-seeking behavior by inviting RNs to share feedback. Until a workplace culture is established that validates nurses' input and creates safe opportunities for them to contribute to resident education, the voices of nurses will remain unheard.

Rethinking Our Annual Congress-Meeting the Needs of Specialist Physicians by Partnering With Provincial Simulation Centers

Canada's maintenance of certification programs for physicians has evolved to emphasize assessment activities. Our organization recognized the importance of offering more practice assessment opportunities to our members to enhance their practice and help them comply with a regulation from our provincial professional body related to ongoing continuing education. This led us to rethink our annual congress and enrich the program with a curriculum of interdisciplinary simulation sessions tailored to meet the needs of a broad audience of specialists. Our challenges are similar to those of many national specialty societies having limited access to simulation facilities, instructors, and simulation teams that can cover the breadth and scope of perceived and unperceived simulation needs for their specialty. Our innovative solution was to partner with local experts to develop 22 simulation sessions over the past three years. The response was very positive, drawing 867 participants. Over 95% of participants either agreed or strongly agreed that their simulation session (1) met their learning objectives, (2) was relevant for their practice, and (3) encouraged them to modify their practice. Narrative comments from a survey sent to the 2018 participants four months after their activity indicated several self-reported changes in their practice or patient outcomes. We were able to centralize offers from organizations that had previously worked in silo to develop simulation sessions meeting the needs of our members. Proposing simulation sessions allowed our organization to establish long-term partnerships and to expend our "educational toolbox" to address skill gaps not usually addressed during annual meetings.

Impact of implementing multisource feedback on behaviors of young doctors

Objectives:

Multisource feedback (MSF) is a workplace-based assessment tool that offers 360-degree evaluation of the trainee doctor. Little is known about its receptiveness among stakeholders in Pakistan. This study explores house officers’ perceptions regarding MSF since its implementation in Eye Unit-II, Institute of Ophthalmology, King Edward Medical University/ Mayo Hospital, Lahore.

Methods:

A qualitative case study was conducted from July 2019 to February 2020 in Eye Unit II. A purposive (maximum variation) sample of 12 house surgeons was taken. Two focus group discussions were conducted. Data were transcribed and analyzed thematically.

Results:

The study identified the impact of MSF on house surgeons. Most participants reported positive experiences. The feedback they received increased their motivation, management skills and team working. A number of factors affecting the receptiveness of MSF were also identified which mainly included characteristics of raters and emotional response to MSF.

Conclusion:

Multisource Feedback is a useful tool for feedback that impacts the young doctors in many ways. It contributes to increasing their sense of responsibility, management skills and self-directed learning. The improvement in individual abilities and teamwork also helped in improving patient care.

A Training Intervention through a 360° Multisource Feedback Model

Physicians and other health sciences professionals need continuous training, not only in technical aspects of their activity but also in nontechnical, transversal competencies with a cost-efficient impact on the proper functioning of healthcare. The objective of this paper is to analyze the behavioral change among health professionals at a large public hospital following a training intervention on a set of core nontechnical competencies: Teamwork, Adaptability-Flexibility, Commitment-Engagement, Results Orientation, and Leadership Skills for Supervisors. The 360° Multisource Feedback (MSF) model was applied using three sources of information: supervisors, co-workers, and the workers themselves (self-assessment). A quasi-experimental pretest–post-test single-group design with two points in time was utilized. The training intervention improved the scores of only one of the trained competencies—the “Results Orientation” competency—although the scores were slightly inflated. Moreover, significant discrepancies were detected between the three sources, with supervisors awarding the highest scores. The magnitude of behavioral change was related to certain sociodemographic and organizational variables. The study was not immune to the ceiling effect, despite control measures aimed at avoiding it. The empirical evidence suggests that the 360° MSF model must be maintained over time to enhance and reinforce an evaluation culture for better patient care.

Correlation of attending and patient assessment of resident communication skills in the emergency department

BACKGROUND

Communication and interpersonal skills are one of the Accreditation Council for Graduate Medical Education's six core competencies. Validated methods for assessing these among trainees are lacking. Educators have developed various communication assessment tools from both the supervising attending and the patient perspectives. How these different assessment methods and tools compare with each other remains unknown. The goal of this study was to determine the degree of agreement between attending and patient assessment of resident communication skills.

METHODS

This was a retrospective study of emergency medicine (EM) residents at an academic medical center. From July 2017 to June 2018, residents were assessed on communication skills during their emergency department shifts by both their supervising attending physicians and their patients. The attendings rated residents' communication skills with patients, colleagues, and nursing/ancillary staff using a 1 to 5 Likert scale. Patients completed the modified Communication Assessment Tool (CAT), a 14-item questionnaire based on a 1 to 5 Likert scale. Mean attending ratings and patient CAT scores were calculated for each resident. Means were divided into tertiles due to nonparametric distribution of scores. Agreement between attending and patient ratings of residents were measured using Cohen's kappa for each attending evaluation question. Scores were weighted to assign adjacent tertiles partial agreement.

RESULTS

During the study period, 1,097 attending evaluations and 952 patient evaluations were completed for 26 residents. Attending scores and CAT scores of the residents showed slight to fair agreement in the following three domains: patient communication (κ = 0.21), communication with colleagues (κ = 0.21), and communication with nursing/ancillary staff (κ = 0.26).

CONCLUSIONS

Attending and patient ratings of EM residents' communication skills show slight to fair agreement. The use of different types of raters may be beneficial in fully assessing trainees' communication skills.

Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study

OBJECTIVES

To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.

DESIGN

Qualitative text analysis supplemented by quantitative descriptive analysis.

SETTING

Internal Medicine Departments in Zealand, Denmark.

PARTICIPANTS

48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).

INTERVENTION

This study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either 'no reasons for concern' or 'possibly some concern', thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.

RESULTS

Out of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter-rater correlations between four authors were 0.7-1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.

CONCLUSIONS

A purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees' performances based on the MSF reports.

Doctors' maintenance of professional competence: a qualitative study informed by the theory of planned behaviour

BACKGROUND

Medical regulators worldwide have implemented programmes of maintenance of professional competence (MPC) to ensure that doctors, throughout their careers, are up to date and fit to practice. The introduction of MPC required doctors to adopt a range of new behaviours. Despite high enrolment rates on these programmes, it remains uncertain whether doctors engage in the process because they perceive benefits like improvements in their practice and professional development or if they solely meet the requirements to retain medical registration. In this study, we aimed to explore the relationship between doctors' beliefs, intention and behaviour regarding MPC through the lens of the Theory of Planned Behaviour (TPB) to make explicit the factors that drive meaningful engagement with the process.

METHODS

We conducted a qualitative study using semi-structured interviews. From a pool of 1258 potential participants, we purposively selected doctors from multiple specialities, age groups, and locations across Ireland. We used thematic analysis, and the TPB informed the analytic coding process.

RESULTS

Forty-one doctors participated in the study. The data analysis revealed doctors' intention and behaviour and the factors that shape their engagement with MPC. We found that attitudes and beliefs about the benefits and impact of MPC mediated the nature of doctors' engagement with the process. Some participants perceived positive changes in practice and other gains from participating in MPC, which facilitated committed engagement with the process. Others believed MPC was unfair, unnecessary, and lacking any benefit, which negatively influenced their intention and behaviour, and that was demonstrated by formalistic engagement with the process. Although participants with positive and negative attitudes shared perceptions about barriers to participation, such perceptions did not over-ride strongly positive beliefs about the benefits of MPC. While the requirements of the regulator strongly motivated doctors to participate in MPC, beliefs about patient expectations appear to have had less impact on intention and behaviour.

CONCLUSIONS

The findings of this study broaden our understanding of the determinants of doctors' intention and behaviour regarding MPC, which offers a basis for designing targeted interventions. While the barriers to engagement with MPC resonate with previous research findings, our findings challenge critical assumptions about enhancing doctors' engagement with the process. Overall, our results suggest that focused policy initiatives aimed at strengthening the factors that underpin the intention and behaviour related to committed engagement with MPC are warranted.

Assessing anaesthesiology and intensive care specialty physicians: An Italian language multisource feedback system

BACKGROUND

Physician professionalism, including anaesthesiologists and intensive care doctors, should be continuously assessed during training and subsequent clinical practice. Multi-source feedback (MSF) is an assessment system in which healthcare professionals are assessed on several constructs (e.g., communication, professionalism, etc.) by multiple people (medical colleagues, coworkers, patients, self) in their sphere of influence. MSF has gained widespread acceptance for both formative and summative assessment of professionalism for reflecting on how to improve clinical practice.

METHODS

Instrument development and psychometric analysis (feasibility, reliability, construct validity via exploratory factor analysis) for MSF questionnaires in a postgraduate specialty training in Anaesthesiology and intensive care in Italy. Sixty-four residents at the Università del Piemonte Orientale (Italy) Anesthesiology Residency Program. Main outcomes assessed were: development and psychometric testing of 4 questionnaires: self, medical colleague, coworker and patient assessment.

RESULTS

Overall 605 medical colleague questionnaires (mean of 9.3 ±1.9) and 543 coworker surveys (mean 8.4 ±1.4) were collected providing high mean ratings for all items (> 4.0 /5.0). The self-assessment item mean score ranged from 3.1 to 4.3. Patient questionnaires (n = 308) were returned from 31 residents (40%; mean 9.9 ± 6.2). Three items had high percentages of "unable to assess" (> 15%) in coworker questionnaires. Factor analyses resulted in a two-factor solution: clinical management with leadership and accountability accounting for at least 75% of the total variance for the medical colleague and coworker's survey with high internal consistency reliability (Cronbach's α > 0.9). Patient's questionnaires had a low return rate, a limited exploratory analysis was performed.

CONCLUSIONS

We provide a feasible and reliable Italian language MSF instrument with evidence of construct validity for the self, coworkers and medical colleague. Patient feedback was difficult to collect in our setting.

Feedback Credibility in Healthcare Education: a Systematic Review and Synthesis

PURPOSE

The purpose of this study was to systematically review and synthesize factors that influence learners' perceptions of credibility when feedback is provided by an authority figure in a healthcare environment.

METHODS

This study reviewed literature from medicine, psychology, and education using systematic review and qualitative synthesis methods. In a multi-step process, major electronic bibliographic databases were searched for relevant studies until October 2020.

RESULTS

The search identified 9216 articles. A total of 134 abstracts underwent full-text review. Of these, 22 articles met inclusion criteria. The studies were heterogenous and the majority utilized a qualitative design with interviews and focus groups. A few studies employed mixed methodology (n = 2) and two studies used a quantitative design. Four main themes were identified: feedback characteristics, context of feedback, source credibility, and recipient characteristics.

CONCLUSION

As programs implement major educational change initiatives to create more formative assessment practices, feedback will become even more crucial. The four main themes identified are important factors that contribute to the perception of feedback credibility. While the factors are described independently, they may be viewed as interrelated and the association between these factors and feedback may be driven more by learning culture than each characteristic.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-020-01167-w.

Role of Technology in Self-Assessment and Feedback Among Hospitalist Physicians: Semistructured Interviews and Thematic Analysis

Background

Lifelong learning is embedded in the culture of medicine, but there are limited tools currently available for many clinicians, including hospitalists, to help improve their own practice. Although there are requirements for continuing medical education, resources for learning new clinical guidelines, and developing fields aimed at facilitating peer-to-peer feedback, there is a gap in the availability of tools that enable clinicians to learn based on their own patients and clinical decisions.

Objective

The aim of this study was to explore the technologies or modifications to existing systems that could be used to benefit hospitalist physicians in pursuing self-assessment and improvement by understanding physicians’ current practices and their reactions to proposed possibilities.

Methods

Semistructured interviews were conducted in two separate stages with analysis performed after each stage. In the first stage, interviews (N=12) were conducted to understand the ways in which hospitalist physicians are currently gathering feedback and assessing their practice. A thematic analysis of these interviews informed the prototype used to elicit responses in the second stage.

Results

Clinicians actively look for feedback that they can apply to their practice, with the majority of the feedback obtained through self-assessment. The following three themes surrounding this aspect were identified in the first round of semistructured interviews: collaboration, self-reliance, and uncertainty, each with three related subthemes. Using a wireframe, the second round of interviews led to identifying the features that are currently challenging to use or could be made available with technology.

Conclusions

Based on each theme and subtheme, we provide targeted recommendations for use by relevant stakeholders such as institutions, clinicians, and technologists. Most hospitalist self-assessments occur on a rolling basis, specifically using data in electronic medical records as their primary source. Specific objective data points or subjective patient relationships lead clinicians to review their patient cases and to assess their own performance. However, current systems are not built for these analyses or for clinicians to perform self-assessment, making this a burdensome and incomplete process. Building a platform that focuses on providing and curating the information used for self-assessment could help physicians make more accurately informed changes to their own clinical practice and decision-making.

A german-language competency-based multisource feedback instrument for residents: development and validity evidence

BACKGROUND

In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor's performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity.

METHODS

We developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, "unable to comment" ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents' learning goals and the progress as reported via MSF.

RESULTS

Our resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician's competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach's alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself.

CONCLUSIONS

To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.

A randomised controlled trial of feedback to improve patient satisfaction and consultation skills in medical students

BACKGROUND

The use of feedback has been integral to medical student learning, but rigorous evidence to evaluate its education effect is limited, especially in the role of patient feedback in clinical teaching and practice improvement. The aim of the Patient Teaching Associate (PTA) Feedback Study was to evaluate whether additional written consumer feedback on patient satisfaction improved consultation skills among medical students and whether multisource feedback (MSF) improved student performance.

METHODS

In this single site, double-blinded randomised controlled trial, 71 eligible medical students from two universities in their first clinical year were allocated to intervention or control and followed up for one semester. They participated in five simulated student-led consultations in a teaching clinic with patient volunteers living with chronic illness. Students in the intervention group received additional written feedback on patient satisfaction combined with guided self-reflection. The control group received usual immediate formative multisource feedback from tutors, patients and peers. Student characteristics, baseline patient-rated satisfaction scores and tutor-rated consultation skills were measured.

RESULTS

Follow-up assessments were complete in 70 students attending the MSF program. At the final consultation episodes, both groups improved patient-rated rapport (P = 0.002), tutor-rated patient-centeredness and tutor-rated overall consultation skills (P = 0.01). The intervention group showed significantly better tutor-rated patient-centeredness (P = 0.003) comparing with the control group. Distress relief, communication comfort, rapport reported by patients and tutor-rated clinical skills did not differ significantly between the two groups.

CONCLUSIONS

The innovative multisource feedback program effectively improved consultation skills in medical students. Structured written consumer feedback combined with guided student reflection further improved patient-centred practice and effectively enhanced the benefit of an MSF model. This strategy might provide a valuable adjunct to communication skills education for medical students.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry Number ACTRN12613001055796 .

A framework and toolkit of interventions to enhance reflective learning among health-care professionals: the PEARL mixed-methods study

Background

Although most health care is high quality, many patients and members of staff can recall episodes of a lack of empathy, respect or effective communication from health-care staff. In extreme form, this contributes to high-profile organisational failures. Reflective learning is a universally promoted technique for stimulating insight, constructive self-appraisal and empathy; however, its efficacy tends to be assumed rather than proven. The Patient Experience And Reflective Learning (PEARL) project has used patient and staff experience to co-design a novel reflective learning framework that is based on theories of behaviour and learning.

Objective

To create a toolkit to help health-care staff obtain meaningful feedback to stimulate effective reflective learning that will promote optimal patient-, family- and colleague-focused behaviours.

Design

A 3-year developmental mixed-methods study with four interlinked workstreams and 12 facilitated co-design meetings. The Capability, Opportunity, Motivation – Behaviour framework was used to describe factors influencing the behaviour of reflection.

Setting

This took place at five acute medical units and three intensive care units in three urban acute hospital trusts in England.

Participants

Patients and relatives, medical and nursing staff, managers and researchers took part.

Data sources

Two anonymous surveys, one for patients and one for staff, were developed from existing UK-validated instruments, administered locally and analysed centrally. Ethnographers undertook interviews and observed clinical care and reflective learning activities in the workplace, as well as in the co-design meetings, and fed back their observations in plenary workshops.

Main outcome measures

Preliminary instruments were rated by participants for effectiveness and feasibility to derive a final set of tools. These are presented in an attractively designed toolbox with multiple sections, including the theoretical background of reflection, mini guides for obtaining meaningful feedback and for reflecting effectively, guides for reflecting ‘in-action’ during daily activities, and a set of resources.

Results

Local project teams (physicians, nurses, patients, relatives and managers) chaired by a non-executive director found the quarterly reports of feedback from the patient and staff surveys insightful and impactful. Patient satisfaction with care was higher for intensive care units than for acute medical units, which reflects contextual differences, but in both settings quality of communication was the main driver of satisfaction. Ethnographers identified many additional forms of experiential feedback. Those that generated an emotional response were particularly effective as a stimulus for reflection. These sources of data were used to supplement individual participant experiences in the nine local co-design meetings and four workshops to identify barriers to and facilitators of effective reflection, focusing on capability, opportunity and motivation. A logic model was developed combining the Capability, Opportunity, Motivation – Behaviour framework for reflection and theories of learning to link patient and staff experience to changes in downstream behaviours. Participants proposed practical tools and activities to enhance reflection ‘in-action’ and ‘on-action’. These tools were developed iteratively by the local and central project teams.

Limitations

Paper-based surveys were burdensome to administer and analyse.

Conclusions

Patients and health-care staff collaborated to produce a novel reflective learning toolkit.

Future work

The toolkit requires evaluating in a cluster randomised controlled trial.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 8, No. 32. See the NIHR Journals Library website for further project information.

Survey of physician attitudes to using multisource feedback for competence assessment in Alberta

BACKGROUND

The use of multisource feedback (MSF) for assessing physician performance is widespread and rapidly growing. Findings from early very small research studies using highly selected participants suggest high levels of satisfaction and support. However, after nearly two decades of experience using MSF to evaluate all physicians in Alberta, we are sceptical of this.

OBJECTIVES

To determine physicians' actual opinions of MSF using the entire physician population of Alberta, Canada DESIGN: Online survey.

SETTING

Alberta, Canada.

PARTICIPANTS

All physicians with a full licence to practice in Alberta in 2015.

INTERVENTIONS

All participants were asked to grade how well they thought MSF was at assessing various aspects of physician performance using a 10-point Likert-type scale. There was also a text response field for written comments.

OUTCOMES

Mean responses to quantitative questions. Qualitative content and thematic analysis of open-ended text responses.We analysed the data using SPSS V.23 and NVivo V.11 and built a multivariate model highlighting the predictors of high and low opinions of MSF.

RESULTS

Survey response rate was high for physicians with 2215 responses (25%). The mean rating for how successful MSF was at assessing a variety of dimensions, varied from a low of 5.03/10 for medical knowledge to a high of 6.38/10 for professionalism and communication. Canadian-trained MDs rated MSF significantly lower on every dimension by approximately 20% compared with non-Canadian-trained MDs.

CONCLUSIONS

Alberta physicians have much lower opinions about the ability of MSF to measure any dimension of their performance than what has been suggested in the literature. Canadian-trained MDs have a particularly low opinion of MSF for reasons that remain unclear. The results of this survey offer a serious challenge to the effectiveness of a programme that is designed to promote self-reflection and performance improvement.

Assessment methods in respiratory medicine training in Europe: current status and future needs

There is a perceived need for harmonisation of training standards in medicine across Europe. Assessment methodology is a fundamental part of this harmonisation. Assessment may be: summative, measuring competency at the end of training, usually in a pass/fail mode; or formative, assessing strengths and weaknesses during training with a view to optimising performance. A survey of experienced respiratory medicine trainers in Europe was undertaken to discover the formative and summative assessment methods being used, and to explore the use of workplace-based assessments (WBAs). Structured interviews were sought with experienced trainers in adult and paediatric respiratory medicine. 35 trainers from 22 (71%) out of 31 countries were interviewed. The number of types of summative assessments required at the end of training varied from 0 to 4 including clinical (25%), written (44%) and oral (47%) examinations. Four respondents required a research thesis as a proof of clinical competence. WBA was not commonly used. Only 14% of respondents reported using a formal case-based discussion, 20% used a form of multisource feedback and 25% described some form of formal assessment of procedural skills. However, 77% of all respondents expressed a wish to have access to case-based discussion and 72% wanted to have a facility for multisource feedback. The majority also wanted training in the use of these tools. Almost half of the respondents had received no formal training in educational supervision and 80% of all respondents expressed a wish to receive such training. The findings suggest that there is no adequate process of ensuring uniform standards for specialist accreditation in Europe, and demonstrate a need and desire among trainers for more WBA tools and training in their use to be made available.

Understanding unwarranted variation in clinical practice: a focus on network effects, reflective medicine and learning health systems

In the past decades, extensive research has been performed on the phenomenon of unwarranted clinical variation in clinical practice. Many studies have been performed on signaling, describing and visualizing clinical variation. We argue that it is time for next steps in practice variation research. In addition to describing and signaling variation patterns, we argue that a better understanding of causes of variation should be gained. Moreover, target points for improving and decreasing clinical variation should be created. Key elements in this new focus should be research on the complex interaction of networks, reflective medicine, patient beliefs and objective criteria for treatment choices. By combining these different concepts, alternative research objectives and new targets for improving and reducing unwarranted variation may be defined. In this perspective, we reflect on these concepts and propose target points for future research.

Internal Medicine Resident Professionalism Assessments: Exploring the Association With Patients' Overall Satisfaction With Their Hospital Stay

PURPOSE

Successful training of internal medicine (IM) residents requires accurate assessments. Patients could assess IM residents in a hospital setting, but medical educators must understand how contextual factors may affect assessments. The objective was to investigate relationships between patient, resident, and hospital-encounter characteristics and the results of patient assessments of IM resident professionalism.

METHOD

The authors performed a prospective cohort study of postgraduate year 1 (PGY-1) IM residents and their patients at 4 general medicine inpatient teaching services at Mayo Clinic Hospital-Rochester, Saint Marys Campus in Rochester, Minnesota, from July 1, 2015, through June 30, 2016. Patient assessments of resident professionalism were adapted from validated instruments. Multivariable modeling with generalized estimating equations was used to determine associations between patient assessment scores and characteristics of residents, residents' clinical performance and evaluations (including professionalism assessments in other settings), patients, and hospital encounters and to account for repeated assessments of residents.

RESULTS

A total of 409 patients assessed 72 PGY-1 residents (mean [SD], 5.7 [3.0] patient assessments per resident). In the multivariable model, only the highest rating out of 5 levels for overall satisfaction with hospital stay was significantly associated with patient assessment scores of resident professionalism (β [SE], 0.80 [0.08]; P < .001). Hospitalized patients' assessment scores of resident professionalism were not significantly correlated with assessment scores of resident professionalism in other clinical settings.

CONCLUSIONS

Hospitalized patients' assessment scores of in-hospital resident professionalism were strongly correlated with overall patient satisfaction with hospital stay but were not correlated with resident professionalism in other settings. The limitations of patient evaluations should be considered before incorporating these evaluations into programs of assessment.

Exploring Perspectives from Internal Medicine Clerkship Directors in the USA on Effective Narrative Evaluation: Results from the CDIM National Survey

PURPOSE

Clinical performance evaluations play a critical role in determining medical school clerkship grades. This study aimed to provide clarification from clerkship directors in internal medicine on what constitutes an effective and informative narrative description of student performance.

METHODS

In September 2016, the Clerkship Directors in Internal Medicine (CDIM) electronically administered its annual, voluntary, and confidential cross-sectional survey of its US membership. One section of the survey asked six questions regarding the helpful components of an effective narrative evaluation. Respondents were asked to rate the effectiveness of elements contained within narrative evaluations of students.

RESULTS

Ninety-five CDIM members responded to the survey with an overall response rate of 74.2%. Descriptions of skills and behaviors were felt to be the most important, followed by a description of the overall synthetic or global assessment level of the student. Descriptions of personality and attitude were the next highest rated feature followed by adjectives describing performance. Length was felt to be the least important component. In free-text comments, several respondents indicated that direct observation of performance and specific examples of skills and behaviors are also desirable.

CONCLUSIONS

Narrative evaluations of students that explicitly comment on skills, behaviors, and an overarching performance level of the learner are strongly preferred by clerkship directors. Direct observation of clinical performance and giving specific examples of such behaviors give evaluations even more importance. Faculty development on evaluation and assessment should include instruction on these narrative assessment characteristics.

An Examination of Self-Reported Assessment Activities Documented by Specialist Physicians for Maintenance of Certification

INTRODUCTION

Specialists in a Maintenance of Certification program are required to participate in assessment activities, such as chart audit, simulation, knowledge assessment, and multisource feedback. This study examined data from five different specialties to identify variation in participation in assessment activities, examine differences in the learning stimulated by assessment, assess the frequency and type of planned changes, and assess the association between learning, discussion, and planned changes.

METHODS

E-portfolio data were categorized and analyzed descriptively. Chi-squared tests examined associations.

RESULTS

A total of 2854 anatomical pathologists, cardiologists, gastroenterologists, ophthalmologists, and orthopedic surgeons provided data about 6063 assessment activities. Although there were differences in the role that learning played by discipline and assessment type, the most common activities documented across all specialties were self-assessment programs (n = 2122), feedback on teaching (n = 1078), personal practice assessments which the physician did themselves (n = 751), annual reviews (n = 682), and reviews by third parties (n = 661). Learning occurred for 93% of the activities and was associated with change. For 2126 activities, there were planned changes. Activities in which there was a discussion with a peer or supervisor were more likely to result in a change.

CONCLUSIONS AND DISCUSSION

Although specialists engaged in many types of assessment activities to meet the Maintenance of Certification program requirements, there was variability in how assessment stimulated learning and planned changes. It seems that peer discussion may be an important component in fostering practice change and forming plans for improvement which bears further study.

Perceptions of people with respiratory problems on physician performance evaluation-A qualitative study

BACKGROUND

Despite increasing calls for patient and public involvement in health-care quality improvement, the question of how patient evaluations can contribute to physician learning and performance assessment has received scant attention.

OBJECTIVE

The objective of this study was to explore, amid calls for patient involvement in quality assurance, patients' perspectives on their role in the evaluation of physician performance and to support physicians' learning and decision making on professional competence.

DESIGN

A qualitative study based on semi-structured interviews.

SETTING AND PARTICIPANTS

The study took place in a secondary care setting in the Netherlands. The authors selected 25 patients from two Dutch hospitals and through the Dutch Lung Foundation, using purposive sampling.

METHODS

Data were analysed according to the principles of template analysis, based on an a priori coding framework developed from the literature about patient empowerment, feedback and performance assessment.

RESULTS

The analysis unearthed three predominant patient perspectives: the proactive perspective, the restrained perspective and the outsider perspective. These perspectives differed in terms of perceived power dynamics within the doctor-patient relationship, patients' perceived ability, and willingness to provide feedback and evaluate their physician's performance. Patients' perspectives thus affected the role patients envisaged for themselves in evaluating physician performance.

DISCUSSION AND CONCLUSION

Although not all patients are equally suitable or willing to be involved, patients can play a role in evaluating physician performance and continuing training through formative approaches. To involve patients successfully, it is imperative to distinguish between different patient perspectives and empower patients by ensuring a safe environment for feedback.

Seeing is believing: an exploration of what residents value when they receive feedback

OBJECTIVE

The primary purpose of this paper is to report on psychiatry residents' perceptions of what is important when receiving feedback from evaluators.

METHODS

In January 2018, as part of the Harvard South Shore Psychiatry Residency Training Program's (HSS) ongoing local quality improvement efforts to enhance the delivery and effectiveness of feedback that residents receive from faculty during training, the authors disseminated a survey to psychiatry residents (n = 31) at HSS. Residents rated the level of importance of 17 statements pertaining to the way feedback is delivered. Two open-ended prompts also allowed respondents to share examples of growth-oriented and unhelpful feedback they have received during residency.

RESULTS

Twenty-seven residents responded (87% response rate). Eighty-one percent rated "the evaluator models the same behavior they're encouraging" as "extremely important" when receiving feedback. Many residents also rated the following survey items as "extremely important": "confidence in the evaluator's clinical and interpersonal skills" (63.0%), "amount of time the evaluator observed the resident" (51.9%), "there is a way to fix a performance deficit" (51.8%), and "specific feedback based off the resident's work" (48.1%). Conversely, only 11.1% of residents rated the feedback sandwich as "extremely important."

CONCLUSIONS

Despite a small sample size, this project demonstrated that, when receiving feedback, the majority of psychiatry residents strongly value when evaluators model the targeted behavior. The feedback sandwich was least important to residents. This project underscores the importance of evaluators serving as role models in the context of feedback, and findings can be used in faculty development activities focused on feedback delivery best practices.

Perceived Communication Skills Among Tertiary Care Physicians

OBJECTIVES

Self-perception in clinical skills, including communication skills (CS), has been found to provide insights on strengths, weaknesses, and opportunities for skill improvement. The present study is aimed at exploring perceived CS among physicians working at a tertiary healthcare level.

METHODS

Physicians working at a tertiary hospital based in Saudi Arabia were invited to complete a modified self-questionnaire that assesses CS. Descriptive and association studies were performed. Psychometric properties of the questionnaire were determined.

RESULTS

Out of the 101 participating physicians, 57.2% rated their CS in the range of very good and excellent, but only 30.7% rated themselves as overall excellent. The question item with the highest mean score (score range, 1 to 5) was related to encouraging patients to ask questions (4.2 ± 0.9), while the lowest was for the item that assessed information disclosure (3.8 ± 0.8). Males rated themselves higher than females in the item related to explaining things to patients (p < 0.05), whereas physicians with non-surgical specialties rated themselves better than those with surgical specialties in the item related to expressing interest in patients (p < 0.05).

CONCLUSIONS

Based on physicians' self-rated assessment, less than the third of tertiary care physicians considered themselves as maintaining an excellent level of CS. Future studies are encouraged to examine CS through a multisystem assessment and promote the need for CS training for physicians working at a tertiary care level.

Patient experience and reflective learning (PEARL): a mixed methods protocol for staff insight development in acute and intensive care medicine in the UK

INTRODUCTION

Patient and staff experiences are strongly influenced by attitudes and behaviours, and provide important insights into care quality. Patient and staff feedback could be used more effectively to enhance behaviours and improve care through systematic integration with techniques for reflective learning. We aim to develop a reflective learning framework and toolkit for healthcare staff to improve patient, family and staff experience.

METHODS & ANALYSIS

Local project teams including staff and patients from the acute medical units (AMUs) and intensive care units (ICUs) of three National Health Service trusts will implement two experience surveys derived from existing instruments: a continuous patient and relative survey and an annual staff survey. Survey data will be supplemented by ethnographic interviews and observations in the workplace to evaluate barriers to and facilitators of reflective learning. Using facilitated iterative co-design, local project teams will supplement survey data with their experiences of healthcare to identify events, actions, activities and interventions which promote personal insight and empathy through reflective learning. Outputs will be collated by the central project team to develop a reflective learning framework and toolkit which will be fed back to the local groups for review, refinement and piloting. The development process will be mapped to a conceptual theory of reflective learning which combines psychological and pedagogical theories of learning, alongside theories of behaviour change based on capability, opportunity and motivation influencing behaviour. The output will be a locally-adaptable workplace-based toolkit providing guidance on using reflective learning to incorporate patient and staff experience in routine clinical activities.

ETHICS & DISSEMINATION

The PEARL project has received ethics approval from the London Brent Research Ethics Committee (REC Ref 16/LO/224). We propose a national cluster randomised step-wedge trial of the toolkit developed for large-scale evaluation of impact on patient outcomes.

Does evidence-based medicine training improve doctors' knowledge, practice and patient outcomes? A systematic review of the evidence

Courses in Evidence-Based Medicine (EBM) for doctors have consistently demonstrated short-term improvements in knowledge. However, there is no strong evidence linking EBM training to changes in clinical practice or patient outcomes. This systematic review investigates whether EBM training leads to sustained improvements in doctors' knowledge and practice behaviors that may also facilitate changes in patient outcomes and experiences. A literature search was undertaken in Ovid Medline, Ovid Embase, The Cochrane Library, ERIC and Scopus. Studies published from 1997 to 2016 that assessed outcomes of EBM educational interventions amongst doctors and used measures of knowledge, skills, attitudes, practice or patient outcomes were included. Fifteen studies were included in the analysis: four randomized controlled trials (RCTs), three non-RCTs, and eight before-after (longitudinal cohort) studies. Heterogeneity among studies prevented meaningful comparisons. Varying degrees of bias due to the use of subjective measures were identified, limiting study validity. Results showed that EBM interventions can improve short-term knowledge and skills, but there is little reliable evidence of changes in long-term knowledge, attitudes, and clinical practice. No study measured improvement in patient outcomes or experiences. EBM training for medical practitioners needs to incorporate measures of behavioral changes while incorporating patient outcomes and experience measures.

Exploring residents' reactions to and use of parent feedback in a pediatric emergency department: A grounded theory study

BACKGROUND

Competency-based medical education (CBME) involves workplace-based assessment. In pediatrics, patients' parents can participate in this assessment and generate feedback for residents. Prior to routinely collecting parent feedback, it is important to investigate residents' perspectives on it.

AIM

To explore residents' reactions to and use of written parent feedback.

METHODS

Using a grounded theory approach, we interviewed residents who received written parent feedback at the mid- and end-points of a pediatric emergency training rotation.

RESULTS

Twenty-five residents participated. The residents reacted positively to the feedback. They thought that it complements educators' feedback, can elucidate parents' perspectives and needs, and is something that residents want and need. Although the residents thought that non-specific negative parent feedback is not useful, they believed non-specific positive and constructive parent feedback to be encouraging and useful. They delineated how they use non-specific positive parent feedback to boost their self-confidence and reassure themselves that parents perceive their clinical practices as appropriate. They also elucidated how they use constructive parent feedback to understand what is important to parents, become aware of their own behaviors, and modify their clinical practices.

CONCLUSIONS

The findings encourage educators in pediatrics to include parents in resident assessment, especially in the CBME era.

Optimizing Multisource Feedback Implementation for Australasian Physicians

INTRODUCTION

Medical regulatory bodies in Australasia are encouraging greater use of continuing professional development activities, such as multisource feedback (MSF), which are practice-based, include facilitated feedback, and improve performance. The aim of this study was to explore the feasibility, effectiveness, and sustainability of an MSF process that includes a telephone/videoconference debrief, to better design future MSF implementation.

METHOD

Thirty-seven Australasian physician participants sought feedback from patients and colleagues and debriefed their feedback report with a trained facilitator. The impact was evaluated using quantitative and qualitative measures including surveys and semistructured interviews.

RESULTS

The feasibility of the MSF process was impacted by the level of support for the participant to complete the MSF, the nature and location of participants' work, and the use of telephone/videoconference to deliver the debrief. Regarding effectiveness, the MSF stimulated reflection on practice and action on areas identified for improvement. In addition, the quality of rater feedback and the inclusion of a debrief impacted participant and facilitator reports of effectiveness. The absence of a culture of feedback, the formative nature of the MSF, and the confidentiality of the results were factors impacting sustainability.

DISCUSSION

Optimizing MSF is important to encourage broad uptake in the wider medical community in Australasia. Although several factors were identified as having an impact, it is clear that inclusion of a quality debrief will increase the perceived value and the effectiveness of MSF. Delivering that debrief through telephone/videoconference can be effective and will increase the overall feasibility and sustainability.

The impact of patient feedback on the medical performance of qualified doctors: a systematic review

BACKGROUND

Patient feedback is considered integral to quality improvement and professional development. However, while popular across the educational continuum, evidence to support its efficacy in facilitating positive behaviour change in a postgraduate setting remains unclear. This review therefore aims to explore the evidence that supports, or refutes, the impact of patient feedback on the medical performance of qualified doctors.

METHODS

Electronic databases PubMed, EMBASE, Medline and PsycINFO were systematically searched for studies assessing the impact of patient feedback on medical performance published in the English language between 2006-2016. Impact was defined as a measured change in behaviour using Barr's (2000) adaptation of Kirkpatrick's four level evaluation model. Papers were quality appraised, thematically analysed and synthesised using a narrative approach.

RESULTS

From 1,269 initial studies, 20 articles were included (qualitative (n=8); observational (n=6); systematic review (n=3); mixed methodology (n=1); randomised control trial (n=1); and longitudinal (n=1) design). One article identified change at an organisational level (Kirkpatrick level 4); six reported a measured change in behaviour (Kirkpatrick level 3b); 12 identified self-reported change or intention to change (Kirkpatrick level 3a), and one identified knowledge or skill acquisition (Kirkpatrick level 2). No study identified a change at the highest level, an improvement in the health and wellbeing of patients. The main factors found to influence the impact of patient feedback were: specificity; perceived credibility; congruence with physician self-perceptions and performance expectations; presence of facilitation and reflection; and inclusion of narrative comments. The quality of feedback facilitation and local professional cultures also appeared integral to positive behaviour change.

CONCLUSION

Patient feedback can have an impact on medical performance. However, actionable change is influenced by several contextual factors and cannot simply be guaranteed. Patient feedback is likely to be more influential if it is specific, collected through credible methods and contains narrative information. Data obtained should be fed back in a way that facilitates reflective discussion and encourages the formulation of actionable behaviour change. A supportive cultural understanding of patient feedback and its intended purpose is also essential for its effective use.

A qualitative exploration: questioning multisource feedback in residency education

BACKGROUND

Multisource feedback (MSF), involves the collection of feedback from multiple groups of assessors, including those without a traditional hierarchal responsibility to evaluate doctors. Allied healthcare professionals (AHCPs), administrative staff, peers, patients and their families may all contribute to the formative assessment of physicians. Theoretically, this feedback provides a thorough view of physician performance; however, the ability of MSF programs to consistently impact physician behavior remains in question. Therefore, the objective of this study was to explore perceptions and prerequisites to an effective MSF program in postgraduate medical education from the perspectives of both pediatric residents and AHCPs.

METHODS

This exploratory study was conducted in a pediatric inpatient unit prior to implementation of a MSF program. Focus groups were conducted with purposefully recruited participants from three distinct groups: junior pediatric residents, senior pediatric residents, and AHCPs. Discussions were audio recorded, transcribed verbatim and analyzed using thematic analysis.

RESULTS

Both residents and AHCPs expressed a strong interest in the concept of MSF. However, more in depth discussions identified barriers to residents' acceptance of, and AHCPs' provision of feedback. Roles and responsibilities, perceptions of expertise, hospital culture/interprofessionalism and power dynamics were identified as barriers to the acceptance and provision of feedback. All groups expressed interest in opportunities to engage in bi-directional feedback.

CONCLUSIONS

The identified barriers and prerequisites to providing and accepting MSF suggest limits to the efficacy of the MSF process. Our findings suggest that these factors should be considered in the design and implementation of MSF programs.

Prevention of and dealing with poor performance: an interview study about how professional associations aim to support healthcare professionals

Objective

To explore how professional associations of nine healthcare professions aim to support professionals to prevent and deal with poor performance.

Design

Qualitative interview study.

Setting

The Netherlands.

Participants

Representatives of professional associations for dentists, general practitioners, medical specialists, midwives, nurses, pharmacists, physiotherapists, psychologists and psychotherapists.

Interventions

During nine face-to-face semi-structured interviews we asked how associations aim to support professionals in prevention of and dealing with poor performance. Following the first interview, we monitored new initiatives in support over a 2.5-year period, after which we conducted a second interview. Interviews were analysed using thematic analysis.

Main outcome measures

Available policy and support regarding poor performance.

Results

Three themes emerged from our data (i.e. elaborating on professional performance, performance insight and dealing with poor performance) for which we identified a total of 10 categories of support. Support concerned professional codes, guidelines and codes of conduct, quality registers, individual performance assessment, peer consultation, practice evaluation, helpdesk and expert counselling, a protocol for dealing with poor performance, a place for support and to report poor performance, and internal disciplinary procedures.

Conclusions

This study provides an overview of support given to nine healthcare professions by their associations regarding poor performance, and identifies gaps that associations could follow up on, such as clarifying what to do when confronted with a poorly performing colleague, supporting professionals that poorly perform, and developing methods for individual performance assessment to gain performance insight. A next step would be to evaluate the use and effect of different types of support.

The Evaluation of Physicians' Communication Skills From Multiple Perspectives

PURPOSE

To examine how family physicians', patients', and trained clinical raters' assessments of physician-patient communication compare by analysis of individual appointments.

METHODS

Analysis of survey data from patients attending face-to-face appointments with 45 family physicians at 13 practices in England. Immediately post-appointment, patients and physicians independently completed a questionnaire including 7 items assessing communication quality. A sample of videotaped appointments was assessed by trained clinical raters, using the same 7 communication items. Patient, physician, and rater communication scores were compared using correlation coefficients.

RESULTS

Included were 503 physician-patient pairs; of those, 55 appointments were also evaluated by trained clinical raters. Physicians scored themselves, on average, lower than patients (mean physician score 74.5; mean patient score 94.4); 63.4% (319) of patient-reported scores were the maximum of 100. The mean of rater scores from 55 appointments was 57.3. There was a near-zero correlation coefficient between physician-reported and patient-reported communication scores (0.009, P = .854), and between physician-reported and trained rater-reported communication scores (-0.006, P = .69). There was a moderate and statistically significant association, however, between patient and trained-rater scores (0.35, P = .042).

CONCLUSIONS

The lack of correlation between physician scores and those of others indicates that physicians' perceptions of good communication during their appointments may differ from those of external peer raters and patients. Physicians may not be aware of how patients experience their communication practices; peer assessment of communication skills is an important approach in identifying areas for improvement.

Doctors on the move: a European case study on the key characteristics of national recertification systems

OBJECTIVES

With increased cross-border movement, ensuring safe and high-quality healthcare has gained primacy. The purpose of recertification is to ensure quality of care through periodically attesting doctors' professional proficiency in their field. Professional migration and facilitated cross-border recognition of qualifications, however, make us question the fitness of national policies for safeguarding patient care and the international accountability of doctors.

DESIGN AND SETTING

We performed document analyses and conducted 19 semistructured interviews to identify and describe key characteristics and effective components of 10 different European recertification systems, each representing one case (collective case study). We subsequently compared these systems to explore similarities and differences in terms of assessment criteria used to determine process quality.

RESULTS

Great variety existed between countries in terms and assessment formats used, targeting cognition, competence and performance (Miller's assessment pyramid). Recertification procedures and requirements also varied significantly, ranging from voluntary participation in professional development modules to the mandatory collection of multiple performance data in a competency-based portfolio. Knowledge assessment was fundamental to recertification in most countries. Another difference concerned the stakeholders involved in the recertification process: while some systems exclusively relied on doctors' self-assessment, others involved multiple stakeholders but rarely included patients in assessment of doctors' professional competence. Differences between systems partly reflected different goals and primary purposes of recertification.

CONCLUSION

Recertification systems differ substantially internationally with regard to the criteria they apply to assess doctors' competence, their aims, requirements, assessment formats and patient involvement. In the light of professional mobility and associated demands for accountability, we recommend that competence assessment includes patients' perspectives, and recertification practices be shared internationally to enhance transparency. This can help facilitate cross-border movement, while guaranteeing high-quality patient care.

How Residents Learn From Patient Feedback: A Multi-Institutional Qualitative Study of Pediatrics Residents' Perspectives

BACKGROUND

Residents may view feedback from patients and their families with greater skepticism than feedback from supervisors and peers. While discussing patient and family feedback with faculty may improve residents' acceptance of feedback and learning, specific strategies have not been identified.

OBJECTIVE

We explored pediatrics residents' perspectives of patient feedback and identified strategies that promote residents' reflection on and learning from feedback.

METHODS

In this multi-institutional, qualitative study conducted in June and July 2016, we conducted focus groups with a purposive sample of pediatrics residents after their participation in a randomized controlled trial in which they received written patient feedback and either discussed it with faculty or reviewed it independently. Focus group transcripts were audiorecorded, transcribed, and analyzed for themes using the constant comparative approach associated with grounded theory.

RESULTS

Thirty-six of 92 (39%) residents participated in 7 focus groups. Four themes emerged: (1) residents valued patient feedback but felt it may lack the specificity they desire; (2) discussing feedback with a trusted faculty member was helpful for self-reflection; (3) residents identified 5 strategies faculty used to facilitate their openness to and acceptance of patient feedback (eg, help resident overcome emotional responses to feedback and situate feedback in the context of lifelong learning); and (4) residents' perceptions of feedback credibility improved when faculty observed patient encounters and solicited feedback on the resident's behalf prior to discussions.

CONCLUSIONS

Discussing patient feedback with faculty provided important scaffolding to enhance residents' openness to and reflection on patient feedback.

Using parent feedback: A qualitative study of residents' and physician-educators' perspectives

INTRODUCTION

Patients and family members can contribute to resident assessment in competency-based medical education. However, few studies have examined the use of patient/family member feedback generated from questionnaire-based assessments. To implement appropriate assessment strategies and optimize feedback use, we need to understand how residents and physician-educators would use feedback from these stakeholders. This study aimed to understand how paediatric residents and physician-educators would use parent feedback generated from questionnaire-based assessments.

METHODS

This study was conducted at a paediatric academic health science centre. We held dyadic interviews with six residents and six physician-educators. A three-step approach was used to analyze the data: data reduction, data display, and conclusions/verifications. We developed an initial coding scheme, conducted an in-depth review of the data and coded it, finalized our coding scheme, and identified categories.

RESULTS

Participants described that they would use parent feedback to: (a) provide additional direct observations of residents' performances, (b) teach and coach residents, (c) assess residents' overall performance and progression, and (d) encourage resident self-assessment and behaviour change.

DISCUSSION

Parents directly observe residents as they interact with them and their children and, therefore, can provide feedback on residents' performances. Residency programs should include parent feedback and promote and facilitate its use by residents and physician-educators.

CONCLUSION

This study provides an initial understanding of how paediatric residents and physician-educators would use parent feedback if they were to receive it. This information, combined with future research, can inform the development and implementation of parent feedback strategies in competency-based medical education.

Multisource Feedback and Narrative Comments: Polarity, Specificity, Actionability, and CanMEDS Roles

INTRODUCTION

Multisource feedback is a questionnaire-based assessment tool that provides physicians with data about workplace behaviors and may combine numeric and narrative (free-text) comments. Little attention has been paid to wording of requests for comments, potentially limiting its utility to support physician performance. This study tested the phrasing of two different sets of questions.

METHODS

Two sets of questions were tested with family physicians, medical and surgical specialists, and their medical colleague and coworker respondents. One set asked respondents to identify one thing the participant physician does well and one thing the physician could target for action. Set 2 questions asked what does the physician do well and what might the physician do to enhance practice. Resulting free-text comments provided by respondents were coded for polarity (positive, neutral, or negative), specificity (precision and detail), actionability (ability to use the feedback to direct future activity), and CanMEDS roles (competencies) and analyzed descriptively.

RESULTS

Data for 222 physicians (111 physicians per set) were analyzed. A total of 1824 comments (8.2/physician) were submitted, with more comments from coworkers than medical colleagues. Set 1 yielded more comments and were more likely to be positive, semi specific, and very actionable than set 2. However, set 2 generated more very specific comments. Comments covered all CanMEDS roles with more comments for collaborator and leader roles.

DISCUSSION

The wording of questions inviting free-text responses influences the volume and nature of the comments provided. Individuals designing multisource feedback tools should carefully consider wording of items soliciting narrative responses.

Validation of Multisource Feedback in Assessing Medical Performance: A Systematic Review

INTRODUCTION

Over the past 10 years, a number of systematic reviews have evaluated the validity of multisource feedback (MSF) to assess and quality-assure medical practice. The purpose of this study is to synthesize the results from existing reviews to provide a holistic overview of the validity evidence.

METHODS

This review identified eight systematic reviews evaluating the validity of MSF published between January 2006 and October 2016. Using a standardized data extraction form, two independent reviewers extracted study characteristics. A framework of validation developed by the American Psychological Association was used to appraise the validity evidence within each systematic review.

RESULTS

In terms of validity evidence, each of the eight reviews demonstrated evidence across at least one domain of the American Psychological Association's validity framework. Evidence of assessment validity within the domains of "internal structure" and "relationship to other variables" has been well established. However, the domains of content validity (ie, ensuring that MSF tools measure what they are intended to measure); consequential validity (ie, evidence of the intended or unintended consequences MSF assessments may have on participants or wider society), and response process validity (ie, the process of standardization and quality control in the delivery and completion of assessments) remain limited.

DISCUSSION

Evidence for the validity of MSF has, across a number of domains, been well established. However, the size and quality of the existing evidence remains variable. To determine the extent to which MSF is considered a valid instrument to assess medical performance, future research is required to determine the following: (1) how best to design and deliver MSF assessments that address the identified limitations of existing tools and (2) how to ensure that involvement within MSF supports positive changes in practice. Such research is integral if MSF is to continue to inform medical performance and subsequent improvements in the quality and safety of patient care.

The Graduating European Dentist: A New Undergraduate Curriculum Framework

With "The Graduating European Dentist", ADEE provides a new approach that reflects best academic practice for European undergraduate dental education. The new suite of documents sees increased emphasis on important curriculum components such as patient safety, working as a team and patient-centred care. There is also an increased emphasis on teaching excellence, student satisfaction and student preparedness and support. Guidance relating to methods of teaching and learning, and assessment (educational measurement), is also provided. It is anticipated that this new format will further increase accessibility and utility for educators, institutions, societies and regulators, across the sector.

The assessment of medical competencies

OBJECTIVE

To describe the most widely used tools in the assessment of medical competencies, analyse their prevalence of use, their advantages and disadvantages and propose an appropriate model for our context.

METHODS

We conducted a narrative review of articles from MEDLINE, following the PRISM protocol, and analysed a total of 62 articles.

RESULTS

The assessment of competencies is heterogeneous, especially in the educational and professional settings. The specific and technical competencies acquired during university education are mainly assessed using the objective structured clinical assessment. In the professional setting, core competencies are assessed using the 360° technique.

CONCLUSIONS

We need a rigorous empiric comparison of the efficiency of the tools according to the type of competency. We propose a competency management model for the «undergraduate/graduate/active professional» continuum, whose goal is to improve training and professional practice and thereby increase the quality of patient care.