1
|
Azami G, Aazami S, Ebrahimy B, Emami P. Development and psychometric evaluation of a 360-degree evaluation instrument to assess medical students’ performance in clinical settings at the emergency medicine department in Iran: a methodological study. JOURNAL OF EDUCATIONAL EVALUATION FOR HEALTH PROFESSIONS 2024; 21:7. [PMID: 38555955 PMCID: PMC11078574 DOI: 10.3352/jeehp.2024.21.7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/14/2024] [Accepted: 03/18/2024] [Indexed: 04/02/2024]
Abstract
BACKGROUND In the Iranian context, no 360-degree evaluation tool has been developed to assess the performance of prehospital medical emergency students in clinical settings. This article describes the development of a 360-degree evaluation tool and presents its first psychometric evaluation. METHODS There were 2 steps in this study: step 1 involved developing the instrument (i.e., generating the items) and step 2 constituted the psychometric evaluation of the instrument. We performed exploratory and confirmatory factor analyses and also evaluated the instrument’s face, content, and convergent validity and reliability. RESULTS The instrument contains 55 items across 6 domains, including leadership, management, and teamwork (19 items), consciousness and responsiveness (14 items), clinical and interpersonal communication skills (8 items), integrity (7 items), knowledge and accountability (4 items), and loyalty and transparency (3 items). The instrument was confirmed to be a valid measure, as the 6 domains had eigenvalues over Kaiser’s criterion of 1 and in combination explained 60.1% of the variance (Bartlett’s test of sphericity [1,485]=19,867.99, P<0.01). Furthermore, this study provided evidence for the instrument’s convergent validity and internal consistency (α=0.98), suggesting its suitability for assessing student performance. CONCLUSION We found good evidence for the validity and reliability of the instrument. Our instrument can be used to make future evaluations of student performance in the clinical setting more structured, transparent, informative, and comparable.
Collapse
Affiliation(s)
- Golnaz Azami
- Department of Emergency Medical Sciences, Faculty of Paramedical Sciences, Kurdistan University of Medical Sciences, Sanandaj, Iran
| | - Sanaz Aazami
- Department of Nursing and Midwifery, Faculty of Nursing and Midwifery, Ilam University of Medical Science, Ilam, Iran
| | - Boshra Ebrahimy
- Department of Emergency Medical Sciences, Faculty of Paramedical Sciences, Kurdistan University of Medical Sciences, Sanandaj, Iran
| | - Payam Emami
- Department of Emergency Medical Sciences, Faculty of Paramedical Sciences, Kurdistan University of Medical Sciences, Sanandaj, Iran
| |
Collapse
|
2
|
Toews H, Pearce J, Tavares W. Recasting Assessment in Continuing Professional Development as a Person-Focused Activity. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2023; 43:S35-S40. [PMID: 38054490 DOI: 10.1097/ceh.0000000000000538] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/07/2023]
Abstract
ABSTRACT In this article, we examine assessment as conceptualized and enacted in continuing professional development (CPD). Assessment is pervasive throughout the life of an individual health professional, serving many different purposes compounded by varied and unique contexts, each with their own drivers and consequences, usually casting the person as the object of assessment. Assessment is often assumed as an included part in CPD development conceptualization. Research on assessment in CPD is often focused on systems, utility, and quality instead of intentionally examining the link between assessment and the person. We present an alternative view of assessment in CPD as person-centered, practice-informed, situated and bound by capability, and enacted in social and material contexts. With this lens of assessment as an inherently personal experience, we introduce the concept of subjectification, as described by educationalist Gert Biesta. We propose that subjectification may be a fruitful way of examining assessment in a CPD context. Although the CPD community, researchers, and educators consider this further, we offer some early implications of adopting a subjectification lens on the design and enactment of assessment in CPD.
Collapse
Affiliation(s)
- Helen Toews
- Toews: Registered Dietitian, The Wilson Centre, University of Toronto, Toronto, Ontario, Canada. Dr. Pearce: Principal Research Fellow, Specialist and Professional Assessment, Tertiary Education, Australian Council for Educational, Research, Camberwell, Australia. Dr. Tavares: Scientist|Assistant Professor, Department of Health and Society, The Wilson Centre, Department of Medicine, University of Toronto, Scarborough, Ontario, Canada
| | | | | |
Collapse
|
3
|
Lockyer J, Sargeant J. Multisource feedback: an overview of its use and application as a formative assessment. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:30-35. [PMID: 36091727 PMCID: PMC9441111 DOI: 10.36834/cmej.73775] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Multisource feedback (MSF), often termed 360-degree feedback, is a formative performance assessment in which data about an individual's observable workplace behaviors are collected through questionnaires from those interacting with the individual; data are aggregated for anonymity and confidentiality; the aggregated data, along with self-assessment if available, are provided to the individual; and the recipient meets with a trusted individual to review the data and develop an action plan. It is used along the continuum of medical education. This article provides an overview of MSF's utility, its evidence base and cautions.
Collapse
Affiliation(s)
- Jocelyn Lockyer
- Department of Community Health Sciences, Cumming School of Medicine, Alberta, Canada
| | - Joan Sargeant
- Continuing Professional Development and Medical Education, Faculty of Medicine, Dalhousie University, Nova Scotia, Canada
| |
Collapse
|
4
|
Bindels E, van den Goor M, Scherpbier A, Lombarts K, Heeneman S. Sharing Reflections on Multisource Feedback in a Peer Group Setting: Stimulating Physicians' Professional Performance and Development. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:1449-1456. [PMID: 33951680 DOI: 10.1097/acm.0000000000004142] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Reflecting on and using feedback are important for physicians' continuous professional development (CPD). A common format is the discussion of multisource feedback (MSF) in a one-on-one session with a trusted peer or coach. A new approach is to discuss MSF during a peer group session moderated by a professional facilitator. This qualitative study explored how physicians experience participation in these peer group sessions in the context of their CPD. METHOD Between March and July 2018, 26 physicians were interviewed about their experiences in a peer group session. These physicians represented 13 monospecialty physician groups from 5 general hospitals in the Netherlands. Interviews were transcribed verbatim and analyzed iteratively, following the interpretative phenomenological approach. RESULTS Participation was experienced as a process of disclosing and sharing personal reflections with peers while striking a balance between interpersonal proximity to and distance from peers. Sharing reflections with peers rendered the feedback more meaningful, deepened collegial relationships, and created a sense of urgency for improvement. Improvement goals were mostly related to relational fine-tuning in collaboration; goals related to individual career management remained in the background. Influential factors for the perceived effectiveness of the group sessions were related to the facilitator's expertise, group size, continuity and quality of collegial relationships, personal vulnerabilities, and the context of CPD policy. CONCLUSIONS Peer group sessions offered interactivity and established a clear link between individual physicians and their work environments. Sharing reflections on MSF in a peer group setting provided physicians with nuanced insight into their professional performance and fostered a community spirit that supported the implementation of intended changes. Future research should focus on the role of group dynamics and communication strategies and the application of coaching principles, such as drawing up a detailed plan of action and monitoring the follow-up process.
Collapse
Affiliation(s)
- Elisa Bindels
- E. Bindels was a PhD candidate, Amsterdam Center for Professional Performance and Compassionate Care, Department of Medical Psychology, Amsterdam University Medical Centers, Amsterdam, the Netherlands, and Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands, at the time of the study. The author is now a lecturer, Department of Clinical Psychology, Faculty of Social and Behavioural Sciences, Utrecht University, Utrecht, the Netherlands; ORCID: https://orcid.org/0000-0001-6096-4950
| | - Myra van den Goor
- M. van den Goor is CEO, Q3 Company for Physician Development, Den Bosch, the Netherlands; ORCID: https://orcid.org/0000-0002-6361-0178
| | - Albert Scherpbier
- A. Scherpbier is professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0001-9652-0163
| | - Kiki Lombarts
- K. Lombarts is professor, Amsterdam Center for Professional Performance and Compassionate Care, Department of Medical Psychology, Amsterdam University Medical Centers, Amsterdam, the Netherlands; ORCID: https://orcid.org/0000-0001-6167-0620
| | - Sylvia Heeneman
- S. Heeneman is professor, Department of Pathology, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Cardiovascular Research Institute Maastricht, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-6103-8075
| |
Collapse
|
5
|
Bindels E, Boerebach B, Scheepers R, Nooteboom A, Scherpbier A, Heeneman S, Lombarts K. Designing a system for performance appraisal: balancing physicians' accountability and professional development. BMC Health Serv Res 2021; 21:800. [PMID: 34384410 PMCID: PMC8359079 DOI: 10.1186/s12913-021-06818-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2020] [Accepted: 07/23/2021] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND In many healthcare systems, physicians are accustomed to periodically participate in individual performance appraisals to guide their professional development. For the purpose of revalidation, or maintenance of certification, they need to demonstrate that they have engaged with the outcomes of these appraisals. The combination of taking ownership in professional development and meeting accountability requirements may cause undesirable interference of purposes. To support physicians in their professional development, new Dutch legislation requires that they discuss their performance data with a non-hierarchical (peer)coach and draft a personal development plan. In this study, we report on the design of this system for performance appraisal in a Dutch academic medical center. METHODS Using a design-based research approach, a hospital-based research group had the lead in drafting and implementing a performance appraisal protocol, selecting a multisource feedback tool, co-developing and piloting a coaching approach, implementing a planning tool, recruiting peer coaches and facilitating their training and peer group debriefings. RESULTS The system consisted of a two-hour peer-to-peer conversation based on the principles of appreciative inquiry and solution-focused coaching. Sessions were rated as highly motivating, development-oriented, concrete and valuable. Peer coaches were considered suitable, although occasionally physicians preferred a professional coach because of their expertise. The system honored both accountability and professional development purposes. By integrating the performance appraisal system with an already existing internal performance system, physicians were enabled to openly and safely discuss their professional development with a peer, while also being supported by their superior in their self-defined developmental goals. Although the peer-to-peer conversation was mandatory and participation in the process was documented, it was up to the physician whether or not to share its results with others, including their superior. CONCLUSIONS In the context of mandatory revalidation, professional development can be supported when the appraisal process involves three characteristics: the appraisal process is appreciative and explores developmental opportunities; coaches are trustworthy and skilled; and the physician has control over the disclosure of the appraisal output. Although the peer-to-peer conversations were positively evaluated, the effects on physicians' professional development have yet to be investigated in longitudinal research designs.
Collapse
Affiliation(s)
- Elisa Bindels
- Department of Medical Psychology, Amsterdam Center for Professional Performance and Compassionate Care, Amsterdam University Medical Centers, Amsterdam, the Netherlands.
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands.
- Department of Clinical Psychology, Faculty of Social and Behavioral Sciences, Utrecht University, Heidelberglaan 1, H1, 54, 3584 CS, Utrecht, the Netherlands.
| | - Benjamin Boerebach
- Department of Value-Based Healthcare, St. Antonius Hospital, Nieuwegein, the Netherlands
| | - Renée Scheepers
- Department of Socio-Medical Sciences, Erasmus School of Health Policy and Management, Erasmus University of Rotterdam, Rotterdam, the Netherlands
| | - Annemiek Nooteboom
- Department of Intensive Care, Amsterdam University Medical Centers, Amsterdam, the Netherlands
- Nooteboom Consult, Amsterdam, the Netherlands
| | - Albert Scherpbier
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Sylvia Heeneman
- Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Kiki Lombarts
- Department of Medical Psychology, Amsterdam Center for Professional Performance and Compassionate Care, Amsterdam University Medical Centers, Amsterdam, the Netherlands
| |
Collapse
|
6
|
van der Meulen MW, Arah OA, Heeneman S, Oude Egbrink MGA, van der Vleuten CPM, Lombarts KMJMH. When Feedback Backfires: Influences of Negative Discrepancies Between Physicians' Self and Assessors' Scores on Their Subsequent Multisource Feedback Ratings. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2021; 41:94-103. [PMID: 34009839 DOI: 10.1097/ceh.0000000000000347] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
INTRODUCTION With multisource feedback (MSF) physicians might overrate their own performance compared with scores received from assessors. However, there is limited insight into how perceived divergent feedback affects physicians' subsequent performance scores. METHODS During 2012 to 2018, 103 physicians were evaluated twice by 684 peers, 242 residents, 999 coworkers, and themselves in three MSF performance domains. Mixed-effect models quantified associations between the outcome variable "score changes" between first and second MSF evaluations, and the explanatory variable "negative discrepancy score" (number of items that physicians rated themselves higher compared with their assessors' scores) at the first MSF evaluation. Whether associations differed across assessor groups and across a physician's years of experience as a doctor was analyzed too. RESULTS Forty-nine percent of physicians improved their total MSF score at the second evaluation, as assessed by others. Number of negative discrepancies was negatively associated with score changes in domains "organization and (self)management" (b = -0.02; 95% confidence interval [CI], -0.03 to -0.02; SE = 0.004) and "patient-centeredness" (b = -0.03; 95% CI, -0.03 to -0.02; SE = 0.004). For "professional attitude," only negative associations between score changes and negative discrepancies existed for physicians with more than 6-year experience (b6-10yearsofexperience = -0.03; 95% CI, -0.05 to -0.003; SE = 0.01; b16-20yearsofexperience = -0.03; 95% CI, -0.06 to -0.004; SE = 0.01). DISCUSSION The extent of performance improvement was less for physicians confronted with negative discrepancies. Performance scores actually declined when physicians overrated themselves on more than half of the feedback items. PA score changes of more experienced physicians confronted with negative discrepancies and were affected more adversely. These physicians might have discounted feedback due to having more confidence in own performance. Future work should investigate how MSF could improve physicians' performance taking into account physicians' confidence.
Collapse
Affiliation(s)
- Mirja W van der Meulen
- Dr. van der Meulen: is PhD Candidate, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands, and Professional Performance and Compassionate Care Research Group, Department of Medical Psychology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands. Dr. Arah: is professor, Department of Epidemiology, University of California, Los Angeles (UCLA), Los Angeles, the United States of America. Dr. Heeneman: is professor, Department of Pathology, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Cardiovascular Research Institute Maastricht, Maastricht University, Maastricht, the Netherlands. Dr. oude Egbrink: is professor, Department of Physiology, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. Dr. van der Vleuten: is professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. Dr. Lombarts: is professor, Professional Performance and Compassionate Care Research Group, Department of Medical Psychology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands
| | | | | | | | | | | |
Collapse
|
7
|
Hennel EK, Subotic U, Berendonk C, Stricker D, Harendza S, Huwendiek S. A german-language competency-based multisource feedback instrument for residents: development and validity evidence. BMC MEDICAL EDUCATION 2020; 20:357. [PMID: 33046060 PMCID: PMC7552497 DOI: 10.1186/s12909-020-02259-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/22/2019] [Accepted: 09/28/2020] [Indexed: 06/11/2023]
Abstract
BACKGROUND In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor's performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity. METHODS We developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, "unable to comment" ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents' learning goals and the progress as reported via MSF. RESULTS Our resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician's competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach's alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself. CONCLUSIONS To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.
Collapse
Affiliation(s)
- Eva K. Hennel
- Department for Assessment and Evaluation (AAE), Institute for Medical Education, University of Bern, Mittelstrasse 43, 3012 Bern, Switzerland
| | - Ulrike Subotic
- University Children’s Hospital Basel, Spitalstrasse 33, 4056 Basel, Switzerland
| | - Christoph Berendonk
- Department for Assessment and Evaluation (AAE), Institute for Medical Education, University of Bern, Mittelstrasse 43, 3012 Bern, Switzerland
| | - Daniel Stricker
- Department for Assessment and Evaluation (AAE), Institute for Medical Education, University of Bern, Mittelstrasse 43, 3012 Bern, Switzerland
| | - Sigrid Harendza
- Department of Internal Medicine, University Medical Centre Hamburg-Eppendorf, Martinistr. 52, 20246 Hamburg, Germany
| | - Sören Huwendiek
- Department for Assessment and Evaluation (AAE), Institute for Medical Education, University of Bern, Mittelstrasse 43, 3012 Bern, Switzerland
| |
Collapse
|
8
|
Bindels E, Boerebach B, van der Meulen M, Donkers J, van den Goor M, Scherpbier A, Lombarts K, Heeneman S. A New Multisource Feedback Tool for Evaluating the Performance of Specialty-Specific Physician Groups: Validity of the Group Monitor Instrument. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2019; 39:168-177. [PMID: 31306280 DOI: 10.1097/ceh.0000000000000262] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
INTRODUCTION Since clinical practice is a group-oriented process, it is crucial to evaluate performance on the group level. The Group Monitor (GM) is a multisource feedback tool that evaluates the performance of specialty-specific physician groups in hospital settings, as perceived by four different rater classes. In this study, we explored the validity of this tool. METHODS We explored three sources of validity evidence: (1) content, (2) response process, and (3) internal structure. Participants were 254 physicians, 407 staff, 621 peers, and 282 managers of 57 physician groups (in total 479 physicians) from 11 hospitals. RESULTS Content was supported by the fact that the items were based on a review of an existing instrument. Pilot rounds resulted in reformulation and reduction of items. Four subscales were identified for all rater classes: Medical practice, Organizational involvement, Professionalism, and Coordination. Physicians and staff had an extra subscale, Communication. However, the results of the generalizability analyses showed that variance in GM scores could mainly be explained by the specific hospital context and the physician group specialty. Optimization studies showed that for reliable GM scores, 3 to 15 evaluations were needed, depending on rater class, hospital context, and specialty. DISCUSSION The GM provides valid and reliable feedback on the performance of specialty-specific physician groups. When interpreting feedback, physician groups should be aware that rater classes' perceptions of their group performance are colored by the hospitals' professional culture and/or the specialty.
Collapse
Affiliation(s)
- Elisa Bindels
- Ms. Bindels: PhD Candidate, Department of Medical Psychology, Amsterdam Center for Professional Performance and Compassionate Care, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands, and Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. Dr. Boerebach: Staff Advisor, Department of Medical Psychology, Amsterdam Center for Professional Performance and Compassionate Care, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands. Ms. van der Meulen: PhD Candidate, Department of Medical Psychology, Amsterdam Center for Professional Performance and Compassionate Care, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands, and Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. Dr. Donkers: Assistant Professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. Dr. van den Goor: PhD Candidate, Department of Medical Psychology, Amsterdam Center for Professional Performance and Compassionate Care, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands, and Q3 Consult, Zeist, the Netherlands. Dr. Scherpbier: Professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. Dr. Lombarts: Professor, Department of Medical Psychology, Amsterdam Center for Professional Performance and Compassionate Care, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands. Dr. Heeneman: Professor, Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | | | | | | | | | | | | | | |
Collapse
|
9
|
Lockyer JM, Sargeant J, Richards SH, Campbell JL, Rivera LA. Multisource Feedback and Narrative Comments: Polarity, Specificity, Actionability, and CanMEDS Roles. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2018; 38:32-40. [PMID: 29329147 DOI: 10.1097/ceh.0000000000000183] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
INTRODUCTION Multisource feedback is a questionnaire-based assessment tool that provides physicians with data about workplace behaviors and may combine numeric and narrative (free-text) comments. Little attention has been paid to wording of requests for comments, potentially limiting its utility to support physician performance. This study tested the phrasing of two different sets of questions. METHODS Two sets of questions were tested with family physicians, medical and surgical specialists, and their medical colleague and coworker respondents. One set asked respondents to identify one thing the participant physician does well and one thing the physician could target for action. Set 2 questions asked what does the physician do well and what might the physician do to enhance practice. Resulting free-text comments provided by respondents were coded for polarity (positive, neutral, or negative), specificity (precision and detail), actionability (ability to use the feedback to direct future activity), and CanMEDS roles (competencies) and analyzed descriptively. RESULTS Data for 222 physicians (111 physicians per set) were analyzed. A total of 1824 comments (8.2/physician) were submitted, with more comments from coworkers than medical colleagues. Set 1 yielded more comments and were more likely to be positive, semi specific, and very actionable than set 2. However, set 2 generated more very specific comments. Comments covered all CanMEDS roles with more comments for collaborator and leader roles. DISCUSSION The wording of questions inviting free-text responses influences the volume and nature of the comments provided. Individuals designing multisource feedback tools should carefully consider wording of items soliciting narrative responses.
Collapse
Affiliation(s)
- Jocelyn M Lockyer
- Dr. Lockyer: Professor, Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada. Dr. Sargeant: Professor, Medical Education, Dalhousie University, Halifax, Nova Scotia, Canada. Dr. Richards: Academic Unit of Primary Care, Leeds Institute of Health Sciences, University of Leeds, Leeds, United Kingdom. Dr. Campbell: Professor of General Practice and Primary Care and Director, University of Exeter Collaboration for Academic Primary Care (APEx), University of Exeter Medical School, Exeter, United Kingdom. Ms. Rivera: Research Associate, Office of Continuing Medical Education and Professional Development, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
| | | | | | | | | |
Collapse
|
10
|
Ramani S, Könings K, Mann KV, van der Vleuten C. Uncovering the unknown: A grounded theory study exploring the impact of self-awareness on the culture of feedback in residency education. MEDICAL TEACHER 2017; 39:1065-1073. [PMID: 28741446 DOI: 10.1080/0142159x.2017.1353071] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/09/2023]
Abstract
AIM Self-assessment and reflection are essential for meaningful feedback. We aimed to explore whether the well-known Johari window model of self-awareness could guide feedback conversations between faculty and residents and enhance the institutional feedback culture. METHODS We had previously explored perceptions of residents and faculty regarding sociocultural factors impacting feedback. We re-analyzed data targeting themes related to self-assessment, reflection, feedback seeking and acceptance, aiming to generate individual and institutional feedback strategies applicable to each quadrant of the window. RESULTS We identified the following themes for each quadrant: (1) Behaviors known to self and others - Validating the known; (2) Behaviors unknown to self but known to others - Accepting the blind; (3) Behaviors known to self and unknown to others - Disclosure of hidden; and (4) Behaviors unknown to self and others - Uncovering the unknown. Normalizing self-disclosure of limitations, encouraging feedback seeking, training in nonjudgmental feedback and providing opportunities for longitudinal relationships could promote self-awareness, ultimately expanding the "open" quadrant of the Johari window. CONCLUSIONS The Johari window, a model of self-awareness in interpersonal communications, could provide a robust framework for individuals to improve their feedback conversations and institutions to design feedback initiatives that enhance its quality and impact.
Collapse
Affiliation(s)
- Subha Ramani
- a Department of Medicine , Brigham and Women's Hospital, Harvard Medical School , Boston , MA , USA
| | - Karen Könings
- b School of Health Professions Education , Maastricht University , Maastricht , The Netherlands
| | | | - Cees van der Vleuten
- b School of Health Professions Education , Maastricht University , Maastricht , The Netherlands
| |
Collapse
|
11
|
Olsson JE, Wallentin FY, Toth-Pal E, Ekblad S, Bertilson BC. Psychometric analysis of the Swedish version of the General Medical Council's multi source feedback questionnaires. INTERNATIONAL JOURNAL OF MEDICAL EDUCATION 2017; 8:252-261. [PMID: 28704204 PMCID: PMC5511742 DOI: 10.5116/ijme.5948.0bb6] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/31/2017] [Accepted: 06/19/2017] [Indexed: 05/17/2023]
Abstract
OBJECTIVES To determine the internal consistency and the underlying components of our translated and adapted Swedish version of the General Medical Council's multisource feedback questionnaires (GMC questionnaires) for physicians and to confirm which aspects of good medical practice the latent variable structure reflected. METHODS From October 2015 to March 2016, residents in family medicine in Sweden were invited to participate in the study and to use the Swedish version to perform self-evaluations and acquire feedback from both their patients and colleagues. The validation focused on internal consistency and construct validity. Main outcome measures were Cronbach's alpha coefficients, Principal Component Analysis, and Confirmatory Factor Analysis indices. RESULTS A total of 752 completed questionnaires from patients, colleagues, and residents were analysed. Of these, 213 comprised resident self-evaluations, 336 were feedback from residents' patients, and 203 were feedback from residents' colleagues. Cronbach's alpha coefficients of the scores were 0.88 from patients, 0.93 from colleagues, and 0.84 in the self-evaluations. The Confirmatory Factor Analysis validated two models that fit the data reasonably well and reflected important aspects of good medical practice. The first model had two latent factors for patient-related items concerning empathy and consultation management, and the second model had five latent factors for colleague-related items, including knowledge and skills, attitude and approach, reflection and development, teaching, and trust. CONCLUSIONS The current Swedish version seems to be a reliable and valid tool for formative assessment for resident physicians and their supervisors. This needs to be verified in larger samples.
Collapse
Affiliation(s)
| | | | - Eva Toth-Pal
- Division of Family Medicine, Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden
| | - Solvig Ekblad
- Academic Primary Healthcare Centre, Huddinge, Sweden
| | | |
Collapse
|