From the eye of the nurses: 360-degree evaluation of residents

The Ottawa resident observation form for nurses (O-RON): evaluation of an assessment tool's psychometric properties in different specialties

BACKGROUND

Workplace-based assessment (WBA) used in post-graduate medical education relies on physician supervisors' feedback. However, in a training environment where supervisors are unavailable to assess certain aspects of a resident's performance, nurses are well-positioned to do so. The Ottawa Resident Observation Form for Nurses (O-RON) was developed to capture nurses' assessment of trainee performance and results have demonstrated strong evidence for validity in Orthopedic Surgery. However, different clinical settings may impact a tool's performance. This project studied the use of the O-RON in three different specialties at the University of Ottawa.

METHODS

O-RON forms were distributed on Internal Medicine, General Surgery, and Obstetrical wards at the University of Ottawa over nine months. Validity evidence related to quantitative data was collected. Exit interviews with nurse managers were performed and content was thematically analyzed.

RESULTS

179 O-RONs were completed on 30 residents. With four forms per resident, the ORON's reliability was 0.82. Global judgement response and frequency of concerns was correlated (r = 0.627, P < 0.001).

CONCLUSIONS

Consistent with the original study, the findings demonstrated strong evidence for validity. However, the number of forms collected was less than expected. Exit interviews identified factors impacting form completion, which included clinical workloads and interprofessional dynamics.

Leveraging the vantage point - exploring nurses' perception of residents' communication skills: a mixed-methods study

INTRODUCTION

Effective communication is key to a successful patient-doctor interaction and improved healthcare outcomes. However, communication skills training in residency is often subpar, leading to inadequate patient-physician communication. There is a dearth of studies exploring the observations of nurses - key members of healthcare teams with a special vantage point to observe the impact of residents' communication with patients. Thus, we aimed to gauge the perceptions of nurses regarding residents' communication skills expertise.

METHODS

This study employed a sequential mixed-methods design, and was conducted at an academic medical center in South Asia. Quantitative data was collected via a REDCap survey using a structured validated questionnaire. Ordinal logistic regression was applied. For qualitative data, In-depth interviews were conducted with nurses using a semi-structured interview guide.

RESULTS

A total of 193 survey responses were obtained from nurses hailing from various specialties including Family Medicine (n = 16), Surgery (n = 27), Internal Medicine (n = 22), Pediatrics (n = 27), and Obstetrics/Gynecology (n = 93). Nurses rated long working hours, infrastructural deficits, and human failings as the main barriers to effective patient-resident communication. Residents working in in-patient settings were more likely to have inadequate communication skills (P-value = 0.160). Qualitative data analysis of nine in-depth interviews revealed two major themes: existing status-quo of residents' communication skills (including deficient verbal and non-verbal communication, bias in patient counselling and challenging patients) and recommendations for improving patient-resident communication.

CONCLUSION

The findings from this study highlight significant gaps in patient-resident communication from the perception of nurses and identify the need for creating a holistic curriculum for residents to improve patient-physician interaction.

A Unique Lens: Understanding What Nurses Are Best Positioned to Assess About Residents

BACKGROUND

Resident feedback is generally elicited from attending physicians, although nurses can also provide feedback on distinct domains. Physicians may be hesitant to accept feedback from nurses if they perceive that nurses are being asked about areas outside their expertise. Understanding specific resident behaviors that nurses are best suited to assess is critical to successful implementation of feedback from nurses to residents.

OBJECTIVE

To understand specific resident behaviors nurses are uniquely positioned to assess from the perspectives of both nurses and residents.

METHODS

We performed a qualitative study using thematic analysis of 5 focus groups with 20 residents and 5 focus groups with 17 nurses at a large free-standing children's hospital in 2020. Two reviewers developed a codebook and subsequently analyzed all transcripts. Codes were organized into themes and subthemes. Thematic saturation was achieved prior to analyzing the final transcript.

RESULTS

We identified 4 major themes. Nurses are positioned to provide feedback: (1) on residents' interprofessional collaborative practice; (2) on residents' communication with patients and their families; and (3) on behalf of patients and their families. Within each of these, we identified subthemes noting specific behaviors on which nurses can provide feedback. The fourth theme encompassed topics that may not be best suited for nursing feedback-medical decision-making and resident responsiveness.

CONCLUSIONS

Nurses and residents described specific resident behaviors that nurses were best positioned to assess.

Continuous multidimensional assessment system of medical residents in teaching hospitals

Background and aims

Evaluation of learners is considered as one of the most important principles in education, especially in the clinical fields. Continuous assessment can be used to provide appropriate feedback to students about their strengths and weaknesses. In this regard, this study is aimed to design a system of continuous assessment of medical residents (SCMARs).

Methods

This study was performed using a combination of qualitative methods, including focus group discussion, expert panel, and Delphi technique. The study population consisted of all the stakeholders involved in the evaluation process of medical residents in Imam Reza Hospital Complex (IRHC) in Iran. This study was conducted in three phases, including identification of subthemes and indicators, production of a primary framework for the SCMAR, and agreement on the subthemes of the SCMAR. The nominal group technique was used for generating priority information. Data analysis was performed during the agreement review stage with the Excel software version 2016.

Results

The finalized SCMAR consisted of 10 main themes and 38 subthemes. The themes included objectives, evaluators, areas, and indicators of evaluation, evaluation periods, evaluation requirements, data collection, data sources, point assignment and data analysis methods, reporting, and feedback dissemination methods. Five areas of evaluation and 11 indicators of evaluation were proposed.

Conclusion

A comprehensive assessment system that continuously evaluates the performance of Medical Residents can be used as a stimulus to improve the quality of educational processes. The present study was aimed to address this need by designing a framework for such a system.

Unobserved Observers: Nurses' Perspectives About Sharing Feedback on the Performance of Resident Physicians

PURPOSE

Postgraduate training programs are incorporating feedback from registered nurses (RNs) to facilitate holistic assessments of resident performance. RNs are a potentially rich source of feedback because they often observe trainees during clinical encounters when physician supervisors are not present. However, RN perspectives about sharing feedback have not been deeply explored. This study investigated RN perspectives about providing feedback and explored the facilitators and barriers influencing their engagement.

METHOD

Constructivist grounded theory methodology was used in interviewing 11 emergency medicine and 8 internal medicine RNs at 2 campuses of a tertiary care academic medical center in Ontario, Canada, between July 2019 and March 2020. Interviews explored RN experiences working with and observing residents in clinical practice. Data collection and analysis were conducted iteratively. Themes were identified using constant comparative analysis.

RESULTS

RNs felt they could observe authentic day-to-day behaviors of residents often unwitnessed by supervising physicians and offer unique feedback related to patient advocacy, communication, leadership, collaboration, and professionalism. Despite a strong desire to contribute to resident education, RNs were apprehensive about sharing feedback and reported barriers related to hierarchy, power differentials, and a fear of overstepping professional boundaries. Although infrequent, a key stimulus that enabled RNs to feel safe in sharing feedback was an invitation from the supervising physician to provide input.

CONCLUSIONS

Perceived hierarchy in academic medicine is a critical barrier to engaging RNs in feedback for residents. Accessing RN feedback on authentic resident behaviors requires dismantling the negative effects of hierarchy and fostering a collaborative interprofessional working environment. A critical step toward this goal may require supervising physicians to model feedback-seeking behavior by inviting RNs to share feedback. Until a workplace culture is established that validates nurses' input and creates safe opportunities for them to contribute to resident education, the voices of nurses will remain unheard.

The Ottawa Resident Observation Form for Nurses (O-RON): Assessment of Resident Performance through the Eyes of the Nurses

OBJECTIVE

Most work-place based assessment relies on physician supervisors making observations of residents. Many areas of performance are not directly observed by physicians but rather by other healthcare professionals, most often nurses. Assessment of resident performance by nurses is captured with multi-source feedback tools. However, these tools combine the assessments of nurses with other healthcare professionals and so their perspective can be lost. A novel tool was developed and implemented to assess resident performance on a hospital ward from the perspective of the nurses.

DESIGN

Through a nominal group technique, nurses identified dimensions of performance that are reflective of high-quality physician performance on a hospital ward. These were included as items in the Ottawa Resident Observation Form for Nurses (O-RON). The O-RON was voluntarily completed during an 11-month period. Validity evidence related to quantitative and qualitative data was collected.

SETTING

The Orthopedic Surgery Residency Program at the University of Ottawa.

PARTICIPANTS

49 nurses on the Orthopedic Surgery wards at The Ottawa Hospital (tertiary care).

RESULTS

The O-RON has 15 items rated on a 3-point frequency scale, one global judgment yes/no question regarding whether they would want the resident on their team and a space for comments. 1079 O-RONs were completed on 38 residents. There was an association between the response to the global judgment question and the frequency of concerns (p < 0.01). With 8 forms per resident, the reliability of the O-RON was 0.80. Open-ended responses referred to aspects of interpersonal skills, responsiveness, dependability, communication skills, and knowledge.

CONCLUSIONS

The O-RON demonstrates promise as a work-place based assessment tool to provide residents and training programs with feedback on aspects of their performance on a hospital ward through the eyes of the nurses. It appears to be easy to use, has solid evidence for validity and can provide reliable data with a small number of completed forms.

Factors influencing the rating of sonographer students' clinical performance

INTRODUCTION

Little is known about the factors influencing clinical supervisor-assessors' ratings of sonographer students' performance. This study identifies these influential factors and relates them to professional competency standards, with the aim of raising awareness and improving assessment practice.

METHODS

This study used archived written comments from 94 clinical assessors describing 174 sonographer students' performance one month into their initial clinical practice (2015-6). Qualitative mixed method analysis revealed factors influencing assessor ratings of student performance and provided an estimate of the valency, association, and frequency of these factors.

RESULTS

Assessors provided written comments for 93 % (n = 162/174) of students. Comments totaled 7190 words (mean of 44 words/student). One-third of comment paragraphs were wholly positive, two-thirds were equivocal. None were wholly negative. Thematic analysis revealed eleven factors, and eight sub-factors, influencing assessor impressions of five dimensions of performance. Of the factors mentioned, 84.6 % (n = 853/1008) related to professional competencies. While 15.4 % (n = 155/1008) were unrelated to competencies, instead reflecting humanistic factors such as student motivation, disposition, approach to learning, prospects and impact on supervisor and staff. Factors were prioritised and combined independently, although some associated.

CONCLUSION

Clinical assessors formed impressions based on student performance, humanistic behaviours and personal qualities not necessarily outlined in educational outcomes or professional competency standards. Their presence, and interrelations, impact success in clinical practice, through their contribution to, and indication of, competence.

IMPLICATIONS FOR PRACTICE

Sonographer student curricula and assessor training should raise awareness of the factors influencing performance ratings and judgement of clinical competence, particularly the importance of humanistic factors. Inclusion of narrative comments, multiple assessors, and broad performance dimensions would enhance clinical assessment of sonographer student performance.

Multisource Feedback in the Trauma Context: Priorities and Perspectives

OBJECTIVES

Trauma resuscitations require competence in both clinical and nonclinical skills but these can be difficult to observe and assess. Multisource feedback (MSF) is workplace-based, involves the direct observation of learners, and can provide feedback on nonclinical skills. We sought to compare and contrast the priorities of multidisciplinary trauma team members when assessing resident trauma team captain (TTC) performance. Additionally, we aimed to describe the nature of the assessment and perceived the utility of incorporating MSF into the trauma context.

METHODS

A convenience sample of 10 trauma team activations were observed. Following each activation, the attending physician trauma team leader (TTL), TTC, and a registered nurse (RN) participated in a semistructured interview. MSF was not provided to the TTC for the purpose of this study because MSF was not part of the assessment process of TTCs at the time of this study and maintaining anonymity may have encouraged more honest interview responses. Transcripts from each assessor group (TTL, TTC, RN) were coded and assigned to one of the five crisis resource management skills: leadership, communication, situational awareness, resource utilization, and problem-solving. Comments were also coded as positive, negative, or neutral as interpreted by the coder.

RESULTS

All assessor groups mentioned communication skills most frequently. After communication, the RN and TTC groups commented on situational awareness most frequently, comprising 15 and 29% of their total responses, respectively, whereas 31% of the TTL comments focused on leadership skills. The RN and TTL groups provided positive assessments, with 51 and 42% of their respective comments coded as positive. Forty-five percent of self-assessment comments in the TTC group were negative. All (100%) of the TTC and TTL respondents felt that incorporating MSF would add to the quality of feedback, only 66% of the RN group felt that way.

CONCLUSIONS

We found that each assessor group brings a unique focus and perspective to the assessment of resident TTC performance. The future inclusion of MSF in the trauma team context has the potential to enhance the learning environment in a clinical arena that is difficult to directly observe and assess.

Off to a Jump Start: Using Immersive Activities to Integrate Continuity Clinic and Advocacy

BACKGROUND

Training in advocacy is an important component of graduate medical education. Several models have been implemented by residency programs to address this objective. Little has been published regarding application of immersive advocacy activities integrated into continuity clinic.

OBJECTIVE

To create an Integrated Community Health and Child Advocacy Curriculum (ICHCA) by integrating advocacy activities that were immersive and contextualized in a continuity clinic setting and to familiarize interns with continuity clinic immediately at the beginning of their training.

METHODS

We utilized a socio-constructivist lens, Kern's Six-step curriculum development and a published curriculum mapping tool to create the curriculum. Twenty residents completed ICHCA in 2019. Evaluations from key stakeholders including participants, support staff and attendings were analyzed on four levels of Kirkpatrick's model. We compared results before intervention, immediately following intervention and ten months following intervention.

RESULTS

We demonstrated improvement in learner satisfaction, knowledge and behaviors with respect to advocacy in the clinical environment. Response rate was 70% (7/10) for attendings, 75% for support staff (15/20) and 72.5% for residents (29/40). Our intervention was feasible, no cost, and required no additional materials or training as it relied on learning in real time.

CONCLUSIONS

An integrated advocacy curriculum utilizing the mapping tool for curricular design and evaluation is feasible and has value demonstrated by improvements in reaction, knowledge, and behaviors. This model improves understanding of social responsibility and can be implemented similarly in other residency programs.

Evaluating the effectiveness of undergraduate clinical education programs

Medical schools should use a variety of measures to evaluate the effectiveness of their clinical curricula. Both outcome measures and process measures should be included, and these can be organized according to the four-level training evaluation model developed by Donald Kirkpatrick. Managing evaluation data requires the institution to employ deliberate strategies to monitor signals in real-time and aggregate data so that informed decisions can be made. Future steps in program evaluation includes increased emphasis on patient outcomes and multi-source feedback, as well as better integration of existing data sources.

Faculty Assessments in a Military Medical Field Practicum: Rater Experience and Gender Do Not Appear to Influence Scoring

INTRODUCTION

Any implicit and explicit biases that exist may alter our interpretation of people and events. Within the context of assessment, it is important to determine if biases exist and to decrease any existing biases, especially when rating student performance to provide meaningful, fair, and useful input. The purpose of this study was to determine if the experience and gender of faculty members contribute to their ratings of students in a military medical field practicum. This information is important for fair ratings of students. Three research questions were addressed: Were there differences between new versus experienced faculty raters? Were there differences in assessments provided by female and male faculty members? Did gender of faculty raters impact ratings of female and male students?.

MATERIALS AND METHODS

This study examined trained faculty evaluators' ratings of three cohorts of medical students during 2015-2017 during a medical field practicum. Female (n = 80) and male (n = 161) faculty and female (n = 158) and male (n = 311) students were included. Within this dataset, there were 469 students and 241 faculty resulting in 5,599 ratings for each of six outcome variables that relate to overall leader performance, leader competence, and leader communication. Descriptive statistics were computed for all variables for the first four observations of each student. Descriptive analyses were performed for evaluator experience status and gender differences by each of the six variables. A multivariate analyses of variance was performed to examine whether there were differences between gender of faculty and gender of students.

RESULTS

Descriptive analyses of the experience status of faculty revealed no significant differences between means on any of the rating elements. Descriptive analyses of faculty gender revealed no significant differences between female and male faculty ratings of the students. The overall MANOVA analyses found no statistically significant difference between female and male students on the combined dependent variables of leader performance for any of the four observations.

CONCLUSIONS

The study revealed that there were no differences in ratings of student leader performance based on faculty experience. In addition, there were no differences in ratings of student leader performance based on faculty gender.

Exploring Gender Bias in Nursing Evaluations of Emergency Medicine Residents

OBJECTIVES

Nursing evaluations are an important component of residents' professional development as nurses are present for interactions with patients and nonphysician providers. Despite this, there has been few prior studies on the benefits, harms, or effectiveness of using nursing evaluations to help guide emergency medicine residents' development. We hypothesized that gender bias exists in nursing evaluations and that female residents, compared to their male counterparts, would receive more negative feedback on the perception of their interpersonal communication skills.

METHODS

Data were drawn from nursing evaluations of residents between March 2013 and April 2016. All comments were coded if they contained words falling into four main categories: standout, ability, grindstone, and interpersonal. This methodology and the list of words that guided coding were based on the work of prior scholars. Names and gendered pronouns were obscured and each comment was manually reviewed and coded for valence (positive, neutral, negative) and strength (certain or tentative) by at least two members of the research team. Following the qualitative coding, quantitative analysis was performed to test for differences. To evaluate whether any measurable differences in ability between male and female residents existed, we compiled and compared American Board of Emergency Medicine in-training examination scores and relevant milestone evaluations between female and male residents from the same period in which the residents were evaluated by nursing staff.

RESULTS

Of 1,112 nursing evaluations, 30% contained comments. Chi-square tests on the distribution of valence (positive, neutral, or negative) indicated statistically significant differences in ability and grindstone categories based on the gender of the resident. A total of 51% of ability comments about female residents were negative compared to 20% of those about male residents (χ²  = 11.83, p < 0.01). A total of 57% of grindstone comments about female residents were negative as opposed 24% of those about male residents (χ²  = 6.03, p < 0.01).

CONCLUSIONS

Our findings demonstrate that, despite the lack of difference in ability or competence as measured by in-service examination scores and milestone evaluations, nurses evaluate female residents lower in their abilities and work ethic compared to male residents.

The Feedback Tango: An Integrative Review and Analysis of the Content of the Teacher-Learner Feedback Exchange

PURPOSE

To conduct an integrative review and analysis of the literature on the content of feedback to learners in medical education.

METHOD

Following completion of a scoping review in 2016, the authors analyzed a subset of articles published through 2015 describing the analysis of feedback exchange content in various contexts: audiotapes, clinical examination, feedback cards, multisource feedback, videotapes, and written feedback. Two reviewers extracted data from these articles and identified common themes.

RESULTS

Of the 51 included articles, about half (49%) were published since 2011. Most involved medical students (43%) or residents (43%). A leniency bias was noted in many (37%), as there was frequently reluctance to provide constructive feedback. More than one-quarter (29%) indicated the feedback was low in quality (e.g., too general, limited amount, no action plans). Some (16%) indicated faculty dominated conversations, did not use feedback forms appropriately, or provided inadequate feedback, even after training. Multiple feedback tools were used, with some articles (14%) describing varying degrees of use, completion, or legibility. Some articles (14%) noted the impact of the gender of the feedback provider or learner.

CONCLUSIONS

The findings reveal that the exchange of feedback is troubled by low-quality feedback, leniency bias, faculty deficient in feedback competencies, challenges with multiple feedback tools, and gender impacts. Using the tango dance form as a metaphor for this dynamic partnership, the authors recommend ways to improve feedback for teachers and learners willing to partner with each other and engage in the complexities of the feedback exchange.

The International Council of Ophthalmology 360-degree assessment tool: development and validation

BACKGROUND

The Accreditation Council for Graduate Medical Education and other organizations recommend 360-degree assessments for evaluation of interpersonal and communication skills, professional behaviours, and some aspects of patient care and system-based practice. No such tool has been developed for ophthalmology or received international content validation.

OBJECTIVE

To develop a valid, internationally applicable, ophthalmology-specific 360-degree assessment tool.

DESIGN

Exploratory study.

METHODS

A literature review was conducted. Individual 360-degree evaluation items from several publications were catalogued and classified according to different groups of assessors. A panel of international authors reviewed the list and voted on items that were most appropriate for international use. The list was trimmed to reduce redundancy and to make it as brief as possible while still capturing the essential components for each category. A second panel of international ophthalmic educators reviewed the international applicability and appropriateness of this collated list; relevant comments and suggestions were incorporated.

RESULTS

A tool for the evaluation of interpersonal and communication skills, professionalism, and system-based practice was developed. The tool has face and content validity.

CONCLUSION

This assessment tool can be used internationally for giving formative feedback based on the opinions of the different groups of people who interact with residents.

Critical Deficiency Ratings in Milestone Assessment: A Review and Case Study

PURPOSE

The Accreditation Council for Graduate Medical Education (ACGME) requires programs to report learner progress using specialty-specific milestones. It is unclear how milestones can best identify critical deficiencies (CDs) in trainee performance. Specialties developed milestones independently of one another; not every specialty included CDs within milestones ratings. This study examined the proportion of ACGME milestone sets that include CD ratings, and describes one residency program's experiences using CD ratings in assessment.

METHOD

The authors reviewed ACGME milestones for all 99 specialties in November 2015, determining which rating scales contained CDs. The authors also reviewed three years of data (July 2012-June 2015) from the University of Cincinnati Medical Center (UCMC) internal medicine residency assessment system based on observable practice activities mapped to ACGME milestones. Data were analyzed by postgraduate year, assessor type, rotation, academic year, and core competency. The Mantel-Haenszel chi-square test was used to test for changes over time.

RESULTS

Specialties demonstrated heterogeneity in accounting for CDs in ACGME milestones, with 22% (22/99) of specialties having no language describing CDs in milestones assessment. Thirty-three percent (63/189) of UCMC internal medicine residents received at least one CD rating, with CDs accounting for 0.18% (668/364,728) of all assessment ratings. The authors identified CDs across multiple core competencies and rotations.

CONCLUSIONS

Despite some specialties not accounting for CDs in milestone assessment, UCMC's experience demonstrates that a significant proportion of residents may be rated as having a CD during training. Identification of CDs may allow programs to develop remediation and improvement plans.

An evaluation of the competencies of primary health care clinic nursing managers in two South African provinces

Background

Managerial competencies to enhance individual and organisational performance have gained currency in global efforts to strengthen health systems. Competent managers are essential in the implementation of primary health care (PHC) reforms that aim to achieve universal health coverage.

Objective

To evaluate the competencies of PHC clinic nursing managers in two South African provinces.

Design

A cross-sectional study was conducted in two South African provinces. Using stratified random sampling, 111 PHC clinic nursing managers were selected. All supervisors (n=104) and subordinate nurses (n=383) were invited to participate in the survey on the day of data collection. Following informed consent, the nursing managers, their supervisors, and subordinate nurses completed a 40-item, 360-degree competency assessment questionnaire, with six domains: communication, leadership and management, staff management, financial management, planning and priority setting, and problem-solving. Standard deviations, medians, and inter-quartile ranges (IQRs) were computed separately for PHC nursing managers, supervisors, and subordinate nurses for competencies in the six domains. The Tinsley and Weiss index was used to assess agreement between each of the three possible pairs of raters.

Results

A 95.4% response rate was obtained, with 105 nursing managers in Gauteng and Free State completing the questionnaires. There was a lack of agreement about nursing managers’ competencies among the three groups of raters. Overall, clinic nursing managers rated themselves high on the five domains of communication (8.6), leadership and management (8.67), staff management (8.75), planning and priority setting (8.6), and problem-solving (8.83). The exception was financial management with a median score of 7.94 (IQR 6.33–9.11). Compared to the PHC clinic managers, the supervisors and subordinate nurses gave PHC nursing managers lower ratings on all six competency domains, with the lowest rating for financial management (supervisor median rating 6.56; subordinate median rating 7.31).

Conclusion

The financial management competencies of PHC clinic nursing managers need to be prioritised in continuing professional development programmes.

Usefulness of 360 degree evaluation in evaluating nursing students in Iran

PURPOSE

This study aimed to evaluate the clinical nursing students using 360 degree evaluation.

METHODS

In this descriptive cross-sectional study that conducted between September 2014 and February 2015, 28 students who were selected by census from those who were passing the last semester of the Nursing BSc program in Rafsanjan University of Medical Sciences. Data collection tools included demographic questionnaire and students' evaluation questionnaire, to evaluate "professional behavior" and "clinical skills" in pediatric ward. Every student got evaluated from clinical instructor, students, peers, clinical nurses, and children's mothers' point of view. Data analysis was done with descriptive and analytic statistics test including Pearson coefficient using SPSS version 18.0.

RESULTS

The evaluation mean scores were as following: students, 89.74±6.17; peers, 94.12±6.87; children's mothers, 92.87±6.21; clinical instructor, 84.01±8.81; and the nurses, 94.87±6.35. The results showed a significant correlation between evaluation scores of peers, clinical instructor and self-evaluation (Pearson coefficient, p<0.001), but the correlation between the nurses' evaluation score and that of the clinical instructor was not significant (Pearson coefficient, p=0.052).

CONCLUSION

360 Degree evaluation can provide additional useful information on student performance and evaluation of different perspectives of care. The use of this method is recommended for clinical evaluation of nursing students.

Perceptions of Peer-to-Peer Interprofessional Feedback Among Students in the Health Professions

PURPOSE

Interprofessional teamwork should include interprofessional feedback to optimize performance and collaboration. Social identity theory predicts that hierarchy and stereotypes may limit receptiveness to interprofessional feedback, but literature on this is sparse. This study explores perceptions among health professions students regarding interprofessional peer feedback received after a team exercise.

METHOD

In 2012-2013, students from seven health professions schools (medicine, pharmacy, nursing, dentistry, physical therapy, dietetics, and social work) participated in a team-based interprofessional exercise early in clinical training. Afterward, they wrote anonymous feedback comments for each other. Each student subsequently completed an online survey to rate the usefulness and positivity (on five-point scales) of feedback received and guessed each comment's source. Data analysis included analysis of variance to examine interactions (on usefulness and positivity ratings) between profession of feedback recipients and providers.

RESULTS

Of 353 study participants, 242 (68.6%) accessed the feedback and 221 (62.6%) completed the survey. Overall, students perceived the feedback as useful (means across professions = 3.84-4.27) and positive (means = 4.17-4.86). There was no main effect of profession of the feedback provider, and no interactions between profession of recipient and profession of provider regardless of whether the actual or guessed provider profession was entered into the analysis.

CONCLUSIONS

These findings suggest that students have positive perceptions of interprofessional feedback without systematic bias against any specific group. Whether students actually use interprofessional feedback for performance improvement and remain receptive toward such feedback as they progress in their professional education deserves further study.

Gender Bias in Nurse Evaluations of Residents in Obstetrics and Gynecology

PURPOSE

We examined the evaluations given by nurses to obstetrics and gynecology residents to estimate whether gender bias was evident.

BACKGROUND

Women receive more negative feedback and evaluations than men-from both sexes. Some suggest that, to be successful in traditionally male roles such as surgeon, women must manifest a warmth-related (communal) rather than competence-related (agentic) demeanor. Compared with male residents, female residents experience more interpersonal difficulties and less help from female nurses. We examined feedback provided to residents by female nurses.

METHODS

We examined Professional Associate Questionnaires (2006-2014) using a mixed-methods design. We compared scores per training year by gender using Mann-Whitney and linear regression adjusting for resident and nurse cohorts. Using grounded theory analysis, we developed a coding system for blinded comments based on principles of effective feedback, medical learners' evaluation, and impression management. χ examined the proportions of negative and positive and communal and agentic comments between genders.

RESULTS

We examined 2,202 evaluations: 397 (18%) for 10 men and 1,805 (82%) for 34 women. Twenty-three compliments (eg, "Great resident!") were excluded. Evaluations per training year varied: men n=77-134; women n=384-482. Postgraduate year (PGY)-1, PGY-2, and PGY-4 women had lower mean ratings (P<.035); when adjusted, the difference remained significant in PGY-2 (MWomen=1.5±0.6 compared with MMen=1.7±0.5; P=.001). Postgraduate year-1 women received disproportionately fewer positive and more negative agentic comments than PGY-1 men (positive=17.3% compared with 40%, negative=17.3% compared with 3.3%, respectively; P=.041).

CONCLUSION

Evidence of gender bias in evaluations emerged; albeit subtle, women received harsher feedback as lower-level residents than men. Training in effective evaluation and gender bias management is warranted.

Assessing interprofessional teamwork: pilot test of a new assessment module for practicing physicians

INTRODUCTION

Teamwork is a basic component of all health care, and substantial research links the quality of teamwork to safety and quality of care. The TEAM (Teamwork Effectiveness Assessment Module) is a new Web-based teamwork assessment module for practicing hospital physicians. The module combines self-assessment, multisource feedback from members of other professions and specialties with whom the physician exercises teamwork, and a structured review of those data with a peer to develop an improvement plan.

METHODS

We conducted a pilot test of this module with hospitalist physicians to evaluate the feasibility and usefulness of the module in practice, focusing on these specific questions: Would physicians in hospitals of different types and sizes be able to use the module; would the providers identified as raters respond to the request for feedback; would the physicians be able to identify one or more "trusted peers" to help analyze the feedback; and how would physicians experience the module process overall?

RESULTS

20 of 25 physicians who initially volunteered for the pilot completed all steps of the TEAM, including identifying interprofessional teammates, soliciting feedback from their team, and identifying a peer to help review data. Module users described the feedback they received as helpful and actionable, and indicated this was information they would not have otherwise received.

CONCLUSIONS

The results suggest that a module combining self-assessment, multisource feedback, and a guided process for interpreting these data can provide help practicing hospital physicians to understand and potentially improve their interprofessional teamwork skills and behaviors.

Reliability of the interprofessional collaborator assessment rubric (ICAR) in multi source feedback (MSF) with post-graduate medical residents

BACKGROUND

Increased attention on collaboration and teamwork competency development in medical education has raised the need for valid and reliable approaches to the assessment of collaboration competencies in post-graduate medical education. The purpose of this study was to evaluate the reliability of a modified Interprofessional Collaborator Assessment Rubric (ICAR) in a multi-source feedback (MSF) process for assessing post-graduate medical residents' collaborator competencies.

METHODS

Post-graduate medical residents (n = 16) received ICAR assessments from three different rater groups (physicians, nurses and allied health professionals) over a four-week rotation. Internal consistency, inter-rater reliability, inter-group differences and relationship between rater characteristics and ICAR scores were analyzed using Cronbach's alpha, one-way and two-way repeated measures ANOVA, and logistic regression.

RESULTS

Missing data decreased from 13.1% using daily assessments to 8.8% utilizing an MSF process, p = .032. High internal consistency measures were demonstrated for overall ICAR scores (α = .981) and individual assessment domains within the ICAR (α = .881 to .963). There were no significant differences between scores of physician, nurse, and allied health raters on collaborator competencies (F2,5 = 1.225, p = .297, η2 = .016). Rater gender was the only significant factor influencing scores with female raters scoring residents significantly lower than male raters (6.12 v. 6.82; F1,5 = 7.184, p = .008, η 2 = .045).

CONCLUSION

The study findings suggest that the use of the modified ICAR in a MSF assessment process could be a feasible and reliable assessment approach to providing formative feedback to post-graduate medical residents on collaborator competencies.

"Who writes what?" Using written comments in team-based assessment to better understand medical student performance: a mixed-methods study

BACKGROUND

Observation of the performance of medical students in the clinical environment is a key part of assessment and learning. To date, few authors have examined written comments provided to students and considered what aspects of observed performance they represent. The aim of this study was to examine the quantity and quality of written comments provided to medical students by different assessors using a team-based model of assessment, and to determine the aspects of medical student performance on which different assessors provide comments.

METHODS

Medical students on a 7-week General Surgery & Anesthesiology clerkship received written comments on 'Areas of Excellence' and 'Areas for Improvement' from physicians, residents, nurses, patients, peers and administrators. Mixed-methods were used to analyze the quality and quantity of comments provided and to generate a conceptual framework of observed student performance.

RESULTS

1,068 assessors and 127 peers provided 2,988 written comments for 127 students, a median of 188 words per student divided into 26 "Areas of Excellence" and 5 "Areas for Improvement". Physicians provided the most comments (918), followed by patients (692) and peers (586); administrators provided the fewest (91). The conceptual framework generated contained four major domains: 'Student as Physician-in-Training', 'Student as Learner', 'Student as Team Member', and 'Student as Person.'

CONCLUSIONS

A wide range of observed medical student performance is recorded in written comments provided by members of the surgical healthcare team. Different groups of assessors provide comments on different aspects of student performance, suggesting that comments provided from a single viewpoint may potentially under-represent or overlook some areas of student performance. We hope that the framework presented here can serve as a basis to better understand what medical students do every day, and how they are perceived by those with whom they work.

Training fellows for retinopathy of prematurity care: a Web-based survey

PURPOSE

To characterize the training received by pediatric ophthalmology and retina fellows in retinopathy of prematurity (ROP) management.

METHODS

Pediatric ophthalmology and retina fellowship programs were emailed a Web-based survey to assess fellowship training in ROP management.

RESULTS

Of 140 programs contacted, 42 (30%) participated, resulting in 87 surveys for analysis. Of the 87 respondents, 25 (29%) reported that two-thirds or less of ROP examinations performed by fellows were also seen by an attending. When stratified by specialty, this trend was statistically different between pediatric ophthalmology and retina fellows (P = 0.03). Additionally, pediatric ophthalmology fellows performed fewer laser photocoagulation procedures than retina fellows (P < 0.001). Regarding fellows' perceived competency in ROP management, 3 of 51 (6%) felt competent at the start of their fellowship and 43 of 51 (84%) felt competent at the time of the survey. Only 7% of respondents reported the use of formal evaluations at their programs to assess fellow competence in ROP examination.

CONCLUSIONS

Training programs for fellows in pediatric ophthalmology and retina vary greatly with respect to ROP training and the quality of clinical care. Many clinical ROP examinations are being performed by pediatric ophthalmology and retina fellows without involvement and/or direct supervision by attending ophthalmologists. Our findings have important implications for the development of a future workforce for ROP management.