Variance Components of Job Performance Ratings

Evaluation of the Psychometric Properties of Relational Insights 360 Among Baccalaureate Nursing Students With the Use of Standardized Patients

BACKGROUND AND PURPOSE

The purpose of this study was to evaluate reliability and validity of the relational competency assessment instrument, Relational Insights 360 (RI-360) (Koloroutis & Trout, 2012), among 104 baccalaureate nursing students and 10 standardized patients (SPs) upon completion of a communication simulation.

METHODS

The reliability of the RI-360 was determined by Cronbach's alpha coefficient. Interrater reliability of the RI-360 was measured between the SPs and the primary investigator with percentages. Factor analysis was run to evaluate the construct validity.

RESULTS

The RI-360 was internally consistent with an alpha coefficient of 0.93. Interrater reliability for all items on the RI-360 was 42% between students' scores and SPs' scores and 38.9% between the Primary Investigator's scores and SPs' scores. Exploratory factor analysis showed that factor loadings ranged from 0.29 to 0.84.

CONCLUSIONS

The RI-360 appears to be a valid and reliable scale for use in measuring relational competency among nursing students and SPs.

Inter-rater variability as mutual disagreement: identifying raters' divergent points of view

Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting 'idiosyncratic rater variance' is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters' comments and justifications. This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from "most consistent with my impression" to "most contrary to my impression". Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians' divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.

An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners

Applications of interrater agreement (IRA) statistics for Likert scales are plentiful in research and practice. IRA may be implicated in job analysis, performance appraisal, panel interviews, and any other approach to gathering systematic observations. Any rating system involving subject-matter experts can also benefit from IRA as a measure of consensus. Further, IRA is fundamental to aggregation in multilevel research, which is becoming increasingly common in order to address nesting. Although, several technical descriptions of a few specific IRA statistics exist, this paper aims to provide a tractable orientation to common IRA indices to support application. The introductory overview is written with the intent of facilitating contrasts among IRA statistics by critically reviewing equations, interpretations, strengths, and weaknesses. Statistics considered include r_wg, rwg*, r′_wg, r_wg(p), average deviation (AD), a_wg, standard deviation (S_wg), and the coefficient of variation (CV_wg). Equations support quick calculation and contrasting of different agreement indices. The article also includes a “quick reference” table and three figures in order to help readers identify how IRA statistics differ and how interpretations of IRA will depend strongly on the statistic employed. A brief consideration of recommended practices involving statistical and practical cutoff standards is presented, and conclusions are offered in light of the current literature.

Comparing multisource ratings: can I believe what they say about my leadership behaviour?

Purpose

This paper aims to investigate whether the feedback managers receive from their work colleagues is based on a shared understanding of their leadership behaviours. This similarity in perception is called conceptual equivalence. The study also took into account the managers’ and their work colleagues’ expectations.

Design/methodology/approach

A total of 441 managers participated in a 360-degree feedback programme that sought responses from their bosses (n = 380), peers (n = 1,621) and subordinates (n = 1,680). The instrument used was the Competing Values Framework 16-item survey, which describes the leadership behaviours that managers must display to be effective. It also measures the behaviours they consider important. Confirmatory factor analysis was conducted on the rating scores of the managers and the three hierarchical levels of their work colleagues to determine conceptual equivalence between self and work colleagues’ ratings. Conceptual models were formulated and compared by testing for factor form equivalence.

Findings

Conceptual equivalence was found across all rater groups for both leadership behaviours displayed and the importance of leadership behaviours.

Practical implications

This paper provides managers and human resource professionals with useful insights on how to improve the use of 360-degree feedback processes by ensuring that conceptual equivalence is established for feedback supplied by raters from different hierarchical levels. The findings also stress the importance of including expectations when receiving feedback on leadership behaviours.

Originality/value

This study contributes to the literature on the benefits of determining the conceptual equivalence of feedback received by managers from various work colleagues. A second contribution is the inclusion of expectations in the feedback process, as none of the previous work has included such a measure.