A new frame for frame-of-reference training: enhancing the construct validity of assessment centers

Does effectiveness in performance appraisal improve with rater training?

Performance appraisal is a complex process by which an organization can determine the extent to which employees are performing their work effectively. However, this appraisal may not be accurate if there is no reduction in the impact of problems caused by possibly subjective rater judgements. The main objective of this work is to check the effectiveness—separately and jointly—of the following four training programmes in the extant literature aimed at improving the accuracy of performance assessment: 1) Performance Dimension Training, 2) Frame-of-Reference, 3) Rater Error Training, and 4) Behavioural Observation Training. Based on these training strategies, three programmes were designed and applied separately. A fourth programme was a combination of the other three. We analyzed two studies using different samples (85 students and 42 employees) for the existence of differences in the levels of knowledge of performance and its dimensions, rater errors, observational accuracy, and accuracy of task and citizenship performance appraisal, according to the type of training raters receive. First, the main results show that training based on performance dimensions and the creation of a common framework, in addition to the training that includes the four programmes (Training_4_programmes), increases the level of knowledge of performance and its dimensions. Second, groups that receive training in rater error score higher in knowledge of biases than the other groups, whether or not they have received training. Third, participants’ observational accuracy improves with each new moment measure (post-training and follow-up), though not because of the type of training received. Fourth, participants who receive training through the programme that combine the other four gave a task performance appraisal that was closer to the one undertaken by the judges-experts than the other groups. And finally, students’ citizenship performance appraisal does not vary according to type of training or to different moment measures, whereas the group of employees who received all four types of training gave a more accurate citizenship performance assessment.

Toward a Better Understanding of Assessment Centers: A Conceptual Review

Assessment centers (ACs) are employed for selecting and developing employees and leaders. They are interpersonal at their core because they consist of interactive exercises. Minding this perspective, this review focuses on the role of the assessee, the assessor, and the AC design, as well as their interplay in the interpersonal situation of the AC. Therefore, it addresses which conceptual perspectives have increased our understanding of ACs in this context. Building on this, we review relevant empirical findings. On this basis, the review contributes to an empirically driven understanding of the interpersonal nature of ACs and provides directions for practice and future research avenues on this topic as well as on technology in ACs and cross-cultural applications.

Converting values awareness to values enactment through frame-of-reference training

Purpose

This paper aims to investigate whether values enactment could be increased through frame-of-reference (FOR) training configured around values prototyping and behavioural domain training for managers within an Australian public sector organisation.

Design/methodology/approach

Employees from an Australian public sector organisation were studied to ascertain the effect of values training and development via a three-way longitudinal design with a control group.

Findings

The findings indicate that FOR training can increase employee values enactment clarity and, thereby, have a positive impact upon organisational values enactment.

Practical implications

The application of FOR training constitutes a new approach to supporting the development of employee values clarity, which, in turn, can support the achievement of organisational values enactment. Through FOR training, employees can learn to apply organisational values in their decision-making and other behaviours irrespective of whether they are highly congruent with their personal values.

Originality/value

Empirical research into values management is limited and there is a lack of consensus to what is needed to create a values-driven organisation. The article shows that FOR training can be a beneficial component of a broader human resource strategy aimed at increasing organisational values enactment. With reference to the resource-based view of the firm, it is argued that values enactment constitutes a distinctive capability that may confer sustained organisational advantage.

Retention of Assessment Center Rater Training

Abstract. The purpose of this research was to examine frame-of-reference (FOR) training retention in an assessment center (AC) rater training context. In this study, we extended Gorman and Rentsch’s (2009) research showing FOR training effects on performance schemas by examining the effects immediately after training and again after a two-week nonuse period. We examined the retention effects of FOR training on performance ratings and on performance schema accuracy. The results indicated that the FOR training condition, compared to a control condition, yielded performance ratings and performance schemas more similar to expert ratings and to an expert schema, respectively. FOR training also had positive effects on ratings and performance schema accuracy assessed two weeks after training. These results support and extend the theory of FOR training, which posits that the instructed theory of performance replaces the preexisting rater schemas ( Lievens, 2001 ), and they contribute to the research on FOR training within AC contexts.

The Modeling and Assessment of Work Performance

Individual work role performance drives the entire economy. It is organizational psychology and organizational behavior’s (OP/OB’s) most crucial dependent variable. In this review, alternative specifications for the definition and latent structure of individual performance are reviewed and summarized. Setting aside differences in terminology, the alternatives are remarkably similar. The Campbell (2012) model is offered as a synthesized description of the content of the latent structure. Issues pertaining to performance dynamics are then reviewed, along with the role played by individual adaptability to changing performance requirements. Using the synthesized model of the latent content structure and dynamics of performance as a backdrop, issues pertaining to the assessment of performance are summarized. The alternative goals of performance assessment, general measurement issues, and the construct validity of specific methods (e.g., ratings, simulations) are reviewed and described. Cross-cultural issues and future research needs are noted.

Why Assessment Centers Do Not Work the Way They Are Supposed To

Assessment centers (ACs) are often designed with the intent of measuring a number of dimensions as they are assessed in various exercises, but after 25 years of research, it is now clear that AC ratings that are completed at the end of each exercise (commonly known as postexercise dimension ratings) substantially reflect the effects of the exercises in which they were completed and not the dimensions they were designed to reflect. This is the crux of the long-standing “construct validity problem” for AC ratings. I review the existing research on AC construct validity and conclude that (a) contrary to previous notions, AC candidate behavior is inherently cross-situationally (i.e., cross-exercise) specific, not cross-situationally consistent as was once thought, (b) assessors rather accurately assess candidate behavior, and (c) these facts should be recognized in the redesign of ACs toward task- or role-based ACs and away from traditional dimension-based ACs.

Where Have We Been, How Did We Get There, and Where Shall We Go?

Commentators expressed a wide variety of views on my evaluation of the state of assessment center (AC) research and practice. In this response, I first trace the evolution of the construct validity paradox “urban legend.” Next, I consider the commentators’ comments as they relate to (a) my recommendation to abandon dimension-based ACs in lieu of task- or role-based structures (b) my recommendation to discontinue design fix attempts toward making ACs conform to multitrait–multimethod construct validity criteria, and (c) considerations of construct validity and validation evidence. Finally, I offer some directions for future AC research and practice.

Are Improvements in Assessment Center Construct-Related Validity Paralleled by Improvements in Criterion-Related Validity?

Previous studies have found that factors that improved assessment center (AC) construct-related validity also had beneficial effects on criterion-related validity. However, some factors might have diverging effects on construct- and criterion-related validity. Accordingly, we followed recent calls to evaluate construct- and criterion-related validity of ACs simultaneously by examining the effects of exercise similarity on both aspects of validity within a single study. Data were collected in an AC (N = 92) that consisted of two different types of exercises. Convergent validity was better for similar exercises than it was for dissimilar exercises. However, regarding criterion-related validity, we did not find differences between similar and dissimilar exercises. Hence, this study revealed that improvements in AC construct-related validity are not necessarily paralleled by improvements in criterion-related validity.

Seeing the same thing differently: mechanisms that contribute to assessor differences in directly-observed performance assessments

Assessors' scores in performance assessments are known to be highly variable. Attempted improvements through training or rating format have achieved minimal gains. The mechanisms that contribute to variability in assessors' scoring remain unclear. This study investigated these mechanisms. We used a qualitative approach to study assessors' judgements whilst they observed common simulated videoed performances of junior doctors obtaining clinical histories. Assessors commented concurrently and retrospectively on performances, provided scores and follow-up interviews. Data were analysed using principles of grounded theory. We developed three themes that help to explain how variability arises: Differential Salience-assessors paid attention to (or valued) different aspects of the performances to different degrees; Criterion Uncertainty-assessors' criteria were differently constructed, uncertain, and were influenced by recent exemplars; Information Integration-assessors described the valence of their comments in their own unique narrative terms, usually forming global impressions. Our results (whilst not precluding the operation of established biases) describe mechanisms by which assessors' judgements become meaningfully-different or unique. Our results have theoretical relevance to understanding the formative educational messages that performance assessments provide. They give insight relevant to assessor training, assessors' ability to be observationally "objective" and to the educational value of narrative comments (in contrast to numerical ratings).

The evaluation of a frame-of-reference training programme for intern psychometrists

Orientation: The use of assessment centres (ACs) has drastically increased over the past decade. However, ACs are constantly confronted with the lack of construct validity. One aspect of ACs that could improve the construct validity significantly is that of assessor training. Unfortunately untrained or poorly trained assessors are often used in AC processes. Research purpose: The purpose of this research was to evaluate a frame-of-reference (FOR) programme to train intern psychometrists as assessors at an assessment centre. Motivation of study: The role of an assessor is important in an AC; therefore it is vital for an assessor to be able to evaluate and observe candidates’ behaviour adequately. Commencing with this training in a graduate psychometrist programme gives the added benefit of sending skilled psychometrists to the workplace. Research design, approach and method: A quantitative research approach was implemented, utilising a randomised pre-test-post-test comparison group design. Industrial Psychology postgraduate students (N = 22) at a South African university were used and divided into an experimental group (n = 11) and control group (n = 11). Three typical AC simulations were utilised as pre- and post-tests, and the ratings obtained from both groups were statistically analysed to determine the effect of the FOR training programme. Main findings: The data indicated that there was a significant increase in the familiarity of the participants with the one-on-one simulation and the group discussion simulation. Practical/managerial implications: Training intern psychometrists in a FOR programme could assist organisations in the appointment of more competent assessors. Contribution/value-add: To design an assessor training programme using FOR training for intern psychometrists in the South African context, specifically by incorporating this programme into the training programme for Honours students at universities.

Personnel Selection

We review developments in personnel selection since the previous review by Hough & Oswald (2000) in the Annual Review of Psychology. We organize the review around a taxonomic structure of possible bases for improved selection, which includes (a) better understanding of the criterion domain and criterion measurement, (b) improved measurement of existing predictor methods or constructs, (c) identification and measurement of new predictor methods or constructs, (d) improved identification of features that moderate or mediate predictor-criterion relationships, (e) clearer understanding of the relationship between predictors or between predictors and criteria (e.g., via meta-analytic synthesis), (f) identification and prediction of new outcome variables, (g) improved ability to determine how well we predict the outcomes of interest, (h) improved understanding of subgroup differences, fairness, bias, and the legal defensibility, (i) improved administrative ease with which selection systems can be used, (j) improved insight into applicant reactions, and (k) improved decision-maker acceptance of selection systems.

A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings

Two recent reviews have attempted to summarize findings quantitatively regarding assessment center (AC) construct-related validity (i.e., Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001). Unlike these previous studies, which reanalyzed individual multitrait-multimethod (MTMM) matrices from previously published research, the authors recoded and combined past matrices into a single MTMM matrix. This matrix, comprised of 6 dimensions each measured by 6 exercises, was then analyzed, providing a more generalizable set of results. Both dimensions and exercises were found to contribute substantially to AC ratings. Specific dimensions (i.e., communication, influencing others, organizing and planning, and problem solving) appeared more construct valid than others (i.e., consideration/awareness of others and drive). Implications for AC design and practice are discussed.

Revised Estimates of Dimension and Exercise Variance Components in Assessment Center Postexercise Dimension Ratings

The authors reanalyzed assessment center (AC) multitrait-multimethod (MTMM) matrices containing correlations among postexercise dimension ratings (PEDRs) reported by F. Lievens and J. M. Conway (2001). Unlike F. Lievens and J. M. Conway, who used a correlated dimension-correlated uniqueness model, we used a different set of confirmatory-factor-analysis-based models (1-dimension-correlated Exercise and 1-dimension-correlated uniqueness models) to estimate dimension and exercise variance components in AC PEDRs. Results of reanalyses suggest that, consistent with previous narrative reviews, exercise variance components dominate over dimension variance components after all. Implications for AC construct validity and possible redirections of research on the validity of ACs are discussed.

Assessor Cognitive Processes in an Operational Assessment Center

The purpose of this study was (a) to provide additional tests of C. E. Lance, Newbolt, et al.'s (2000) situational specificity (vs. method bias) interpretation of exercise effects on assessment center postexercise dimension ratings and (b) to provide competitive tests of salient dimension versus general impression models of assessor within-exercise evaluations of candidate performance. Results strongly support the situational specificity hypothesis and the general impression model of assessor cognitive processes in which assessors first form overall evaluations of candidate performance that then drive more specific dimensional ratings.