USING WEB-BASED FRAME-OF-REFERENCE TRAINING TO DECREASE BIASES IN PERSONALITY-BASED JOB ANALYSIS: AN EXPERIMENTAL FIELD STUDY

A Conceptual Framework for Investigating and Mitigating Machine-Learning Measurement Bias (MLMB) in Psychological Assessment

Given significant concerns about fairness and bias in the use of artificial intelligence (AI) and machine learning (ML) for psychological assessment, we provide a conceptual framework for investigating and mitigating machine-learning measurement bias (MLMB) from a psychometric perspective. MLMB is defined as differential functioning of the trained ML model between subgroups. MLMB manifests empirically when a trained ML model produces different predicted score levels for different subgroups (e.g., race, gender) despite them having the same ground-truth levels for the underlying construct of interest (e.g., personality) and/or when the model yields differential predictive accuracies across the subgroups. Because the development of ML models involves both data and algorithms, both biased data and algorithm-training bias are potential sources of MLMB. Data bias can occur in the form of nonequivalence between subgroups in the ground truth, platform-based construct, behavioral expression, and/or feature computing. Algorithm-training bias can occur when algorithms are developed with nonequivalence in the relation between extracted features and ground truth (i.e., algorithm features are differentially used, weighted, or transformed between subgroups). We explain how these potential sources of bias may manifest during ML model development and share initial ideas for mitigating them, including recognizing that new statistical and algorithmic procedures need to be developed. We also discuss how this framework clarifies MLMB but does not reduce the complexity of the issue.

Leveraging historiometry to better understand teams in context

The prevalence of teams in organizational settings has dramatically increased over the last 50 years, and as such, researchers have made much progress in understanding the conditions and intra-team dynamics that facilitate successful team performance. However, much remains to be learned due to the complexity of teams. This complexity often makes it difficult to study teams operating in context, especially when trying to examine longitudinal aspects of teams. Adding to this difficulty, studying teams in context is resource intensive and access is often a key barrier, especially if the focus is on teams that are elite or that operate in extreme environments. This drives a need to look outside the traditional methodological tools typically utilized to study teams. Thereby, the purpose of this manuscript is to highlight a method that while not typically utilized in the team literature can offer benefits when exploring team dynamics in context—historiometry.

Trait Activation Theory: A Review of the Literature and Applications to Five Lines of Personality Dynamics Research

Extending interactionist principles and targeting situational specificity of trait–performance linkages, trait activation theory (TAT) posits personality traits are expressed as valued work behavior in response to trait-relevant situational cues, subject to constraints and other factors, all operating at the task, social, and organizational levels. Review of 99 key sources citing TAT spanning 2011–2019 reveals diverse applications (e.g., bidirectionality, trait specificity, team building) and an overall 60% significance rate for 262 TAT-based moderator effects reported in 60 of 75 empirical studies. Applying five key aspects of TAT (e.g., behavior/performance distinction, need-based motivation) to five lines of personality dynamics research (e.g., personality states, self-regulation models of motivation) supports TAT as a vehicle for advancing understanding of within-person variability over brief and extended timelines. Critical research needs include personality-oriented work analysis, longitudinal study of trait-situation processes, trait activation in teams, within-job bidirectionality, and situation relevance as a unifying principle in advancing person–workplace fit.

Does effectiveness in performance appraisal improve with rater training?

Performance appraisal is a complex process by which an organization can determine the extent to which employees are performing their work effectively. However, this appraisal may not be accurate if there is no reduction in the impact of problems caused by possibly subjective rater judgements. The main objective of this work is to check the effectiveness—separately and jointly—of the following four training programmes in the extant literature aimed at improving the accuracy of performance assessment: 1) Performance Dimension Training, 2) Frame-of-Reference, 3) Rater Error Training, and 4) Behavioural Observation Training. Based on these training strategies, three programmes were designed and applied separately. A fourth programme was a combination of the other three. We analyzed two studies using different samples (85 students and 42 employees) for the existence of differences in the levels of knowledge of performance and its dimensions, rater errors, observational accuracy, and accuracy of task and citizenship performance appraisal, according to the type of training raters receive. First, the main results show that training based on performance dimensions and the creation of a common framework, in addition to the training that includes the four programmes (Training_4_programmes), increases the level of knowledge of performance and its dimensions. Second, groups that receive training in rater error score higher in knowledge of biases than the other groups, whether or not they have received training. Third, participants’ observational accuracy improves with each new moment measure (post-training and follow-up), though not because of the type of training received. Fourth, participants who receive training through the programme that combine the other four gave a task performance appraisal that was closer to the one undertaken by the judges-experts than the other groups. And finally, students’ citizenship performance appraisal does not vary according to type of training or to different moment measures, whereas the group of employees who received all four types of training gave a more accurate citizenship performance assessment.

Personality oriented job analysis to identify non-cognitive factors for a doctor of physical therapy program in the United States

It aimed to conduct a personality oriented job analysis to identify non-cognitive factors that may predict successful performance or performance difficulties in doctorate in physical therapy (DPT) students. The study employed focus groups and a survey with 9 DPT subject matter experts. Focus group participants including 3 DPT faculty members and 4 recent graduates of the DPT program identified 22 non-cognitive factors. Out of them, 15 factors were possibly associated with successful performance and 7 factors were possibly associated with performance difficulties. The questionnaire employing the Combination Job Analysis Method resulted in 12 factors which could be used in selection, and 10 which could be incorporated into training. The present study employed an established job analysis method using subject matter experts to identify a broad array of factors that go beyond what previous studies have examined, and which may predict success or difficulties in a DPT program.

Applicant Ethnicity Affects Which Questions Are Asked in a Job Interview

Abstract. Three experiments on professional recruiters explored how applicants’ ethnicity affects questions prepared for a job interview and the implications of this. Study 1 revealed that outgroup applicants prompt recruiters to focus more on whether applicants have integrated cultural norms and values fitting the ingroup norms (person-culture fit), as well as the match between the applicants and their would-be work team (person-group fit). When applicants were from the ethnic ingroup, recruiters focused more on questions pertaining to the match between the applicants’ abilities and the specific demands of the job (person-job fit). Studies 2 and 3 revealed that questions prepared for outgroup applicants were rated as less useful for hireability decisions, and that summaries emphasizing person-job fit were perceived as more useful.

Tools for fairness: Increased structure in the selection process reduces discrimination

Employment discrimination causes problems at the labor market, and is hard to combat. Can increasing the degree of structure when selecting applicants increase fairness? Students were asked to perform a computerized selection task and were either provided with tools for systematizing information about the applicants (structured selection) or no such tools (unstructured selection). We hypothesized and found that a structured process, where employing recruitment tools rather than the recruiter’s impressionistic judgment is key, improves the ability to identify job-relevant criteria and hence selecting more qualified applicants, even when in-group favoritism is tempting (e.g. when the outgroup applicants are more competent). Increasing structure helped recruiters select more competent applicants and reduced ethnic discrimination. Increasing the motivation to carefully follow the structured procedure strengthened these effects further. We conclude that structure pays off, and that motivational factors should be taken into account in order for it to have the optimal effect.

The application of subjective job task analysis techniques in physically demanding occupations: evidence for the presence of self-serving bias

The aim of this study was to determine if perceptions of physically demanding job tasks are biased by employee demographics and employment profile characteristics including: age, sex, experience, length of tenure, rank and if they completed or supervised a task. Surveys were administered to 427 Royal Australian Navy personnel who characterised 33 tasks in terms of physical effort, importance, frequency, duration and vertical/horizontal distance travelled. Results showed no evidence of bias resulting from participant characteristics, however participants who were actively involved in both task participation and supervision rated these tasks as more important than those involved only in the supervision of that task. This may indicate self-serving bias in which participants that are more actively involved in a task had an inflated perception of that task's importance. These results have important implications for the conduct of job task analyses, especially the use of subjective methodologies in the development of scientifically defensible physical employment standards. Practitioner Summary: To examine the presence of systematic bias in subjective job task analysis methodologies, a survey was conducted on a sample of Royal Australian Navy personnel. The relationship between job task descriptions and participant's demographic and job profile characteristics revealed the presence of self-serving bias affecting perceptions of task importance.

Converting values awareness to values enactment through frame-of-reference training

Purpose

This paper aims to investigate whether values enactment could be increased through frame-of-reference (FOR) training configured around values prototyping and behavioural domain training for managers within an Australian public sector organisation.

Design/methodology/approach

Employees from an Australian public sector organisation were studied to ascertain the effect of values training and development via a three-way longitudinal design with a control group.

Findings

The findings indicate that FOR training can increase employee values enactment clarity and, thereby, have a positive impact upon organisational values enactment.

Practical implications

The application of FOR training constitutes a new approach to supporting the development of employee values clarity, which, in turn, can support the achievement of organisational values enactment. Through FOR training, employees can learn to apply organisational values in their decision-making and other behaviours irrespective of whether they are highly congruent with their personal values.

Originality/value

Empirical research into values management is limited and there is a lack of consensus to what is needed to create a values-driven organisation. The article shows that FOR training can be a beneficial component of a broader human resource strategy aimed at increasing organisational values enactment. With reference to the resource-based view of the firm, it is argued that values enactment constitutes a distinctive capability that may confer sustained organisational advantage.

Increasing systematicity leads to better selection decisions: Evidence from a computer paradigm for evaluating selection tools

A computerized paradigm was created to allow for testing in the laboratory whether increasing systematicity helps the recruiter make better selection decisions. Participants were introduced to the job and the applicants on the computer screen and asked to select who they thought should be considered for the job and who should not. Level of systematicity, i.e. the extent to which the recruitment is methodical and uses prepared tools, was manipulated between subjects. Depending on experimental condition participants were helped by means of a tool for extracting judgment criteria (job analysis) and a tool for making judgments related to selected criteria (including calculation of a final score). The general prediction that increased systematicity leads to the selection of more qualified candidates was supported by the results, particularly when the motivation to put time and effort into the task was higher. The results support the claim from Industrial/Organizational psychologists that systematicity is a desirable characteristic in selection processes. The fact that increasing systematicity led to better selection decisions in a controlled laboratory experiment, along with process-related measures, suggests that this kind of paradigm could be useful when evaluating new tools for improving selection decisions, before they are tested in large (and costly) field studies of actual personnel selection.

Retention of Assessment Center Rater Training

Abstract. The purpose of this research was to examine frame-of-reference (FOR) training retention in an assessment center (AC) rater training context. In this study, we extended Gorman and Rentsch’s (2009) research showing FOR training effects on performance schemas by examining the effects immediately after training and again after a two-week nonuse period. We examined the retention effects of FOR training on performance ratings and on performance schema accuracy. The results indicated that the FOR training condition, compared to a control condition, yielded performance ratings and performance schemas more similar to expert ratings and to an expert schema, respectively. FOR training also had positive effects on ratings and performance schema accuracy assessed two weeks after training. These results support and extend the theory of FOR training, which posits that the instructed theory of performance replaces the preexisting rater schemas ( Lievens, 2001 ), and they contribute to the research on FOR training within AC contexts.

Self-serving bias effects on job analysis ratings

The purpose of this study was to investigate whether worker-oriented job analysis importance ratings were influenced by subject matter experts' (SME) standing (as measured by self-rated performance) on a competency. This type of relationship (whereby SMEs indicate that the traits they have are important for successful job performance) is an example of the self-serving bias (which is widely described in the social cognition literature and rarely described in the industrial/organizational psychology literature). An archival dataset covering 57 clerical and technical occupations with 26,682 participants was used. Support was found for the relationship between self-rated performance and importance ratings. Significant relationships (typically in the .30s) were observed for all 31 competencies that were studied. Controls were taken to account for common method bias and differences in the competencies required for each of the 57 occupations. Past research has demonstrated the effects of the self-serving bias on personality-based job analysis ratings. This study was the first to extend these findings to traditional job analysis, which covers other competencies in addition to personality. In addition, this study is the first to use operational field data instead of laboratory data.

The Rise and Fall of Job Analysis and the Future of Work Analysis

This review begins by contrasting the importance ascribed to the study of occupational requirements observed in the early twentieth-century beginnings of industrial-organizational psychology with the diminishing numbers of job analysis articles appearing in top journals in recent times. To highlight the many pending questions associated with the job-analytic needs of today's organizations that demand further inquiry, research on the three primary types of job analysis data, namely work activities, worker attributes, and work context, is reviewed. Research on competencies is also reviewed along with the goals of a potential research agenda for the emerging trend of competency modeling. The cross-fertilization of job analysis research with research from other domains such as the meaning of work, job design, job crafting, strategic change, and interactional psychology is proposed as a means of responding to the demands of today's organizations through new forms of work analysis.

Debunking Myths and Urban Legends About Meta-Analysis

Meta-analysis is the dominant approach to research synthesis in the organizational sciences. We discuss seven meta-analytic practices, misconceptions, claims, and assumptions that have reached the status of myths and urban legends (MULs). These seven MULs include issues related to data collection (e.g., consequences of choices made in the process of gathering primary-level studies to be included in a meta-analysis), data analysis (e.g., effects of meta-analytic choices and technical refinements on substantive conclusions and recommendations for practice), and the interpretation of results (e.g., meta-analytic inferences about causal relationships). We provide a critical analysis of each of these seven MULs, including a discussion of why each merits being classified as an MUL, their kernels of truth value, and what part of each MUL represents misunderstanding. As a consequence of discussing each of these seven MULs, we offer best-practice recommendations regarding how to conduct meta-analytic reviews.

Customer-Centric Science: Reporting Significant Research Results With Rigor, Relevance, and Practical Impact in Mind

In response to the ongoing concern regarding a science-practice gap, we propose a customer-centric approach to reporting significant research results that involves a sequence of three interdependent steps. The first step involves setting an alpha level (i.e., a priori Type I error rate) that considers the relative seriousness of falsely rejecting a null hypothesis of no effect or relationship (i.e., Type I error) relative to not detecting an existing effect or relationship (i.e., Type II error) and reporting the actual observed p value (i.e., probability that the data would be obtained if the null hypothesis is true). The second step involves reporting estimates of the size of the effect or relationship, which indicate the extent to which an outcome is explained or predicted. The third step includes reporting results of a qualitative study to gather evidence regarding the practical significance of the effect or relationship. Our proposal to report research results with rigor, relevance, and practical impact involves important changes in how we report research results with the goal to bridge the science-practice gap.

Benefits of training and development for individuals and teams, organizations, and society

This article provides a review of the training and development literature since the year 2000. We review the literature focusing on the benefits of training and development for individuals and teams, organizations, and society. We adopt a multidisciplinary, multilevel, and global perspective to demonstrate that training and development activities in work organizations can produce important benefits for each of these stakeholders. We also review the literature on needs assessment and pretraining states, training design and delivery, training evaluation, and transfer of training to identify the conditions under which the benefits of training and development are maximized. Finally, we identify research gaps and offer directions for future research.