Ethnic group differences in measures of job performance: a new meta-analysis

Administrative records-based criterion measures

This paper covers considerations in using criterion measures based on administrative data. We begin with a conceptual framework for understanding and evaluating administrative criterion measures as "objective" rather than (ratings-based) assessments of job performance. We then describe the associated advantages (e.g., availability) and disadvantages (e.g., contamination) of using administrative data for criterion-related validation purposes. Best practices in the use of administrative data for validation purposes, including procedures for (a) handling missing data, (b) performing data checks, and (c) reporting detailed decision rules so future researchers can replicate the analyses are also described. Finally, we discuss "modern data management" approaches for improving administrative data for supporting organizational decision-making.

Identification and evaluation of criterion measurement methods

Criterion measures vary greatly in terms of their psychometric quality and ease of use. This paper serves two purposes. First, it provides a general summary of different approaches to criterion measurement in a military context. Second, it provides an extensive review of 16 specific types of criterion measurement methods (e.g., job performance rating scales, self-report questionnaires, job knowledge tests) on nine psychometric and ease-of-use evaluation factors. Eight criterion measurement experts read a summary of extant research and made ratings to evaluate each measurement method on the evaluation factors. Rater intra-class correlations (ICCs) were high, ranging from .75 to .95 across the evaluation dimensions with a median of .91. Data showed a quality-feasibility tradeoff, where criterion data that are easy to obtain often have technical flaws. Recommendations for military services and future directions in criterion measurement (e.g., applications of machine learning) are discussed.

Reducing Black–White Racial Differences on Intelligence Tests Used in Hiring for Public Safety Jobs

This paper explores whether a diversity and inclusion strategy focused on using modern intelligence tests can assist public safety organizations in hiring a talented diverse workforce. Doing so may offer strategies for mitigating the issues of systematic racism with which these occupations have historically struggled. Past meta-analytic research shows that traditional forms of intelligence tests, which are often used in this sector, have not consistently demonstrated predictive validity but have negatively impacted Black candidates. As an alternative, we examine a modern intelligence test that consists of novel unfamiliar cognitive problems that test takers must solve without relying on their prior experience. Across six studies of varying public safety jobs (e.g., police, firefighter) in different organizations, we found a pattern of results that supports the criterion-related validity of the modern intelligence test. In addition to consistently predicting job performance and training success, the modern intelligence test also substantially mitigated the observed Black–White group differences. The implications of these findings are discussed in terms of how to alter the legacy of I/O psychology and human resource fields when it comes to our impact on facilitating employment opportunities for Black citizens, particularly in public safety positions.

The AAMC Standardized Video Interview and the Electronic Standardized Letter of Evaluation in Emergency Medicine: A Comparison of Performance Characteristics

PURPOSE

To compare the performance characteristics of the electronic Standardized Letter of Evaluation (eSLOE), a widely used structured assessment of emergency medicine (EM) residency applicants, and the Association of American Medical Colleges (AAMC) Standardized Video Interview (SVI), a new tool designed by the AAMC to assess interpersonal and communication skills and professionalism knowledge.

METHOD

The authors matched EM residency applicants with valid SVI total scores and completed eSLOEs in the 2018 Match application cycle. They examined correlations and group differences for both tools, United States Medical Licensing Examination (USMLE) Step exam scores, and honor society memberships.

RESULTS

The matched sample included 2,884 applicants. SVI score and eSLOE global assessment ratings demonstrated small positive correlations approaching r = 0.20. eSLOE ratings had higher correlations with measures of academic ability (USMLE scores, academic honor society membership) than did SVI scores. Group differences were minimal for the SVI, with scores slightly favoring women (d = -0.21) and US-MD applicants (d = 0.23-0.42). Group differences in eSLOE ratings were small, favoring women over men (approaching d = -0.20) and white applicants over black applicants (approaching d = 0.40).

CONCLUSIONS

Small positive correlations between SVI score and eSLOE global assessment ratings, alongside varying correlations with academic ability indicators, suggest these are complementary tools. Findings suggest the eSLOE is subject to similar sources and degrees of bias as other common assessments; these group differences were not observed with the SVI. Further examination of both tools is necessary to understand their ability to predict clinical performance.

Estimating Subgroup Differences in Staffing Research When the Selection Mechanism Is Unknown: A Response to Li’s Case IV Correction

When estimating subgroup differences on incumbents, range restriction may bias estimates. Bobko, Roth, and Bobko recognized this problem and developed a Case II and Case III correction for Cohen’s d. Subsequently, Li developed a Case IV correction, which seeks to estimate group differences on a predictor using only incumbent data but must assume that group membership (e.g., ethnicity) plays no role in selection decisions. In this paper, we extend Li’s correction and relax this assumption. In addition, this new correction allows for the estimation of subgroup differences on both the criterion and predictor. Using Monte Carlo simulation, we study the performance of both estimators under situations where Li’s assumptions are violated and demonstrate that this new procedure almost always outperforms Li’s Case IV correction and does so with greater precision. We also provide R code to assist applied researchers in using these corrections.

The Indirect Effect of Justice Perceptions on Job Satisfaction Among Hispanic Employees

The notion that justice perceptions greatly influence behaviors and attitudes at work has been supported in the organizational behavior literature. Given the significant increase of Hispanic employees in the U.S. workforce in the last two decades, more research is needed to understand how justice relates to important outcomes in this population. The present study uses social exchange theory to examine conflict as a mediator of the relationship between justice and overall job satisfaction, and three individual facets of job satisfaction in a sample of 154 working Hispanic young adults in a variety of jobs. Findings indicate that conflict mediates the relationship between each dimension of justice and overall job satisfaction, and between two out of three examined facets of job satisfaction. The results provide evidence for the importance of justice perceptions in determining the attitudes and behaviors of Hispanic employees in the workplace.

The Legal Context of the Management of Human Resources

Laws designed to reduce employment discrimination and to regulate labor standards have a strong impact on the management of human resources in organizations. This article examines in detail the development and enforcement of antidiscrimination laws in the United States; it also considers comparable laws and policies in the European Union (EU). A significant body of research focuses on the standards that are used to determine whether particular policies or practices are discriminatory, and if so, whether they are sufficiently job-related to be legally permissible; here, I examine key themes in this research. This article also examines emerging issues, such as determining who is an applicant and who is an employee, and it explores the role of the legal environment in impeding the application of scientific knowledge to advance the practice of human resource management (HRM).

Multi-Ethnic Workforce and Business Performance: Review and Synthesis of the Empirical Literature

This article provides an updated review and synthesis of the empirical literature of the determinants of business performance, and discusses directions for future research aimed at developing better theories and advancing knowledge of the specific influence of ethnic minorities on firm financial achievements. The review is intended to overcome the weaknesses in previous reviews by focusing exclusively on studies that concentrate on ethnic minority–related determinants of business performance. This article suggests that future research should take a more holistic approach to analyzing the impact of a multicultural workforce on the performance of medium-sized enterprises (SMEs). This article proposes a tentative research model that links six different ethnic-related concept groups directly to the financial and export performance of the firm.

Differential Validity and Differential Prediction of Cognitive Ability Tests: Understanding Test Bias in the Employment Context

Substantial mean score differences and significant adverse impact have long motivated the question of whether cognitive ability tests are biased against certain non-White subgroups. This article presents a framework for understanding the interrelated issues of adverse impact and test bias, with particular focus on two forms of test bias especially relevant for personnel selection: differential validity and differential prediction. Ethical and legal reasons that organizations should be concerned about differential validity/prediction are discussed. This article also serves as a critical review of the research literature on differential validity/prediction. The general conclusion is that available evidence supports the existence of differential validity/prediction in the form of correlation/slope and intercept differences between White and non-White subgroups. Implications for individuals and organizations are outlined, and a future research agenda is proposed highlighting the need for new, better data; new, better methods of testing for differential validity/prediction; and investigation of substantive factors causing differential validity/prediction.

Explaining Extremity in Evaluation of Group Members

A meta-analysis that included more than 1,100 effect sizes tested the predictions of three theoretical perspectives that explain evaluative extremity in social judgment: complexity-extremity theory, subjective group dynamics model, and expectancy-violation theory. The work seeks to understand the ways in which group-based information interacts with person-based information to influence extremity in evaluations. Together, these three theories point to the valence of person-based information, group membership of the evaluated targets relative to the evaluator, status of the evaluators’ ingroup, norm consistency of the person-based information, and incongruency of person-based information with stereotype-based expectations as moderators. Considerable support, but some limiting conditions, were found for each theoretical perspective. Implications of the results are discussed.

The making of inclusion as structuration: empirical evidence of a multinational company

Purpose

– The purpose of this paper is to theoretically and empirically analyse the question how organizations become inclusive – with special regard to migrants – and the potential limits to inclusion.

Design/methodology/approach

– The paper develops a theoretical framework based on Giddens’ structuration theory. By a firm-level case study, the paper empirically examines the theoretical propositions.

Findings

– The paper proposes that inclusion bears specific kinds of the structural dimensions signification, domination and legitimation on which organizational actors draw to reproduce the inclusive organization. The empirical case reveals three areas of organizational practices – personnel recruitment and selection; training and development; meals and parties – in the making of inclusion. But the interplay of specific rules and resources also contains social practices of differentiation and hierarchization that limit inclusion.

Research limitations/implications

– Future studies would benefit from considering additional socio-demographic characteristics and intersectionalities. An ethnographic approach on the basis of participant observation is also recommendable. A longitudinal empirical design focusing on causal relationships would expand the papers descriptive approach.

Practical implications

– The findings suggest that organizational actors can shape the structural dimensions corresponding to an inclusive organization by acting themselves accordingly and inciting others to do so. They should be aware of processes of differentiation and hierarchization that go along with practices of inclusion.

Originality/value

– Applying key arguments of structuration theory, the paper develops a comprehensive framework that considers corresponding rules and resources in detail. The empirical case study demonstrates the fruitfulness of the theoretical framework and reveals the ambivalence of organizational practices that promote inclusion.

Core personal competencies important to entering students' success in medical school: what are they and how could they be assessed early in the admission process?

Assessing applicants' personal competencies in the admission process has proven difficult because there is not an agreed-on set of personal competencies for entering medical students. In addition, there are questions about the measurement properties and costs of currently available assessment tools. The Association of American Medical College's Innovation Lab Working Group (ILWG) and Admissions Initiative therefore engaged in a multistep, multiyear process to identify personal competencies important to entering students' success in medical school as well as ways to measure them early in the admission process. To identify core personal competencies, they conducted literature reviews, surveyed U.S and Canadian medical school admission officers, and solicited input from the admission community. To identify tools with the potential to provide data in time for pre-interview screening, they reviewed the higher education and employment literature and evaluated tools' psychometric properties, group differences, risk of coaching/faking, likely applicant and admission officer reactions, costs, and scalability. This process resulted in a list of nine core personal competencies rated by stakeholders as very or extremely important for entering medical students: ethical responsibility to self and others; reliability and dependability; service orientation; social skills; capacity for improvement; resilience and adaptability; cultural competence; oral communication; and teamwork. The ILWG's research suggests that some tools hold promise for assessing personal competencies, but the authors caution that none are perfect for all situations. They recommend that multiple tools be used to evaluate information about applicants' personal competencies in deciding whom to interview.

Workaholism and work-life imbalance: does cultural origin influence the relationship?

In recent years, workaholism has become prevalent throughout organizations and has captured the attention of organizational leaders as well as the academic and scientific communities. Most research in this area has focused on the negative consequences of workaholism, specifically work-life imbalance. One area of research that has largely been ignored is the potential influence of demographic variables on the relationship between workaholism and work-life imbalance. Therefore, the current study focused on how cultural origin might influence the intensity of this relationship. Based on relative deprivation theory and previous empirical work, it was expected that cultural origin would moderate the relationship between workaholism and work-life imbalance. Specifically, it was predicted that Caucasian participants would score higher on levels of workaholism than Black participants, and that the relationship between workaholism and work-life imbalance would be stronger for Caucasians than for Blacks. The results revealed that high levels of workaholism were significantly correlated with high levels of work-life imbalance. However, results also indicated that cultural origin did not moderate the relationship between workaholism and work-life imbalance, and there was no significant mean difference between Caucasian and Black participants on our measure of workaholism. These findings are important in that it is essential for employers to be aware of workaholic tendencies so they can better handle the negative consequences that result for the organization, and to also help promote the well-being of their employees. En los últimos años la adicción al trabajo se ha vuelto muy popular en las organizaciones y ha capturado la atención tanto de líderes organizacionales, así como de las comunidades científicas y académicas. La mayoría de investigaciones en esta área se han focalizado en las consequencias negativas de la adicción al trabajo (workoholismo), específicamente en el desequilibrio entre trabajo-vida. Una área de investigación que ha sido ignorada por mucho tiempo es la influencia potencial de las variables demográficas sobre la relación entre adicción al trabajo y el desequilibrio trabajo-vida. Por este motivo es que el presente estudio se focalizó en como el origen cultural puede influenciar la intensidad de esta relación. Tomando como base la teoría de la deprivación relativa y estudios empíricos previos, se esperó que el origen cultural moderaría la relación entre adicción al trabajo y desequilibrio trabajo-vida. Específicamente se pudo predecir que los participantes caucásicos mostraron más elevados puntajes en los niveles de adicción al trabajo que los participantes negros, y que la relación entre adicción al trabajo y desequilibrio trabajo-vida era más intensa entre los caucásicos que entre los negros. Los resultados revelaron que elevados niveles de adicción al trabajo correlacionaron significativamente con altos niveles de desequilibrio trabajo-vida. Sin embargo los resultados también revelaron que el origen cultural no moderaba la relación entre adicción al trabajo y desequilibrio trabajo-vida, y que además no había una diferencia significativa de medias entre los participantes caucásicos y negros respecto de nuestras mediciones de adicción al trabajo. Estos resultados son importantes en la medida en que es esencial para los empleadores el ser conscientes respecto de las tendencias de adicción al trabajo, lo que les permitirá manejar con mayor efectividad las consequencias negativas para la empresa resultantes de estas tendencias, y por otro lado apoyar a sus empleados en la promoción de su bienestar. Au cours des dernières années, le travaillolisme (workaholism) a pris de l'ampleur dans les organisations et a capté l'attention à la fois des dirigeants d'organisations et des communautés académique et scientifique. La majorité de la recherche dans ce domaine fut orientée sur les conséquences négatives du travaillolisme et, plus spécifiquement, sur le conflit travail-vie. Un champ de recherche qui a été largement ignoré est l'influence potentielle des variables démographiques sur la relation entre le travaillolisme et le conflit travail-vie. La présente étude visait à comprendre comment l'origine culturelle peut influencer l'intensité de cette relation. Se basant sur la théorie de la privation relative et sur les travaux empiriques antérieurs, il était attendu que l'origine culturelle allait jouer un rôle modérateur dans la relation entre le travaillolisme et le conflit travail-vie. Plus spécifiquement, il était prédit que les participants caucasiens allaient obtenir des scores plus élevés de travaillolisme comparativement aux participants noirs, et que la relation entre le travaillolisme et le conflit travail-vie allait être plus forte pour les caucasiens que pour les noirs. Les résultats ont révélé que des niveaux élevés de travaillolisme étaient significativement corrélés avec des niveaux élevés de conflit travail-vie. Cependant, les résultats ont aussi indiqué que l'origine culturelle ne modère pas la relation entre le travaillolisme et le conflit travail-vie. De plus, il n'y avait pas de différence significative entre les participants caucasiens et les participants noirs sur notre mesure de travaillolisme. Ces résultats sont importants puisqu'il est essentiel pour les employeurs d'être à l'affût des tendances travaillolistes pour qu'ils puissent mieux en gérer les conséquences négatives pour l'organisation, ainsi que pour aider à promouvoir le bien-être de leurs employés. diferenciaban en algunos aspectos.

The Language of Performance Evaluations

Performance evaluations of male and female junior attorneys in a Wall Street law firm were analyzed. Male supervisors judged male attorneys more favorably than female attorneys on numerical ratings that mattered for promotion but offered narrative comments that showed either no sex effects or greater favorability toward women. Judgments of male attorneys were more consistent overall than they were for female attorneys, and predictors of numerical ratings differed by sex: Narrative ratings of technical competence mattered more for men than women, and narrative ratings of interpersonal warmth mattered more for women than men. Open-ended use of positive performance words—the only outcome that favored women—did not translate into positive numerical ratings for women. The data suggest subtle patterns of gender bias, in which women were harmed by not meeting gendered expectations of interpersonal warmth but were less benefited than men by meeting masculine standards of high technical competence.

Test Bias, Differential Prediction, and a Revised Approach for Determining the Suitability of a Predictor in a Selection Context

The most commonly used and accepted model of assessing bias in a selection context is that proposed by Cleary in which predictor-criterion regression lines are tested for both slope and intercept equality. With this approach, any difference in intercepts or slopes is considered an indication of bias. We argue that differing regression lines intercepts is indicative of differential prediction but not test bias. We describe several fundamentally different potential causes of differences in groups’ regression line intercepts, many of which are unrelated to test properties. We argue that differential prediction because of such sources should not preclude the use of the test in selection contexts. We propose a new procedure to potentially identify the source of regression line differences and illustrate this framework using a job incumbent sample.