“Transforming” Personality Scale Development: Illustrating the Potential of State-of-the-Art Natural Language Processing

Natural language processing (NLP) techniques are becoming increasingly popular in industrial and organizational psychology. One promising area for NLP-based applications is scale development; yet, while many possibilities exist, so far these applications have been restricted—mainly focusing on automated item generation. The current research expands this potential by illustrating an NLP-based approach to content analysis, which manually categorizes scale items by their measured constructs. In NLP, content analysis is performed as a text classification task whereby a model is trained to automatically assign scale items to the construct that they measure. Here, we present an approach to text classification—using state-of-the-art transformer models—that builds upon past approaches. We begin by introducing transformer models and their advantages over alternative methods. Next, we illustrate how to train a transformer to content analyze Big Five personality items. Then, we compare the models trained to human raters, finding that transformer models outperform human raters and several alternative models. Finally, we present practical considerations, limitations, and future research directions.

Investigating the promise and pitfalls of pulse surveys

Despite the growing popularity and marketing of pulse surveys, there is little research concerning this practice. To this end, this practice forum reports the results of a four-wave pulse survey that was conducted in a health care organization. Pulse surveys provided reliable estimates of overall engagement, but scores remained stable across 8 months. Practically no differences in group scores or trends could be found despite high participation (≍ 50%). Item responses displayed little differences between groups, ICC(1) ranging from .03 to .18, and poor discriminant validity. Based on these results, pulse surveys may be adequate for estimating overall employee sentiment but not useful for detecting change over time or differences between groups. These limitations should be considered when designing or implementing pulse surveys.