Do we measure leadership effectively? Articulating and evaluating scale development psychometrics for best practice

Feasibility of virtual reality and machine learning to assess personality traits in an organizational environment

Introduction

Personality plays a crucial role in shaping an individual's interactions with the world. The Big Five personality traits are widely used frameworks that help describe people's psychological behaviours. These traits predict how individuals behave within an organizational setting.

Methods

In this article, we introduce a virtual reality (VR) strategy for relatively scoring an individual's personality to evaluate the feasibility of predicting personality traits from implicit measures captured from users interacting in VR simulations of different organizational situations. Specifically, eye-tracking and decision-making patterns were used to classify individuals according to their level in each of the Big Five dimensions using statistical machine learning (ML) methods. The virtual environment was designed using an evidence-centered design approach.

Results

The dimensions were assessed using NEO-FFI inventory. A random forest ML model provided 83% accuracy in predicting agreeableness. A k-nearest neighbour ML model provided 75%, 75%, and 77% accuracy in predicting openness, neuroticism, and conscientiousness, respectively. A support vector machine model provided 85% accuracy for predicting extraversion. These analyses indicated that the dimensions could be differentiated by eye-gaze patterns and behaviours during immersive VR.

Discussion

Eye-tracking measures contributed more significantly to this differentiation than the behavioural metrics. Currently, we have obtained promising results with our group of participants, but to ensure the robustness and generalizability of our findings, it is imperative to replicate the study with a considerably larger sample. This study demonstrates the potential of VR and ML to recognize personality traits.

Small but Nontrivial: A Comparison of Six Strategies to Handle Cross-Loadings in Bifactor Predictive Models

The bifactor model is a promising alternative to traditional modeling techniques for studying the predictive validity of hierarchical constructs. However, no study to date has systematically examined the influence of cross-loadings on the estimation of regression coefficients in bifactor predictive models. Therefore, we present a systematic examination of the statistical performance of six modeling strategies to handle cross-loadings in bifactor predictive models: structural equation modeling (SEM), exploratory structural equation modeling (ESEM) with target rotation, Bayesian structural equation modeling (BSEM), and each of the three with augmentation. Results revealed four clear patterns: 1) forcing even small cross-loadings to zero was detrimental to empirical identification, estimation bias, power and Type I error rates; 2) the performance of ESEM with target rotation was unexpectedly weak; 3) augmented BSEM had satisfactory performance in an absolute sense and outperformed the other five strategies across most conditions; 4) augmentation improved the performance of ESEM and SEM, although the degree of improvement was not as substantial as that of BSEM. In addition, we also presented an empirical example to show the feasibility of the proposed approach. Overall, these findings can help users of bifactor predictive models design better studies, choose more appropriate analytical strategies, and obtain more reliable results. Implications, limitations, and future directions are discussed.

Evaluation of a leadership development impact assessment toolkit: a comparative case study of experts' perspectives in three Canadian provinces

PURPOSE

This paper aims to explore users' perceptions of whether the Leadership Development Impact Assessment (LDI) Toolkit is valid, reliable, simple to use and cost-effective as a guide to its quality improvement.

DESIGN/METHODOLOGY/APPROACH

The Canadian Health Leadership Network codesigned and codeveloped the LDI Toolkit as a theory-driven and evidence-informed resource that aims to assist health-care organizational development practitioners to evaluate various programs at five levels of impact: reaction, learning, application, impact and return on investment (ROI) and intangible benefits. A comparative evaluative case study was conducted using online questionnaires and semistructured telephone interviews with three health organizations where robust leadership development programs were in place. A total of seven leadership consultants and specialists participated from three Canadian provinces. Data were analyzed sequentially in two stages involving descriptive statistical analysis augmented with a qualitative content analysis of key themes.

FINDINGS

Users perceived the toolkit as cost-effective in terms of direct costs, indirect costs and intangibles; they found it easy-to-use in terms of clarity, logic and structure, ease of navigation with a coherent layout; and they assessed the sources of the evidence-informed tools and guides as appropriate. Users rated the toolkit highly on their perceptions of its validity and reliability. The analysis also informed the refinement of the toolkit.

ORIGINALITY/VALUE

The refined LDI Toolkit is a comprehensive online collection of various tools to support health organizations to evaluate the leadership development investments effectively and efficiently at five impact levels including ROI.

Combining Virtual Reality and Machine Learning for Leadership Styles Recognition

The aim of this study was to evaluate the viability of a new selection procedure based on machine learning (ML) and virtual reality (VR). Specifically, decision-making behaviours and eye-gaze patterns were used to classify individuals based on their leadership styles while immersed in virtual environments that represented social workplace situations. The virtual environments were designed using an evidence-centred design approach. Interaction and gaze patterns were recorded in 83 subjects, who were classified as having either high or low leadership style, which was assessed using the Multifactor leadership questionnaire. A ML model that combined behaviour outputs and eye-gaze patterns was developed to predict subjects' leadership styles (high vs low). The results indicated that the different styles could be differentiated by eye-gaze patterns and behaviours carried out during immersive VR. Eye-tracking measures contributed more significantly to this differentiation than behavioural metrics. Although the results should be taken with caution as the small sample does not allow generalization of the data, this study illustrates the potential for a future research roadmap that combines VR, implicit measures, and ML for personnel selection.

A comparative evaluation of factor- and component-based structural equation modelling approaches under (in)correct construct representations

Structural equation modelling (SEM) has evolved into two domains, factor-based and component-based, dependent on whether constructs are statistically represented as common factors or components. The two SEM domains are conceptually distinct, each assuming their own population models with either of the statistical construct proxies, and statistical SEM approaches should be used for estimating models whose construct representations correspond to what they assume. However, SEM approaches have often been evaluated and compared only under population factor models, providing misleading conclusions about their relative performance. This is partly because population component models and their relationships have not been clearly formulated. Also, it is of fundamental importance to examine how robust SEM approaches can be to potential misrepresentation of constructs because researchers may often lack clear theories to determine whether a factor or component is more representative of a given construct. Addressing these issues, this study begins by clarifying several population component models and their relationships and then provides a comprehensive evaluation of four SEM approaches - the maximum likelihood approach and factor score regression for factor-based SEM as well as generalized structured component analysis (GSCA) and partial least squares path modelling (PLSPM) for component-based SEM - under various experimental conditions. We confirm that the factor-based SEM approaches should be preferred for estimating factor models, whereas the component-based SEM approaches should be chosen for component models. Importantly, the component-based approaches are generally more robust to construct misrepresentation than the factor-based ones. Of the component-based approaches, GSCA should be chosen over PLSPM, regardless of whether or not constructs are misrepresented.

Coachability and the development of the coachability scale

Purpose

The purpose of this research was to develop a conceptualization and measure of workplace coachability.

Design/methodology/approach

Using four independent samples of employed adults, we developed a short and long version of the Coachability Scale. We followed standard scale development practices, presenting evidence of the scales’ factor structure, reliability and validity.

Findings

With the first two samples, we derived an initial three-dimensional version of the Coachability Scale and provided evidence of convergent validity. With Samples 3 and 4, we expanded the scale with additional dimensions related to coaching feedback processes and accumulated additional evidence of the scale's validity, and provided evidence of convergence between the two versions of the Coachability Scale.

Research limitations/implications

We encourage continued research on the Coachability Scale, as well as research on coachability in formal coaching relationships and with more diverse populations and cultures. It is also important to examine how coachability relates to specific coachee behaviors and outcomes. Although common method bias may be a limitation, we used temporally separated measurements to minimize method bias in Sample 4.

Practical implications

Knowledge about coachability can inform coaching practice decisions and help tailor the coaching engagement to better fit the coachee's needs.

Social implications

Measuring how individuals respond to coaching and coaching relationships has important implications for managerial behavior and the quality of work life.

Originality/value

This is one of the first studies to develop valid scales for assessing workplace coachability.

Harmful Leader Behaviors: Toward an Increased Understanding of How Different Forms of Unethical Leader Behavior Can Harm Subordinates

Research on unethical leadership has predominantly focused on interpersonal and high-intensity forms of harmful leader behavior such as abusive supervision. Other forms of harmful leader behavior such as excessively pressuring subordinates or acting in self-centered ways have received less attention, despite being harmful and potentially occurring more frequently. We propose a model of four types of harmful leader behavior (HLB) varying in intensity (high vs low) and orientation (people/relationships or tasks/goals): Intimidation, Lack of Care, Self-Centeredness, and Excessive Pressure for Results. We map out how these relate to other constructs in the unethical leader behavior field in order to integrate the existing work on how leaders can cause harm to followers. Next, in five studies (N = 35, N = 218, N = 352, N = 160, N = 1921 in 196 teams), we develop and test a new survey instrument measuring the four proposed types of perceived HLB. We provide initial validity evidence for this new measure, establish its psychometric properties, and examine its nomological network by linking the four types of HLB to related leadership constructs and soft and hard outcome correlates at the individual and team level. We find that HLB is negatively related to constructive forms of leadership (e.g., ethical and transformational) and positively to unethical ones (e.g., abusive supervision). HLB is also related in the expected direction to job satisfaction, engagement, psychological safety, knowledge sharing, knowledge hiding, deviance, and objectively recorded team-level stress-related absenteeism.

A Systematic Review of Leadership Definitions, Competencies, and Assessment Methods in Pharmacy Education

Objective. To characterize leadership definitions, competencies, and assessment methods used in pharmacy education, based on a systematic review of the literature. Findings. After undergoing title, abstract, and full-text review, 44 (10%) of 441 articles identified in the initial search were included in this report. Leadership or an aspect of leadership was defined in 37 (84%) articles, and specific leadership competencies were listed or described in 40 (91%) articles. The most common definitions of leadership involved motivating others toward the achievement of a specific goal and leading organizational change. Definitions of leadership in some articles required that individuals hold a formal leadership position whereas others did not. Only two leadership competencies were related to specific areas of knowledge. Most of the competencies identified were interpersonal and self-management skills. In terms of assessment, only one (2.3%) article assessed leadership effectiveness, and none assessed leadership development. Of the remaining 24 (55%) articles that included some type of assessment, most involved behavioral-based tools assessing individual attributes conceptually related to leadership (eg, strengths, emotional intelligence), or self-assessments regarding whether learning objectives in a leadership course had been met. Summary. Definitions for leadership in pharmacy varied considerably, as did leadership competencies. Most conceptualizations of leadership resembled a combination of established approaches rather than being grounded in a specific theory. If leadership development is to remain a focus within accreditation standards for Doctor of Pharmacy education, a consistent framework for operationalizing it is needed.