Linear Factor Analytic Thurstonian Forced-Choice Models: Current Status and Issues

Thurstonian forced-choice modeling is considered to be a powerful new tool to estimate item and person parameters while simultaneously testing the model fit. This assessment approach is associated with the aim of reducing faking and other response tendencies that plague traditional self-report trait assessments. As a result of major recent methodological developments, the estimation of normative trait scores has become possible in addition to the computation of only ipsative scores. This opened up the important possibility of comparisons between individuals with forced-choice assessment procedures. With item response theory (IRT) methods, a multidimensional forced-choice (MFC) format has also been proposed to estimate individual scores. Customarily, items to assess different traits are presented in blocks, often triplets, in applications of the MFC, which is an efficient form of item presentation but also a simplification of the original models. The present study provides a comprehensive review of the present status of Thurstonian forced-choice models and their variants. Critical features of the current models, especially the block models, are identified and discussed. It is concluded that MFC modeling with item blocks is highly problematic and yields biased results. In particular, the often-recommended presentation of blocks with items that are keyed in different directions of a trait proves to be counterproductive considering the goal to reduce response tendencies. The consequences and implications of the highlighted issues are further discussed.

Though Forced, Still Valid: Examining the Psychometric Performance of Forced-Choice Measurement of Personality in Children and Adolescents

Unveiling the roles personality plays during childhood and adolescence necessitates its accurate measurement, commonly using traditional Likert-type (LK) scales. However, this format is susceptible to various response biases, which can be particularly prevalent in children and adolescents, thus likely undermining measurement accuracy. Forced-choice (FC) scales appear to be a promising alternative because they are largely free from these biases by design. However, some argue that the FC format may not perform satisfactorily in children and adolescents due to its complexity. Little empirical evidence exists regarding the suitability of the FC format for children and adolescents. As such, the current study examined the psychometric performance of an FC measure of the Big Five personality factors in three children and adolescent samples: 5th to 6th graders (N = 428), 7th to 8th graders (N = 449), and 10th to 11th graders (N = 555). Across the three age groups, the FC scale demonstrated a better fit to the Big Five model and better discriminant validity in comparison to the LK counterpart. Personality scores from the FC scale also converged well with those from the LK scale and demonstrated high reliability as well as sizable criterion-related validity. Furthermore, the FC scale had more invariant statements than its LK counterpart across age groups. Overall, we found good evidence showing that FC measurement of personality is suitable for children and adolescents.

Moving beyond Likert and Traditional Forced-Choice Scales: A Comprehensive Investigation of the Graded Forced-Choice Format

The graded forced-choice (FC) format has recently emerged as an alternative that may preserve the advantages and overcome the issues of the dichotomous FC measures. The current study presented the first large-scale evaluation of the performance of three types of FC measures (FC2, FC4 and FC5 with 2, 4 and 5 response options, respectively) and compared their performance to their Likert (LK) counterparts (LK2, LK4, and LK5) on (1) psychometric properties, (2) respondent reactions, and (3) susceptibility to response styles. Results showed that, compared to LK measures with the same number of response options, the three FC scales provided better support for the hypothesized factor structure, were perceived as more faking-resistant and cognitive demanding, and were less susceptible to response styles. FC4/5 and LK4/5 demonstrated similarly good reliability, while LK2 provided more reliable scores than FC2. When compared across the three FC measures, FC4 and FC5 displayed comparable psychometric performance and respondent reactions. FC4 exhibited a moderate presence of extreme response style, while FC5 had a weak presence of both extreme and middle response styles. Based on these findings, the study recommends the use of graded FC over dichotomous FC and LK, particularly FC5 when extreme response style is a concern.

Multidimensional IRT for forced choice tests: A literature review

The Multidimensional Forced Choice (MFC) test is frequently utilized in non-cognitive evaluations because of its effectiveness in reducing response bias commonly associated with the conventional Likert scale. Nonetheless, it is critical to recognize that the MFC test generates ipsative data, a type of measurement that has been criticized due to its limited applicability for comparing individuals. Multidimensional item response theory (MIRT) models have recently sparked renewed interest among academics and professionals. This is largely due to the development of several models that make it easier to collect normative data from forced-choice tests. The paper introduces a modeling framework made up of three key components: response format, measurement model, and decision theory. Under this paradigm, four IRT models were chosen as examples. Following that, a comprehensive study is carried out to compare and characterize the parameter estimation techniques used in MFC-IRT models. This work then examines empirical research on the concept by analyzing three distinct domains: parameter invariance testing, computerized adaptive testing (CAT), and validity investigation. Finally, it is recommended that future research initiatives follow four distinct paths: modeling, parameter invariance testing, forced-choice CAT, and validity studies.

Antimicrobial Prescription Behavior in Equine Asthma Cases: An International Survey

Equine asthma is considered a non-infectious respiratory disease characterized by inflammation and hypersensitivity. Given the importance of antimicrobial stewardship, an international survey was designed to investigate the use of antimicrobials in asthmatic horses and the factors influencing prescription behavior. An online survey was distributed in six languages by international mailing lists and social media from December 2020 to January 2022. Of the 434 responses recorded, 249 veterinarians working in 25 countries finished the survey. These included 79 ECEIM/ACVIM diplomats. A total of 204 respondents confirmed national regulations concerning antimicrobial use in their country. Knowledge of the asthma definitions as presented in the revised ACVIM consensus statement was greater in veterinarians treating over 95% of equine patients compared to veterinarians treating more species, based on 10 questions (answers consistent with the consensus statement in 7 (IQR 5-9) and 4 (IQR 3-6) questions, respectively, (p < 0.001)). A total of 131 respondents stated to use antimicrobials (at least 'sometimes') in at least one of the three presented cases consistent with equine asthma. Trimethoprim-Sulfa combinations, penicillin(s), and tetracyclines were prescribed most (by 105, 53 and 38 veterinarians, respectively). Aminoglycosides, cephalosporins and fluoroquinolones were also selected (22, 15, 9 veterinarians respectively). Tetracyclines and cephalosporins are prescribed more often by veterinarians working without national regulations (p 0.019 and p 0.035, respectively). The most selected factors influencing prescription behavior were: 'tracheal wash culture' (62% of 131 veterinarians using antimicrobials in these cases), 'whether other horses in the barn have similar complaints' (60%), and 'response to other forms of therapy' (53%). In conclusion, insight into prescription behavior is the first step towards minimizing and optimizing antimicrobial use.

Estimating and Using Block Information in the Thurstonian IRT Model

Multidimensional forced-choice (MFC) tests are increasing in popularity but their construction is complex. The Thurstonian item response model (Thurstonian IRT model) is most often used to score MFC tests that contain dominance items. Currently, in a frequentist framework, information about the latent traits in the Thurstonian IRT model is computed for binary outcomes of pairwise comparisons, but this approach neglects stochastic dependencies. In this manuscript, it is shown how to estimate Fisher information on the block level. A simulation study showed that the observed and expected standard errors based on the block information were similarly accurate. When local dependencies for block sizes [Formula: see text] were neglected, the standard errors were underestimated, except with the maximum a posteriori estimator. It is shown how the multidimensional block information can be summarized for test construction. A simulation study and an empirical application showed small differences between the block information summaries depending on the outcome considered. Thus, block information can aid the construction of reliable MFC tests.

Multidimensional Forced-Choice CAT With Dominance Items: An Empirical Comparison With Optimal Static Testing Under Different Desirability Matching

Several forced-choice (FC) computerized adaptive tests (CATs) have emerged in the field of organizational psychology, all of them employing ideal-point items. However, despite most items developed historically follow dominance response models, research on FC CAT using dominance items is limited. Existing research is heavily dominated by simulations and lacking in empirical deployment. This empirical study trialed a FC CAT with dominance items described by the Thurstonian Item Response Theory model with research participants. This study investigated important practical issues such as the implications of adaptive item selection and social desirability balancing criteria on score distributions, measurement accuracy and participant perceptions. Moreover, nonadaptive but optimal tests of similar design were trialed alongside the CATs to provide a baseline for comparison, helping to quantify the return on investment when converting an otherwise-optimized static assessment into an adaptive one. Although the benefit of adaptive item selection in improving measurement precision was confirmed, results also indicated that at shorter test lengths CAT had no notable advantage compared with optimal static tests. Taking a holistic view incorporating both psychometric and operational considerations, implications for the design and deployment of FC assessments in research and practice are discussed.

Investigating the Normativity of Trait Estimates from Multidimensional Forced-Choice Data

The Thurstonian item response model (Thurstonian IRT model) allows deriving normative trait estimates from multidimensional forced-choice (MFC) data. In the MFC format, persons must rank-order items that measure different attributes according to how well the items describe them. This study evaluated the normativity of Thurstonian IRT trait estimates both in a simulation and empirically. The simulation investigated normativity and compared Thurstonian IRT trait estimates to those using classical partially ipsative scoring, from dichotomous true-false (TF) data and rating scale data. The results showed that, with blocks of opposite keyed items, Thurstonian IRT trait estimates were normative in contrast to classical partially ipsative estimates. Unbalanced numbers of items per trait, few opposite keyed items, traits correlated positively or assessing fewer traits did not decrease measurement precision markedly. Measurement precision was lower than that of rating scale data. The empirical study investigated whether relative MFC responses provide a better differentiation of behaviors within persons than absolute TF responses. However, criterion validity was equal and construct validity (with constructs measured by rating scales) lower in MFC. Thus, Thurstonian IRT modeling of MFC data overcomes the drawbacks of classical scoring, but gains in validity may depend on eliminating common method biases from the comparison.

Small but Nontrivial: A Comparison of Six Strategies to Handle Cross-Loadings in Bifactor Predictive Models

The bifactor model is a promising alternative to traditional modeling techniques for studying the predictive validity of hierarchical constructs. However, no study to date has systematically examined the influence of cross-loadings on the estimation of regression coefficients in bifactor predictive models. Therefore, we present a systematic examination of the statistical performance of six modeling strategies to handle cross-loadings in bifactor predictive models: structural equation modeling (SEM), exploratory structural equation modeling (ESEM) with target rotation, Bayesian structural equation modeling (BSEM), and each of the three with augmentation. Results revealed four clear patterns: 1) forcing even small cross-loadings to zero was detrimental to empirical identification, estimation bias, power and Type I error rates; 2) the performance of ESEM with target rotation was unexpectedly weak; 3) augmented BSEM had satisfactory performance in an absolute sense and outperformed the other five strategies across most conditions; 4) augmentation improved the performance of ESEM and SEM, although the degree of improvement was not as substantial as that of BSEM. In addition, we also presented an empirical example to show the feasibility of the proposed approach. Overall, these findings can help users of bifactor predictive models design better studies, choose more appropriate analytical strategies, and obtain more reliable results. Implications, limitations, and future directions are discussed.

Can Faking Be Measured With Dedicated Validity Scales? Within-Subject Trifactor Mixture Modeling Applied to BIDR Responses

A sample of 516 participants responded to the Balanced Inventory of Desirable Responding (BIDR) under answer honest and instructed faking conditions in a within-subjects design. We analyze these data with a novel application of trifactor modeling that models the two substantive factors measured by the BIDR-self-deceptive enhancement (SDE) and impression management (IM), condition-related common factors, and item-specific factors. The model permits examination of invariance and change within subjects across conditions. Participants were able to significantly increase their SDE and IM in the instructed faking condition relative to the honest response condition. Mixture modeling confirmed the existence of a theoretical two-class solution comprised of approximately two thirds of "compliers" and one third of "noncompliers." Factor scores had good determinacy and correlations with observed scores were near unity for continuous scoring, supporting observed score interpretations of BIDR scales in high-stakes settings. Correlations were somewhat lower for the dichotomous scoring protocol. Overall, results show that the BIDR scales function similarly as measures of socially desirable functioning in low- and high-stakes conditions. We discuss conditions under which we expect these results will and will not generalize to other validity scales.

Detecting DIF in Multidimensional Forced Choice Measures Using the Thurstonian Item Response Theory Model

Although modern item response theory (IRT) methods of test construction and scoring have overcome ipsativity problems historically associated with multidimensional forced choice (MFC) formats, there has been little research on MFC differential item functioning (DIF) detection, where item refers to a block, or group, of statements presented for an examinee’s consideration. This research investigated DIF detection with three-alternative MFC items based on the Thurstonian IRT (TIRT) model, using omnibus Wald tests on loadings and thresholds. We examined constrained and free baseline model comparisons strategies with different types and magnitudes of DIF, latent trait correlations, sample sizes, and levels of impact in an extensive Monte Carlo study. Results indicated the free baseline strategy was highly effective in detecting DIF, with power approaching 1.0 in the large sample size and large magnitude of DIF conditions, and similar effectiveness in the impact and no-impact conditions. This research also included an empirical example to demonstrate the viability of the best performing method with real examinees and showed how a DIF and a DTF effect size measure can be used to assess the practical significance of MFC DIF findings.

On the Statistical and Practical Limitations of Thurstonian IRT Models

Forced-choice questionnaires have been proposed to avoid common response biases typically associated with rating scale questionnaires. To overcome ipsativity issues of trait scores obtained from classical scoring approaches of forced-choice items, advanced methods from item response theory (IRT) such as the Thurstonian IRT model have been proposed. For convenient model specification, we introduce the thurstonianIRT R package, which uses Mplus, lavaan, and Stan for model estimation. Based on practical considerations, we establish that items within one block need to be equally keyed to achieve similar social desirability, which is essential for creating forced-choice questionnaires that have the potential to resist faking intentions. According to extensive simulations, measuring up to five traits using blocks of only equally keyed items does not yield sufficiently accurate trait scores and inter-trait correlation estimates, neither for frequentist nor for Bayesian estimation methods. As a result, persons' trait scores remain partially ipsative and, thus, do not allow for valid comparisons between persons. However, we demonstrate that trait scores based on only equally keyed blocks can be improved substantially by measuring a sizable number of traits. More specifically, in our simulations of 30 traits, scores based on only equally keyed blocks were non-ipsative and highly accurate. We conclude that in high-stakes situations where persons are motivated to give fake answers, Thurstonian IRT models should only be applied to tests measuring a sizable number of traits.

Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures

Forced-choice (FC) measures are gaining popularity as an alternative assessment format to single-statement (SS) measures. However, a fundamental question remains to be answered: Do FC and SS instruments measure the same underlying constructs? In addition, FC measures are theorized to be more cognitively challenging, so how would this feature influence respondents’ reactions to FC measures compared to SS? We used both between- and within-subjects designs to examine the equivalence of the FC format and the SS format. As the results illustrate, FC measures scored by the multi-unidimensional pairwise preference (MUPP) model and SS measures scored with the generalized graded unfolding model (GGUM) showed strong equivalence. Specifically, both formats demonstrated similar marginal reliabilities and test-retest reliabilities, high convergent validities, good discriminant validities, and similar criterion-related validities with theoretically relevant criteria. In addition, the formats had little differential impact on respondents’ general emotional and cognitive reactions except that the FC format was perceived to be slightly more difficult and more time-saving.

The sound of beauty: How complexity determines aesthetic preference

Stimulus complexity is an important determinant of aesthetic preference. An influential idea is that increases in stimulus complexity lead to increased preference up to an optimal point after which preference decreases (inverted-U pattern). However, whereas some studies indeed observed this pattern, most studies instead showed an increased preference for more complexity. One complicating issue is that it remains unclear how to define complexity. To address this, we approached complexity and its relation to aesthetic preference from a predictive coding perspective. Here, low- and high-complexity stimuli would correspond to low and high levels of prediction errors, respectively. We expected participants to prefer stimuli which are neither too easy to predict (low prediction error), nor too difficult (high prediction error). To test this, we presented two sequences of tones on each trial that varied in predictability from highly regular (low prediction error) to completely random (high prediction error), and participants had to indicate which of the two sequences they preferred in a two-interval forced-choice task. The complexity of each tone sequence (amount of prediction error) was estimated using entropy. Results showed that participants tended to choose stimuli with intermediate complexity over those of high or low complexity. This confirms the century-old idea that stimulus complexity has an inverted-U relationship to aesthetic preference.

Taking the Test Taker's Perspective: Response Process and Test Motivation in Multidimensional Forced-Choice Versus Rating Scale Instruments

The multidimensional forced-choice (MFC) format has been proposed as an alternative to the rating scale (RS) response format. However, it is unclear how changing the response format may affect the response process and test motivation of participants. In Study 1, we investigated the MFC response process using the think-aloud technique. In Study 2, we compared test motivation between the RS format and different versions of the MFC format (presenting 2, 3, 4, and 5 items simultaneously). The response process to MFC item blocks was similar to the RS response process but involved an additional step of weighing the items within a block against each other. The RS and MFC response format groups did not differ in their test motivation. Thus, from the test taker's perspective, the MFC format is somewhat more demanding to respond to, but this does not appear to decrease test motivation.