1
|
Guo Z, Wang D, Cai Y, Tu D. An Item Response Theory Model for Incorporating Response Times in Forced-Choice Measures. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT 2024; 84:450-480. [PMID: 38756463 PMCID: PMC11095319 DOI: 10.1177/00131644231171193] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/18/2024]
Abstract
Forced-choice (FC) measures have been widely used in many personality or attitude tests as an alternative to rating scales, which employ comparative rather than absolute judgments. Several response biases, such as social desirability, response styles, and acquiescence bias, can be reduced effectively. Another type of data linked with comparative judgments is response time (RT), which contains potential information concerning respondents' decision-making process. It would be challenging but exciting to combine RT into FC measures better to reveal respondents' behaviors or preferences in personality measurement. Given this situation, this study aims to propose a new item response theory (IRT) model that incorporates RT into FC measures to improve personality assessment. Simulation studies show that the proposed model can effectively improve the estimation accuracy of personality traits with the ancillary information contained in RT. Also, an application on a real data set reveals that the proposed model estimates similar but different parameter values compared with the conventional Thurstonian IRT model. The RT information can explain these differences.
Collapse
Affiliation(s)
| | - Daxun Wang
- Jiangxi Normal University, Nanchang, China
| | - Yan Cai
- Jiangxi Normal University, Nanchang, China
| | - Dongbo Tu
- Jiangxi Normal University, Nanchang, China
| |
Collapse
|
2
|
Bäckström M, Björklund F. Why Forced-Choice and Likert Items Provide the Same Information on Personality, Including Social Desirability. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT 2024; 84:549-576. [PMID: 38756462 PMCID: PMC11095325 DOI: 10.1177/00131644231178721] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 05/18/2024]
Abstract
The forced-choice response format is often considered superior to the standard Likert-type format for controlling social desirability in personality inventories. We performed simulations and found that the trait information based on the two formats converges when the number of items is high and forced-choice items are mixed with regard to positively and negatively keyed items. Given that forced-choice items extract the same personality information as Likert-type items do, including socially desirable responding, other means are needed to counteract social desirability. We propose using evaluatively neutralized items in personality measurement, as they can counteract social desirability regardless of response format.
Collapse
|
3
|
Zhang B, Luo J, Li J. Moving beyond Likert and Traditional Forced-Choice Scales: A Comprehensive Investigation of the Graded Forced-Choice Format. MULTIVARIATE BEHAVIORAL RESEARCH 2024; 59:434-460. [PMID: 37652572 DOI: 10.1080/00273171.2023.2235682] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/02/2023]
Abstract
The graded forced-choice (FC) format has recently emerged as an alternative that may preserve the advantages and overcome the issues of the dichotomous FC measures. The current study presented the first large-scale evaluation of the performance of three types of FC measures (FC2, FC4 and FC5 with 2, 4 and 5 response options, respectively) and compared their performance to their Likert (LK) counterparts (LK2, LK4, and LK5) on (1) psychometric properties, (2) respondent reactions, and (3) susceptibility to response styles. Results showed that, compared to LK measures with the same number of response options, the three FC scales provided better support for the hypothesized factor structure, were perceived as more faking-resistant and cognitive demanding, and were less susceptible to response styles. FC4/5 and LK4/5 demonstrated similarly good reliability, while LK2 provided more reliable scores than FC2. When compared across the three FC measures, FC4 and FC5 displayed comparable psychometric performance and respondent reactions. FC4 exhibited a moderate presence of extreme response style, while FC5 had a weak presence of both extreme and middle response styles. Based on these findings, the study recommends the use of graded FC over dichotomous FC and LK, particularly FC5 when extreme response style is a concern.
Collapse
Affiliation(s)
- Bo Zhang
- School of Labor and Employment Relations, University of Illinois Urbana-Champaign
- Department of Psychology, University of Illinois Urbana-Champaign
| | - Jing Luo
- Feinberg School of Medicine, Northwestern University
| | - Jian Li
- Faculty of Psychology, Beijing Normal University
| |
Collapse
|
4
|
Sun L, Qin Z, Wang S, Tian X, Luo F. Contributions to Constructing Forced-Choice Questionnaires Using the Thurstonian IRT Model. MULTIVARIATE BEHAVIORAL RESEARCH 2024; 59:229-250. [PMID: 37776890 DOI: 10.1080/00273171.2023.2248979] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/02/2023]
Abstract
Forced-choice questionnaires involve presenting items in blocks and asking respondents to provide a full or partial ranking of the items within each block. To prevent involuntary or voluntary response distortions, blocks are usually formed of items that possess similar levels of desirability. Assembling forced-choice blocks is not a trivial process, because in addition to desirability, both the direction and magnitude of relationships between items and the traits being measured (i.e., factor loadings) need to be carefully considered. Based on simulations and empirical studies using item pairs, we provide recommendations on how to construct item pairs matched by desirability. When all pairs contain items keyed in the same direction, score reliability is improved by maximizing within-block loading differences. Higher reliability is obtained when even a small number of pairs consist of unequally keyed items.
Collapse
Affiliation(s)
- Luning Sun
- The Psychometrics Centre, University of Cambridge
| | - Zijie Qin
- Faculty of Psychology, Beijing Normal University
| | - Shan Wang
- Faculty of Psychology, Beijing Normal University
| | - Xuetao Tian
- Faculty of Psychology, Beijing Normal University
| | - Fang Luo
- Faculty of Psychology, Beijing Normal University
| |
Collapse
|
5
|
Frick S, Brown A, Wetzel E. Investigating the Normativity of Trait Estimates from Multidimensional Forced-Choice Data. MULTIVARIATE BEHAVIORAL RESEARCH 2023; 58:1-29. [PMID: 34464217 DOI: 10.1080/00273171.2021.1938960] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The Thurstonian item response model (Thurstonian IRT model) allows deriving normative trait estimates from multidimensional forced-choice (MFC) data. In the MFC format, persons must rank-order items that measure different attributes according to how well the items describe them. This study evaluated the normativity of Thurstonian IRT trait estimates both in a simulation and empirically. The simulation investigated normativity and compared Thurstonian IRT trait estimates to those using classical partially ipsative scoring, from dichotomous true-false (TF) data and rating scale data. The results showed that, with blocks of opposite keyed items, Thurstonian IRT trait estimates were normative in contrast to classical partially ipsative estimates. Unbalanced numbers of items per trait, few opposite keyed items, traits correlated positively or assessing fewer traits did not decrease measurement precision markedly. Measurement precision was lower than that of rating scale data. The empirical study investigated whether relative MFC responses provide a better differentiation of behaviors within persons than absolute TF responses. However, criterion validity was equal and construct validity (with constructs measured by rating scales) lower in MFC. Thus, Thurstonian IRT modeling of MFC data overcomes the drawbacks of classical scoring, but gains in validity may depend on eliminating common method biases from the comparison.
Collapse
Affiliation(s)
- Susanne Frick
- Department of Psychology, School of Social Sciences, University of Mannheim
| | - Anna Brown
- Department of Psychology, University of Kent
| | - Eunike Wetzel
- Department of Psychology, Otto-von-Guericke University Magdeburg
- Department of Psychology, University of Koblenz-Landau
| |
Collapse
|
6
|
Bürkner PC. On the Information Obtainable from Comparative Judgments. PSYCHOMETRIKA 2022; 87:1439-1472. [PMID: 35133553 PMCID: PMC9636126 DOI: 10.1007/s11336-022-09843-z] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/02/2021] [Revised: 11/04/2021] [Indexed: 06/14/2023]
Abstract
Personality tests employing comparative judgments have been proposed as an alternative to Likert-type rating scales. One of the main advantages of a comparative format is that it can reduce faking of responses in high-stakes situations. However, previous research has shown that it is highly difficult to obtain trait score estimates that are both faking resistant and sufficiently accurate for individual-level diagnostic decisions. With the goal of contributing to a solution, I study the information obtainable from comparative judgments analyzed by means of Thurstonian IRT models. First, I extend the mathematical theory of ordinal comparative judgments and corresponding models. Second, I provide optimal test designs for Thurstonian IRT models that maximize the accuracy of people's trait score estimates from both frequentist and Bayesian statistical perspectives. Third, I derive analytic upper bounds for the accuracy of these trait estimates achievable through ordinal Thurstonian IRT models. Fourth, I perform numerical experiments that complement results obtained in earlier simulation studies. The combined analytical and numerical results suggest that it is indeed possible to design personality tests using comparative judgments that yield trait scores estimates sufficiently accurate for individual-level diagnostic decisions, while reducing faking in high-stakes situations. Recommendations for the practical application of comparative judgments for the measurement of personality, specifically in high-stakes situations, are given.
Collapse
|
7
|
Lee H. Reduction of faking with the use of a forced‐choice personality test: Cross‐cultural comparisons between South Korea and the United States. INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT 2022. [DOI: 10.1111/ijsa.12408] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Affiliation(s)
- HyeSun Lee
- Psychology Department California State University Channel Islands Camarillo California USA
| |
Collapse
|
8
|
Bunji K, Okada K. Linear Ballistic Accumulator Item Response Theory Model for Multidimensional Multiple-Alternative Forced-Choice Measurement of Personality. MULTIVARIATE BEHAVIORAL RESEARCH 2022; 57:658-678. [PMID: 33750245 DOI: 10.1080/00273171.2021.1896351] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
There has been a growing interest in psychological measurements that use the multiple-alternative forced-choice (MAFC) response format for its resistance to response biases. Although several models have been proposed for the data obtained from such measurements, none have succeeded in incorporating the response time information. Given that currently, many psychological measurements are performed via computers, it would be beneficial to develop a joint model involving an MAFC item response and response time. The present study proposes the first model that combines a cognitive process model that underlies the observed response time and the forced-choice item response model. Specifically, the proposed model is based on the linear ballistic accumulator model of response time, which is substantially extended by reformulating its parameters so as to incorporate the MAFC item responses. The model parameters are estimated by the Markov chain Monte Carlo (MCMC) algorithm. A simulation study confirmed that the proposed approach could appropriately recover the parameters. Two empirical applications are reported to demonstrate the use of the proposed model and compare it with existing models. The results showed that the proposed model could be a useful tool for jointly modeling the MAFC item responses and response times.
Collapse
Affiliation(s)
- Kyosuke Bunji
- Graduate School of Education, The University of Tokyo
- Japan Society for the Promotion of Science
| | - Kensuke Okada
- Graduate School of Education, The University of Tokyo
| |
Collapse
|
9
|
Frick S. Modeling Faking in the Multidimensional Forced-Choice Format: The Faking Mixture Model. PSYCHOMETRIKA 2022; 87:773-794. [PMID: 34927219 PMCID: PMC9166892 DOI: 10.1007/s11336-021-09818-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/23/2020] [Revised: 08/31/2021] [Accepted: 10/15/2021] [Indexed: 06/14/2023]
Abstract
The multidimensional forced-choice (MFC) format has been proposed to reduce faking because items within blocks can be matched on desirability. However, the desirability of individual items might not transfer to the item blocks. The aim of this paper is to propose a mixture item response theory model for faking in the MFC format that allows to estimate the fakability of MFC blocks, termed the Faking Mixture model. Given current computing capabilities, within-subject data from both high- and low-stakes contexts are needed to estimate the model. A simulation showed good parameter recovery under various conditions. An empirical validation showed that matching was necessary but not sufficient to create an MFC questionnaire that can reduce faking. The Faking Mixture model can be used to reduce fakability during test construction.
Collapse
Affiliation(s)
- Susanne Frick
- Department of Psychology, School of Social Sciences, Mannheim, Germany.
| |
Collapse
|
10
|
Lee P, Joo SH, Zhou S, Son M. Investigating the impact of negatively keyed statements on multidimensional forced-choice personality measures: A comparison of partially ipsative and IRT scoring methods. PERSONALITY AND INDIVIDUAL DIFFERENCES 2022. [DOI: 10.1016/j.paid.2022.111555] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
11
|
Martínez A, Salgado JF. A Meta-Analysis of the Faking Resistance of Forced-Choice Personality Inventories. Front Psychol 2021; 12:732241. [PMID: 34659043 PMCID: PMC8511514 DOI: 10.3389/fpsyg.2021.732241] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2021] [Accepted: 09/01/2021] [Indexed: 11/13/2022] Open
Abstract
This study presents a comprehensive meta-analysis on the faking resistance of forced-choice (FC) inventories. The results showed that (1) FC inventories show resistance to faking behavior; (2) the magnitude of faking is higher in experimental contexts than in real-life selection processes, suggesting that the effects of faking may be, in part, a laboratory phenomenon; and (3) quasi-ipsative FC inventories are more resistant to faking than the other FC formats. Smaller effect sizes were found for conscientiousness when the quasi-ipsative format was used (δ = 0.49 vs. δ = 1.27 for ipsative formats). Also, the effect sizes were smaller for the applicant samples than for the experimental samples. Finally, the contributions and practical implications of these findings are discussed.
Collapse
Affiliation(s)
- Alexandra Martínez
- Department of Political Science and Sociology, Faculty of Labor Relations, University of Santiago de Compostela, Santiago de Compostela, Spain
| | | |
Collapse
|
12
|
Speer AB, Delacruz AY. Introducing a supervised alternative to forced‐choice personality scoring: A test of validity and resistance to faking. INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT 2021. [DOI: 10.1111/ijsa.12345] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Affiliation(s)
- Andrew B. Speer
- Department of Psychology Wayne State University Detroit Michigan USA
| | - Angie Y. Delacruz
- Department of Psychology Wayne State University Detroit Michigan USA
| |
Collapse
|
13
|
Martínez A, Moscoso S, Lado M. Faking Effects on the Factor Structure of a Quasi-Ipsative Forced-Choice Personality Inventory. REVISTA DE PSICOLOGÍA DEL TRABAJO Y DE LAS ORGANIZACIONES 2021. [DOI: 10.5093/jwop2021a7] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
14
|
Otero I, Cuadrado D, Martínez A. Convergent and Predictive Validity of the Big Five Factors Assessed with SingleStimulus and Quasi-Ipsative Questionnaires. REVISTA DE PSICOLOGÍA DEL TRABAJO Y DE LAS ORGANIZACIONES 2020. [DOI: 10.5093/jwop2020a17] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
15
|
Lee P, Joo SH, Stark S. Detecting DIF in Multidimensional Forced Choice Measures Using the Thurstonian Item Response Theory Model. ORGANIZATIONAL RESEARCH METHODS 2020. [DOI: 10.1177/1094428120959822] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Although modern item response theory (IRT) methods of test construction and scoring have overcome ipsativity problems historically associated with multidimensional forced choice (MFC) formats, there has been little research on MFC differential item functioning (DIF) detection, where item refers to a block, or group, of statements presented for an examinee’s consideration. This research investigated DIF detection with three-alternative MFC items based on the Thurstonian IRT (TIRT) model, using omnibus Wald tests on loadings and thresholds. We examined constrained and free baseline model comparisons strategies with different types and magnitudes of DIF, latent trait correlations, sample sizes, and levels of impact in an extensive Monte Carlo study. Results indicated the free baseline strategy was highly effective in detecting DIF, with power approaching 1.0 in the large sample size and large magnitude of DIF conditions, and similar effectiveness in the impact and no-impact conditions. This research also included an empirical example to demonstrate the viability of the best performing method with real examinees and showed how a DIF and a DTF effect size measure can be used to assess the practical significance of MFC DIF findings.
Collapse
|
16
|
Wetzel E, Frick S, Greiff S. The Multidimensional Forced-Choice Format as an Alternative for Rating Scales. EUROPEAN JOURNAL OF PSYCHOLOGICAL ASSESSMENT 2020. [DOI: 10.1027/1015-5759/a000609] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Affiliation(s)
- Eunike Wetzel
- Department of Psychology, Otto-von-Guericke University Magdeburg, Germany
| | - Susanne Frick
- Department of Psychology, University of Mannheim, Germany
| | - Samuel Greiff
- Institute of Cognitive Science and Assessment, University of Luxembourg, Luxembourg
| |
Collapse
|
17
|
Lee H, Smith WZ. Fit Indices for Measurement Invariance Tests in the Thurstonian IRT Model. APPLIED PSYCHOLOGICAL MEASUREMENT 2020; 44:282-295. [PMID: 32536730 PMCID: PMC7262996 DOI: 10.1177/0146621619893785] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
This study examined whether cutoffs in fit indices suggested for traditional formats with maximum likelihood estimators can be utilized to assess model fit and to test measurement invariance when a multiple group confirmatory factor analysis was employed for the Thurstonian item response theory (IRT) model. Regarding the performance of the evaluation criteria, detection of measurement non-invariance and Type I error rates were examined. The impact of measurement non-invariance on estimated scores in the Thurstonian IRT model was also examined through accuracy and efficiency in score estimation. The fit indices used for the evaluation of model fit performed well. Among six cutoffs for changes in model fit indices, only ΔCFI > .01 and ΔNCI > .02 detected metric non-invariance when the medium magnitude of non-invariance occurred and none of the cutoffs performed well to detect scalar non-invariance. Based on the generated sampling distributions of fit index differences, this study suggested ΔCFI > .001 and ΔNCI > .004 for scalar non-invariance and ΔCFI > .007 for metric non-invariance. Considering Type I error rate control and detection rates of measurement non-invariance, ΔCFI was recommended for measurement non-invariance tests for forced-choice format data. Challenges in measurement non-invariance tests in the Thurstonian IRT model were discussed along with the direction for future research to enhance the utility of forced-choice formats in test development for cross-cultural and international settings.
Collapse
Affiliation(s)
- HyeSun Lee
- California State University Channel Islands, Camarillo, USA
| | | |
Collapse
|
18
|
Ng V, Lee P, Ho MHR, Kuykendall L, Stark S, Tay L. The Development and Validation of a Multidimensional Forced-Choice Format Character Measure: Testing the Thurstonian IRT Approach. J Pers Assess 2020; 103:224-237. [DOI: 10.1080/00223891.2020.1739056] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Affiliation(s)
- Vincent Ng
- Department of Psychology, University of Houston, Houston, Texas
| | - Philseok Lee
- Department of Psychology, George Mason University, Fairfax, Virginia
| | - Moon-Ho Ringo Ho
- School of Humanities and Social Sciences, Nanyang Technological University, Singapore
| | - Lauren Kuykendall
- Department of Psychology, George Mason University, Fairfax, Virginia
| | - Stephen Stark
- Department of Psychology, University of South Florida, Tampa, Florida
| | - Louis Tay
- Department of Psychological Sciences, Purdue University, West Lafayette, Indiana
| |
Collapse
|
19
|
Pelt DHM, Van der Linden D, Dunkel CS, Born MP. The Motivation and Opportunity for Socially Desirable Responding Does Not Alter the General Factor of Personality. Assessment 2019; 28:1376-1396. [PMID: 31619053 PMCID: PMC8167912 DOI: 10.1177/1073191119880960] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Socially desirable responding may affect the factor structure of personality questionnaires and may be one of the reasons for the common variance among personality traits. In this study, we test this hypothesis by investigating the influence of the motivational test-taking context (development vs. selection) and the opportunity to distort responses (forced-choice vs. Likert response format) on personality questionnaire scores. Data from real selection and assessment candidates (total N = 3,980) matched on gender, age, and educational level were used. Mean score differences were found between the selection and development groups, with smaller differences for the FC version. Yet, exploratory structural equation models showed that the overall factor structures as well as the general factor were highly similar across the four groups. Thus, although socially desirable responding may affect mean scores on personality traits, it does not appear to affect factor structures. This study further suggests that the common variance in personality questionnaires is consistent and appears to be little influenced by motivational pressures for response distortion.
Collapse
Affiliation(s)
- Dirk H M Pelt
- Erasmus University Rotterdam, Rotterdam, Netherlands.,Ixly, Utrecht, Netherlands
| | | | | | - Marise Ph Born
- Erasmus University Rotterdam, Rotterdam, Netherlands.,North-West University, Vanderbijlpark, South Africa
| |
Collapse
|
20
|
Bürkner PC, Schulte N, Holling H. On the Statistical and Practical Limitations of Thurstonian IRT Models. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT 2019; 79:827-854. [PMID: 31488915 PMCID: PMC6713979 DOI: 10.1177/0013164419832063] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
Forced-choice questionnaires have been proposed to avoid common response biases typically associated with rating scale questionnaires. To overcome ipsativity issues of trait scores obtained from classical scoring approaches of forced-choice items, advanced methods from item response theory (IRT) such as the Thurstonian IRT model have been proposed. For convenient model specification, we introduce the thurstonianIRT R package, which uses Mplus, lavaan, and Stan for model estimation. Based on practical considerations, we establish that items within one block need to be equally keyed to achieve similar social desirability, which is essential for creating forced-choice questionnaires that have the potential to resist faking intentions. According to extensive simulations, measuring up to five traits using blocks of only equally keyed items does not yield sufficiently accurate trait scores and inter-trait correlation estimates, neither for frequentist nor for Bayesian estimation methods. As a result, persons' trait scores remain partially ipsative and, thus, do not allow for valid comparisons between persons. However, we demonstrate that trait scores based on only equally keyed blocks can be improved substantially by measuring a sizable number of traits. More specifically, in our simulations of 30 traits, scores based on only equally keyed blocks were non-ipsative and highly accurate. We conclude that in high-stakes situations where persons are motivated to give fake answers, Thurstonian IRT models should only be applied to tests measuring a sizable number of traits.
Collapse
|
21
|
Morillo D, Abad FJ, Kreitchmann RS, Leenen I, Hontangas P, Ponsoda V. The Journey from Likert to Forced-Choice Questionnaires: Evidence of the Invariance of Item Parameters. REVISTA DE PSICOLOGÍA DEL TRABAJO Y DE LAS ORGANIZACIONES 2019. [DOI: 10.5093/jwop2019a11] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
22
|
Lee P, Joo SH, Lee S. Examining stability of personality profile solutions between Likert-type and multidimensional forced choice measure. PERSONALITY AND INDIVIDUAL DIFFERENCES 2019. [DOI: 10.1016/j.paid.2019.01.022] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
|
23
|
Walton KE, Cherkasova L, Roberts RD. On the Validity of Forced Choice Scores Derived From the Thurstonian Item Response Theory Model. Assessment 2019; 27:706-718. [PMID: 31007043 DOI: 10.1177/1073191119843585] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
Abstract
Forced choice (FC) measures may be a desirable alternative to single stimulus (SS) Likert items, which are easier to fake and can have associated response biases. However, classical methods of scoring FC measures lead to ipsative data, which have a number of psychometric problems. A Thurstonian item response theory (TIRT) model has been introduced as a way to overcome these issues, but few empirical validity studies have been conducted to ensure its effectiveness. This was the goal of the current three studies, which used FC measures of domains from popular personality frameworks including the Big Five and HEXACO, and both statement and adjective item stems. We computed TIRT and ipsative scores and compared their validity estimates. Convergent and discriminant validity of the scores were evaluated by correlating them with SS scores, and test-criterion validity evidence was evaluated by examining their relationships with meaningful outcomes. In all three studies, there was evidence for the convergent and test-criterion validity of the TIRT scores, though at times this was on par with the validity of the ipsative scores. The discriminant validity of the TIRT scores was problematic and was often worse than the ipsative scores.
Collapse
Affiliation(s)
| | | | - Richard D Roberts
- Research and Assessment Design (RAD): Science Solution, Philadelphia, PA, USA
| |
Collapse
|
24
|
Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures. ORGANIZATIONAL RESEARCH METHODS 2019. [DOI: 10.1177/1094428119836486] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
Abstract
Forced-choice (FC) measures are gaining popularity as an alternative assessment format to single-statement (SS) measures. However, a fundamental question remains to be answered: Do FC and SS instruments measure the same underlying constructs? In addition, FC measures are theorized to be more cognitively challenging, so how would this feature influence respondents’ reactions to FC measures compared to SS? We used both between- and within-subjects designs to examine the equivalence of the FC format and the SS format. As the results illustrate, FC measures scored by the multi-unidimensional pairwise preference (MUPP) model and SS measures scored with the generalized graded unfolding model (GGUM) showed strong equivalence. Specifically, both formats demonstrated similar marginal reliabilities and test-retest reliabilities, high convergent validities, good discriminant validities, and similar criterion-related validities with theoretically relevant criteria. In addition, the formats had little differential impact on respondents’ general emotional and cognitive reactions except that the FC format was perceived to be slightly more difficult and more time-saving.
Collapse
|