Validity of Three IRT Models for Measuring and Controlling Extreme and Midpoint Response Styles

Correcting for Extreme Response Style: Model Choice Matters

Extreme response style (ERS), the tendency of participants to select extreme item categories regardless of the item content, has frequently been found to decrease the validity of Likert-type questionnaire results. For this reason, various item response theory (IRT) models have been proposed to model ERS and correct for it. Comparisons of these models are however rare in the literature, especially in the context of cross-cultural comparisons, where ERS is even more relevant due to cultural differences between groups. To remedy this issue, the current article examines two frequently used IRT models that can be estimated using standard software: a multidimensional nominal response model (MNRM) and a IRTree model. Studying conceptual differences between these models reveals that they differ substantially in their conceptualization of ERS. These differences result in different category probabilities between the models. To evaluate the impact of these differences in a multigroup context, a simulation study is conducted. Our results show that when the groups differ in their average ERS, the IRTree model and MNRM can drastically differ in their conclusions about the size and presence of differences in the substantive trait between these groups. An empirical example is given and implications for the future use of both models and the conceptualization of ERS are discussed.

Studies in the Mentality of Literates: 3. Conceptual Structure and Nonsense of Personality Testing

In this study, third in a series of studies of the relationships between the dominant type of the Word Meaning Structure (WMS) and various psychic processes, response patterns on personality questionnaires with Likert-type response format of individuals with different levels of education (including adult illiterates) in Brazil (N = 102) and in Estonia (N = 520) were assessed with person oriented methods of data analysis. We found that responses to two personality questionnaires (International Personality Item Pool Questionnaire, IPIP-Q60 and Estonian Collectivism Scale, ESTCOL) are inconsistent and do not correspond to theories that underlie construction and interpretation of such assessment tools. Two novel ways to assess inconsistent response patterns were developed. The Consistency Index (CI) characterizes between-item inconsistency and the Determinacy Index (DI) characterizes within-item inconsistency. The dominant type of the WMS and the level of education were related to both CI and DI. Higher level of between-item inconsistency characterizes everyday conceptual thinkers with lower levels of education and higher level of within-item inconsistency was observed among logical conceptual thinkers with higher levels of education. Systematic relationships between WMS and inconsistent patterns of responses indicate that responses on personality questionnaires cannot be interpreted in terms of personality characteristics. The results of our study also provide further support to the idea that dominant type of the WMS is a pervasive characteristic of the psyche and determines qualitatively possibilities and limits of the psychic processes. The results of this study are in agreement with the idea that WMS defines the "Great Divide."

The Impact of Different Methods to Correct for Response Styles on the External Validity of Self-Reports

Abstract. Response styles (RSs) such as acquiescence represent systematic respondent behaviors in self-report questionnaires beyond the actual item content. They distort trait estimates and contribute to measurement bias in questionnaire-based research. Although various approaches were proposed to correct the influence of RSs, little is known about their relative performance. Because different correction methods formalize the latent traits differently, it is unclear how model choice affects the external validity of the corrected measures. Therefore, the present study on N = 1,000 Dutch respondents investigated the impact of correcting responses to measures of self-esteem and the need for cognition using structural equation models with structured residuals, multidimensional generalized partial credit models, and multinomial processing trees. The study considered three RSs: extreme, midpoint, and acquiescence RS. The results showed homogeneous correlation patterns among the modeled latent and external variables, especially if they were not themselves subject to RSs. In that case, the IRT-based models, including an uncorrected model, still yielded consistent results. Nevertheless, the strength of the effect sizes showed variation.

Modeling Changes in Response Style with Longitudinal IRTree Models

Traditional psychometric modeling focuses on observed categorical item responses, which can over-simplify the respondent cognitive response process. A further weakness is that analysis of ordinal responses has been primarily limited to a single substantive trait at one time point. We propose a significant expansion of this modeling framework to account for complex response processes across multiple waves of data collection using the beneficial item response tree framework. This study proposes a novel model, the longitudinal IRTree, for response processes in longitudinal studies, and investigates whether the response style changes are proportional to changes in the substantive trait of interest. A simulation study demonstrates adequate item parameter recovery in a Bayesian framework, especially with larger sample sizes of 2000. The longitudinal change parameters were recovered similarly well, with improved recovery using informative priors over default priors in Mplus. The empirical application demonstrates that relatively stable observed scores are due to a decrease in response styles offsetting an increase in the latent trait of interest.

The Relationship of Insufficient Effort Responding and Response Styles: An Online Experiment

While self-report data is a staple of modern psychological studies, they rely on participants accurately self-reporting. Two constructs that impede accurate results are insufficient effort responding (IER) and response styles. These constructs share conceptual underpinnings and both utilized to reduce cognitive effort when responding to self-report scales. Little research has extensively explored the relationship of the two constructs. The current study explored the relationship of the two constructs across even-point and odd-point scales, as well as before and after data cleaning procedures. We utilized IRTrees, a statistical method for modeling response styles, to examine the relationship between IER and response styles. To capture the wide range of IER metrics available, we employed several forms of IER assessment in our analyses and generated IER factors based on the type of IER being detected. Our results indicated an overall modest relationship between IER and response styles, which varied depending on the type of IER metric being considered or type of scale being evaluated. As expected, data cleaning also changed the relationships of some of the variables. We posit the difference between the constructs may be the degree of cognitive effort participants are willing to expend. Future research and applications are discussed.