Dimensionality assessment in the presence of wording effects: A network psychometric and factorial approach

This study proposes a procedure for substantive dimensionality estimation in the presence of wording effects, the inconsistent response to regular and reversed self-report items. The procedure developed consists of subtracting an approximate estimate of the wording effects variance from the sample correlation matrix and then estimating the substantive dimensionality on the residual correlation matrix. This is achieved by estimating a random intercept factor with unit loadings for all the regular and unrecoded reversed items. The accuracy of the procedure was evaluated through an extensive simulation study that manipulated nine relevant variables and employed the exploratory graph analysis (EGA) and parallel analysis (PA) retention methods. The results indicated that combining the proposed procedure with EGA or PA achieved high accuracy in estimating the substantive latent dimensionality, but that EGA was superior. Additionally, the present findings shed light on the complex ways that wording effects impact the dimensionality estimates when the response bias in the data is ignored. A tutorial on substantive dimensionality estimation with the R package EGAnet is offered, as well as practical guidelines for applied researchers.

Review of the psychometric properties of the Patient Health Questionnaire-9 (PHQ-9) Spanish version in a sample of Puerto Rican workers

Background

This study aimed to examine the internal structure and assess the psychometric properties of the Patient Health Questionnaire (PHQ-9) in a Puerto Rican sample of workers. This instrument is a nine-item questionnaire, which was conceptualized as a unidimensional structure; however, there are mixed results regarding this internal structure. This measure is used in the occupational health psychology context in organizations in Puerto Rico; nevertheless, there is little evidence of its psychometric properties with samples of workers.

Materials and methods

A total of 955 samples from two different study samples were used in this cross-sectional study design in which the PHQ-9 was used. We conducted confirmatory factor analysis, bifactor analysis, and random intercept item factor analysis to examine the internal structure of the PHQ-9. Moreover, a two-factor model was examined by randomly assigning items to the two factors. Measurement invariance across sex and the relationship with other constructs were examined.

Results

The best-fitted model was the bifactor model followed by the random intercept item factor. The five sets of two-factor models with items randomly assigned obtained acceptable and similar fit indices regardless of the items.

Conclusion

The results suggest that the PHQ-9 appears to be a reliable and valid instrument to measure depression. The more parsimonious interpretation of its scores, for now, is a unidimensional structure. Comparison across sex appears to be useful in occupational health psychology research settings since the results suggest that the PHQ-9 is invariant regarding this variable.

Response tendencies due to item wording using eye-tracking methodology accounting for individual differences and item characteristics

The discrepancy of the scores on responses to negatively and positively worded items has led to hypotheses of inattention, confusion, difficulty, and differential processing of negatively worded items. The present study, utilizing eye-tracking methodology, aimed to fill an explanatory gap regarding response behavior, providing observations on the item-level response process. It experimentally examined characteristics of the items (wording type, self-relevance) and characteristics of the respondents (neuroticism, verbal abilities, and mood) for their impact on response outcomes. A sample of 87 university students completed a computerized version of a questionnaire with items presented in four alternative wording types: positive, negative, negated positive, and negated negative; half of the items referred to attitudes toward the self and the other half to attitudes toward others. Participants' eye movements during item completion were recorded with the Gazepoint-GP3-HD desk-mounted eye tracker. In linear mixed effects models, wording type and self-relevance were found to relate to response time, time of viewing and revisits to the body of the items and the response options, indicating that there were effects at the stages of comprehension and selection of response. Neuroticism was associated with differential item responses, suggesting a role in later levels of the response process, the retrieval, judgment, and response selection stages. Eye-tracking measures can enhance the examination of response tendencies with regards to item content, item wording, and person characteristics.

Using Constrained Factor Mixture Analysis to Validate Mixed-Worded Psychological Scales: The Case of the Rosenberg Self-Esteem Scale in the Dominican Republic

A common method to collect information in the behavioral and health sciences is the self-report. However, the validity of self-reports is frequently threatened by response biases, particularly those associated with inconsistent responses to positively and negatively worded items of the same dimension, known as wording effects. Modeling strategies based on confirmatory factor analysis have traditionally been used to account for this response bias, but they have recently become under scrutiny due to their incorrect assumption of population homogeneity, inability to recover uncontaminated person scores or preserve structural validities, and their inherent ambiguity. Recently, two constrained factor mixture analysis (FMA) models have been proposed by Arias et al. (2020) and Steinmann et al. (2021) that can be used to identify and screen inconsistent response profiles. While these methods have shown promise, tests of their performance have been limited and they have not been directly compared. Thus the objective of the current study was to assess and compare their performance with data from the Dominican Republic of the Rosenberg Self-Esteem Scale (N = 632). Additionally, as this scale had not yet been studied for this population, another objective was to show how using constrained FMAs could help in the validation of mixed-worded scales. The results indicated that removing the inconsistent respondents identified by both FMAs (≈8%) reduced the amount of wording effects in the database. However, whereas the Steinmann et al. method only cleaned the data partially, the Arias et al. (2020) method was able to remove the great majority of the wording effects variance. Based on the screened data with the Arias et al. method, we evaluated the psychometric properties of the RSES for the Dominican population, and the results indicated that the scores had good validity and reliability properties. Given these findings, we recommend that researchers incorporate constrained FMAs into their toolbox and consider using them to screen out inconsistent respondents to mixed-worded scales.

Modeling Wording Effects Does Not Help in Recovering Uncontaminated Person Scores: A Systematic Evaluation With Random Intercept Item Factor Analysis

The item wording (or keying) effect consists of logically inconsistent answers to positively and negatively worded items that tap into similar (but polarly opposite) content. Previous research has shown that this effect can be successfully modeled through the random intercept item factor analysis (RIIFA) model, as evidenced by the improvements in the model fit in comparison to models that only contain substantive factors. However, little is known regarding the capability of this model in recovering the uncontaminated person scores. To address this issue, the study analyzes the performance of the RIIFA approach across three types of wording effects proposed in the literature: carelessness, item verification difficulty, and acquiescence. In the context of unidimensional substantive models, four independent variables were manipulated, using Monte Carlo methods: type of wording effect, amount of wording effect, sample size, and test length. The results corroborated previous findings by showing that the RIIFA models were consistently able to account for the variance in the data, attaining an excellent fit regardless of the amount of bias. Conversely, the models without the RIIFA factor produced increasingly a poorer fit with greater amounts of wording effects. Surprisingly, however, the RIIFA models were not able to better estimate the uncontaminated person scores for any type of wording effect in comparison to the substantive unidimensional models. The simulation results were then corroborated with an empirical dataset, examining the relationship between learning strategies and personality with grade point average in undergraduate studies. The apparently paradoxical findings regarding the model fit and the recovery of the person scores are explained, considering the properties of the factor models examined.

Revised Short Screening Version of the Profile of Mood States (POMS) From the German General Population

The present study was conducted with the aim of constructing and validating a short form of the Profile of Mood States (POMS). The POMS is a widely-applied measure for the assessment of an individual's mood. Thus, it is of great relevance for many research questions in clinical and social psychology. To develop the short scale, we first examined psychometric properties and found the optimal 16-item solution among all valid combinations of the full POMS in an exploratory subsample (n = 1,029) of our complete representative sample of the German general population. We then validated this model in a confirmatory subsample (n = 977). Additionally, we examined its invariance across age groups and sex, as well as its reliability. Our results indicate that the POMS-16 is a valid and reliable measure of mood states with minimal losses compared to the 35-item version. Particularly where brevity and an economical assessment is desired, the POMS-16 should be considered.

A reanalysis of the center for epidemiological studies depression scale (CES-D) using non-parametric item response theory

The "Center for Epidemiological Studies Depression Scale" (CES-D; Radloff, 1977) is a questionnaire used world-wide to measure depressive symptoms. Although the original four-factor-structure has been widely accepted and replicated, some studies point to other factor-structures like a one- and two-factor-structure. The goal of the current study was to evaluate the factor structure of the CES-D (one-, two- and four-factor-structure), which was found using classical test theory (CTT), with two non-parametric item-response-theory-models (Mokken-Scaling; Monotone-homogeneity-model; MHM and Double-monotonicity-model; DMM). To this end, a representative German sample was analyzed (N = 2,507). Regarding the one-factor-model, neither the MHM nor the DMM were supported; the two-factor-model and the four-factor-model, however, satisfied the assumptions of the MHM but not of the stricter DMM. Sum scores therefore constitute ordinal scales for the two-factor and four-factor-structure. This justifies, for example, the use of percentile ranks or medians and associated statistical techniques. Especially the use of the 16 negative formulated items seems appealing. Further studies should explore, whether the use of these 16 instead of the original 20 items is of advantage regarding the practical validity of the CES-D (e.g. higher sensitivity and specificity in screening applications and better temporal stability).