A study of polytomous IRT methods and item wording directionality effects on perceived stress items

Modeling insufficient effort responses in mixed-worded scales

Surveys often add reverse-coded questions to monitor respondents with insufficient effort responses (IERs) but often wrongly assume that all respondents consistently answer all questions with full effort. By contrast, this study expanded the mixture model for IERs and ran a simulation via LatentGOLD to show the harmful consequences of ignoring IERs to positively and negatively worded questions: less test reliability, bias and less accuracy in slope and intercept parameters. We showed its practical application to two public data sets: Machiavellianism (five-point scale) and self-reported depression (four-point scale).

Influence of respondents' Differentiation of subjective response on water knowledge stock test scale: Evaluation based on two-parameter-multidimensional IRT model

Insufficient awareness of water issues is a crucial bottleneck restricting the sustainable utilization of water resources. To accurately measure citizens' water knowledge stock and overcome the differences between scales and respondents' characteristic levels on test results, the research focuses on developing and evaluating water knowledge stock test scales. The mechanism for identifying indicators is designed based on the grounded theory, and as a result, the water knowledge stock test indicator system is derived. The data was collected by the form of survey questionnaire developed with the test indicator system. A two-parameter multidimensional item response theoretical model is constructed based on item parameter estimation, data model fitting, and item information function. The survey data and optimization model are used to optimize the water knowledge stock test scale and verify the fitting degree with the characteristics of the respondents. The test information function and standard error function indicate that the scale is most informative for individuals with characteristic levels ranging from -2 to 3, resulting in a highly reliable test effect. The research has established a measurement indicator system, methodology, and presented results that serve as a foundation for measuring the stock of water knowledge.

Comparing the Psychometric Properties of a Scale Across Three Likert and Three Alternative Formats: An Application to the Rosenberg Self-Esteem Scale

Zhang and Savalei proposed an alternative scale format to the Likert format, called the Expanded format. In this format, response options are presented in complete sentences, which can reduce acquiescence bias and method effects. The goal of the current study was to compare the psychometric properties of the Rosenberg Self-Esteem Scale (RSES) in the Expanded format and in two other alternative formats, relative to several versions of the traditional Likert format. We conducted two studies to compare the psychometric properties of the RSES across the different formats. We found that compared with the Likert format, the alternative formats tend to have a unidimensional factor structure, less response inconsistency, and comparable validity. In addition, we found that the Expanded format resulted in the best factor structure among the three alternative formats. Researchers should consider the Expanded format, especially when creating short psychological scales such as the RSES.

Relationship quality among dating adolescents: Development and validation of the Relationship Quality Inventory for Adolescents

Relationship quality has implications for individuals’ and couples’ wellbeing, such as higher couple functioning and perceived quality of life. In adolescence, low relationship quality has been associated with poor mental health and relational outcomes. However, given the lack of instruments to assess satisfaction in dating relationships, most studies have relied on measures of marital satisfaction. The current study aimed to address this gap by elaborating and validating the Relationship Quality Inventory for Adolescents (RQI-A). Exploratory and confirmatory factor analyses were conducted among two samples of French-speaking dating adolescents (n₁ = 310; n₂ = 335). The two-factor structure (Connectedness and Commitment) was cross-validated, and dimensions showed high internal consistency coefficients (ω = 0.86–0.89). Results also provide evidence of convergent validity of the scale with related measures. The RQI-A may help study predictors and correlates of dating relationship quality.

Mindful acceptance, not awareness, associated with lower food susceptibility

PURPOSE

Food susceptibility refers to an individual's thoughts, feelings, and motivations when highly palatable foods are available. Mindfulness, or the practice of paying attention, non-judgmentally, in the present moment, is a key element in acceptance-based programs, which have been shown to benefit those with high food susceptibility. This study examined the relationship between food susceptibility and (1) trait mindfulness and (2) mindfulness facets (i.e., awareness, acceptance) in daily life.

METHODS

Participants were 108 adults with overweight/obesity (45.56 ± 11.41 years old, 75.9% white, 72.2% female) enrolled in a weight loss trial (Clinical Trials.gov Identifier: NCT02786238). Food susceptibility was measured with the Power of Food Scale (PFS). Mindfulness was assessed using the Philadelphia Mindfulness Scale (PHMS) and its two subscales: PHMS-Awareness and PHMS-Acceptance. Two regressions examined the associations of (1) total PHMS on PFS, and (2) simultaneous PHMS subscales on PFS. Covariates were age, sex, race, and education.

RESULTS

Regression results revealed, after adjustment for covariates, that Total PHMS was significantly negatively associated with PFS scores (β = - 0.258, p = 0.001), but only one of the PHMS subscales, Acceptance, was significantly associated with PFS scores (β = - 0.328, p < 0.001). PHMS-Awareness was not related to PFS scores.

CONCLUSION

Greater levels of mindfulness were associated with lower food susceptibility in treatment-seeking adults with overweight/obesity. Mindful acceptance may be the driving factor in this relationship, suggesting that awareness alone is not sufficient for promoting healthier appetite regulation. Interventions aimed to reduce food susceptibility and improve coping with cravings may benefit from an enhanced focus on teaching mindful-acceptance skills.

LEVEL OF EVIDENCE

Level III, observational cohort study.

On the Relationship Between Item Stem Formulation and Criterion Validity of Multiple-Component Measuring Instruments

The possible dependency of criterion validity on item formulation in a multicomponent measuring instrument is examined. The discussion is concerned with evaluation of the differences in criterion validity between two or more groups (populations/subpopulations) that have been administered instruments with items having differently formulated item stems. The case of complex item stems involving two stimuli description sentences (double-barreled questions) is thereby compared with the setting where items contained a single sentence. Using empirical data, the latent criterion validity differences are evaluated across three groups that are randomly assigned to conditions characterized by item stems with differing number of stimuli. The results indicate that validity of an instrument can be influenced by the specific way item stem is formulated. Implications for empirical educational, behavioral, and social science research are discussed.

The Impact of Item Feature and Response Preference in a Mixed-Format Design

A combination of positively and negatively worded items (termed a mixed-format design) has been widely adopted in personality and attitude assessments. While advocates claim that the inclusion of positively and negatively worded items will encourage respondents to process the items more carefully and avoid response preference, others have reported that negatively worded (NW) items may induce a nuisance factor and contaminate scale scores. The present study examined the extent of the impact of the NW-item feature and further investigated whether a mixed-format design could effectively control acquiescence and the preference for extreme response options using two datasets (Attitude toward Peace Walls, and International Personality Item Pool). A proposed multidimensional item response model was implemented to simultaneously estimate the impact of item feature and response preference. The results suggested that NW items induced an impact on item responses and that affirmative preference was negligible, regardless of the proportion of NW items in a scale. However, participants' extremity preference was large in both balanced and imbalanced mixed-format designs. It concludes that the impact of the NW-item feature is not negligible in a mixed-format scale, which exhibits good control of acquiescence but not extremity preference.

To Reverse Item Orientation or Not to Reverse Item Orientation, That Is the Question

To investigate the effect of using negatively oriented items, we wrote semantic reversals of the items in the Rosenberg Self Esteem Scale, UCLA Loneliness Scale, and the General Belongingness Scale and used them to create four experimental conditions. Participants (N = 2,019) were recruited through Amazon's Mechanical Turk. Data were assessed for dimensionality, item functioning, instrument properties, and associations with other variables. Regarding dimensionality, although a two-factor model (positively vs. negatively oriented factors) exhibits better fit than a unidimensional model across all conditions, bifactor indices were used to argue that a unidimensional interpretation of the data can be employed. With respect to item functioning, factor loadings were found to be nearly invariant across conditions, but thresholds were not. Concerning instrument properties, inclusion of negatively oriented items results in lower mean scores and higher score variances. Instruments with both positively and negatively oriented items demonstrated lower reliability estimates than those with only one orientation. For associations with other variables, path coefficients in a model where loneliness mediates the effects of belongingness on life satisfaction and self-esteem were found to vary across conditions. Findings suggest that negatively oriented items have minor impact on instrument quality, but influence measurement model and path coefficients.