Measuring individuals' response quality in self-administered psychological tests: an introduction to Gendre's functional method

Impact of social desirability bias on the CoPsoQ-ISTAS 21 psychosocial risk assessment questionnaire

BACKGROUND

Self-reported test is one of the main psychosocial risk assessment tools. However, this test it is susceptible to certain sources of error, including social desirability. Since psychosocial risks are emerging, there are not many studies on their assessment.

OBJECTIVE

The aim of this work is to analyze the impact of social desirability on the short version of the CopSoq-ISTAS 21 assessment tool.

METHOD

A total of 563 workers (45.10% women and 54.90% men) participated in this study. The short version of the CoPsoQ-Istas21 questionnaire with four Likert scale questions as markers, which correspond to the Eysenck Personality Lie Scale Questionnaire Revised (EPQ-r), were used. The sample was divided into two halves, and both a confirmatory analysis and an exploratory analysis were carried out to find out the factorial structure of the scale and, with it, apply the bias filtering method.

RESULTS

The results indicate that 10% of the scale is biased due to social desirability, and that there are significant differences between the group with bias clean scores and the group with scores without bias control.

CONCLUSIONS

The effects of social desirability on the scale are verified, so it is concluded that in a psychosocial risk assessment is not enough to apply a self-report test and interpret its results, being necessary to minimize the sources of error.

Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices

With the development of online data collection and instruments such as Amazon's Mechanical Turk (MTurk), the appearance of malicious software that generates responses to surveys in order to earn money represents a major issue, for both economic and scientific reasons. Indeed, even if paying one respondent to complete one questionnaire represents a very small cost, the multiplication of botnets providing invalid response sets may ultimately reduce study validity while increasing research costs. Several techniques have been proposed thus far to detect problematic human response sets, but little research has been undertaken to test the extent to which they actually detect nonhuman response sets. Thus, we proposed to conduct an empirical comparison of these indices. Assuming that most botnet programs are based on random uniform distributions of responses, we present and compare seven indices in this study to detect nonhuman response sets. A sample of 1,967 human respondents was mixed with different percentages (i.e., from 5% to 50%) of simulated random response sets. Three of the seven indices (i.e., response coherence, Mahalanobis distance, and person-total correlation) appear to be the best estimators for detecting nonhuman response sets. Given that two of those indices-Mahalanobis distance and person-total correlation-are calculated easily, every researcher working with online questionnaires could use them to screen for the presence of such invalid data.

All metrics are equal, but some metrics are more equal than others: A systematic search and review on the use of the term 'metric'

OBJECTIVE

To examine the use of the term 'metric' in health and social sciences' literature, focusing on the interval scale implication of the term in Modern Test Theory (MTT).

MATERIALS AND METHODS

A systematic search and review on MTT studies including 'metric' or 'interval scale' was performed in the health and social sciences literature. The search was restricted to 2001-2005 and 2011-2015. A Text Mining algorithm was employed to operationalize the eligibility criteria and to explore the uses of 'metric'. The paradigm of each included article (Rasch Measurement Theory (RMT), Item Response Theory (IRT) or both), as well as its type (Theoretical, Methodological, Teaching, Application, Miscellaneous) were determined. An inductive thematic analysis on the first three types was performed.

RESULTS

70.6% of the 1337 included articles were allocated to RMT, and 68.4% were application papers. Among the number of uses of 'metric', it was predominantly a synonym of 'scale'; as adjective, it referred to measurement or quantification. Three incompatible themes 'only RMT/all MTT/no MTT models can provide interval measures' were identified, but 'interval scale' was considerably more mentioned in RMT than in IRT.

CONCLUSION

'Metric' is used in many different ways, and there is no consensus on which MTT metric has interval scale properties. Nevertheless, when using the term 'metric', the authors should specify the level of the metric being used (ordinal, ordered, interval, ratio), and justify why according to them the metric is at that level.

Psychiatric symptoms and response quality to self-rated personality tests: Evidence from the PsyCoLaus study

Despite the fact that research has demonstrated consistent associations between self-rated measures of personality dimensions and mental disorders, little has been undertaken to investigate the relation between psychiatric symptoms and response patterns to self-rated tests. The aim of this study was to investigate the association between psychiatric symptoms and response quality using indices from our functional method. A sample of 1,784 participants from a Swiss population-based cohort completed a personality inventory (NEO-FFI) and a symptom checklist of 90 items (SCL-90-R). Different indices of response quality were calculated based on the responses given to the NEO-FFI. Associations among the responses to indices of response quality, sociodemographic characteristics and the SCL-90-R dimensions were then established. Psychiatric symptoms were associated with several important differences in response quality, questioning subjects' ability to provide valid information using self-rated instruments. As suggested by authors, psychiatric symptoms seem associated with differences in personality scores. Nonetheless, our study shows that symptoms are also related to differences in terms of response patterns as sources of differences in personality scores. This could constitute a bias for clinical assessment. Future studies could still determine whether certain subpopulations of subjects are more unable to provide valid information to self-rated questionnaires than others.

Do bipolar subjects' responses to personality questionnaires lack reliability? Evidence from the PsyCoLaus study

Differences in personality scores between subjects with and without mood disorders might result from response biases rather than specific personality traits per se. The aim of this study was to compare subjects with bipolar disorders (BPD) to non-bipolar subjects in terms of response quality to the NEO-FFI. Using data from the population-based cohort study PsyCoLaus, subjects were compared in terms of responses to the NEO-FFI, and indices of response quality were calculated. Hierarchical regression analyses were performed and controlled for sociodemographic factors, depressive episodes, dysthymia, anxiety disorders and substance use disorders. Consistent with the literature, subjects with BPD had higher scores in neuroticism and openness, and lower scores in conscientiousness. However, significant differences were measured for response reliability and validity. In particular, the indices of response quality including response reliability were lower in subjects with BPD suggesting that bipolar subjects might have more difficulty in providing consistent answers throughout questionnaires. However, regression models resulted in small associations between mania/hypomania and response quality, and showed that differences in response quality were mainly attributable to correlates of BPD instead of the presence of mania/hypomania itself. The current findings suggest that bipolar subjects' responses to questionnaires are biased, making them less reliable.