Controlling the effect of stimulus context change on attitude statements using michell's binary tree procedure

Ordering motivation and Likert scale ratings: When a numeric scale is not necessarily better

Measuring psychological attributes, such as motivation, typically involves rating scales, assuming that an attribute can be ordered, and that ratings represent this order. Previously, only the first assumption had been tested, albeit limited. First, we checked the ordinal structure of motivation, looking at whether people can establish transitive relations between motivation levels in pairwise comparisons; and we found different ordering patterns: strict transitive, weak transitive, changing order, and intransitivity. The rate of intransitivity was similar to that found previously and was somewhat higher than we obtained when we asked participants to compare definitely quantitative attributes (such as weight). Second, we checked if specific ordering patterns were related to individual interpretations of the statements that deviated from expected motivation types. Indeed, about a third of participants miscategorized statements, and these deviant interpretations were related to intransitivity as well as weak transitivity. Third, we checked whether Likert ratings represent the order of motives obtained from pairwise comparisons. We found rather homomorphic representation: ratings correlated with the order, but they did not differentiate between different ordering patterns and hierarchies of motives. We conclude that the Likert rating scale provides less information about respondents than pairwise ordering. The findings question the mainstream practice of using rating scales without testing underlying assumptions.

Item Response Models, Pathological Science and the Shape of Error

There is nothing in Borsboom and Mellenbergh’s (2004) response that refutes my thesis that psychometrics is a pathology of science. They seek to defend item response models from my charge of pathological science without apparently realizing that my charge relates to psychometricians, not to models. They appeal to the Quine-Duhem thesis in an attempt to argue that item response models do not allow the hypothesis that psychological attributes are quantitative to be tested in isolation, but their argument is based upon a misinterpretation of Duhem. In any experiment, what is being tested depends on what the experimenter already takes to be true, and it is possible that a psychometrician could be testing just one of the hypotheses constituting an item response model. Furthermore, using the theory of conjoint measurement, it is possible to isolate predictions that depend upon psychological attributes being quantitative, as opposed to merely ordinal. Despite this, Borsboom and Mellenbergh agree with the first part of my thesis. They do not discuss the second part, but an examination of textbooks on item response models shows that psychometricians disguise their failure to test the hypothesis that psychological attributes are quantitative by simply declining to mention that this hypothesis is presumed in their models. Claims to measure psychological attributes based upon these models depend exclusively upon the weakest part of these models: the hypothesis that the distribution of ‘errors’ takes a specific form.

The Rasch paradox, conjoint measurement, and psychometrics: Response to Humphry and Sijtsma

This response to Humphry (2013) and Sijtsma (2012) is confined to three issues: the nature of the Rasch paradox, the relevance of the theory of conjoint measurement to psychometrics, and the relationship between test items on the one hand and the character of the attributes they assess on the other. First, contrary to Sijtsma’s view, it is argued that the Rasch model does involve a genuine paradox. Second, typical of psychometricians generally, both Humphry and Sijtsma misunderstand the role the theory of conjoint measurement is able to play in psychometrics. It is argued that in conjunction with item response theory models, it has a significant role. Finally, complementary to Humphry’s and Sijtsma’s insistence upon the importance of theories of the attribute, I argue that features of attribute structure can be inferred from the character of test items and briefly sketch an argument that in the first instance at least, the attributes tests assess contain a feature logically incompatible with quantitative structure.

Descriptive theories of behaviour may allow for the scientific measurement of psychological attributes

Trendler (2009) argued that psychological attributes cannot be measured, as the experimental manipulation and control necessary for the application of measurement theory cannot be achieved. It is argued that Trendler’s conclusion ignored deeper issues. The scientific measurement of psychological attributes depends not only upon adequate stimulus control, but also upon descriptive theories of psychological systems and the demonstration of pure differences in degree (magnitude) within attributes hypothesized to be quantitative. For some classes of stimuli, where descriptive theories of the response process exist and where quantitative features in the stimuli themselves can be empirically manipulated, the demonstration of pure differences in degree is plausible and the scientific measurement of the relevant attributes credible. Where attribute differences between stimuli have identified qualitative causes, stimuli cannot be engineered to produce equivalent magnitudes of the relevant attribute. Here is where Trendler’s Millean Quantity Objection has force.