Ordering motivation and Likert scale ratings: When a numeric scale is not necessarily better

Measuring psychological attributes, such as motivation, typically involves rating scales, assuming that an attribute can be ordered, and that ratings represent this order. Previously, only the first assumption had been tested, albeit limited. First, we checked the ordinal structure of motivation, looking at whether people can establish transitive relations between motivation levels in pairwise comparisons; and we found different ordering patterns: strict transitive, weak transitive, changing order, and intransitivity. The rate of intransitivity was similar to that found previously and was somewhat higher than we obtained when we asked participants to compare definitely quantitative attributes (such as weight). Second, we checked if specific ordering patterns were related to individual interpretations of the statements that deviated from expected motivation types. Indeed, about a third of participants miscategorized statements, and these deviant interpretations were related to intransitivity as well as weak transitivity. Third, we checked whether Likert ratings represent the order of motives obtained from pairwise comparisons. We found rather homomorphic representation: ratings correlated with the order, but they did not differentiate between different ordering patterns and hierarchies of motives. We conclude that the Likert rating scale provides less information about respondents than pairwise ordering. The findings question the mainstream practice of using rating scales without testing underlying assumptions.

“Are psychological attributes quantitative?” is not an empirical question: Conceptual confusions in the measurement debate

Critics of psychological measurement have accused quantitative psychologists of ignoring the empirical hypothesis that psychological phenomena are quantitative (Michell), or have claimed that it is impossible in principle to find out whether psychological phenomena are actually quantitative (Trendler). By drawing on Bennett and Hacker (2003), I argue that both criticisms do not go far enough because they sidestep the fundamental conceptual problem of the measurement debate: It is impossible to give concrete formulations of the question “Are psychological attributes quantitative?” without transgressing the boundaries of meaningful language. Conceptual confusions and questionable philosophical assumptions have contributed to the misguided idea that the quantity of psychological phenomena must or can be demonstrated empirically. First, the measurement debate is characterized by misleading examples and ambiguous terminology. Second, the idea of psychological measurement is inherently Cartesian. In summary, psychological measurement is even more problematic than Michell and Trendler have argued.

All metrics are equal, but some metrics are more equal than others: A systematic search and review on the use of the term 'metric'

OBJECTIVE

To examine the use of the term 'metric' in health and social sciences' literature, focusing on the interval scale implication of the term in Modern Test Theory (MTT).

MATERIALS AND METHODS

A systematic search and review on MTT studies including 'metric' or 'interval scale' was performed in the health and social sciences literature. The search was restricted to 2001-2005 and 2011-2015. A Text Mining algorithm was employed to operationalize the eligibility criteria and to explore the uses of 'metric'. The paradigm of each included article (Rasch Measurement Theory (RMT), Item Response Theory (IRT) or both), as well as its type (Theoretical, Methodological, Teaching, Application, Miscellaneous) were determined. An inductive thematic analysis on the first three types was performed.

RESULTS

70.6% of the 1337 included articles were allocated to RMT, and 68.4% were application papers. Among the number of uses of 'metric', it was predominantly a synonym of 'scale'; as adjective, it referred to measurement or quantification. Three incompatible themes 'only RMT/all MTT/no MTT models can provide interval measures' were identified, but 'interval scale' was considerably more mentioned in RMT than in IRT.

CONCLUSION

'Metric' is used in many different ways, and there is no consensus on which MTT metric has interval scale properties. Nevertheless, when using the term 'metric', the authors should specify the level of the metric being used (ordinal, ordered, interval, ratio), and justify why according to them the metric is at that level.

Heating up the measurement debate: What psychologists can learn from the history of physics

Discussions of psychological measurement are largely disconnected from issues of measurement in the natural sciences. We show that there are interesting parallels and connections between the two, by focusing on a real and detailed example (temperature) from the history of science. More specifically, our novel approach is to study the issue of validity based on the history of measurement in physics, which will lead to three concrete points that are relevant for the validity debate in psychology. First of all, studying the causal mechanisms underlying the measurements can be crucial for evaluating whether the measurements are valid. Secondly, psychologists would benefit from focusing more on the robustness of measurements. Finally, we argue that it is possible to make good science based on (relatively) bad measurements, and that the explanatory success of science can contribute to justifying the validity of measurements.

Measurement in psychology: A case of ignoramus et ignorabimus? A rejoinder

The Millean Quantity Objection—that is, the claim that the measurement of psychological attributes is impossible (Trendler, 2009)—has been countered with partly vigorous opposition (Kyngdon, 2013; Markus & Borsboom, 2012; Saint-Mont, 2012). Kyngdon’s response is of particular interest, since he asserts that measurement may already have been established. If correct, it would definitely invalidate any quantity objection and end the century-long discussion about the measurability of psychological attributes. Therefore the focus of the rejoinder will be on the question of when measurement is reached. First the meaning of measurement is elaborated. On this basis, criteria for the successful establishment of measurement are formulated and it is outlined how these are satisfied in the case of intensive quantities. It is concluded that the evidence presented by Kyngdon is insufficient and inadequate. The approach will also serve as background to discuss objections raised against the Millean Quantity Objection.