Rhetoric of psychological measurement theory and practice

Metascience scholars have long been concerned with tracking the use of rhetorical language in scientific discourse, oftentimes to analyze the legitimacy and validity of scientific claim-making. Psychology, however, has only recently become the explicit target of such metascientific scholarship, much of which has been in response to the recent crises surrounding replicability of quantitative research findings and questionable research practices. The focus of this paper is on the rhetoric of psychological measurement and validity scholarship, in both the theoretical and methodological and empirical literatures. We examine various discourse practices in published psychological measurement and validity literature, including: (a) clear instances of rhetoric (i.e., persuasion or performance); (b) common or rote expressions and tropes (e.g., perfunctory claims or declarations); (c) metaphors and other "literary" styles; and (d) ambiguous, confusing, or unjustifiable claims. The methodological approach we use is informed by a combination of conceptual analysis and exploratory grounded theory, the latter of which we used to identify relevant themes within the published psychological discourse. Examples of both constructive and useful or misleading and potentially harmful discourse practices will be given. Our objectives are both to contribute to the critical methodological literature on psychological measurement and connect metascience in psychology to broader interdisciplinary examinations of science discourse.

Accuracy and precision of responses to visual analog scales: Inter- and intra-individual variability

Visual analog scales (VASs) are gaining popularity for collecting responses in computer administration of psychometric tests and surveys. The VAS format consists of a line marked at its endpoints with the minimum and maximum positions that it covers for respondents to place a mark at their selected location. Creating the line with intermediate marks along its length was discouraged, but no empirical evidence has ever been produced to show that their absence does any good. We report a study that asked respondents to place marks at pre-selected locations on a 100-unit VAS line, first when it only had numerical labels (0 and 100) at its endpoints and then when intermediate locations (from 0 to 100 in steps of 20) were also labeled. The results show that settings are more accurate and more precise when the VAS line has intermediate tick marks: The average absolute error decreased from 3.02 units without intermediate marks to 0.82 units with them. Provision of intermediate tick marks also reduced substantially inter- and intra-individual variability in accuracy and precision: The standard deviation of absolute error decreased from 0.87 units without tick marks to 0.25 units with them and the standard deviation of signed distance to target decreased from 1.16 units without tick marks to 0.24 units with them. These results prompt the recommendation that the design of VASs includes intermediate tick marks along the length of the line.

Validation of the Swedish Patient-Reported Outcomes in Parkinson's Disease Scale in Outpatients

BACKGROUND

Successful treatment of Parkinson's disease (PD) requires symptom monitoring. Patient-Reported Outcomes in Parkinson's Disease (PRO-PD) is a broad scale that covers 35 motor and nonmotor symptoms, but its validation is limited.

OBJECTIVE

The aim was to validate PRO-PD in a random sample of outpatients with PD.

METHODS

Of 2123 PD patients who visited outpatient clinics in West Sweden over a 12-month period, 25% were randomly selected and invited to participate in a longitudinal observational study. Included patients were assessed at baseline, 1 year, and 3 years, with a subset also assessed at 3 to 6 months. The assessments included PRO-PD, other patient-reported scales, and Clinical Impression of Severity Index for Parkinson's Disease (CISI-PD).

RESULTS

The study included 286 PD patients. PRO-PD ratings were available from 716 (96%) of 747 study visits. All PRO-PD items exhibited positive skewness without ceiling effects. Internal consistency at baseline was excellent (Cronbach's α: 0.93). Six-month test-retest reliability was good (intraclass correlation coefficient: 0.87). Convergent validity was good, with correlation coefficients between total PRO-PD and the 8-Item Parkinson's Disease Questionnaire of 0.70, the Non-Motor Symptoms Questionnaire of 0.70, EuroQoL Five-Dimension Five-Level Scale of 0.71, and CISI-PD of 0.69. Median PRO-PD score at baseline was 995 (interquartile range: 613-1399), with a median yearly increase of 71 (interquartile range: -21 to 111). Items representing axial motor symptoms increased most over time. The minimal clinically important change in total score was 119.

CONCLUSIONS

PRO-PD was found reliable and valid for monitoring symptoms in a representative sample of outpatients with PD. © 2023 The Authors. Movement Disorders published by Wiley Periodicals LLC on behalf of International Parkinson and Movement Disorder Society.

Rating scales institutionalise a network of logical errors and conceptual problems in research practices: A rigorous analysis showing ways to tackle psychology's crises

This article explores in-depth the metatheoretical and methodological foundations on which rating scales-by their very conception, design and application-are built and traces their historical origins. It brings together independent lines of critique from different scholars and disciplines to map out the problem landscape, which centres on the failed distinction between psychology's study phenomena (e.g., experiences, everyday constructs) and the means of their exploration (e.g., terms, data, scientific constructs)-psychologists' cardinal error. Rigorous analyses reveal a dense network of 12 complexes of problematic concepts, misconceived assumptions and fallacies that support each other, making it difficult to be identified and recognised by those (unwittingly) relying on them (e.g., various forms of reductionism, logical errors of operationalism, constructification, naïve use of language, quantificationism, statisticism, result-based data generation, misconceived nomotheticism). Through the popularity of rating scales for efficient quantitative data generation, uncritically interpreted as psychological measurement, these problems have become institutionalised in a wide range of research practices and perpetuate psychology's crises (e.g., replication, confidence, validation, generalizability). The article provides an in-depth understanding that is needed to get to the root of these problems, which preclude not just measurement but also the scientific exploration of psychology's study phenomena and thus its development as a science. From each of the 12 problem complexes; specific theoretical concepts, methodologies and methods are derived as well as key directions of development. The analyses-based on three central axioms for transdisciplinary research on individuals, (1) complexity, (2) complementarity and (3) anthropogenicity-highlight that psychologists must (further) develop an explicit metatheory and unambiguous terminology as well as concepts and theories that conceive individuals as living beings, open self-organising systems with complementary phenomena and dynamic interrelations across their multi-layered systemic contexts-thus, theories not simply of elemental properties and structures but of processes, relations, dynamicity, subjectivity, emergence, catalysis and transformation. Philosophical and theoretical foundations of approaches suited for exploring these phenomena must be developed together with methods of data generation and methods of data analysis that are appropriately adapted to the peculiarities of psychologists' study phenomena (e.g., intra-individual variation, momentariness, contextuality). Psychology can profit greatly from its unique position at the intersection of many other disciplines and can learn from their advancements to develop research practices that are suited to tackle its crises holistically.

Sense and nonsense in psychological measurement: A case of problem and method passing one another by

There exists conceptual confusion in the psychological measurement literature. One such example is the nonsense of the question, “are psychological attributes quantitative?” when referring to nontechnical psychological phenomena. Rather than debating whether this is an empirical or conceptual question, the issue of concern should be whether it makes sense to ask it. As illuminated by Wittgenstein almost a century ago, the concepts of psychology are everyday concepts rather than technical ones. However, the enduring comparison of psychology with the physical sciences leads to the adoption of technical language that loses its meaning when used with the aim of explaining everyday concepts. For this reason, asking whether psychological attributes are quantitative is, in many cases, a nonsensical question to ask.

Is measurement in psychology an empirical or a conceptual issue? A comment on David Franz

David Franz’s (2022) claim that it is not possible to give an empirically meaningful answer to the question “Are psychological attributes quantitative?” is rejected as incorrect. First, the ambiguous terminology identified by Franz in the measurement debate is not inherent to the debate itself, but reflects a biased understanding of the empirical meaning of axioms of measurement. Second, the problem of “hidden” or “unobservable” phenomena described by Franz as the root of conceptual confusion is not unique to psychology, but can also be found in physics. Therefore, it cannot be the cause for the failure to establish measurement in psychology. This comment concludes with the statement that, though both conceptual and empirical analyses are useful, Franz’s general claim that the former supersedes the latter must be rejected as inaccurate.

Psychological measurement is highly questionable but the details remain controversial: A response to Tafreshi, Michell, and Trendler

The profound and stimulating comments by Michell (2022), Tafreshi (2022), and Trendler (2022) push the measurement debate forward. Tafreshi provides arguments that are in line with my own position. While Michell is right in claiming that conceptualizations of psychological measurement are not necessarily Cartesian, I argue that the continuing influence of Cartesian thinking on conceptualizations of psychological measurement is evident in current definitions of psychological constructs. Moreover, I show that, contrary to Michell, it is meaningless to ask whether intelligence is quantitative. Michell and Trendler could counter this claim, but only at the cost of contradicting their stance that conceptual questions are not primary to the empirical questions surrounding psychological measurement. Beyond that, I contend that Trendler’s epistemological distinction of extensive versus intensive quantities does not compensate for the conceptual confusions inherent in his and Michell’s examples. Finally, I emphasize that all disputants agree that psychological measurement is a highly questionable idea.

Denying Descartes and wary of Wittgenstein: Response to Franz

In his article, “‘Are Psychological Attributes Quantitative?’ is not an Empirical Question: Conceptual Confusions in the Measurement Debate,” Franz (2022) concludes that psychological measurement does not rest on empirical hypotheses but rather on linguistic deceptions. His major premise is that psychometrics is inherently Cartesian. History shows otherwise: the mantras of operationism and the rituals of construct validity were intended to exorcise psyche from psychometrics. These mainstays of psychometrics ensured that theoretical constructs were more frequently dispositional concepts than they were mental concepts. It is with the latter, however, especially with attempts to measure currently occurring mental states, such as anxiety, that Franz’s argument looks more promising, but nevertheless it fails because it rests upon Wittgenstein’s views about the grammar of mental discourse. I conclude that conceptual analysis, realistically construed and applied to mental concepts, may show that they exclude quantitative structure. Despite that, it is always possible that empirical research might elicit quantitative-friendly revisions of mental concepts.