Rhetoric of psychological measurement theory and practice

Metascience scholars have long been concerned with tracking the use of rhetorical language in scientific discourse, oftentimes to analyze the legitimacy and validity of scientific claim-making. Psychology, however, has only recently become the explicit target of such metascientific scholarship, much of which has been in response to the recent crises surrounding replicability of quantitative research findings and questionable research practices. The focus of this paper is on the rhetoric of psychological measurement and validity scholarship, in both the theoretical and methodological and empirical literatures. We examine various discourse practices in published psychological measurement and validity literature, including: (a) clear instances of rhetoric (i.e., persuasion or performance); (b) common or rote expressions and tropes (e.g., perfunctory claims or declarations); (c) metaphors and other "literary" styles; and (d) ambiguous, confusing, or unjustifiable claims. The methodological approach we use is informed by a combination of conceptual analysis and exploratory grounded theory, the latter of which we used to identify relevant themes within the published psychological discourse. Examples of both constructive and useful or misleading and potentially harmful discourse practices will be given. Our objectives are both to contribute to the critical methodological literature on psychological measurement and connect metascience in psychology to broader interdisciplinary examinations of science discourse.

A Multidimensional Model to Facilitate Within Person Comparison of Attributes

In psychological research and practice, a person's scores on two different traits or abilities are often compared. Such within-person comparisons require that measurements have equal units (EU) and/or equal origins: an assumption rarely validated. We describe a multidimensional SEM/IRT model from the literature and, using principles of conjoint measurement, show that its expected response variables satisfy the axioms of additive conjoint measurement for measurement on a common scale. In an application to Quality of Life data, the EU analysis is used as a pre-processing step to derive a simple structure Quality of Life model with three dimensions expressed in equal units. The results are used to address questions that can only be addressed by scores expressed in equal units. When the EU model fits the data, scores in the corresponding simple structure model will have added validity in that they can address questions that cannot otherwise be addressed. Limitations and the need for further research are discussed.

CancerSupportSource-Spanish: Development of a Distress Screening Measure for Spanish-Speaking Hispanic and Latino Individuals with Cancer

Introduction: CancerSupportSource (CSS), a distress screening and referral program, identifies unmet needs of people with cancer and links them to resources and support. We developed and validated a Spanish-language version (CSS-Spanish) to better serve Hispanic and Latino communities and promote health equity. Methods: The 25-item CSS-Spanish was created leveraging rigorous translation methods and cognitive interviews to ensure cultural relevance and topical breadth. A total of 210 Spanish-speaking Hispanic and Latino individuals completed CSS-Spanish and comparison measures. Psychometric analyses examined dimensionality and statistical validation, and determined scoring thresholds for depression and anxiety risk subscales. Results: CSS-Spanish represented key concerns across five factors and exhibited strong internal consistency and test-retest reliability, convergent validity, and known-groups validity. Risk subscales demonstrated adequate sensitivity. Conclusion: CSS-Spanish is a reliable, valid multidimensional distress screener that rapidly assesses needs of Hispanic and Latino individuals. Embedded depression and anxiety risk flags can support staff in identifying those with high-acuity needs.

Predictive validity of a selection instrument measuring resilience: The INSPIRE resilience scale

Military personnel needs to be resilient to be able to remain effective, motivated, and in good mental and physical health. Military organizations select on resilience to determine whether candidates are psychologically fit for their job. The INSPIRE Resilience Scale (IRS) is such a selection instrument that aims to assess the psychological resilience potential of candidates in high-risk professions. A longitudinal predictive validity study was conducted in five European Defense organizations and in the Dutch National Police. The IRS was submitted in selection (N = 11,404), and criterion data about performance and health were collected in the second half of the first training year (N = 726). The results based on correlational and regression analyses showed that the IRS scores significantly predicted the criterion measures. Emotional stability, part of the IRS, turned out to be the best predictor. Results also showed that candidates who dropped out of training had significantly lower means on the IRS in selection than candidates who were still in training in the second half of the first training year. Overall, the IRS proved to be a valid instrument to assess resilience potential in candidates for high-risk professions. Selecting on resilience may therefore contribute to training success and reduction of health problems.

Measuring young individuals' responses to climate change: validation of the Slovenian versions of the climate anxiety scale and the climate change worry scale

Introduction

While increasing awareness of climate change is needed to address this threat to the natural environment and humanity, it may simultaneously negatively impact mental health. Previous studies suggest that climate-specific mental health phenomena, such as climate anxiety and worry, tend to be especially pronounced in youth. To properly understand and address these issues, we need valid measures that can also be used in non-Anglophone samples. Therefore, in the present paper, we aimed to validate Slovenian versions of the Climate Anxiety Scale (CAS) and the Climate Change Worry Scale (CCWS) among Slovenian youth.

Method

We conducted an online survey in which 442 young individuals (18-24 years) from Slovenia filled out the two central questionnaires and additional instruments capturing other relevant constructs (e.g., general anxiety, neuroticism, and behavioral engagement).

Results

The confirmatory factor analyses results supported the hypothesized factorial structure of the CAS (two factors) and the CCWS (one factor). Both scales also demonstrated great internal reliability. Moreover, the analyses exploring both constructs' nomological networks showed moderate positive associations with similar measures, such as anxiety and stress (convergent validity), and very weak associations with measures they should not be particularly related to, such as narcissism (discriminant validity). Lastly, we found that the CAS and, even more so, the CCWS have unique predictive value in explaining outcomes such as perceived threat, support for climate policies, and behavioral engagement (incremental validity).

Discussion

Overall, Slovenian versions of the CAS and the CCWS seem to be valid, reliable, and appropriate for future studies tackling young individuals' responses to climate change. Limitations of the study and areas for future research are discussed.

Study of item text in the Chinese Symptom Checklist-90

Symptom Checklist-90 (SCL-90) is the popular and widely used instrument, whether in mental health screening, psychological consultation, or the diagnosis and estimate of mental illness. In 1984, it was translated from theEnglish version into Chinese and then SCL-90 has been widely used in China. It is a pity that the item text of Chinese version has not been revised since the birth of it until today. We analyzed the Chinese version of the 90-item text from 3 new perspectives: translation, semantic, and cross-cultural, and thought that 18 items should be revised. This study' results have taken one step forward on the basis of previous studies, which will play an important role in improving the quality of Chinese version SCL-90 and improving the mental health level of Chinese people.

Psychometric Properties of the Climate Change Worry Scale

Climate change worry involves primarily verbal-linguistic thoughts about the changes that may occur in the climate system and the possible effects of these changes. Such worry is one of several possible psychological responses (e.g., fear, anxiety, depression, and trauma) to climate change. Within this article, the psychometric development of the ten-item Climate Change Worry Scale (CCWS) is detailed in three studies. The scale was developed to assess proximal worry about climate change rather than social or global impacts. Study 1 provided evidence that the CCWS items were internally consistent, constituted a single factor, and that the facture structure of the items was invariant for men and women. The results from Study 1 also indicated a good fit with a Rasch model of the items. Study 2 affirmed the internal consistency of the CCWS items and indicated that peoples’ responses to the measure were temporally stable over a two-week test–retest interval (r = 0.91). Study 3 provided support for the convergent and divergent validity of the CCWS through its pattern of correlations with several established clinical and weather-related measures. The limitations of the studies and the possible uses of the CCWS were discussed. The current work represents a starting point.

Using Internet-Based Psychological Measurement to Capture the Deteriorating Community Mental Health Profile During COVID-19: Observational Study

BACKGROUND

The coronavirus disease (COVID-19) is expected to have widespread and pervasive implications for mental health in terms of deteriorating outcomes and increased health service use, leading to calls for empirical research on mental health during the pandemic. Internet-based psychological measurement can play an important role in collecting imperative data, assisting to guide evidence-based decision making in practice and policy, and subsequently facilitating immediate reporting of measurement results to participants.

OBJECTIVE

The aim of this study is to use an internet-based mental health measurement platform to compare the mental health profile of community members during COVID-19 with community members assessed before the pandemic.

METHODS

This study uses an internet-based self-assessment tool to collect data on psychological distress, mental well-being, and resilience in community cohorts during (n=673) and prior to the pandemic (two cohorts, n=1264 and n=340).

RESULTS

Our findings demonstrate significantly worse outcomes on all mental health measures for participants measured during COVID-19 compared to those measured before (P<.001 for all outcomes, effect sizes ranging between Cohen d=0.32 to Cohen d=0.81. Participants who demonstrated problematic scores for at least one of the mental health outcomes increased from 58% (n=197/340) before COVID-19 to 79% (n=532/673) during COVID-19, leading to only 21% (n=141) of measured participants displaying good mental health during the pandemic.

CONCLUSIONS

The results clearly demonstrate deterioration in mental health outcomes during COVID-19. Although further research is needed, our findings support the serious mental health implications of the pandemic and highlight the utility of internet-based data collection tools in providing evidence to innovate and strengthen practice and policy during and after the pandemic.

The measurement of state engagement in actions beyond basic professional duties

BACKGROUND

The goal of this paper is to present the phenomenon of engagement in actions that go beyond basic professional duties, and the factor structure and other psychometric properties of the daily measurements of this construct, which corresponds with the growing interest of researchers in the phenomenon of employee engagement, extra-role behaviors and the factors that affect them on a daily basis.

MATERIAL AND METHODS

The research was carried out among 62 full-time or part-time working students of the humanities and social sciences (79% of whom were female), using a 5-day on-line daily survey. The main measure was a modified Polish version of the 9-item questionnaire consistent with the <i>Utrecht Work Engagement Scale</i> (UWES), which was prepared on the basis of the adaptation of the <i>State Work Engagement</i> (SWE) questionnaire proposed by Breevaart et al. (2012) to measure daily work engagement.

RESULTS

The obtained results have shown that the adapted version of the SWE questionnaire is a reliable and valid measure which can be used to measure daily fluctuations in beyond-duties engagement. The 3-factor multilevel model has a satisfactory fit.

CONCLUSIONS

The study contributes to the current body of literature by examining state engagement in actions that go beyond basic professional duties and the validity of its measurement. It also shows that giving employees more autonomy and feedback may positively influence their beyond-duties engagement. Med Pr. 2020;71(2):127-36.

Development of a New Instrument for Depression With Cognitive Diagnosis Models

Most existing instruments for depression are developed based on classical test theory, factor analysis, or sometimes, item response theory, and focus on the accurate measurement of the severity of depressive disorder. Nevertheless, they tend to be less useful in supporting the decision based on ICD-10 or DSM-5 because of the lack of detailed information for symptoms. To gain rich and valid information at the symptom level, this article developed a depression test under the framework of cognitive diagnosis models (CDMs), referred to as CDMs-D. A total of 1,181 individuals were finally recruited and their responses were used to examine the psychometric properties of CDMs-D. After excluding poor items for statistical reasons (e.g., low discrimination, poor model-fit or having DIF), 56 items were included in the CDMs-D. The CDMs-D measures all ten symptom criteria for depression defined in ICD-10 and covers five domains of depression defined by Gibbons et al. (2012). Comparing with the existing self-report measures (such as PHQ-9, SDS, CES-D and so on), a distinguishing feature of the CDMs-D is that it can provide both overall information about the severity of depressive disorder and the assessment information about specific symptoms, which could be useful for diagnostic and interventional purposes.

Improving Psychological Measurement: Does It Make a Difference? A Comment on Nesselroade and Molenaar (2016)

Nesselroade and Molenaar advocate the use of an idiographic filter approach. This is a fixed-effects approach, which may limit the number of individuals that can be simultaneously modeled, and it is not clear how to model the presence of subpopulations. Most important, Nesselroade and Molenaar's proposal appears to be best suited for modeling long time series on a few variables for a few individuals. Long time series are not common in psychological applications. Can it be applied to the usual longitudinal data we face? These are characterized by short time series (four to five points in time), hundreds of individuals, and dozens of variables. If so, what do we gain? Applied settings most often involve between-individual decisions. I conjecture that their approach will not outperform common, simpler, methods. However, when intraindividual decisions are involved, their approach may have an edge.

Sharing feelings online: studying emotional well-being via automated text analysis of Facebook posts

Digital traces of activity on social network sites represent a vast source of ecological data with potential connections with individual behavioral and psychological characteristics. The present study investigates the relationship between user-generated textual content shared on Facebook and emotional well-being. Self-report measures of depression, anxiety, and stress were collected from 201 adult Facebook users from North Italy. Emotion-related textual indicators, including emoticon use, were extracted form users’ Facebook posts via automated text analysis. Correlation analyses revealed that individuals with higher levels of depression, anxiety expressed negative emotions on Facebook more frequently. In addition, use of emoticons expressing positive emotions correlated negatively with stress level. When comparing age groups, younger users reported higher frequency of both emotion-related words and emoticon use in their posts. Also, the relationship between online emotional expression and self-report emotional well-being was generally stronger in the younger group. Overall, findings support the feasibility and validity of studying individual emotional well-being by means of examination of Facebook profiles. Implications for online screening purposes and future research directions are discussed.

Achieving across-laboratory replicability in psychophysical scaling

It is well known that, although psychophysical scaling produces good qualitative agreement between experiments, precise quantitative agreement between experimental results, such as that routinely achieved in physics or biology, is rarely or never attained. A particularly galling example of this is the fact that power function exponents for the same psychological continuum, measured in different laboratories but ostensibly using the same scaling method, magnitude estimation, can vary by a factor of three. Constrained scaling (CS), in which observers first learn a standardized meaning for a set of numerical responses relative to a standard sensory continuum and then make magnitude judgments of other sensations using the learned response scale, has produced excellent quantitative agreement between individual observers’ psychophysical functions. Theoretically it could do the same for across-laboratory comparisons, although this needs to be tested directly. We compared nine different experiments from four different laboratories as an example of the level of across experiment and across-laboratory agreement achievable using CS. In general, we found across experiment and across-laboratory agreement using CS to be significantly superior to that typically obtained with conventional magnitude estimation techniques, although some of its potential remains to be realized.

Comparison of depressive symptoms in type 2 diabetes using a two-stage survey design

OBJECTIVE

To test the validity of the Patient Health Questionnaire-9 (PHQ-9) in adults with newly diagnosed Type 2 diabetes mellitus (T2DM) and compare the distribution of PHQ-9 items in those with and without Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) depression.

METHODS

A two-stage survey design was used in primary care centers (n = 96). In Stage 1, participants were administered the PHQ-9 and biopsychosocial measures. In Stage 2, PHQ-9 positives (score ≥ 10) and randomly selected PHQ-9 negatives (score <10) had a diagnostic interview, Schedule for Clinical Assessment in Neuropsychiatry 2.1. The sensitivity, specificity and receiver operating characteristics of the PHQ-9 were calculated. The means of PHQ-9 items were compared.

RESULTS

Stage 1: Prevalence of PHQ-9 positives (completing Stage 2) was 12.1% (n = 182). PHQ-9 positives were younger (mean [standard deviation] age = 52.8 [9.47] versus 56.2 [11.50] years, p < .001) and had higher body mass index (32.7 [7.08] versus 31.6 [6.25] kg/m(2), p = .028) than PHQ-9 negatives (n = 1278). Stage 2: Prevalence of DSM-IV depression was 5.6% (n = 84). The optimal cutoff was ≥ 12 (sensitivity = 86.9%, specificity = 80.3%). PHQ-9 true positives scored significantly higher than PHQ-9 false positives on all items, excluding sleep (mean [standard deviation] score = 2.4 [0.98] versus 2.2 [1.06]), fatigue (2.5 [0.80] versus 2.3 [0.86]), and appetite (1.6 [1.23] versus 1.5 [1.26]).

CONCLUSIONS

Over-identification of depression by PHQ-9 in T2DM may be driven by presence of symptoms in keeping with an organic origin.

Some methodological considerations in theory-based health behavior research

As this special issue shows, much research in social and personality psychology is directly relevant to health psychology. In this brief commentary, we discuss three topics in research methodology that may be of interest to investigators involved in health-related psychological research. The first topic is statistical analysis of mediated and moderated effects. The second is measurement of latent constructs. The third is the Multiphase Optimization Strategy, a framework for translation of innovations from social and personality psychology into behavioral interventions.

The attack of the psychometricians

This paper analyzes the theoretical, pragmatic, and substantive factors that have hampered the integration between psychology and psychometrics. Theoretical factors include the operationalist mode of thinking which is common throughout psychology, the dominance of classical test theory, and the use of "construct validity" as a catch-all category for a range of challenging psychometric problems. Pragmatic factors include the lack of interest in mathematically precise thinking in psychology, inadequate representation of psychometric modeling in major statistics programs, and insufficient mathematical training in the psychological curriculum. Substantive factors relate to the absence of psychological theories that are sufficiently strong to motivate the structure of psychometric models. Following the identification of these problems, a number of promising recent developments are discussed, and suggestions are made to further the integration of psychology and psychometrics.

When a Psychometric Advance Falls in the Forest

Borsboom (2006) attacks psychologists for failing to incorporate psychometric advances in their work, discusses factors that contribute to this regrettable situation, and offers suggestions for ameliorating it. This commentary applauds Borsboom for calling the field to task on this issue and notes additional problems in the field regarding measurement that he could add to his critique. It also chastises Borsboom for occasionally being unnecessarily perjorative in his critique, noting that negative rhetoric is unlikely to make converts of offenders. Finally, it exhorts psychometricians to make their work more accessible and points to Borsboom, Mellenbergh, and Van Heerden (2003) as an excellent example of how this can be done.