Content validity of measures of theoretical constructs in health psychology: Discriminant content validity is needed

Linking measures to mechanisms of action in behavior change: A qualitative analysis of expert views

RATIONALE

Testing mechanisms of action (MoAs) hypothesized to underlie behavior change can enhance intervention effectiveness. Rigorous measurement of putative mechanisms is critical to this effort, but measures are rarely validated with respect to target MoAs.

OBJECTIVE

This study aimed to elucidate challenges of linking measures to putative MoAs and to identify priorities for future research.

METHOD

This study was a systematic exploration of written comments by experts in behavioral intervention research and theories of behavior change (N = 20) capturing their opinions about a task querying whether self-report measures from the Science Of Behavior Change (SOBC) Measures Repository were related to a set of MoAs identified by the Human Behaviour Change Project (HBCP).

RESULTS

Six themes were identified: 1) Study Value, 2) Measure Properties, 3) Mechanism Properties, 4) Miscellaneous Measure Concerns, 5) Conceptual Challenges, and 6) Approaches to Developing Measure-Mechanism Links. Experts noted challenges such as lack of measure validation, poor measure properties (e.g., double-barreled items), overly broad MoA definitions that limited their utility, lack of clarity around the term "related," and more. Nonetheless, experts expressed the importance of the exercise. Suggestions included developing and refining measures that are validate for assessing MoAs, clarifying and elaborating MoA definitions, and conducting further, more granular research.

CONCLUSION

This systematic examination of expert comments highlights issues that need further investigation to advance behavioral science, specifically pertaining to identifying valid measures of MoAs in behavioral and process research. This study highlights the challenges and opportunities for future research on linking measures and MoA in behavioral science and subsequently enhancing the efficacy of behavioral interventions.

Measures of Subjective Memory for People with Epilepsy: A Systematic Review of Measurement Properties

People with epilepsy frequently express concern about the burden of memory problems in their everyday lives. Self-report memory questionnaires may provide valuable insight into individuals' perceptions of their everyday memory performance and changes over time. Yet, despite their potential utility, the measurement properties of self-report memory questionnaires have not been evaluated in epilepsy. This systematic review aimed to provide a critical appraisal of the measurement properties of self-report memory questionnaires for adults with epilepsy. Following protocol registration (PROSPERO CRD42020210967), a systematic search of PubMed, EMBASE, Web of Science, CINAHL, and PsychInfo from database inception until 27 May 2021 was conducted. Eligible studies were published in English-language peer-reviewed journals, recruited adults with epilepsy, and reported on the development or evaluation of the measurement properties of a self-report memory questionnaire. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) methodology was used to evaluate each study of a measurement property, and results were qualitatively synthesised. In total, 80 articles and one test manual were located containing 153 studies of measurement properties pertinent to 23 self-report memory questionnaires. Overall, no scale could be recommended outright for the evaluation of subjective memory symptoms in adults with epilepsy. This was due to the near absence of dedicated content validation studies relevant to this population and shortcomings in the methodology and scientific reporting of available studies of structural validity. Recommendations to support the advancement and psychometric validation of self-report memory questionnaires for people with epilepsy are provided.

Translation and validation of the Work-Related Quality of Life Scale (WRQoLS-2) in a nursing cohort

OBJECTIVES

The purpose of this study was to translate the Work-Related Quality of Life Scale (WRQoLS-2) into Chinese and validate the capacity of the tool to effectively measure this concept in a cohort of nursing professionals from mainland China.

METHODS

The Chinese version of the WRQoLS-2 (WRQoLS-2C) was developed using forward and backward language translation. In total, 639 nurses were invited to complete the WRQoLS-2C. Two weeks later, 79 (12.4%) nurses were retested. Construct validity was analysed using exploratory and confirmatory factor analysis (EFA, CFA). Cronbach's α and the intraclass correlation coefficient (ICC) were used to assess internal consistency, reliability and test-retest reliability. Correlation between the WRQoLS-2C and the Quality of Nursing Work Life scale (QNWL) total score was used to assess criterion-relation validity.

RESULTS

A seven-factor structure was revealed and confirmed using EFA (explaining 70.3% of the variance) and CFA (χ²= 680.39, df = 413, χ²/df = 1.65, p < 0.001). The goodness-of-fit index was 0.88, and adjusted goodness-of-fit index 0.86 indicating a reliable model. The internal consistency (Cronbach's α = 0.94) and test-retest reliability (ICC = 0.84) of the WRQoLS-2C were high. The correlation coefficient between the WRQoLS-2C and QNWL total scores was 0.79 (p < 0.01).

CONCLUSION

The WRQoLS-2C was a reliable and valid instrument that can be used to assess WRQoL in the mainland China nursing profession.

IMPACT STATEMENT

There are few options available to assess work related quality of life in Chinese language. This study has confirmed that the WRQoLs-2C is an effective instrument to measure this concept in nurses from mainland China.

PLAIN LANGUAGE SUMMARY (PLS)

Work related quality of life is an important predictor of workplace turnover intension. Managers can take measures to improve work related quality of life and reduce employee attrition. There are very few tools to measure work related quality of life and fewer in Chinese language. We translated the WRQoLS-2 into Chinese according to Brislin's translation model, following cross-cultural adaption guidelines, and verified its reliability and validity in a cohort of mainland Chinese nurses. The translated instrument has good reliability and validity in nurses, but has not yet been verified in other occupational groups.

Validity and reliability of the novel three‐item occupational violence patient risk assessment tool

Aim

To develop and psychometrically test an occupational violence (OV) risk assessment tool in the emergency department (ED).

Design

Three studies were conducted in phases: content validity, predictive validity and inter‐rater reliability from June 2019 to March 2021.

Methods

For content validity, ED end users (mainly nurses) were recruited to rate items that would appropriately assess for OV risk. Subsequently, a risk assessment tool was developed and tested for its predictive validity and inter‐rater reliability. For predictive validity, triage notes of ED presentations in a month with the highest OV were assessed for presence of OV risk. Each presentation was then matched with events recorded in the OV incident register. Sensitivity and specificity values were calculated. For inter‐rater reliability, two assessors—trained and untrained—independently assessed the triage notes for presence of OV risk. Cohen's kappa was calculated.

Results

Two rounds of content validity with a total of N = 81 end users led to the development of a three‐domain tool that assesses for OV risk using aggression history, behavioural concerns (i.e., angry, clenched fist, demanding, threatening language or resisting care) and clinical presentation concerns (i.e., alcohol/drug intoxication and erratic cognition). Recommended risk ratings are low (score = 0 risk domain present), moderate (score = 1 risk domain present) and high (score = 2–3 risk domains present), with an area under the curve of 0.77 (95% confidence interval 0.7–0.81, p < .01). Moderate risk rating had a 61% sensitivity and 91% specificity, whereas high risk rating had 37% sensitivity and 97% specificity. Inter‐rater reliability ranged from 0.67 to 0.75 (p < .01), suggesting moderate agreement.

Conclusions

The novel three‐domain OV risk assessment tool was shown to be appropriate and relevant for application in EDs. The tool, developed through a rigorous content validity process, demonstrates acceptable predictive validity and inter‐rater reliability.

Impact

The developed tool is currently piloted in a single hospital ED, with a view to extend to inpatient settings and other hospitals.

What do alexithymia items measure? A discriminant content validity study of the Toronto-alexithymia-scale-20

Background

Questions have been raised about whether items of alexithymia scales assess the construct alexithymia and its key features, and no other related constructs. This study assessed the (discriminant) content validity of the most widely used alexithymia scale, i.e., the Toronto Alexithymia Scale (TAS-20).

Methods

Participants (n = 81) rated to what extent TAS-20 items and items of related constructs were relevant for assessing the constructs ‘alexithymia’, ‘difficulty identifying feelings’, ‘difficulty describing feelings’, ‘externally-oriented thinking’, ‘limited imaginal capacity’, ‘anxiety’, ‘depression’, and ‘health anxiety’.

Results

Results revealed that, overall, the TAS-20 did only partly measure ‘alexithymia’. Only the subscales ‘difficulty identifying feelings’ and ‘difficulty describing feelings’ represented ‘alexithymia’ and their intended construct, although some content overlap between these subscales was found. In addition, some items assessed (health) anxiety equally well or even better.

Conclusions

Revision of the TAS-20 is recommended to adequately assess all key features of alexithymia. Findings with the TAS-20 need to be interpreted with caution in people suffering from medical conditions.

Let's talk about pain catastrophizing measures: an item content analysis

BACKGROUND

Concerns have been raised about whether self-report measures of pain catastrophizing reflect the construct as defined in the cognitive-behavioral literature. We investigated the content of these self-report measures; that is, whether items assess the construct 'pain catastrophizing' and not other theoretical constructs (i.e., related constructs or pain outcomes) using the discriminant content validity method.

METHOD

Items (n = 58) of six pain catastrophizing measures were complemented with items (n = 34) from questionnaires measuring pain-related worrying, vigilance, pain severity, distress, and disability. Via an online survey, 94 participants rated to what extent each item was relevant for assessing pain catastrophizing, defined as "to view or present pain or pain-related problems as considerably worse than they actually are" and other relevant constructs (pain-related worrying, vigilance, pain severity, distress, and disability).

RESULTS

Data were analyzed using Bayesian hierarchical models. The results revealed that the items from pain-related worrying, vigilance, pain severity, distress, and disability questionnaires were distinctively related to their respective constructs. This was not observed for the items from the pain catastrophizing questionnaires. The content of the pain catastrophizing measures was equally well, or even better, captured by pain-related worrying or pain-related distress.

CONCLUSION

Based upon current findings, a recommendation may be to develop a novel pain catastrophizing questionnaire. However, we argue that pain catastrophizing cannot be assessed by self-report questionnaires. Pain catastrophizing requires contextual information, and expert judgment, which cannot be provided by self-report questionnaires. We argue for a person-centered approach, and propose to rename 'pain catastrophizing' measures in line with what is better measured: 'pain-related worrying'.