Shame for disrespecting evidence: the personal consequences of insufficient respect for structural equation model testing

Determinants of Beijing Residents' Intentions to Take Protective Behaviors against Smog: An Application of the Health Belief Model

Severe smog, a form of air pollution, has become a threat to public health in Beijing, China. To examine Beijing residents' protective behavioral intentions against smog, we proposed a conceptual model, which applies the health belief model (HBM) and specifies the roles of three distal predictors: exposure to news, discussion, and worry. The proposed model was tested in the context of protective behavioral intentions (i.e., intention to wear facemask & intention to use air purifier). Data were collected from Beijing residents during the period from 2/27 to 3/7 in 2017. Structural-equation-modeling (SEM) analyses of valid cases (N = 523) found support for the health belief model regarding the positive effects of perceived threat, perceived benefit, and perceived self-efficacy on intention to wear facemask or intention to use air purifier. Perceived barrier has a negative effect on intention to use air purifier, but is not related to intention to wear facemask. Neither exposure nor discussion is related to perceived threat. The effect of worry on intention to wear facemaskor intention to use air purifier is mediated by perceived threat. This proposed mediating mechanism is superior to the reverse mechanism (that worry mediates perceived threat). Implications of findings were discussed.

Investigation of measurement invariance in longitudinal health-related quality of life in preemptive or previously dialyzed kidney transplant recipients

PURPOSE

Kidney transplantation (KT) can impact patients' evaluation of health-related quality of life (HRQoL) as they adapt to their new life with a graft and its changes. Patients may adapt to KT in a different way, depending on whether they were on dialysis prior to transplantation or not (i.e. preemptive group). This may result in lack of measurement invariance between these patients' groups and/or over time (i.e. response shift, RS) which may invalidate the between-group comparison of HRQoL change scores. The aim of this study was to investigate and compare RS before and after KT between these two patients' groups. Measurement invariance was investigated between groups and over time with three measurement occasions.

METHODS

Adult patients completed the SF-36 at the last visit before KT, and 3, 6 months after. A structural equation model-based procedure was used to (i) detect and take into account measurement non-invariance between groups and RS, if appropriate, (ii) identify the period of occurrence of RS, (iii) study the heterogeneity of RS between the two groups.

RESULTS

Before KT (i.e. baseline), measurement invariance was not rejected between dialyzed (n = 196) and preemptive (n = 178) patients' groups. Between baseline and 3 months after KT, similar uniform recalibration was detected on the general health domain in both groups. Uniform recalibration was found between 3- and 6 months after KT on the vitality domain for preemptive patients only.

CONCLUSION

HRQoL, adjusted for RS, increased overall for preemptive and dialyzed kidney transplant patients after transplantation. RS may reflect differing adaptation processes following KT.

Drop-the-p: Bayesian CFA of the Multidimensional Scale of Perceived Social Support in Australia

AIM

We aimed to investigate whether the 12-item Multidimensional Scale of Perceived Social Support (MSPSS) constitutes a valid and reliable measure of social support for the general adult Australian population.

METHODS

Data were from Australia's National Survey of Adult Oral Health 2004-2006 and included 3899 participants aged 18 years old and over. The psychometric properties were evaluated with Bayesian confirmatory factor analysis. One-, two-, and three-factor (Significant Other, Family and Friends) structures were tested. Model fit was assessed with the posterior predictive p-value (PPPχ2), Bayesian root mean square error of approximation (BRMSEA), and Bayesian comparative fit index (BCFI). Dimensionality was tested by comparing competing factorial structures with the Bayes factor (BF). Reliability was evaluated with the Bayesian Ω H . Convergent validity was investigated with the Perceived Stress Scale (PSS) and discriminant validity with the Perceived Dental Control scale (PDC-3).

RESULTS

The theoretical three-factor model (Significant Other, Family, and Friends) provided a good fit to the data [PPPχ2 < 0.001, BRMSEA = 0.089-95% credible interval (CrI) (0.088, 0.089); BCFI = 0.963-95% CrI (0.963, 0.964)]. The BF provided decisive support for the three-factor structure in relation to the other structures. The SO [BΩ H = 0.95 - 95% CrI (0.90, 0.99)], FA (BΩ H = 0.92 - 95% CrI (0.87, 0.97), and FR (BΩ H = 0.92 - 95% CrI (0.88, 0.97)] subscales displayed excellent reliability. The MSPSS displayed initial evidence of convergent and discriminant validity.

CONCLUSION

The MSPSS demonstrated good psychometric properties and excellent reliability in a large Australian sample. This instrument can be applied in national surveys and provide evidence of the role of social support in the Australian population.

Model of Post-traumatic Growth in Newly Traumatized vs. Retraumatized Adolescents

Background: In our analysis of adolescents affected by the 2016 Fort McMurray wildfire, we observed many negative mental health effects in individuals with a prior history of psychological trauma. Elevated rates of depression and markers of post-traumatic stress disorder (PTSD) were observed, consistent with the hypothesis that prior trauma may reduce sensitivity thresholds for later psychopathology (stress sensitization). Surprisingly, levels of anxiety did not differ based on prior trauma history, nor were retraumatized individuals at increased risk for recent (past month) suicidal ideation. These results are more suggestive of inoculation by prior trauma than stress sensitization. This led us to consider whether individuals with a prior trauma history showed evidence of Post-Traumatic Growth (PTG), a condition in which the experience of a previous trauma leads to areas of sparing or even improvement. Method: To investigate this issue, we generated a structural equation model (SEM) exploring the role of anxiety in previously traumatized (n = 295) and wildfire trauma alone (n = 740) groups. Specifically, models were estimated to explore the relationship between hopelessness, anxiety, PTSD symptoms, self-efficacy and potential protective factors such as friend and family support in both groups. The model was tested using a cross-sectional sample of affected youth, comparing effects between the two groups. Results: While both models produced relatively good fit, differences in the effects and chi-squared values led us to conclude that the groups are subject to different causal specifications in a number of areas, although details warrant caution pending additional investigation. Discussion: We found that adolescents with a prior trauma history appear to have a more realistic appraisal of potential difficulties associated with traumatic events, and seem less reactive to potentially unsettling PTSD symptoms. They also seemed less prone to overconfidence as they got older, an effect seen in the adolescents without a history of trauma. Our findings provide preliminary evidence that the construct of anxiety may work differently in newly traumatized and retraumatized individuals, particularly in the context of mass trauma events.

Explicating the South African Psychological Ownership Questionnaire’s confirmatory factor analysis model fit: A Bayesian structural equation modelling approach

Orientation: The rigid application of conventional confirmatory factor analysis (CFA) techniques, the overreliance on global model fit indices and the dismissal of the chi-square statistic appear to have an adverse impact on the research of psychological ownership measures.Research purpose: The purpose of this study was to explicate the South African Psychological Ownership Questionnaire’s (SAPOS’s) CFA model fit using the Bayesian structural equation modelling (BSEM) technique.Motivation for the study: The need to conduct this study derived from a renewed awareness of the incorrect use of the chi-square statistic and global fit indices of CFA in social sciences research.Research approach/design and method: The SAPOS measurement model fit was explicated on two study samples consisting, respectively, of 712 and 254 respondents who worked in various organisations in South Africa. A Bayesian approach to CFA was used to evaluate if local model misspecifications were substantive and justified the rejection of the SAPOS model.Main findings: The findings suggested that a rejection of the SAPOS measurement model based on the results of the chi-square statistic and global fit indices would be unrealistic and unfounded in terms of substantive test theory.Practical/managerial implications: BSEM appeared to be a valuable diagnostic tool to pinpoint and evaluate local CFA model misspecifications and their effect on a measurement model.Contribution/value-add: This study showed the importance of considering local misspecifications rather than only relying the chi-square statistic and global fit indices when evaluating model fit.

Modeling lifetime abuse and cardiovascular disease risk among women

BACKGROUND

Cardiovascular disease (CVD) is one of the most significant health challenges facing women today. Abuse is a serious gendered issue also affecting the health of women. Despite beginning evidence that abuse may increase the risk of CVD among women, causal pathways linking abuse to CVD have received little attention. Our purpose was to test Scott-Storey's conceptual model showing direct and indirect pathways through which lifetime abuse severity may affect women's CVD risk.

METHODS

Using data collected from a community sample of 227 Canadian women who had left an abusive partner, we conducted structural equation modeling with latent growth curve analysis using a phantom variable approach to test the direct effects of severity of lifetime abuse on CVD risk (indicated by measures of systolic and diastolic blood pressure) as well as its indirect effects through CVD risk behaviors and through women's initial level of depressive symptoms and the observed rate of change in their depressive symptoms over time.

RESULTS

Women in this sample had above average CVD risk factors (i.e., smoking, overweight/obesity, depressive symptoms, high blood pressure) in comparison to women in the general population. Further, CVD risk behaviors increased with severity of lifetime abuse and remained present long after leaving the abusive relationship. Results of the tested model provide preliminary evidence supporting many of the hypothesized pathways by which severity of lifetime abuse can increase CVD risk among women; the model fit the data reasonably well explaining 41% of the variance in CVD risk.

CONCLUSIONS

Findings support the growing recognition of the long-term effects of lifetime abuse on cardiovascular health, suggest important implications for clinicians working with women, and provide a novel approach for studying the concept of cumulative lifetime abuse through the use of a phantom variable.

Fusion Validity: Theory-Based Scale Assessment via Causal Structural Equation Modeling

Fusion validity assessments employ structural equation models to investigate whether an existing scale functions in accordance with theory. Fusion validity parallels criterion validity by depending on correlations with non-scale variables but differs from criterion validity because it requires at least one theorized effect of the scale, and because both the scale and scaled-items are included in the model. Fusion validity, like construct validity, will be most informative if the scale is embedded in as full a substantive context as theory permits. Appropriate scale functioning in a comprehensive theoretical context greatly enhances a scale's validity. Inappropriate scale functioning questions the scale but the scale's theoretical embedding encourages detailed diagnostic investigations potentially challenging specific items, the procedure used to calculate scale values, or aspects of the theory, but also possibly recommends incorporating additional items into the scale. The scaled items should have survived prior content and methodological assessments but the items may or may not reflect a common factor because items having diverse causal backgrounds can sometimes fuse to form a unidimensional entity. Though items reflecting a common cause can be assessed for fusion validity, we illustrate fusion validity in the more challenging context of a scale comprised of diverse items and embedded in a complicated theory. Specifically we consider the Leadership scale from the Alberta Context Tool with care aides working in Canadian long-term care homes.

Analysis of risk factors of metabolic syndrome using a structural equation model: a cohort study

PURPOSE

We aimed to use a structural equation model (SEM) to determine the interrelations between various risk factors, including latent variables, involved in the development of metabolic syndrome(MetS).

METHODS

This study used data derived from the MJ Longitudinal Health Check-up Population Database for participants aged 20 to 70 years, who were asymptomatic for MetS at enrollment and were followed up for 5 years. A SEM was applied to investigate the attributions of MetS and the interrelations between different risk factors.

RESULTS

Socioeconomic status (SES), living habits, components of metabolic syndrome (COMetS), and blood pressure had a diverse impact on the onset of MetS, directly and (or) indirectly. When investigating the latent risk factors and the interrelations between different risk factors. The standardized total effect (the sum of the direct and indirect effects, β_t) of SES, living habits, blood pressure and COMetS on the onset of MetS was 0.084, -0.179, 0.154, and 0.353, respectively. SES, as a distal risk factor, directly influenced living habits, blood pressure, and COMetS with standardized regression coefficients (β_r) of -0.079 (P < 0.001), 0.200 (P < 0.001), and -0.163 (P < 0.001) respectively. Unfavorable living habits exerted an inverse effect on blood pressure and COMetS (β_r = -0.101, P < 0.001; β_r = -0.463, P < 0.001), which was an important path way for developing MetS.

CONCLUSIONS

These results demonstrate that individuals with a higher level of SES are susceptible to high blood pressure and are at increased risk for MetS. Additionally, there is a decrease in exercise and an increase in smoking and consumption of alcohol corresponded to an increase in metabolic risk factors.

Should We Use F-Tests for Model Fit Instead of Chi-Square in Overidentified Structural Equation Models?

Debate continues about whether the likelihood ratio test ( T ML) or goodness-of-fit indices are most appropriate for assessing data-model fit in structural equation models. Though potential advantages and disadvantages of these methods with large samples are often discussed, shortcomings concomitant with smaller samples are not. This article aims to (a) highlight the broader small sample issues with both approaches to data-model fit assessment, (b) note that what constitutes a small sample is common in empirical studies (approximately 20% to 50% in review studies, depending on the definition of “small”), and (c) more widely introduce F-tests as a desirable alternative than the traditional T ML tests, small-sample corrections, or goodness-of-fit indices with smaller samples. Both goodness-of-fit indices and comparing T ML to a chi-square distribution at smaller samples leads to overrejection of well-fitting models. Simulations and example analyses show that F-tests yield more desirable statistical properties—with or without normality—than standard approaches like chi-square tests or goodness-of-fit indices with smaller samples, roughly defined as N < 200 or N: df < 3.

Morningness-eveningness and social anxiety symptoms: the influence of depression symptoms on the indirect effect through punishment sensitivity and experiential avoidance

Social anxiety has recently been linked to morningness-eveningness; however, the psychological mechanisms underlying this relationship are not well known. As such, the purpose of the current study is to propose a model by which morningness-eveningness is related to social anxiety symptoms through punishment sensitivity and experiential avoidance within an adult American, community sample recruited via Amazon's Mechanical Turk (MTurk). It was hypothesized that experiential avoidance and punishment sensitivity would be associated with increased social anxiety symptoms and that morningness-eveningness would be negatively related to social anxiety symptoms. Furthermore, eveningness was hypothesized to be associated with increased punishment sensitivity and in turn, greater experiential avoidance. Lastly, the relationship between morningness-eveningness and social anxiety was hypothesized to be mediated by punishment sensitivity among the group with high depression levels, but not among the group with lesser depression symptoms. The results indicated that eveningness was related to social anxiety symptoms through experiential avoidance, and that depression symptoms influenced the relationship between morningness-eveningness and punishment sensitivity such that, in those high in depression symptoms, there was a significant association between eveningness and punishment sensitivity, but not among those with lower depression levels. The study findings build upon existing chronobiological research and addresses inconsistencies in previous literature.

Measurement Invariance of the Modified Yale Food Addiction Scale 2.0 Across Gender and Racial Groups

Objective: Food addiction reflects a substance use disorder framework, suggesting certain foods (e.g., high-fat, high-sugar foods) may trigger an addictive-like eating response in vulnerable individuals. This study explored whether the Modified Yale Food Addiction Scale 2.0 (mYFAS 2.0), a newly validated and shortened measure of food addiction, is appropriate for use in heterogeneous samples. Assessment of mYFAS 2.0 invariance is essential, as this measure was developed for use as a brief screener in large epidemiological samples that are likely demographically diverse. Method: Multigroup confirmatory factor analysis examined measurement invariance across racial/ethnic (White, Black, and Hispanic) and gender (male, female) groups. Participants were recruited through Qualtrics qBus, which uses demographic quotas to recruit a sample representative of the U.S. census reference population. Individuals were included in analyses if they identified their race/ethnicity as White, Black, or Hispanic ( N = 923). Results: Results supported full and partial measurement invariance across racial and gender groups, respectively. Discussion: Results increase confidence in the generalizability of findings using the mYFAS 2.0 and indicate that observed differences in prevalence rates, such as the higher rates of food addiction observed for women and Hispanic individuals, are likely due to true differences in the population rather than due to measurement bias.

Selecting the "Best" Factor Structure and Moving Measurement Validation Forward: An Illustration

Despite the broad literature base on factor analysis best practices, research seeking to evaluate a measure's psychometric properties frequently fails to consider or follow these recommendations. This leads to incorrect factor structures, numerous and often overly complex competing factor models and, perhaps most harmful, biased model results. Our goal is to demonstrate a practical and actionable process for factor analysis through (a) an overview of six statistical and psychometric issues and approaches to be aware of, investigate, and report when engaging in factor structure validation, along with a flowchart for recommended procedures to understand latent factor structures; (b) demonstrating these issues to provide a summary of the updated Posttraumatic Stress Disorder Checklist (PCL-5) factor models and a rationale for validation; and (c) conducting a comprehensive statistical and psychometric validation of the PCL-5 factor structure to demonstrate all the issues we described earlier. Considering previous research, the PCL-5 was evaluated using a sample of 1,403 U.S. Air Force remotely piloted aircraft operators with high levels of battlefield exposure. Previously proposed PCL-5 factor structures were not supported by the data, but instead a bifactor model is arguably more statistically appropriate.

The Thorny Relation Between Measurement Quality and Fit Index Cutoffs in Latent Variable Models

Latent variable modeling is a popular and flexible statistical framework. Concomitant with fitting latent variable models is assessment of how well the theoretical model fits the observed data. Although firm cutoffs for these fit indexes are often cited, recent statistical proofs and simulations have shown that these fit indexes are highly susceptible to measurement quality. For instance, a root mean square error of approximation (RMSEA) value of 0.06 (conventionally thought to indicate good fit) can actually indicate poor fit with poor measurement quality (e.g., standardized factors loadings of around 0.40). Conversely, an RMSEA value of 0.20 (conventionally thought to indicate very poor fit) can indicate acceptable fit with very high measurement quality (standardized factor loadings around 0.90). Despite the wide-ranging effect on applications of latent variable models, the high level of technical detail involved with this phenomenon has curtailed the exposure of these important findings to empirical researchers who are employing these methods. This article briefly reviews these methodological studies in minimal technical detail and provides a demonstration to easily quantify the large influence measurement quality has on fit index values and how greatly the cutoffs would change if they were derived under an alternative level of measurement quality. Recommendations for best practice are also discussed.

A graphical perspective of marginal structural models: An application for the estimation of the effect of physical activity on blood pressure

Estimating causal effects requires important prior subject-matter knowledge and, sometimes, sophisticated statistical tools. The latter is especially true when targeting the causal effect of a time-varying exposure in a longitudinal study. Marginal structural models are a relatively new class of causal models that effectively deal with the estimation of the effects of time-varying exposures. Marginal structural models have traditionally been embedded in the counterfactual framework to causal inference. In this paper, we use the causal graph framework to enhance the implementation of marginal structural models. We illustrate our approach using data from a prospective cohort study, the Honolulu Heart Program. These data consist of 8006 men at baseline. To illustrate our approach, we focused on the estimation of the causal effect of physical activity on blood pressure, which were measured at three time points. First, a causal graph is built to encompass prior knowledge. This graph is then validated and improved utilizing structural equation models. We estimated the aforementioned causal effect using marginal structural models for repeated measures and guided the implementation of the models with the causal graph. By employing the causal graph framework, we also show the validity of fitting conditional marginal structural models for repeated measures in the context implied by our data.

Improving measurement-invariance assessments: correcting entrenched testing deficiencies

Background

Factor analysis historically focused on measurement while path analysis employed observed variables as though they were error-free. When factor- and path-analysis merged as structural equation modeling, factor analytic notions dominated measurement discussions – including assessments of measurement invariance across groups. The factor analytic tradition fostered disregard of model testing and consequently entrenched this deficiency in measurement invariance assessments.

Discussion

Applying contemporary model testing requirements to the so-called configural model initiating invariance assessments will improve future assessments but a substantial backlog of deficient assessments remain to be overcome.

This article

summarizes the issues,

demonstrates the problem using a recent example,

illustrates a superior model assessment strategy,

and documents disciplinary entrenchment of inadequate testing as exemplified by the journal Organizational Research Methods.

Summary

Employing the few methodologically and theoretically best, rather than precariously-multiple, indicators of latent variables increases the likelihood of achieving properly causally specified structural equation models capable of displaying measurement invariance. Just as evidence of invalidity trumps reliability, evidence of configural model misspecification trumps invariant estimates of misspecified coefficients.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0230-3) contains supplementary material, which is available to authorized users.

Measuring Depression-Severity in Critically Ill Patients' Families with the Patient Health Questionnaire (PHQ): Tests for Unidimensionality and Longitudinal Measurement Invariance, with Implications for CONSORT

CONTEXT

Families of intensive care unit patients are at risk for depression and are important targets for depression-reducing interventions. Multi-item scores for evaluating such interventions should meet criteria for unidimensionality and longitudinal measurement invariance. The Patient Health Questionnaire (PHQ), widely used for measuring depression severity, provides standard nine-, eight-, and two-item scores. However, published studies often report no (or weak) evidence of these scores' unidimensionality/invariance, and no tests have evaluated them as measures of depression severity in intensive care unit patients' families.

OBJECTIVES

To identify multi-item PHQ constructs with promise for evaluating change in depression severity among family members of critically ill patients.

METHODS

Structural equation models with rigorous fit criterion (χ(2), P ≥ 0.05) tested the standard nine-, eight-, and two-item PHQ, and other item subsets, for unidimensionality and longitudinal invariance, using data from a trial evaluating an intervention to reduce depressive symptoms in family members.

RESULTS

Neither the standard nine-item nor the eight-item PHQ construct showed longitudinal invariance, although the standard two-item construct and other item subsets did.

CONCLUSION

The longer eight- and nine-item PHQ scores appear inappropriate for assessing depression severity in this population, with constructs based on smaller subsets of items being more promising targets for future trials. The Consolidated Standards of Reporting Trials requirement for prespecified trial outcomes is problematic because unidimensionality/invariance testing must occur after trial completion. Consolidated Standards of Reporting Trials could be strengthened by endorsing rigorous assessment of composite scores and encouraging use of the most appropriate substitute, should trial-based evidence challenge the legitimacy of prespecified multi-item scores.

Structural and construct validity of the Leeds Multiple Sclerosis Quality of Life scale

PURPOSE

The Leeds Multiple Sclerosis Quality of Life (LMSQOL) scale was designed as an 8-item, unidimensional disease-targeted measure of quality of life (QOL). This study (1) tested the unidimensionality of the LMSQOL using confirmatory factor analysis (CFA) and (2) examined the construct validity of scores based on a nomological network.

METHODS

The sample (N = 292) included persons with multiple sclerosis (MS) who were recruited from the Midwest region of the USA. Participants completed questionnaires that were delivered and returned using US Postal Service. We tested the fit of a single-factor model for the LMSQOL using CFA with the diagonally weighted least squares estimator in R. We examined the construct validity of the LMSQOL scores using Spearman rank-order correlations (r s) in SPSS.

RESULTS

The single-factor model had a reasonable fit for the 8-item LMSQOL [χ (2) (20) = 83.19, p < .001, CFI = .97, SRMR = .07], but provided an excellent fit for a 7-item version of the LMSQOL [χ (2) (14) = 23.63, p > .05, CFI = .99, SRMR = .05]. LMSQOL scores demonstrated strong correlations with measures of psychological well-being (|r s| = .53-.74) and weak-to-moderate correlations with measures of physical functioning and disability (|r s| = .23-.57).

CONCLUSION

The unidimensional model provides a good fit for the LMSQOL, and its scores provide a valid measure of QOL in MS.

A cautionary note on testing latent variable models

The article tackles the practice of testing latent variable models. The analysis covered recently published studies from 11 psychology journals varying in orientation and impact. Seventy-five studies that matched the criterion of applying some of the latent modeling techniques were reviewed. Results indicate the presence of a general tendency to ignore the model test (χ²) followed by the acceptance of approximate fit hypothesis without detailed model examination yielding relevant empirical evidence. Due to reduced sensitivity of such a procedure to confront theory with data, there is an almost invariable tendency to accept the theoretical model. This absence of model test consequences, manifested in frequently unsubstantiated neglect of evidence speaking against the model, thus implies the perilous question of whether such empirical testing of latent structures (the way it is widely applied) makes sense at all.