Experimental demonstrations of the earned dogmatism effect using a variety of optimal manipulations: Commentary and response to Calin-Jageman (2018)

A Validity-Based Framework for Understanding Replication in Psychology

In recent years, psychology has wrestled with the broader implications of disappointing rates of replication of previously demonstrated effects. This article proposes that many aspects of this pattern of results can be understood within the classic framework of four proposed forms of validity: statistical conclusion validity, internal validity, construct validity, and external validity. The article explains the conceptual logic for how differences in each type of validity across an original study and a subsequent replication attempt can lead to replication “failure.” Existing themes in the replication literature related to each type of validity are also highlighted. Furthermore, empirical evidence is considered for the role of each type of validity in non-replication. The article concludes with a discussion of broader implications of this classic validity framework for improving replication rates in psychological research.

An Evaluation of the Belief in Science Scale

The Belief in Science Scale (BISS) is a unidimensional measure that assesses the degree to which science is valued as a source of superior knowledge. Due to increased academic interest in the concept of belief in science, the BISS has emerged as an important measurement instrument. Noting an absence of validation evidence, the present paper, via two studies, evaluated the scale’s factorial structure. Both studies drew on data collected from previous research. Study 1 (N = 686), using parallel analysis and exploratory factor analysis, identified a unidimensional solution accounting for 56.43% of the observed variance. Study 2 (N = 535), using an independent sample, tested the unidimensional solution using confirmatory factor analysis (CFA). Data-model fit was good (marginal for RMSEA): CFI = 0.93, TLI = 0.91, RMSEA = 0.09 (90% CI of 0.08 to 0.10), SRMR = 0.04. Invariance testing across gender supported invariance of form, factor structure, and item intercepts for this one-factor model. BISS at the overall level correlated negatively with the reality testing dimension of the Inventory of Personality Organization (IPO-RT), demonstrating convergent validity. Researchers often use the IPO-RT as an indirect index of preference for experiential processing (intuitive thinking). In this context, only BISS scores above the median (second quartile) produced a reduction in experiential-based thinking. The authors discuss these findings in the context of belief in science as a psychometric construct.