Psychometric problems with the method of correlated vectors applied to item scores (including some nonsensical results)

Measuring State and Trait Anxiety: An Application of Multidimensional Item Response Theory

The State-Trait Inventory for Cognitive and Somatic Anxiety (STICSA) is a widely used measure of state and trait anxiety. Within the Classical Testing Theory model, consistent findings provide support for its multidimensional factor structure, discriminant, convergent, and nomological validity, as well as age and gender invariance, across healthy and clinical samples. Nevertheless, some issues regarding STICSA dimensionality and item-scale composition remain unresolved (e.g., both bifactor and two-factor models were found to fit data equally well). The goal of this study was to investigate the STICSA's dimensionality within the Item Response Theory, and to assess the tenability of the bifactor model as a plausible model over the multidimensional model. The sample consisted of 3338 Italian participants (58.21% females; 41.79% males) with an average age of 35.65 years (range: 18-99; SD = 20.25). Both bifactor and two-correlated dimensions of the STICSA scales were confirmed to fit data by applying the multidimensional Item Response Theory (mIRT). While the bifactor model showed better fit indices, the multidimensional model was more accurate and precise (0.86-0.88) in estimating state and trait latent anxiety. A further comparison between multidimensional item parameters revealed that the multidimensional and bifactor models were equivalent. Findings showed that the STICSA is an accurate and precise instrument for measuring somatic and cognitive symptomatology dimensions within state and trait anxiety. The use of the state/trait total score requires special attention from the clinicians and researchers to avoid bias in the psychodiagnostic assessment.

The Negative Religiousness-IQ Nexus is a Jensen Effect on Individual-Level Data: A Refutation of Dutton et al.'s 'The Myth of the Stupid Believer'

A recent study by Dutton et al. (J Relig Health 59:1567-1579. https://doi.org/10.1007/s10943-019-00926-3 , 2020) found that the religiousness-IQ nexus is not on g when comparing different groups with various degrees of religiosity and the non-religious. It suggested, accordingly, that the nexus related to the relationship between specialized analytic abilities on the IQ test and autism traits, with the latter predicting atheism. The study was limited by the fact that it was on group-level data, it used only one measure of religiosity that measure may have been confounded by the social element to church membership and it involved relatively few items via which a Jensen effect could be calculated. Here, we test whether the religiousness-IQ nexus is on g with individual-level data using archival data from the Vietnam Experience Study, in which 4462 US veterans were subjected to detailed psychological tests. We used multiple measures of religiosity-which we factor-analysed to a religion-factor-and a large number of items. We found, contrary to the findings of Dutton et al. (2020), that the IQ differences with regard to whether or not subjects believed in God are indeed a Jensen effect. We also uncovered a number of anomalies, which we explore.

Between-Group Mean Differences in Intelligence in the United States Are >0% Genetically Caused: Five Converging Lines of Evidence

The past 30 years of research in intelligence has produced a wealth of knowledge about the causes and consequences of differences in intelligence between individuals, and today mainstream opinion is that individual differences in intelligence are caused by both genetic and environmental influences. Much more contentious is the discussion over the cause of mean intelligence differences between racial or ethnic groups. In contrast to the general consensus that interindividual differences are both genetic and environmental in origin, some claim that mean intelligence differences between racial groups are completely environmental in origin, whereas others postulate a mix of genetic and environmental causes. In this article I discuss 5 lines of research that provide evidence that mean differences in intelligence between racial and ethnic groups are partially genetic. These lines of evidence are findings in support of Spearman’s hypothesis, consistent results from tests of measurement invariance across American racial groups, the mathematical relationship that exists for between-group and within-group sources of heritability, genomic data derived from genome-wide association studies of intelligence and polygenic scores applied to diverse samples, and admixture studies. I also discuss future potential lines of evidence regarding the causes of average group differences across racial groups. However, the data are not fully conclusive, and the exact degree to which genes influence intergroup mean differences in intelligence is not known. This discussion applies only to native English speakers born in the United States and not necessarily to any other human populations.

Differential Item Functioning on Raven’s SPM+ Amongst Two Convenience Samples of Yakuts and Russians

National IQ estimates are based on psychometric measurements carried out in a variety of cultural contexts and are often obtained from Raven’s Progressive Matrices tests. In a series of studies, J. Philippe Rushton et al. have argued that these tests are not biased with respect to ethnicity or race. Critics claimed their methods were inappropriate and suggested differential item functioning (DIF) analysis as a more suitable alternative. In the present study, we conduct a DIF analysis on Raven’s Standard Progressive Matrices Plus (SPM+) tests administered to convenience samples of Yakuts and ethnic Russians. The Yakuts scored lower than the Russians by 4.8 IQ points, a difference that can be attributed to the selectiveness of the Russian sample. Data from the Yakut (n = 518) and Russian (n = 956) samples were analyzed for DIF using logistic regression. Although items B9, B10, B11, B12, and C11 were identified as having uniform DIF, all of these DIF effects can be regarded as negligible (R2 <0.13). This is consistent with Rushton et al.’s arguments that the Raven’s Progressive Matrices tests are ethnically unbiased.

Spearman's hypothesis tested comparing Korean young adults with various other groups of young adults on the items of the Advanced Progressive Matrices

Spearman's hypothesis tested at the subtest level of an IQ battery states that differences between races on the subtests of an IQ battery are a function of the g loadings of these subtests, such that there are small differences between races on subtests with low g loadings and large differences between races on subtests with high g loadings. Jensen (1998) stated that Spearman's hypothesis is a law-like phenomenon. It has also been confirmed many times at the level of items of the Raven's Progressive Matrices. This study hypothesizes that with concern to Spearman's hypothesis, subtests and items function in fundamentally the same way, and tested whether Spearman's hypothesis is confirmed at the item level for White-East Asian comparisons. A group of Korean young adults (N=205) was compared with other groups of young adults from Canada, the US, Russia, Peru and South Africa (total N=4770) who took the Advanced Progressive Matrices. Spearman's hypothesis was strongly confirmed with a sample-size-weighted r with a value of 0.63. Computing the g loadings of the items of the Raven with either the Raven-g or the Wechsler-g led to the same conclusions. Tests of Spearman's hypothesis yielded less-strong outcomes when the 36-item Advanced Progressive Matrices were used than when the 60-item Standard Progressive Matrices were used. There is a substantial correlation between sample size and the outcome of Spearman's hypothesis. So, all four hypotheses were confirmed, showing that a part of the subtest-level nomological net replicates at the item level, strengthening the position that, with concern to Spearman's hypothesis, subtests and items function fundamentally the same. It is concluded that Spearman's hypothesis is still a law-like phenomenon. Detailed suggestions for follow-up research are made.

Sex Differences in 32,347 Jordanian 4th Graders on the National Exam of Mathematics

Abstract. Sex differences in mathematical ability were examined in a nation-wide sample of 32,346 Jordanian 4th graders (age 9–10 year) on a 40-item mathematics test. Overall, boys were found to perform slightly worse ( d = −0.12) but had slightly more variation in scores ( SD = 1.02 and SD = 0.98 for boys and girls, respectively). However, when results were disaggregated by school type, single-sex versus coed (i.e., coeducational), boys were found to perform better than girls in coed schools ( d = 0.27) but worse across single-sex schools ( d = −0.37). Two-parameter item response theory analysis showed that item difficulty was similar across sexes in the full sample. Item loadings exhibited substantial departure from measurement invariance with respect to boys and girls at single-sex schools, though. For boys and girls at coed schools, both the item difficulty and item loading correlations were highly similar, evincing that measurement invariance largely held in this case. Partially consistent with findings from other countries, a correlation between item difficulty and male advantage was observed, r = .57, such that the relative male advantage increased with increased item difficulty. Complicating interpretation, this association did not replicate within coed schools. Item content, Bloom’s cognitive taxonomy category, and item position showed no relation to sex differences.

IGNORING PSYCHOMETRIC PROBLEMS IN THE STUDY OF GROUP DIFFERENCES IN COGNITIVE TEST PERFORMANCE

In a recent study, te Nijenhuis et al. (2017) used a version of Jensen's method of correlated vectors to study the nature of ethnic group differences on Raven's Progressive Matrices test. In this comment, the author points out that this method has been shown to be psychometrically inappropriate in studying group differences in performance on dichotomous (correctly or incorrectly scored) items. Specifically, the method uses item statistics like the item-total correlation that necessarily differ across groups differing in ability and employs a linear model to test inherent non-linear relations. Wicherts (2017) showed that this method can provide correlations far exceeding r=0.44 in cases where the group differences cannot possibly be on g because the items measure different traits across the groups. The psychometric problems with their method cast serious doubts on te Nijenhuis et al.'s conclusions concerning the role of g in the studied group difference in cognitive test performance.

THIS (METHOD) IS (NOT) FINE

In their response to my criticism of their recent article in Journal of Biosocial Science (te Nijenhuis et al., 2017), te Nijenhuis and van den Hoek (2018) raise four points none of which concerns my main point that the method of correlated vectors (MCV) applied to item-level data represents a flawed method. Here, I discuss te Nijenhuis and van den Hoek's four points. First, I argue that my previous application of MCV to item-level data showed that the method can yield nonsensical results. Second, I note that meta-analytic corrections for sampling error, imperfect measures, restriction of range and unreliability of the vectors are futile and cannot help fix the method. Third, I note that even with perfect data, the method can yield negative correlations. Fourth, I highlight the irrelevance of te Nijenhuis and van den Hoek (2018)'s point that my comment had not been published in a peerreviewed journal by referring to my articles in 2009 and 2017 on MCV in peer-reviewed journals.

Comparing Job Applicants to Non–Applicants Using An Item–Level Bifactor Model on the Hexaco Personality Inventory

The present study evaluated the ability of item–level bifactor models (a) to provide an alternative explanation to current theories of higher order factors of personality and (b) to explain socially desirable responding in both job applicant and non–applicant contexts. Participants (46% male; mean age = 42 years, SD = 11) completed the 200–item HEXACO Personality Inventory–Revised either as part of a job application ( n = 1613) or as part of low–stakes research ( n = 1613). A comprehensive set of invariance tests were performed. Applicants scored higher than non–applicants on honesty–humility ( d = 0.86), extraversion ( d = 0.73), agreeableness ( d = 1.06), and conscientiousness ( d = 0.77). The bifactor model provided improved model fit relative to a standard correlated factor model, and loadings on the evaluative factor of the bifactor model were highly correlated with other indicators of item social desirability. The bifactor model explained approximately two–thirds of the differences between applicants and non–applicants. Results suggest that rather than being a higher order construct, the general factor of personality may be caused by an item–level evaluative process. Results highlight the importance of modelling data at the item–level. Implications for conceptualizing social desirability, higher order structures in personality, test development, and job applicant faking are discussed. Copyright © 2017 European Association of Personality Psychology