Dimensional Evaluation of Cognitive Measures: Methodological Confounds and Theoretical Concerns

The current study provides a methodological review of studies supporting a general factor of intelligence as the primary model for contemporary measures of cognitive abilities. A further evaluation is provided by an empirical evaluation that compares statistical estimates using different approaches in a large sample of children (ages 9–13 years, N = 780) administered a comprehensive battery of cognitive measures. Results from this study demonstrate the ramifications of using the bifactor and Schmid–Leiman (BF/SL) technique and suggest that using BF/SL methods limit interpretation of cognitive abilities to only a general factor. The inadvertent use of BF/SL methods is demonstrated to impact both model dimensionality and variance estimates for specific measures. As demonstrated in this study, conclusions from both exploratory and confirmatory studies using BF/SL methods are significantly questioned, especially for studies with a questionable theoretical basis. Guidelines for the interpretation of cognitive test scores in applied practice are discussed.

Análise da Estrutura Interna da Bateria de Avaliação Intelectual de Jovens e Adultos

Resumo O objetivo deste estudo foi analisar a estrutura interna e a relação das habilidades cognitivas avaliadas pela Bateria de Avaliação Intelectual de Adultos (BAIAD). A amostra foi composta por 963 indivíduos (615 F, 348 M), idades entre 15-64 anos (M = 21, DP/SD/DS = 8). A BAID é composta de sete subtestes: sinônimos, antônimos, analogias, pensamento lógico, pensamento viso-espacial, memória visual-auditiva, rapidez de raciocino. A análise de itens permitiu a redução da BAIAD e índices de precisão de 0,99. A análise confirmatória seguiu o modelo CHC (inteligência cristalizada, inteligência fluída e um fator geral - G) e indicou seu ajuste aos dados. A análise de rede que buscou explorar a relação entre os pares dos indicadores cognitivos e o agrupamento das comunidades apoiou o modelo teórico. Conclui-se que a BAIAD possui evidências de validade e índices de precisão adequados para avaliar a inteligência de adultos.

Exploratory Factor Analyses of the Intelligence and Development Scales-2: Implications for Theory and Practice

The factor structure of the intelligence and scholastic skills domains of the Intelligence and Development Scales-2 was examined using exploratory factor analyses with the standardization and validation sample (N = 2,030, aged 5 to 20 years). Results partly supported the seven proposed intelligence group factors. However, the theoretical factors Visual Processing and Abstract Reasoning as well as Verbal Reasoning and Long-Term Memory collapsed, resulting in a five-factor structure for intelligence. Adding the three scholastic skills subtests resulted in an additional factor Reading/Writing and in Logical-Mathematical Reasoning showing a loading on abstract Visual Reasoning and the highest general factor loading. A data-driven separation of intelligence and scholastic skills is not evident. Omega reliability estimates based on Schmid-Leiman transformations revealed a strong general factor that accounted for most of the true score variance both overall and at the group factor level. The possible usefulness of factor scores is discussed.

Construct Validity of the WISC-V in Clinical Cases: Exploratory and Confirmatory Factor Analyses of the 10 Primary Subtests

Independent exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) research with the Wechsler Intelligence Scale for Children-Fifth Edition (WISC-V) standardization sample has failed to provide support for the five group factors proposed by the publisher, but there have been no independent examinations of the WISC-V structure among clinical samples. The present study examined the latent structure of the 10 WISC-V primary subtests with a large (N = 2,512), bifurcated clinical sample (EFA, n = 1,256; CFA, n = 1,256). EFA did not support five factors as there were no salient subtest factor pattern coefficients on the fifth extracted factor. EFA indicated a four-factor model resembling the WISC-IV with a dominant general factor. A bifactor model with four group factors was supported by CFA as suggested by EFA. Variance estimates from both EFA and CFA found that the general intelligence factor dominated subtest variance and omega-hierarchical coefficients supported interpretation of the general intelligence factor. In both EFA and CFA, group factors explained small portions of common variance and produced low omega-hierarchical subscale coefficients, indicating that the group factors were of poor interpretive value.

Investigating the Theoretical Structure of the Differential Ability Scales—Second Edition Through Hierarchical Exploratory Factor Analysis

When the Differential Ability Scales–Second Edition (DAS-II) was developed, the instrument’s content, structure, and theoretical orientation were amended. Despite these changes, the Technical Handbook did not report results from exploratory factor analytic investigations, and confirmatory factor analyses were implemented using selected subtests across the normative age groups from the total battery. To address these omissions, the present study investigated the theoretical structure of the DAS-II using principal axis factoring followed by the Schmid–Leiman procedure with participants from the 5- to 8-year-old age range to determine the degree to which the DAS-II theoretical structure proposed in the Technical Handbook could be replicated. Unlike other age ranges investigated where at most 14 subtests were administered, the entire DAS-II battery was normed on participants aged 5 to 8 years, making it well suited to test the full instrument’s alignment with theory. Results suggested a six-factor solution that was essentially consistent with the Cattell–Horn–Carroll (CHC)-based theoretical structure suggested by the test publisher and simple structure was attained. The only exception involved two subtests (Picture Similarities and Early Number Concepts) that did not saliently load on a group factor. Implications for clinical practice are discussed.

Modeling General and Specific Abilities

The present study examined issues related to structural modeling of abilities by the use of simulated data as well as analysis of the standardization data from the Woodcock–Johnson-III. In both cases, results were evaluated with cross-validation. Simulation results showed that cross-validation with an independent data set was more successful in identifying the model that was used to generate test scores than were several fit indices. Analysis of the Woodcock–Johnson-III standardization data with cross-validation showed that bifactor models provided better fit than hierarchical or correlated factor models. This was true considering both fit indices and cross-validation. General and specific factors shared a considerable amount of variance as evaluated by using the bifactor models to partition variance. The results of the present study suggest that there is a certain degree of ambiguity in determining the exact amount of covariance in test performance accounted for by general and specific factors. This calls in to question the practice of adjusting or controlling for general abilities when evaluating measures of specific abilities. Evidence for the validity of a construct should not be limited to factor analysis of tests purported to measure that construct.

Exploratory Higher Order Analysis of the Luria Interpretive Model on the Kaufman Assessment Battery for Children–Second Edition (KABC-II) School-Age Battery

Higher order factor structure of the Luria interpretive scheme on the Kaufman Assessment Battery for Children–Second Edition (KABC-II) for the 7- to 12-year and the 13- to 18-year age groups in the KABC-II normative sample ( N = 2,025) is reported. Using exploratory factor analysis, multiple factor extraction criteria, and hierarchical exploratory factor analysis not included in the KABC-II manual, two-, three-, and four-factor extractions were analyzed to assess the hierarchical factor structure by sequentially partitioning variance appropriately to higher order and lower order dimensions as recommended by Carroll. No evidence for a four-factor solution was found. Results showed that the largest portions of total and common variance were accounted for by the second-order general factor and that interpretation should focus primarily, if not exclusively, at that level of measurement.

Incremental Criterion Validity of the WJ-III COG Clinical Clusters

The current study examined the incremental validity of the clinical clusters from the Woodcock–Johnson III Tests of Cognitive Abilities (WJ-III COG) for predicting scores on the Woodcock–Johnson III Tests of Achievement (WJ-III ACH). All participants were children and adolescents ( N = 4,722) drawn from the nationally representative WJ-III standardization sample. Hierarchical multiple regression analyses were used to assess for cluster-level effects after controlling for the variance accounted for by the General Intellectual Ability (GIA) composite score. Consistent with previous studies, the GIA accounted for clinically significant portions of WJ-III ACH score variance in all of the regression models with R2 values ranging from .33 to .63. The clinical cluster scores collectively accounted for small to moderate incremental effects with no meaningful effects observed for individual indicators. Potential implications of these results for empirically supported interpretation of the WJ-III COG are discussed.

Exploratory Bifactor Analysis of the WJ-III Achievement at School Age via the Schmid–Leiman Orthogonalization Procedure

The structure of academic achievement measures has been rarely investigated in the literature apart from that which appears in the instruments’ technical manuals. This is concerning, given the widespread use of academic achievement instruments when making educational decisions about children. The Woodcock–Johnson III (WJ-III) Achievement for school-aged children (age 9-19) was investigated using exploratory bifactor analysis via the Schmid–Leiman (SL) orthogonalization procedure. This is the first time the SL has been applied to an academic achievement measure. The results revealed a unidimensional model of academic achievement across the standard 11 subtest battery, a two- or three-factor model at age 9 to 13 in the extended battery, and a two-factor model at age 14 to 19 across the extended battery. Forcing the four-factor fit in the standard battery required extracting eigenvalues as low as 0.67 and yielded areas of both convergence with and divergence from the structure posited in the Technical Manual. Forcing the six-factor fit across the extended battery yielded Heywood Cases, a lack of convergence of the factor solution, and the need to truncate iterations at 2 to force the fit. The results of this study indicate that the WJ-III Achievement is a solid model of general achievement across the 9 to 19 age range. Examination of omega coefficients, the divergent factor structure, and the small amount of variance accounted for by the lower order factors suggest caution when interpreting beyond this level (i.e., the academic clusters). Implications for interpretation of the WJ-III Achievement at age 9 to 19 are discussed.

Incremental validity of WISC-IV(UK) factor index scores with a referred Irish sample: predicting performance on the WIAT-II(UK.)

BACKGROUND

Subtest and factor scores have typically provided little incremental predictive validity beyond the omnibus IQ score.

AIMS

This study examined the incremental validity of Wechsler Intelligence Scale for Children - Fourth UK Edition (WISC-IV(UK) ; Wechsler, 2004a, Wechsler Intelligence Scale for Children - Fourth UK Edition, Harcourt Assessment, London, UK) and factor index scores in predicting academic achievement on the Wechsler Individual Achievement Test - Second UK Edition (WIAT-II(UK) ; Wechsler, 2005a, Wechsler Individual Achievement Test-Second UK Edition, Pearson, London, UK), beyond that predicted by the WISC-IV(UK) FSIQ.

SAMPLE

The sample included 1,014 Irish children (ages 6-0 to 16-9) who were referred for evaluation of learning difficulties.

METHOD

Hierarchical multiple regression analyses were used with the WISC-IV(UK) FSIQ (Block 1) and factor index scores (Block 2) as predictors and WIAT-II(UK) subtest and composite scores as dependent variables.

RESULTS

The WISC-IV(UK) FSIQ accounted for statistically significant and generally large portions of WIAT-II(UK) subtest and composite score variance. WISC-IV(UK) factor index scores combined to provide statistically significant increments in prediction of most WIAT-II(UK) subtest and composite scores over and above the FSIQ; however, the effect sizes were mostly small as previously observed (i.e., Canivez, 2013a, Psychol. Assess., 25, 484; Glutting et al., 2006, J. Spec. Educ., 40, 103; Nelson et al., 2013, Psychol. Assess., 25, 618). Individually, the WISC-IV(UK) factor index scores provided small unique contributions to predicting WIAT-II(UK) scores.

CONCLUSION

This, in combination with studies of apportioned variance from bifactor confirmatory factor analysis (Watkins et al., 2013, Int. J. Sch. Educ. Psychol., 1, 102), indicated that the WISC-IV(UK) FSIQ should retain the greatest weight in WISC-IV(UK) interpretation.

Investigating the Structure of the WJ-III Cognitive in Early School Age Through Two Exploratory Bifactor Analysis Procedures

Two exploratory bifactor methods (e.g., Schmid–Leiman [SL] and exploratory bifactor analysis [EBFA]) were used to investigate the structure of the Woodcock–Johnson III (WJ-III) Cognitive in early school age (age 6-8). The SL procedure is recognized by factor analysts as a preferred method for EBFA. Jennrich and Bentler recently developed an alternative EBFA procedure. They claim that EBFA more readily produces independent cluster structure and overcomes the proportionality constraint experienced by the SL. The results of both analyses support the preeminence of the g factor at age 6 to 8. Examination of omega coefficients, the divergent factor structure, and the small amount of variance accounted for by the lower order factors suggests caution when interpreting beyond the higher order factor. Implications for interpretation of the WJ-III Cognitive at age 6 to 8 are discussed.

Exploratory Bifactor Analysis of the WJ-III Cognitive in Adulthood via the Schmid–Leiman Procedure

The Woodcock–Johnson-III cognitive in the adult time period (age 20 to 90 plus) was analyzed using exploratory bifactor analysis via the Schmid–Leiman orthogonalization procedure. The results of this study suggested possible overfactoring, a different factor structure from that posited in the Technical Manual and a lack of invariance across both age ranges under study. Even when forcing the seven-factor fit, the structure was problematic. The results from the 20 to 39 age group displayed patterns of convergence with and divergence from the Technical Manual’s structure. The results from the 40 and above age group were generally consistent with the Technical Manual’s structure except for retrieval fluency. This study is consistent with the body of exploratory factor analysis structural validity evidence suggesting that contemporary tests of cognitive ability, particularly those based on Cattell–Horn–Carroll theory, are overfactored and lack alignment with their respective Technical Manual’s presented structure.