Simulating the effects of common and specific abilities on test performance: an evaluation of factor analysis

Factor-Analytic Evidence for the Complexity of the Delis-Kaplan Executive Function System (D-KEFS)

This study examined the complexity of the Delis-Kaplan Executive Function System both in terms of the overall number of factors needed to model the entire battery and the complexity of the number of factors needed to model individual tests. The correlations between tests from the Delis-Kaplan Executive Function System standardization sample were modeled with varying numbers of factors. Simulations of test scores were also done that varied the number of factors used to generate the data and considered criteria for factor retention based on eigenvalues, fit indices, and generalization to new data. The results show seven-factor solutions based on generalization of fixed weights to validation samples provided best estimates. Modification indices produced three seven-factor solutions with comparable model fit but markedly different factor loadings. These results are consistent with process overlap theory, which suggest that complex tests are determined my many distinct executive processes. Given uncertainty of factor-analytic solutions, interpretation of the meaning of test scores should be based on evidence that is external to the test items, such as their sensitivity to clinical conditions. Overly simplistic interpretations of executive test performance based on factor-analytic results are not warranted. This is consistent with the cognitive processing approach of the test authors.

How neuroscience can inform the study of individual differences in cognitive abilities

Theories of human mental abilities should be consistent with what is known in neuroscience. Currently, tests of human mental abilities are modeled by cognitive constructs such as attention, working memory, and speed of information processing. These constructs are in turn related to a single general ability. However, brains are very complex systems and whether most of the variability between the operations of different brains can be ascribed to a single factor is questionable. Research in neuroscience suggests that psychological processes such as perception, attention, decision, and executive control are emergent properties of interacting distributed networks. The modules that make up these networks use similar computational processes that involve multiple forms of neural plasticity, each having different time constants. Accordingly, these networks might best be characterized in terms of the information they process rather than in terms of abstract psychological processes such as working memory and executive control.

Evaluation of multidimensional models of WAIS-IV subtest performance

OBJECTIVE

The present study examined the extent to which the covariance structure of the WAIS-IV is best accounted for by models that assume that test performance is the result of group-level factors and multiple independent general factors.

METHOD

Structural models with one to four general factors were evaluated with either four or five group-level factors. Simulations based on four general factors were run to clarify the adequacy of the estimates of the allocation of covariance by the models.

RESULTS

Four independent general factors provided better fit than a single general factor for either model with four or five group-level factors. While one of the general factors had much larger loadings than all other factors, simulation results suggested that this might be an artifact of the statistical procedure rather than a reflection of the nature of individual differences in cognitive abilities.

CONCLUSIONS

These results argue against the contention that clinical interpretation of cognitive test batteries should primarily be at the level of general intelligence. It is a fallacy to assume that factor analysis can reveal the structure of human abilities. Test validity should not be based solely on the results of modeling the covariance of test batteries.

Modeling General and Specific Abilities

The present study examined issues related to structural modeling of abilities by the use of simulated data as well as analysis of the standardization data from the Woodcock–Johnson-III. In both cases, results were evaluated with cross-validation. Simulation results showed that cross-validation with an independent data set was more successful in identifying the model that was used to generate test scores than were several fit indices. Analysis of the Woodcock–Johnson-III standardization data with cross-validation showed that bifactor models provided better fit than hierarchical or correlated factor models. This was true considering both fit indices and cross-validation. General and specific factors shared a considerable amount of variance as evaluated by using the bifactor models to partition variance. The results of the present study suggest that there is a certain degree of ambiguity in determining the exact amount of covariance in test performance accounted for by general and specific factors. This calls in to question the practice of adjusting or controlling for general abilities when evaluating measures of specific abilities. Evidence for the validity of a construct should not be limited to factor analysis of tests purported to measure that construct.