Parameter Invariance and Skill Attribute Continuity in the DINA Model

Relative Robustness of CDMs and (M)IRT in Measuring Growth in Latent Skills

Previous studies have demonstrated evidence of latent skill continuity even in tests intentionally designed for measurement of binary skills. In addition, the assumption of binary skills when continuity is present has been shown to potentially create a lack of invariance in item and latent ability parameters that may undermine applications. In this article, we examine measurement of growth as one such application, and consider multidimensional item response theory (MIRT) as a competing alternative. Motivated by prior findings concerning the effects of skill continuity, we study the relative robustness of cognitive diagnostic models (CDMs) and (M)IRT models in the measurement of growth under both binary and continuous latent skill distributions. We find CDMs to be a less robust way of quantifying growth under misspecification, and subsequently provide a real-data example suggesting underestimation of growth as a likely consequence. It is suggested that researchers should regularly attend to the assumptions associated with the use of latent binary skills and consider (M)IRT as a potentially more robust alternative if unsure of their discrete nature.

The Potential for Interpretational Confounding in Cognitive Diagnosis Models

Binary examinee mastery/nonmastery classifications in cognitive diagnosis models may often be an approximation to proficiencies that are better regarded as continuous. Such misspecification can lead to inconsistencies in the operational definition of "mastery" when binary skills models are assumed. In this paper we demonstrate the potential for an interpretational confounding of the latent skills when truly continuous skills are treated as binary. Using the DINA model as an example, we show how such forms of confounding can be observed through item and/or examinee parameter change when (1) different collections of items (such as representing different test forms) previously calibrated separately are subsequently calibrated together; and (2) when structural restrictions are placed on the relationships among skill attributes (such as the assumption of strictly nonnegative growth over time), among other possibilities. We examine these occurrences in both simulation and real data studies. It is suggested that researchers should regularly attend to the potential for interpretational confounding by studying differences in attribute mastery proportions and/or changes in item parameter (e.g., slip and guess) estimates attributable to skill continuity when the same samples of examinees are administered different test forms, or the same test forms are involved in different calibrations.

A Higher-Order Cognitive Diagnosis Model with Ordinal Attributes for Dichotomous Response Data

Most existing cognitive diagnosis models (CDMs) assume attributes are binary latent variables, which may be oversimplified in practice. This article introduces a higher-order CDM with ordinal attributes for dichotomous response data. The proposed model can either incorporate domain experts' knowledge or learn from the data empirically by regularizing model parameters. A sequential item response model was employed for joint attribute distribution to accommodate the sequential mastery mechanism. The expectation-maximization algorithm was employed for model estimation, and a simulation study was conducted to assess the recovery of model parameters. A set of real data was also analyzed to assess the viability of the proposed model in practice.

A Markov Estimation Strategy for Longitudinal Learning Diagnosis: Providing Timely Diagnostic Feedback

Timely diagnostic feedback is helpful for students and teachers, enabling them to adjust their learning and teaching plans according to a current diagnosis. Motivated by the practical concern that the simultaneity estimation strategy currently adopted by longitudinal learning diagnosis models does not provide timely diagnostic feedback, this study proposes a new Markov estimation strategy, which follows the Markov property. A simulation study was conducted to explore and compare the performance of four estimation strategies: the simultaneity, the Markov, the anchor-item, and the separated estimation strategies. The results show that their performance was highly consistent, and they presented in the following relative order: simultaneity > Markov > anchor-item ≥ separated. Overall, although accuracy in parameter estimation is sacrificed slightly with the proposed strategy, it can provide timely diagnostic feedback to practitioners, which is in line with the concept of "assessment for learning" and the needs of formative assessment.