An Exploratory Diagnostic Model for Ordinal Responses with Binary Attributes: Identifiability and Estimation

Restricted Latent Class Models for Nominal Response Data: Identifiability and Estimation

Restricted latent class models (RLCMs) provide an important framework for diagnosing and classifying respondents on a collection of multivariate binary responses. Recent research made significant advances in theory for establishing identifiability conditions for RLCMs with binary and polytomous response data. Multiclass data, which are unordered nominal response data, are also widely collected in the social sciences and psychometrics via forced-choice inventories and multiple choice tests. We establish new identifiability conditions for parameters of RLCMs for multiclass data and discuss the implications for substantive applications. The new identifiability conditions are applicable to a wealth of RLCMs for polytomous and nominal response data. We propose a Bayesian framework for inferring model parameters, assess parameter recovery in a Monte Carlo simulation study, and present an application of the model to a real dataset.

Sufficient and Necessary Conditions for the Identifiability of DINA Models with Polytomous Responses

Cognitive diagnosis models (CDMs) provide a powerful statistical and psychometric tool for researchers and practitioners to learn fine-grained diagnostic information about respondents' latent attributes. There has been a growing interest in the use of CDMs for polytomous response data, as more and more items with multiple response options become widely used. Similar to many latent variable models, the identifiability of CDMs is critical for accurate parameter estimation and valid statistical inference. However, the existing identifiability results are primarily focused on binary response models and have not adequately addressed the identifiability of CDMs with polytomous responses. This paper addresses this gap by presenting sufficient and necessary conditions for the identifiability of the widely used DINA model with polytomous responses, with the aim to provide a comprehensive understanding of the identifiability of CDMs with polytomous responses and to inform future research in this field.

Diagnostic Classification Models for Testlets: Methods and Theory

Diagnostic classification models (DCMs) have seen wide applications in educational and psychological measurement, especially in formative assessment. DCMs in the presence of testlets have been studied in recent literature. A key ingredient in the statistical modeling and analysis of testlet-based DCMs is the superposition of two latent structures, the attribute profile and the testlet effect. This paper extends the standard testlet DINA (T-DINA) model to accommodate the potential correlation between the two latent structures. Model identifiability is studied and a set of sufficient conditions are proposed. As a byproduct, the identifiability of the standard T-DINA is also established. The proposed model is applied to a dataset from the 2015 Programme for International Student Assessment. Comparisons are made with DINA and T-DINA, showing that there is substantial improvement in terms of the goodness of fit. Simulations are conducted to assess the performance of the new method under various settings.

Going Deep in Diagnostic Modeling: Deep Cognitive Diagnostic Models (DeepCDMs)

Cognitive diagnostic models (CDMs) are discrete latent variable models popular in educational and psychological measurement. In this work, motivated by the advantages of deep generative modeling and by identifiability considerations, we propose a new family of DeepCDMs, to hunt for deep discrete diagnostic information. The new class of models enjoys nice properties of identifiability, parsimony, and interpretability. Mathematically, DeepCDMs are entirely identifiable, including even fully exploratory settings and allowing to uniquely identify the parameters and discrete loading structures (the " Q -matrices") at all different depths in the generative model. Statistically, DeepCDMs are parsimonious, because they can use a relatively small number of parameters to expressively model data thanks to the depth. Practically, DeepCDMs are interpretable, because the shrinking-ladder-shaped deep architecture can capture cognitive concepts and provide multi-granularity skill diagnoses from coarse to fine grained and from high level to detailed. For identifiability, we establish transparent identifiability conditions for various DeepCDMs. Our conditions impose intuitive constraints on the structures of the multiple Q -matrices and inspire a generative graph with increasingly smaller latent layers when going deeper. For estimation and computation, we focus on the confirmatory setting with known Q -matrices and develop Bayesian formulations and efficient Gibbs sampling algorithms. Simulation studies and an application to the TIMSS 2019 math assessment data demonstrate the usefulness of the proposed methodology.

A sequential exploratory diagnostic model using a Pólya-gamma data augmentation strategy

Cognitive diagnostic models provide a framework for classifying individuals into latent proficiency classes, also known as attribute profiles. Recent research has examined the implementation of a Pólya-gamma data augmentation strategy binary response model using logistic item response functions within a Bayesian Gibbs sampling procedure. In this paper, we propose a sequential exploratory diagnostic model for ordinal response data using a logit-link parameterization at the category level and extend the Pólya-gamma data augmentation strategy to ordinal response processes. A Gibbs sampling procedure is presented for efficient Markov chain Monte Carlo (MCMC) estimation methods. We provide results from a Monte Carlo study for model performance and present an application of the model.

Extending exploratory diagnostic classification models: Inferring the effect of covariates

Diagnostic models provide a statistical framework for designing formative assessments by classifying student knowledge profiles according to a collection of fine-grained attributes. The context and ecosystem in which students learn may play an important role in skill mastery, and it is therefore important to develop methods for incorporating student covariates into diagnostic models. Including covariates may provide researchers and practitioners with the ability to evaluate novel interventions or understand the role of background knowledge in attribute mastery. Existing research is designed to include covariates in confirmatory diagnostic models, which are also known as restricted latent class models. We propose new methods for including covariates in exploratory RLCMs that jointly infer the latent structure and evaluate the role of covariates on performance and skill mastery. We present a novel Bayesian formulation and report a Markov chain Monte Carlo algorithm using a Metropolis-within-Gibbs algorithm for approximating the model parameter posterior distribution. We report Monte Carlo simulation evidence regarding the accuracy of our new methods and present results from an application that examines the role of student background knowledge on the mastery of a probability data set.

Generic Identifiability of the DINA Model and Blessing of Latent Dependence

Cognitive diagnostic models are a powerful family of fine-grained discrete latent variable models in psychometrics. Within this family, the DINA model is a fundamental and parsimonious one that has received significant attention. Similar to other complex latent variable models, identifiability is an important issue for CDMs, including the DINA model. Gu and Xu (Psychometrika 84(2):468-483, 2019) established the necessary and sufficient conditions for strict identifiability of the DINA model. Despite being the strongest possible notion of identifiability, strict identifiability may impose overly stringent requirements on designing the cognitive diagnostic tests. This work studies a slightly weaker yet very useful notion, generic identifiability, which means parameters are identifiable almost everywhere in the parameter space, excluding only a negligible subset of measure zero. We propose transparent generic identifiability conditions for the DINA model, relaxing existing conditions in nontrivial ways. Under generic identifiability, we also explicitly characterize the forms of the measure-zero sets where identifiability breaks down. In addition, we reveal an interesting blessing-of-latent-dependence phenomenon under DINA-that is, dependence between the latent attributes can restore identifiability under some otherwise unidentifiable [Formula: see text]-matrix designs. The blessing of latent dependence provides useful practical implications and reassurance for real-world designs of cognitive diagnostic assessments.

Inferring Latent Structure in Polytomous Data with a Higher-Order Diagnostic Model

Researchers continue to develop and advance models for diagnostic research in the social and behavioral sciences. These diagnostic models (DMs) provide researchers with a framework for providing a fine-grained classification of respondents into substantively meaningful latent classes as defined by a multivariate collection of binary attributes. A central concern for DMs is advancing exploratory methods for uncovering the latent structure, which corresponds with the relationship between unobserved binary attributes and observed polytomous items with two or more response options. Multivariate behavioral polytomous data are often collected within a higher-order design where general factors underlying first-order latent variables. This study advances existing exploratory DMs for polytomous data by proposing a new method for inferring the latent structure underlying polytomous response data using a higher-order model to describe dependence among the discrete latent attributes. We report a novel Bayesian formulation that uses variable selection techniques for inferring the latent structure along with a higher-order factor model for attributes. We report evidence of accurate parameter recovery in a Monte Carlo simulation study and present results from an application to the 2012 Programme for International Student Assessment (PISA) problem-solving vignettes to demonstrate the method.

A Note on Weaker Conditions for Identifying Restricted Latent Class Models for Binary Responses

Restricted latent class models (RLCMs) are an important class of methods that provide researchers and practitioners in the educational, psychological, and behavioral sciences with fine-grained diagnostic information to guide interventions. Recent research established sufficient conditions for identifying RLCM parameters. A current challenge that limits widespread application of RLCMs is that existing identifiability conditions may be too restrictive for some practical settings. In this paper we establish a weaker condition for identifying RLCM parameters for multivariate binary data. Although the new results weaken identifiability conditions for general RLCMs, the new results do not relax existing necessary and sufficient conditions for the simpler DINA/DINO models. Theoretically, we introduce a new form of latent structure completeness, referred to as dyad-completeness, and prove identification by applying Kruskal's Theorem for the uniqueness of three-way arrays. The new condition is more likely satisfied in applied research, and the results provide researchers and test-developers with guidance for designing diagnostic instruments.

Learning Latent and Hierarchical Structures in Cognitive Diagnosis Models

Cognitive Diagnosis Models (CDMs) are a special family of discrete latent variable models that are widely used in educational and psychological measurement. A key component of CDMs is the Q-matrix characterizing the dependence structure between the items and the latent attributes. Additionally, researchers also assume in many applications certain hierarchical structures among the latent attributes to characterize their dependence. In most CDM applications, the attribute-attribute hierarchical structures, the item-attribute Q-matrix, the item-level diagnostic models, as well as the number of latent attributes, need to be fully or partially pre-specified, which however may be subjective and misspecified as noted by many recent studies. This paper considers the problem of jointly learning these latent and hierarchical structures in CDMs from observed data with minimal model assumptions. Specifically, a penalized likelihood approach is proposed to select the number of attributes and estimate the latent and hierarchical structures simultaneously. An expectation-maximization (EM) algorithm is developed for efficient computation, and statistical consistency theory is also established under mild conditions. The good performance of the proposed method is illustrated by simulation studies and real data applications in educational assessment.

Scalable Bayesian Approach for the Dina Q-Matrix Estimation Combining Stochastic Optimization and Variational Inference

Diagnostic classification models offer statistical tools to inspect the fined-grained attribute of respondents' strengths and weaknesses. However, the diagnosis accuracy deteriorates when misspecification occurs in the predefined item-attribute relationship, which is encoded into a Q-matrix. To prevent such misspecification, methodologists have recently developed several Bayesian Q-matrix estimation methods for greater estimation flexibility. However, these methods become infeasible in the case of large-scale assessments with a large number of attributes and items. In this study, we focused on the deterministic inputs, noisy "and" gate (DINA) model and proposed a new framework for the Q-matrix estimation to find the Q-matrix with the maximum marginal likelihood. Based on this framework, we developed a scalable estimation algorithm for the DINA Q-matrix by constructing an iteration algorithm that utilizes stochastic optimization and variational inference. The simulation and empirical studies reveal that the proposed method achieves high-speed computation, good accuracy, and robustness to potential misspecifications, such as initial value choices and hyperparameter settings. Thus, the proposed method can be a useful tool for estimating a Q-matrix in large-scale settings.

Direct Estimation of Diagnostic Classification Model Attribute Mastery Profiles via a Collapsed Gibbs Sampling Algorithm

This paper proposes a novel collapsed Gibbs sampling algorithm that marginalizes model parameters and directly samples latent attribute mastery patterns in diagnostic classification models. This estimation method makes it possible to avoid boundary problems in the estimation of model item parameters by eliminating the need to estimate such parameters. A simulation study showed the collapsed Gibbs sampling algorithm can accurately recover the true attribute mastery status in various conditions. A second simulation showed the collapsed Gibbs sampling algorithm was computationally more efficient than another MCMC sampling algorithm, implemented by JAGS. In an analysis of real data, the collapsed Gibbs sampling algorithm indicated good classification agreement with results from a previous study.

Sequential Gibbs Sampling Algorithm for Cognitive Diagnosis Models with Many Attributes

Cognitive diagnosis models (CDMs) are useful statistical tools to provide rich information relevant for intervention and learning. As a popular approach to estimate and make inference of CDMs, the Markov chain Monte Carlo (MCMC) algorithm is widely used in practice. However, when the number of attributes, K, is large, the existing MCMC algorithm may become time-consuming, due to the fact that O(2K) calculations are usually needed in the process of MCMC sampling to get the conditional distribution for each attribute profile. To overcome this computational issue, motivated by Culpepper and Hudson's earlier work in 2018, we propose a computationally efficient sequential Gibbs sampling method, which needs O(K) calculations to sample each attribute profile. We use simulation and real data examples to show the good finite-sample performance of the proposed sequential Gibbs sampling, and its advantage over existing methods.

Exploratory Restricted Latent Class Models with Monotonicity Requirements under PÒLYA-GAMMA Data Augmentation

Restricted latent class models (RLCMs) provide an important framework for supporting diagnostic research in education and psychology. Recent research proposed fully exploratory methods for inferring the latent structure. However, prior research is limited by the use of restrictive monotonicity condition or prior formulations that are unable to incorporate prior information about the latent structure to validate expert knowledge. We develop new methods that relax existing monotonicity restrictions and provide greater insight about the latent structure. Furthermore, existing Bayesian methods only use a probit link function and we provide a new formulation for using the exploratory RLCM with a logit link function that has an additional advantage of being computationally more efficient for larger sample sizes. We present four new Bayesian formulations that employ different link functions (i.e., the logit using the Pòlya-gamma data augmentation versus the probit) and priors for inducing sparsity in the latent structure. We report Monte Carlo simulation studies to demonstrate accurate parameter recovery. Furthermore, we report results from an application to the Last Series of the Standard Progressive Matrices to illustrate our new methods.

Measuring students’ learning progressions in energy using cognitive diagnostic models

This study applied cognitive diagnostic models to assess students’ learning progressions in energy. A Q-matrix (i.e., an item attribute alignment table) was proposed based on existing literature about learning progressions of energy in the physical science domain and the Trends in International Mathematics and Science Study (TIMSS) assessment framework. The Q-matrix was validated by expert review and real data analysis. Then, the deterministic inputs, noisy ‘and’ gate (DINA) model with hierarchical relations was applied to data from three jurisdictions that had stable, defined science curricula (i.e., Australia, Hong Kong, and Ontario). The results suggested that the hypothesized learning progression was consistent with the observed progression in understanding the energy concept. We also found similarities in students’ attribute mastery across the three jurisdictions. In addition, we examined the instructional sensitivity of the selected item. We discuss several curriculum-related issues and student misconceptions that may affect students’ learning progressions and mastery patterns in different regions of the world.

A multiple logistic regression-based (MLR-B) Q-matrix validation method for cognitive diagnosis models:A confirmatory approach

Q-matrix is an essential component specifying the relationship between attributes and items, which plays a key role in cognitive diagnosis assessment. The Q-matrix is usually developed by domain experts and its specifications tend to be subjective and might have misspecifications. Many existing pieces of research concentrate on the validation of Q-matrix; however, few of them can be applied to saturated cognitive diagnosis models. This paper proposes a general and effective Q-matrix validation method by employing multiple logistic regression model. Simulation studies are carried out to investigate the performance of the proposed method and compare it with four existing methods. Simulation results indicate the proposed method outperforms the existing methods in terms of validation accuracy. In addition, a set of real data is used as an example to illustrate its application. Finally, we discuss the limitations of the current study and the directions of future studies.

A Higher-Order Cognitive Diagnosis Model with Ordinal Attributes for Dichotomous Response Data

Most existing cognitive diagnosis models (CDMs) assume attributes are binary latent variables, which may be oversimplified in practice. This article introduces a higher-order CDM with ordinal attributes for dichotomous response data. The proposed model can either incorporate domain experts' knowledge or learn from the data empirically by regularizing model parameters. A sequential item response model was employed for joint attribute distribution to accommodate the sequential mastery mechanism. The expectation-maximization algorithm was employed for model estimation, and a simulation study was conducted to assess the recovery of model parameters. A set of real data was also analyzed to assess the viability of the proposed model in practice.

Inferring the Number of Attributes for the Exploratory DINA Model

Diagnostic classification models (DCMs) are widely used for providing fine-grained classification of a multidimensional collection of discrete attributes. The application of DCMs requires the specification of the latent structure in what is known as the [Formula: see text] matrix. Expert-specified [Formula: see text] matrices might be biased and result in incorrect diagnostic classifications, so a critical issue is developing methods to estimate [Formula: see text] in order to infer the relationship between latent attributes and items. Existing exploratory methods for estimating [Formula: see text] must pre-specify the number of attributes, K. We present a Bayesian framework to jointly infer the number of attributes K and the elements of [Formula: see text]. We propose the crimp sampling algorithm to transit between different dimensions of K and estimate the underlying [Formula: see text] and model parameters while enforcing model identifiability constraints. We also adapt the Indian buffet process and reversible-jump Markov chain Monte Carlo methods to estimate [Formula: see text]. We report evidence that the crimp sampler performs the best among the three methods. We apply the developed methodology to two data sets and discuss the implications of the findings for future research.

On the Sequential Hierarchical Cognitive Diagnostic Model

Model data fit plays an important role in any statistical analysis, and the primary goal is to detect the preferred model based on certain criteria. Under the cognitive diagnostic assessment (CDA) framework, a family of sequential cognitive diagnostic models (CDMs) is introduced to handle polytomously scored data, which are attained by answering constructed-response items sequentially. The presence of attribute hierarchies, which can provide useful information about the nature of attributes, will help understand the relation between attributes and response categories. This article introduces the sequential hierarchical CDM (SH-CDM), which adapts the sequential CDM to deal with attribute hierarchy. Furthermore, model fit analysis for SH-CDMs is assessed using eight model fit indices (i.e., three absolute fit indices and five relative fit indices). Two misfit sources were focused; that is, misspecifying attribute structures and misfitting processing functions. The performances of those indices were evaluated via Monte Carlo simulation studies and a real data illustration.

Measuring Skill Growth and Evaluating Change: Unconditional and Conditional Approaches to Latent Growth Cognitive Diagnostic Models

During the past decade, cognitive diagnostic models (CDMs) have become prevalent in providing diagnostic information for learning. Cognitive diagnostic models have generally focused on single cross-sectional time points. However, longitudinal assessments have been commonly used in education to assess students’ learning progress as well as evaluating intervention effects. Thus, it becomes natural to identify longitudinal growth in skills profiles mastery, which can yield meaningful inferences on learning. This study proposes longitudinal CDMs that incorporate latent growth curve modeling and covariate extensions, with the aim to measure the growth of skills mastery and to evaluate attribute-level intervention effects over time. Using real-world data, this study demonstrates applications of unconditional and conditional latent growth CDMs. Simulation studies show stable parameter recovery and classification of latent classes for different sample sizes. These findings suggest that building on the well-established growth modeling frameworks, applications of covariate-based longitudinal CDM can help understand the effect of explanatory factors and intervention on the change of attribute mastery.

A Constrained Metropolis-Hastings Robbins-Monro Algorithm for Q Matrix Estimation in DINA Models

In diagnostic classification models (DCMs), the Q matrix encodes in which attributes are required for each item. The Q matrix is usually predetermined by the researcher but may in practice be misspecified which yields incorrect statistical inference. Instead of using a predetermined Q matrix, it is possible to estimate it simultaneously with the item and structural parameters of the DCM. Unfortunately, current methods are computationally intensive when there are many attributes and items. In addition, the identification constraints necessary for DCMs are not always enforced in the estimation algorithms which can lead to non-identified models being considered. We address these problems by simultaneously estimating the item, structural and Q matrix parameters of the Deterministic Input Noisy "And" gate model using a constrained Metropolis-Hastings Robbins-Monro algorithm. Simulations show that the new method is computationally efficient and can outperform previously proposed Bayesian Markov chain Monte-Carlo algorithms in terms of Q matrix recovery, and item and structural parameter estimation. We also illustrate our approach using Tatsuoka's fraction-subtraction data and Certificate of Proficiency in English data.