1
|
Lee S, Gu Y. New Paradigm of Identifiable General-response Cognitive Diagnostic Models: Beyond Categorical Data. PSYCHOMETRIKA 2024:10.1007/s11336-024-09983-4. [PMID: 38967857 DOI: 10.1007/s11336-024-09983-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Indexed: 07/06/2024]
Abstract
Cognitive diagnostic models (CDMs) are a popular family of discrete latent variable models that model students' mastery or deficiency of multiple fine-grained skills. CDMs have been most widely used to model categorical item response data such as binary or polytomous responses. With advances in technology and the emergence of varying test formats in modern educational assessments, new response types, including continuous responses such as response times, and count-valued responses from tests with repetitive tasks or eye-tracking sensors, have also become available. Variants of CDMs have been proposed recently for modeling such responses. However, whether these extended CDMs are identifiable and estimable is entirely unknown. We propose a very general cognitive diagnostic modeling framework for arbitrary types of multivariate responses with minimal assumptions, and establish identifiability in this general setting. Surprisingly, we prove that our general-response CDMs are identifiable under Q -matrix-based conditions similar to those for traditional categorical-response CDMs. Our conclusions set up a new paradigm of identifiable general-response CDMs. We propose an EM algorithm to efficiently estimate a broad class of exponential family-based general-response CDMs. We conduct simulation studies under various response types. The simulation results not only corroborate our identifiability theory, but also demonstrate the superior empirical performance of our estimation algorithms. We illustrate our methodology by applying it to a TIMSS 2019 response time dataset.
Collapse
Affiliation(s)
| | - Yuqi Gu
- Department of Statistics, Columbia University, Room 928 SSW, 1255 Amsterdam Avenue, New York, NY, 10027, USA.
| |
Collapse
|
2
|
Liu Y, Wang W. What Can We Learn from a Semiparametric Factor Analysis of Item Responses and Response Time? An Illustration with the PISA 2015 Data. PSYCHOMETRIKA 2024; 89:386-410. [PMID: 37973773 DOI: 10.1007/s11336-023-09936-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/19/2023] [Indexed: 11/19/2023]
Abstract
It is widely believed that a joint factor analysis of item responses and response time (RT) may yield more precise ability scores that are conventionally predicted from responses only. For this purpose, a simple-structure factor model is often preferred as it only requires specifying an additional measurement model for item-level RT while leaving the original item response theory (IRT) model for responses intact. The added speed factor indicated by item-level RT correlates with the ability factor in the IRT model, allowing RT data to carry additional information about respondents' ability. However, parametric simple-structure factor models are often restrictive and fit poorly to empirical data, which prompts under-confidence in the suitablity of a simple factor structure. In the present paper, we analyze the 2015 Programme for International Student Assessment mathematics data using a semiparametric simple-structure model. We conclude that a simple factor structure attains a decent fit after further parametric assumptions in the measurement model are sufficiently relaxed. Furthermore, our semiparametric model implies that the association between latent ability and speed/slowness is strong in the population, but the form of association is nonlinear. It follows that scoring based on the fitted model can substantially improve the precision of ability scores.
Collapse
Affiliation(s)
- Yang Liu
- Department of Human Development and Quantitative Methodology, University of Maryland, 3304R Benjamin Bldg, 3942 Campus Dr, College Park, MD, 20742, USA.
| | - Weimeng Wang
- Department of Human Development and Quantitative Methodology, University of Maryland, 3304R Benjamin Bldg, 3942 Campus Dr, College Park, MD, 20742, USA
| |
Collapse
|
3
|
Xu X, Fang G, Guo J, Ying Z, Zhang S. Diagnostic Classification Models for Testlets: Methods and Theory. PSYCHOMETRIKA 2024:10.1007/s11336-024-09962-9. [PMID: 38528268 DOI: 10.1007/s11336-024-09962-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Indexed: 03/27/2024]
Abstract
Diagnostic classification models (DCMs) have seen wide applications in educational and psychological measurement, especially in formative assessment. DCMs in the presence of testlets have been studied in recent literature. A key ingredient in the statistical modeling and analysis of testlet-based DCMs is the superposition of two latent structures, the attribute profile and the testlet effect. This paper extends the standard testlet DINA (T-DINA) model to accommodate the potential correlation between the two latent structures. Model identifiability is studied and a set of sufficient conditions are proposed. As a byproduct, the identifiability of the standard T-DINA is also established. The proposed model is applied to a dataset from the 2015 Programme for International Student Assessment. Comparisons are made with DINA and T-DINA, showing that there is substantial improvement in terms of the goodness of fit. Simulations are conducted to assess the performance of the new method under various settings.
Collapse
Affiliation(s)
- Xin Xu
- Beijing Normal University, Beijing, China
| | | | - Jinxin Guo
- Minzu University of China, Beijing, China
| | | | - Susu Zhang
- University of Illinois Urbana-Champaign, Champaign, USA.
| |
Collapse
|
4
|
Guo X, Jiao Y, Huang Z, Liu T. Joint Modeling of Response Accuracy and Time in Between-Item Multidimensional Tests Based on Bi-Factor Model. Front Psychol 2022; 13:763959. [PMID: 35478766 PMCID: PMC9035624 DOI: 10.3389/fpsyg.2022.763959] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2021] [Accepted: 03/04/2022] [Indexed: 11/16/2022] Open
Abstract
With the popularity of computer-based testing (CBT), it is easier to collect item response times (RTs) in psychological and educational assessments. RTs can provide an important source of information for respondents and tests. To make full use of RTs, the researchers have invested substantial effort in developing statistical models of RTs. Most of the proposed models posit a unidimensional latent speed to account for RTs in tests. In psychological and educational tests, many tests are multidimensional, either deliberately or inadvertently. There may be general effects in between-item multidimensional tests. However, currently there exists no RT model that considers the general effects to analyze between-item multidimensional test RT data. Also, there is no joint hierarchical model that integrates RT and response accuracy (RA) for evaluating the general effects of between-item multidimensional tests. Therefore, a bi-factor joint hierarchical model using between-item multidimensional test is proposed in this study. The simulation indicated that the Hamiltonian Monte Carlo (HMC) algorithm works well in parameter recovery. Meanwhile, the information criteria showed that the bi-factor hierarchical model (BFHM) is the best fit model. This means that it is necessary to take into consideration the general effects (general latent trait) and the multidimensionality of the RT in between-item multidimensional tests.
Collapse
Affiliation(s)
- Xiaojun Guo
- School of Education Science, Gannan Normal University, Ganzhou, China
| | - Yuyue Jiao
- School of Education Science, Gannan Normal University, Ganzhou, China
| | - ZhengZheng Huang
- School of Humanities, Hubei University of Chinese Medicine, Wuhan, China
- *Correspondence: ZhengZheng Huang,
| | - TieChuan Liu
- School of Education Science, Gannan Normal University, Ganzhou, China
| |
Collapse
|
5
|
Lu J, Zhang J, Zhang Z, Xu B, Tao J. A Novel and Highly Effective Bayesian Sampling Algorithm Based on the Auxiliary Variables to Estimate the Testlet Effect Models. Front Psychol 2021; 12:509575. [PMID: 34456774 PMCID: PMC8386915 DOI: 10.3389/fpsyg.2021.509575] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2019] [Accepted: 07/06/2021] [Indexed: 11/29/2022] Open
Abstract
In this paper, a new two-parameter logistic testlet response theory model for dichotomous items is proposed by introducing testlet discrimination parameters to model the local dependence among items within a common testlet. In addition, a highly effective Bayesian sampling algorithm based on auxiliary variables is proposed to estimate the testlet effect models. The new algorithm not only avoids the Metropolis-Hastings algorithm boring adjustment the turning parameters to achieve an appropriate acceptance probability, but also overcomes the dependence of the Gibbs sampling algorithm on the conjugate prior distribution. Compared with the traditional Bayesian estimation methods, the advantages of the new algorithm are analyzed from the various types of prior distributions. Based on the Markov chain Monte Carlo (MCMC) output, two Bayesian model assessment methods are investigated concerning the goodness of fit between models. Finally, three simulation studies and an empirical example analysis are given to further illustrate the advantages of the new testlet effect model and Bayesian sampling algorithm.
Collapse
Affiliation(s)
- Jing Lu
- Key Laboratory of Applied Statistics of MOE, School of Mathematics and Statistics, Northeast Normal University, Changchun, China
| | - Jiwei Zhang
- Key Lab of Statistical Modeling and Data Analysis of Yunnan Province, School of Mathematics and Statistics, Yunnan University, Kunming, China
| | - Zhaoyuan Zhang
- Department of Statistics, School of Mathematics and Statistics, Northeast Normal University, Changchun, China
| | - Bao Xu
- Institute of Mathematics, Jilin Normal University, Siping, China
| | - Jian Tao
- Key Laboratory of Applied Statistics of MOE, School of Mathematics and Statistics, Northeast Normal University, Changchun, China
| |
Collapse
|
6
|
Yamaguchi K, Okada K. Variational Bayes Inference Algorithm for the Saturated Diagnostic Classification Model. PSYCHOMETRIKA 2020; 85:973-995. [PMID: 33420895 DOI: 10.1007/s11336-020-09739-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/03/2020] [Revised: 10/04/2020] [Accepted: 12/01/2020] [Indexed: 06/12/2023]
Abstract
Saturated diagnostic classification models (DCM) can flexibly accommodate various relationships among attributes to diagnose individual attribute mastery, and include various important DCMs as sub-models. However, the existing formulations of the saturated DCM are not better suited for deriving conditionally conjugate priors of model parameters. Because their derivation is the key in developing a variational Bayes (VB) inference algorithm, in the present study, we proposed a novel mixture formulation of saturated DCM. Based on it, we developed a VB inference algorithm of the saturated DCM that enables us to perform scalable and computationally efficient Bayesian estimation. The simulation study indicated that the proposed algorithm could recover the parameters in various conditions. It has also been demonstrated that the proposed approach is particularly suited to the case when new data become sequentially available over time, such as in computerized diagnostic testing. In addition, a real educational dataset was comparatively analyzed with the proposed VB and Markov chain Monte Carlo (MCMC) algorithms. The result demonstrated that very similar estimates were obtained between the two methods and that the proposed VB inference was much faster than MCMC. The proposed method can be a practical solution to the problem of computational load.
Collapse
Affiliation(s)
- Kazuhiro Yamaguchi
- Department of the Psychological and Quantitative Foundations, University of Iowa, 216 Lindquist Center, 240 S Madison St., Iowa City, IA, 52242, USA.
- Japan Society for the Promotion of Science, Tokyo, Japan.
| | | |
Collapse
|
7
|
Wang W, Song L, Ding S, Wang T, Gao P, Xiong J. A Semi-supervised Learning Method for Q-Matrix Specification Under the DINA and DINO Model With Independent Structure. Front Psychol 2020; 11:2120. [PMID: 33013538 PMCID: PMC7511573 DOI: 10.3389/fpsyg.2020.02120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2020] [Accepted: 07/30/2020] [Indexed: 12/02/2022] Open
Abstract
Cognitive diagnosis assessment (CDA) can be regarded as a kind of formative assessments because it is intended to promote assessment for learning and modify instruction and learning in classrooms by providing the formative diagnostic information about students' cognitive strengths and weaknesses. CDA has two phases, like a statistical pattern recognition. The first phase is feature generation, followed by classification stage. A Q-matrix, which describes the relationship between items and latent skills, corresponds to the feature generation phase in statistical pattern recognition. Feature generation is of paramount importance in any pattern recognition task. In practice, the Q-matrix is difficult to specify correctly in cognitive diagnosis and misspecification of the Q-matrix can seriously affect the accuracy of the classification of examinees. Based on the fact that any columns of a reduced Q-matrix can be expressed by the columns of a reachability R matrix under the logical OR operation, a semi-supervised learning approach and an optimal design for examinee sampling were proposed for Q-matrix specification under the conjunctive and disjunctive model with independent structure. This method only required subject matter experts specifying a R matrix corresponding to a small part of test items for the independent structure in which the R matrix is an identity matrix. Simulation and real data analysis showed that the new method with the optimal design is promising in terms of correct recovery rates of q-entries.
Collapse
Affiliation(s)
- Wenyi Wang
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| | - Lihong Song
- Elementary Education College, Jiangxi Normal University, Nanchang, China
- *Correspondence: Lihong Song
| | - Shuliang Ding
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| | - Teng Wang
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| | - Peng Gao
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| | - Jian Xiong
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| |
Collapse
|
8
|
Lee J, Jung K, Park J. Detecting Conditional Dependence Using Flexible Bayesian Latent Class Analysis. Front Psychol 2020; 11:1987. [PMID: 32903609 PMCID: PMC7438797 DOI: 10.3389/fpsyg.2020.01987] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2019] [Accepted: 07/17/2020] [Indexed: 11/13/2022] Open
Abstract
A fundamental assumption underlying latent class analysis (LCA) is that class indicators are conditionally independent of each other, given latent class membership. Bayesian LCA enables researchers to detect and accommodate violations of this assumption by estimating any number of correlations among indicators with proper prior distributions. However, little is known about how the choice of prior may affect the performance of Bayesian LCA. This article presents a Monte Carlo simulation study that investigates (1) the utility of priors in a range of prior variances (i.e., strongly non-informative to strongly informative priors) in terms of Type I error and power for detecting conditional dependence and (2) the influence of imposing approximate independence on model fit of Bayesian LCA. Simulation results favored the use of a weakly informative prior with large variance-model fit (posterior predictive p-value) was always satisfactory when the class indicators were either independent or dependent. Based on the current findings and the additional literature, this article offers methodological guidelines and suggestions for applied researchers.
Collapse
Affiliation(s)
- Jaehoon Lee
- Department of Educational Psychology and Leadershi, Texas Tech University, Lubbock, TX, United States
| | - Kwanghee Jung
- Department of Educational Psychology and Leadershi, Texas Tech University, Lubbock, TX, United States
| | - Jungkyu Park
- Department of Psychology, Kyungpook National University, Daegu, South Korea
| |
Collapse
|
9
|
De Boeck P, Jeon M. An Overview of Models for Response Times and Processes in Cognitive Tests. Front Psychol 2019; 10:102. [PMID: 30787891 PMCID: PMC6372526 DOI: 10.3389/fpsyg.2019.00102] [Citation(s) in RCA: 59] [Impact Index Per Article: 11.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2018] [Accepted: 01/14/2019] [Indexed: 11/13/2022] Open
Abstract
Response times (RTs) are a natural kind of data to investigate cognitive processes underlying cognitive test performance. We give an overview of modeling approaches and of findings obtained with these approaches. Four types of models are discussed: response time models (RT as the sole dependent variable), joint models (RT together with other variables as dependent variable), local dependency models (with remaining dependencies between RT and accuracy), and response time as covariate models (RT as independent variable). The evidence from these approaches is often not very informative about the specific kind of processes (other than problem solving, information accumulation, and rapid guessing), but the findings do suggest dual processing: automated processing (e.g., knowledge retrieval) vs. controlled processing (e.g., sequential reasoning steps), and alternative explanations for the same results exist. While it seems well-possible to differentiate rapid guessing from normal problem solving (which can be based on automated or controlled processing), further decompositions of response times are rarely made, although possible based on some of model approaches.
Collapse
Affiliation(s)
- Paul De Boeck
- Department of Psychology, Ohio State University, Columbus, OH, United States
- KU Leuven, Leuven, Belgium
| | - Minjeong Jeon
- Graduate School of Education and Information Studies, University of California, Los Angeles, Los Angeles, CA, United States
| |
Collapse
|
10
|
Zhan P, Wang WC, Jiao H, Bian Y. Probabilistic-Input, Noisy Conjunctive Models for Cognitive Diagnosis. Front Psychol 2018; 9:997. [PMID: 29962994 PMCID: PMC6010692 DOI: 10.3389/fpsyg.2018.00997] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2018] [Accepted: 05/28/2018] [Indexed: 11/13/2022] Open
Abstract
Existing cognitive diagnosis models conceptualize attribute mastery status discretely as either mastery or non-mastery. This study proposes a different conceptualization of attribute mastery as a probabilistic concept, i.e., the probability of mastering a specific attribute for a person, and developing a probabilistic-input, noisy conjunctive (PINC) model, in which the probability of mastering an attribute for a person is a parameter to be estimated from data. And a higher-order version of the PINC model is used to consider the associations among attributes. The results of simulation studies revealed a good parameter recovery for the new models using the Bayesian method. The Examination for the Certificate of Proficiency in English (ECPE) data set was analyzed to illustrate the implications and applications of the proposed models. The results indicated that PINC models had better model-data fit, smaller item parameter estimates, and more refined estimates of attribute mastery.
Collapse
Affiliation(s)
- Peida Zhan
- Collaborative Innovation Center of Assessment Toward Basic Education Quality, Beijing Normal University, Beijing, China
| | - Wen-Chung Wang
- Assessment Research Centre, The Education University of Hong Kong, Tai Po, Hong Kong
| | - Hong Jiao
- Measurement, Statistics and Evaluation, Department of Human Development and Quantitative Methodology, University of Maryland, College Park, MD, United States
| | - Yufang Bian
- Collaborative Innovation Center of Assessment Toward Basic Education Quality, Beijing Normal University, Beijing, China
| |
Collapse
|