1
|
Kreitchmann RS, de la Torre J, Sorrel MA, Nájera P, Abad FJ. Improving reliability estimation in cognitive diagnosis modeling. Behav Res Methods 2023; 55:3446-3460. [PMID: 36127563 PMCID: PMC10615987 DOI: 10.3758/s13428-022-01967-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/24/2022] [Indexed: 11/08/2022]
Abstract
Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM reliability indices are based on the posterior probabilities of the estimated attribute profiles. These posteriors are traditionally computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities. This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certificate of Proficiency in English data was analyzed. The effect of sample size was studied by sampling subsets of subjects from the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities, whereas the MI procedure offered more accurate results. Accordingly, practitioners in small educational or clinical settings should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for the MI procedure are made available.
Collapse
Affiliation(s)
- Rodrigo Schames Kreitchmann
- Department of Social Psychology and Methodology, Faculty of Psychology, Universidad Autónoma de Madrid, Calle Iván Pavlov, 6, Ciudad Universitaria de Cantoblanco, 28049, Madrid, Spain.
- School of Science and Technology, IE University, Paseo de la Castellana, 259, Madrid, Spain.
| | - Jimmy de la Torre
- Faculty of Education, The University of Hong Kong, Pokfulam Road, Pok Fu Lam, Hong Kong
| | - Miguel A Sorrel
- Department of Social Psychology and Methodology, Faculty of Psychology, Universidad Autónoma de Madrid, Calle Iván Pavlov, 6, Ciudad Universitaria de Cantoblanco, 28049, Madrid, Spain
| | - Pablo Nájera
- Department of Social Psychology and Methodology, Faculty of Psychology, Universidad Autónoma de Madrid, Calle Iván Pavlov, 6, Ciudad Universitaria de Cantoblanco, 28049, Madrid, Spain
| | - Francisco J Abad
- Department of Social Psychology and Methodology, Faculty of Psychology, Universidad Autónoma de Madrid, Calle Iván Pavlov, 6, Ciudad Universitaria de Cantoblanco, 28049, Madrid, Spain
| |
Collapse
|
2
|
Cognitive diagnostic assessment with ordered multiple-choice items for word problems involving ‘Time’. CURRENT PSYCHOLOGY 2022. [DOI: 10.1007/s12144-022-02965-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
3
|
Qiao X, Jiao H. Explanatory Cognitive Diagnostic Modeling Incorporating Response Times. JOURNAL OF EDUCATIONAL MEASUREMENT 2022. [DOI: 10.1111/jedm.12306] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
4
|
Wang W, Zheng J, Song L, Tu Y, Gao P. Test Assembly for Cognitive Diagnosis Using Mixed-Integer Linear Programming. Front Psychol 2021; 12:623077. [PMID: 33633649 PMCID: PMC7901876 DOI: 10.3389/fpsyg.2021.623077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2020] [Accepted: 01/11/2021] [Indexed: 11/13/2022] Open
Abstract
One purpose of cognitive diagnostic model (CDM) is designed to make inferences about unobserved latent classes based on observed item responses. A heuristic for test construction based on the CDM information index (CDI) proposed by Henson and Douglas (2005) has a far-reaching impact, but there are still many shortcomings. He and other researchers had also proposed new methods to improve or overcome the inherent shortcomings of the CDI test assembly method. In this study, one test assembly method of maximizing the minimum inter-class distance is proposed by using mixed-integer linear programming, which aims to overcome the shortcomings that the CDI method is limited to summarize the discriminating power of each item into a single CDI index while neglecting the discriminating power for each pair of latent classes. The simulation results show that compared with the CDI test assembly and random test assembly, the new test assembly method performs well and has the highest accuracy rate in terms of pattern and attributes correct classification rates. Although the accuracy rate of the new method is not very high under item constraints, it is still higher than the CDI test assembly with the same constraints.
Collapse
Affiliation(s)
- Wenyi Wang
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| | - Juanjuan Zheng
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| | - Lihong Song
- School of Education, Jiangxi Normal University, Nanchang, China
| | - Yukun Tu
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| | - Peng Gao
- School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China
| |
Collapse
|