Cognitive Diagnostic Models for Random Guessing Behaviors

A Mixture Fluency model using responses and response times with cognitive diagnosis model framework

Random guessing behaviors are frequently observed in low-stakes assessments, often attributed to factors such as test-takers lacking motivation or experiencing time constraints and fatigue. Existing research suggests that responses stemming from random guessing behaviors introduce biases into the constructs and relationships of interest. This is particularly problematic when estimating the relationship between speed and ability. This study introduces a Mixture Fluency model designed to account for random guessing behaviors while utilizing valid response accuracy and response time to uncover students' latent attribute profiles. The model directly addresses a limitation present in the Fluency cognitive diagnostic model (Wang & Chen, Psychometrika, 85, 600-629, (2020), which assumes that test-takers consistently employ solution behaviors when answering questions. To investigate the effectiveness of the proposed Mixture Fluency model, we conducted a simulation study encompassing various simulation conditions. Results from this study not only confirm the model's ability to detect potential random guessing behaviors but also demonstrate its capacity to enhance the inference of targeted latent constructs within the assessment. Additionally, we showcase the practical utility of the proposed model through an application to real data.

Testing Replicability and Generalizability of the Time on Task Effect

The time on task (ToT) effect describes the relationship of the time spent on a cognitive task and the probability of successful task completion. The effect has been shown to vary in size and direction across tests and even within tests, depending on the test taker and item characteristics. Specifically, investing more time has a positive effect on response accuracy for difficult items and low ability test-takers, but a negative effect for easy items and high ability test-takers. The present study sought to test the replicability of this result pattern of the ToT effect across samples independently drawn from the same populations of persons and items. Furthermore, its generalizability was tested in terms of differential correlations across ability tests. To this end, ToT effects were estimated for three different reasoning tests and one test measuring natural sciences knowledge in 10 comparable subsamples with a total N = 2640. Results for the subsamples were highly similar, demonstrating that ToT effects are estimated with sufficient reliability. Generally, faster answers tended to be more accurate, suggesting a relatively effortless processing style. However, with increasing item difficulty and decreasing person ability, the effect flipped to the opposite direction, i.e., higher accuracy with longer processing times. The within-task moderation of the ToT effect can be reconciled with an account on effortful processing or cognitive load. By contrast, the generalizability of the ToT effect across different tests was only moderate. Cross-test relations were stronger in relative terms if performance in the respective tasks was more strongly related. This suggests that individual differences in the ToT effect depend on test characteristics such as their reliabilities but also similarities and differences of their processing requirements.

A new person-fit method based on machine learning in CDM in education

Cognitive diagnosis models have become popular in educational assessment and are used to provide more individualized feedback about a student's specific strengths and weaknesses than traditional total scores. However, if the testing data are contaminated by certain biases or aberrant response patterns, such predictions may not be accurate. The current research objective is to develop a new person-fit method that is based on machine learning and improves the functionality of existing person-fit methods. Various simulations were designed under three aberrant conditions: cheating, sleeping and random guessing. Simulation results showed that the new method was more powerful and effective than previous methods, especially for short-length tests.

Test Assembly for Cognitive Diagnosis Using Mixed-Integer Linear Programming

One purpose of cognitive diagnostic model (CDM) is designed to make inferences about unobserved latent classes based on observed item responses. A heuristic for test construction based on the CDM information index (CDI) proposed by Henson and Douglas (2005) has a far-reaching impact, but there are still many shortcomings. He and other researchers had also proposed new methods to improve or overcome the inherent shortcomings of the CDI test assembly method. In this study, one test assembly method of maximizing the minimum inter-class distance is proposed by using mixed-integer linear programming, which aims to overcome the shortcomings that the CDI method is limited to summarize the discriminating power of each item into a single CDI index while neglecting the discriminating power for each pair of latent classes. The simulation results show that compared with the CDI test assembly and random test assembly, the new test assembly method performs well and has the highest accuracy rate in terms of pattern and attributes correct classification rates. Although the accuracy rate of the new method is not very high under item constraints, it is still higher than the CDI test assembly with the same constraints.