Adapting cognitive diagnosis computerized adaptive testing item selection rules to traditional item response theory

Novel item selection strategies for cognitive diagnostic computerized adaptive testing: A heuristic search framework

The computerized adaptive form of cognitive diagnostic testing, CD-CAT, has gained increasing attention in the domain of personalized measurements for its ability to categorize individual mastery status of fine-grained attributes more accurately and efficiently through administering items tailored to one's ability progressively. How to select the next item based on previous response(s) is crucial for the success of CD-CAT. Previous item selection strategies for CD-CAT have often followed a greedy or semi-greedy approach, which makes it difficult to strike a balance between diagnostic performance and item bank utilization. To address this issue, this study takes a graph perspective and transforms the item selection problem in CD-CAT into a path-searching problem, in which paths refer to possible test construction and nodes refer to individual items. A heuristic function is defined to predict the prospect of a path, indicating how well the corresponding test can diagnose the current examinee. Two search mechanisms with different biases towards item exposure control are proposed to approximate the optimal path with the best prospect. The first unused item on the resulting path is selected as the next item. The above components compose a novel CD-CAT item selection framework based on heuristic search. Simulation studies are conducted under a variety of conditions regarding bank designs, bank-quality conditions, and testing scenarios. The results are compared with different types of classic item selection strategies in CD-CAT, showing that the proposed framework can enhance bank utilization at a smaller cost of diagnostic performance.

Development of an Adaptive Computer-Aided Soft Sensor Diagnosis System for Assessment of Executive Functions

The main objective of the present study is to highlight the role of technological (soft sensor) methodologies in the assessment of the neurocognitive dysfunctions specific to neurodevelopmental disorders (for example, autism spectrum disorder (ASD), attention deficit hyperactivity disorder (ADHD), and specific learning disorder). In many cases neurocognitive dysfunctions can be detected in neurodevelopmental disorders, some of them having a well-defined syndrome-specific clinical pattern. A number of evidence-based neuropsychological batteries are available for identifying these domain-specific functions. Atypical patterns of cognitive functions such as executive functions are present in almost all developmental disorders. In this paper, we present a novel adaptation of the Tower of London Test, a widely used neuropsychological test for assessing executive functions (in particular planning and problem-solving). Our version, the Tower of London Adaptive Test, is based on computer adaptive test theory (CAT). Adaptive testing using novel algorithms and parameterized task banks allows the immediate evaluation of the participant's response which in turn determines the next task's difficulty level. In this manner, the subsequent item is adjusted to the participant's estimated capability. The adaptive procedure enhances the original test's diagnostic power and sensitivity. By measuring the targeted cognitive capacity and its limitations more precisely, it leads to more accurate diagnoses. In some developmental disorders (e.g., ADHD, ASD) it could be very useful in improving the diagnosis, planning the right interventions, and choosing the most suitable assistive digital technological service.

cdcatR: An R Package for Cognitive Diagnostic Computerized Adaptive Testing

Cognitive diagnosis models (CDMs) are confirmatory latent class models that provide fine-grained information about skills and cognitive processes. These models have gained attention in the last few years because of their usefulness in educational and psychological settings. Recently, numerous developments have been made to allow for the implementation of cognitive diagnosis computerized adaptive testing (CD-CAT). Despite methodological advances, CD-CAT applications are still scarce. To facilitate research and the emergence of empirical applications in this area, we have developed the cdcatR package for R software. The purpose of this document is to illustrate the different functions included in this package. The package includes functionalities for data generation, model selection based on relative fit information, implementation of several item selection rules (including item exposure control), and CD-CAT performance evaluation in terms of classification accuracy, item exposure, and test length. In conclusion, an R package is made available to researchers and practitioners that allows for an easy implementation of CD-CAT in both simulation and applied studies. Ultimately, this is expected to facilitate the development of empirical applications in this area.

Improving Accuracy and Usage by Correctly Selecting: The Effects of Model Selection in Cognitive Diagnosis Computerized Adaptive Testing

Decisions on how to calibrate an item bank might have major implications in the subsequent performance of the adaptive algorithms. One of these decisions is model selection, which can become problematic in the context of cognitive diagnosis computerized adaptive testing, given the wide range of models available. This article aims to determine whether model selection indices can be used to improve the performance of adaptive tests. Three factors were considered in a simulation study, that is, calibration sample size, Q-matrix complexity, and item bank length. Results based on the true item parameters, and general and single reduced model estimates were compared to those of the combination of appropriate models. The results indicate that fitting a single reduced model or a general model will not generally provide optimal results. Results based on the combination of models selected by the fit index were always closer to those obtained with the true item parameters. The implications for practical settings include an improvement in terms of classification accuracy and, consequently, testing time, and a more balanced use of the item bank. An R package was developed, named cdcatR, to facilitate adaptive applications in this context.

A new approach to assessing shyness of college students using computerized adaptive testing: CAT-Shyness

Assessing shy symptoms via computerized adaptive testing (CAT) provides greater measurement precision coupled with a lower test burden compared to conventional tests. The computerized adaptive test for shyness (CAT-Shyness) was developed based on a large sample of 1400 participants from China. Item bank development included the investigation of unidimensionality, local independence, and exploration of differential item functioning (DIF). CAT simulations based on the real data were carried out to investigate the reliability, validity, and predicted utility (sensitivity and specificity) of the CAT-Shyness. The CAT-Shyness item bank was successfully built and proved to have excellent psychometric properties: high content validity, unidimensionality, local independence, and no DIF. The CAT simulations needed 14 items to achieve a high measurement precision with a reliability of .9. Moreover, the results revealed that the proposed CAT-Shyness had acceptable and reasonable marginal reliability, criterion-related validity, and sensitivity and specificity. It not only had acceptable psychometric properties, but also had a shorter but efficient assessment of shyness, which can save significant test time and reduce the test burden for individuals with less information loss.