The Block Item Pocket Method for Reviewable Multidimensional Computerized Adaptive Testing

Modeling item revisiting behavior in computer-based testing: Exploring the effect of item revisitations as collateral information

Item revisiting behavior is one of the most frequently occurring test-taking strategies, and it can decrease test anxiety and improve test validity. Examinees either confirm the initial answers due to persistence of their beliefs or change to different answers after careful rethought on each part of the questions. Item revisiting sequences as collateral information reveal the examinees' underlying psychological processes, such as motivation, effort, and engagement, which supports policy makers in taking further steps to facilitate instructions for the examinees. Item revisiting behavior is commonly correlated with the latent traits of examinees, and it needs to be properly analyzed in order to make valid statistical inference. In this paper, we proposed a novel item revisiting model, in which a monotonicity assumption is considered based on the observation that examinees are more likely to revisit the current item if more revisiting behavior occurs previously. Three simulation studies were conducted: (1) to evaluate the performance of the proposed Bayesian estimation algorithm for the new model; (2) to show that ignoring item revisiting sequences induces biased parameter estimates; (3) to assess the model fit of the proposed model with the ignorable and nonignorable item revisiting behavior assumptions. The results indicate that item revisiting behavior can be effectively utilized in conjunction with responses and response times to improve parameter estimation precision. A real data example is provided to illustrate the application of the proposed model.

Item Parameter Estimation in Multistage Designs: A Comparison of Different Estimation Approaches for the Rasch Model

There is some debate in the psychometric literature about item parameter estimation in multistage designs. It is occasionally argued that the conditional maximum likelihood (CML) method is superior to the marginal maximum likelihood method (MML) because no assumptions have to be made about the trait distribution. However, CML estimation in its original formulation leads to biased item parameter estimates. Zwitser and Maris (2015, Psychometrika) proposed a modified conditional maximum likelihood estimation method for multistage designs that provides practically unbiased item parameter estimates. In this article, the differences between different estimation approaches for multistage designs were investigated in a simulation study. Four different estimation conditions (CML, CML estimation with the consideration of the respective MST design, MML with the assumption of a normal distribution, and MML with log-linear smoothing) were examined using a simulation study, considering different multistage designs, number of items, sample size, and trait distributions. The results showed that in the case of the substantial violation of the normal distribution, the CML method seemed to be preferable to MML estimation employing a misspecified normal trait distribution, especially if the number of items and sample size increased. However, MML estimation using log-linear smoothing lea to results that were very similar to the CML method with the consideration of the respective MST design.