A generalized longitudinal mixture IRT model for measuring differential growth in learning environments

The Impact of Markov Chain Convergence on Estimation of Mixture IRT Model Parameters

A nonconverged Markov chain can potentially lead to invalid inferences about model parameters. The purpose of this study was to assess the effect of a nonconverged Markov chain on the estimation of parameters for mixture item response theory models using a Markov chain Monte Carlo algorithm. A simulation study was conducted to investigate the accuracy of model parameters estimated with different degree of convergence. Results indicated the accuracy of the estimated model parameters for the mixture item response theory models decreased as the number of iterations of the Markov chain decreased. In particular, increasing the number of burn-in iterations resulted in more accurate estimation of mixture IRT model parameters. In addition, the different methods for monitoring convergence of a Markov chain resulted in different degrees of convergence despite almost identical accuracy of estimation.

Psychometrics of MOOCs: Measuring Learners' Proficiency

Massive open online courses (MOOCs) generate learners' performance data that can be used to understand learners' proficiency and to improve their efficiency. However, the approaches currently used, such as assessing the proportion of correct responses in assessments, are oversimplified and may lead to poor conclusions and decisions because they do not account for additional information on learner, content, and context. There is a need for theoretically grounded data-driven explainable educational measurement approaches for MOOCs. In this conceptual paper, we try to establish a connection between psychometrics, a scientific discipline concerned with techniques for educational and psychological measurement, and MOOCs. First, we describe general principles of traditional measurement of learners' proficiency in education. Second, we discuss qualities of MOOCs which hamper direct application of approaches based on these general principles. Third, we discuss recent developments in measuring proficiency that may be relevant for analyzing MOOC data. Finally, we draw directions in psychometric modeling that might be interesting for future MOOC research.

Measuring growth in students' proficiency in MOOCs: Two component dynamic extensions for the Rasch model

Massive open online courses (MOOCs) are increasingly popular among students of various ages and at universities around the world. The main aim of a MOOC is growth in students' proficiency. That is why students, professors, and universities are interested in the accurate measurement of growth. Traditional psychometric approaches based on item response theory (IRT) assume that a student's proficiency is constant over time, and therefore are not well suited for measuring growth. In this study we sought to go beyond this assumption, by (a) proposing to measure two components of growth in proficiency in MOOCs; (b) applying this idea in two dynamic extensions of the most common IRT model, the Rasch model; (c) illustrating these extensions through analyses of logged data from three MOOCs; and (d) checking the quality of the extensions using a cross-validation procedure. We found that proficiency grows both across whole courses and within learning objectives. In addition, our dynamic extensions fit the data better than does the original Rasch model, and both extensions performed well, with an average accuracy of .763 in predicting students' responses from real MOOCs.

An explanatory item response theory method for alleviating the cold-start problem in adaptive learning environments

Electronic learning systems have received increasing attention because they are easily accessible to many students and are capable of personalizing the learning environment in response to students' learning needs. To that end, using fast and flexible algorithms that keep track of the students' ability change in real time is desirable. Recently, the Elo rating system (ERS) has been applied and studied in both research and practical settings (Brinkhuis & Maris, 2009; Klinkenberg, Straatemeier, & van der Maas in Computers & Education, 57, 1813-1824, 2011). However, such adaptive algorithms face the cold-start problem, defined as the problem that the system does not know a new student's ability level at the beginning of the learning stage. The cold-start problem may also occur when a student leaves the e-learning system for a while and returns (i.e., a between-session period). Because external effects could influence the student's ability level during the period, there is again much uncertainty about ability level. To address these practical concerns, in this study we propose alternative approaches to cold-start issues in the context of the e-learning environment. Particularly, we propose making the ERS more efficient by using an explanatory item response theory modeling to estimate students' ability levels on the basis of their background information and past trajectories of learning. A simulation study was conducted under various conditions, and the results showed that the proposed approach substantially reduces ability estimation errors. We illustrate the approach using real data from a popular learning platform.

A Solution to the Measurement Problem in the Idiographic Approach Using Computer Adaptive Practicing

Molenaar's manifesto on psychology as idiographic science (Molenaar, 2004) brought the N = 1 times series perspective firmly to the attention of developmental scientists. The rich intraindividual variation in complex developmental processes requires the study of these processes at the level of the individual. Yet, the idiographic approach is all but easy in practical research. One major limitation is the collection of short interval times series of high quality data on developmental processes. In this paper, we present a novel measurement approach to this problem. We developed an online practice and monitoring system which is now used by thousands of Dutch primary school children on a daily or weekly basis, providing a new window on cognitive development. We will introduce the origin of this new instrument, called Math Garden, explain its setup, and present and discuss ways to analyze children's individual developmental pathways.

Cognitive Analysis of Educational Games: The Number Game

We analyze the cognitive strategies underlying performance in the Number task, a Math game that requires both arithmetic fluency and mathematical creativity. In this game all elements in a set of numbers (for instance, 2, 5, 9) have to be used precisely once to create a target number (for instance, 27) with basic arithmetic operations (solution: [5-2] × 9). We argue that some instances of this game are NP complete, by showing its relation to the well-known Partition problem. We propose heuristics based on the distinction in forward and backward reasoning. The Number Game is part of Math Garden, a popular online educational platform for practicing and monitoring math skills using innovations in computerized adaptive testing. These educational games generate enormous amounts of rich data on children's cognitive development. We found converging evidence for the use of forward proximity heuristics in the data of Math Garden, consisting of more than 20 million answers to 1,700 items. Item difficulties and the structure of correct answers were analyzed.

The Guttman errors as a tool for response shift detection at subgroup and item levels

PURPOSE

Statistical methods for identifying response shift (RS) at the individual level could be of great practical value in interpreting change in PRO data. Guttman errors (GE) may help to identify discrepancies in respondent's answers to items compared to an expected response pattern and to identify subgroups of patients that are more likely to present response shift. This study explores the benefits of using a GE-based method for RS detection at the subgroup and item levels.

METHODS

The analysis was performed on the SatisQoL study. The number of GE was determined for each individual at each time of measurement (at baseline T0 and 6 months after discharge M6). Individuals showing discrepancies (with many GE) were suspected to interpret the items differently from the majority of the sample. Patients having a large number of GE at M6 only and not at T0 were assumed to present RS. Patients having a small number of GE at T0 and M6 were assumed to present no RS. The RespOnse Shift ALgorithm in Item response theory (ROSALI) was then applied on the whole sample and on both groups.

RESULTS

Different types of RS (non-uniform recalibration, reprioritization) were more prevalent in the group composed of patients assumed to present RS based on GE. On the opposite, no RS was detected on patients having few GE.

CONCLUSIONS

Guttman errors and item response theory models seem to be relevant tools to discriminate individuals affected by RS from the others at the item level.