What Can We Learn from a Semiparametric Factor Analysis of Item Responses and Response Time? An Illustration with the PISA 2015 Data

It is widely believed that a joint factor analysis of item responses and response time (RT) may yield more precise ability scores that are conventionally predicted from responses only. For this purpose, a simple-structure factor model is often preferred as it only requires specifying an additional measurement model for item-level RT while leaving the original item response theory (IRT) model for responses intact. The added speed factor indicated by item-level RT correlates with the ability factor in the IRT model, allowing RT data to carry additional information about respondents' ability. However, parametric simple-structure factor models are often restrictive and fit poorly to empirical data, which prompts under-confidence in the suitablity of a simple factor structure. In the present paper, we analyze the 2015 Programme for International Student Assessment mathematics data using a semiparametric simple-structure model. We conclude that a simple factor structure attains a decent fit after further parametric assumptions in the measurement model are sufficiently relaxed. Furthermore, our semiparametric model implies that the association between latent ability and speed/slowness is strong in the population, but the form of association is nonlinear. It follows that scoring based on the fitted model can substantially improve the precision of ability scores.

Test Format and Local Dependence of Items Revisited: A Case of Two Vocabulary Levels Tests

Local item dependence (LID) is one of the most critical assumption in the Rasch model when it comes to the validity of a test. As the field of vocabulary assessment is calling for more clarity and validity for vocabulary tests, such assumption becomes more important than ever. The article offers a Rasch-based investigation into the issue of LID with the focus on the two popular formats of Vocabulary Levels Tests (VLT): multiple-choice and matching. A Listening Vocabulary Levels Test (LVLT) and an Updated Vocabulary Levels Test (UVLT) were given to a single cohort of 311 university students in an English as a Foreign Language (EFL) context. The analyses of raw score and standardized residuals correlations were conducted. The findings found no relationship between the 4-option, multiple-choice format of the LVLT and item local dependence. However, results from score and standardized residuals correlations analyses showed a strong link between the 3-item-per-cluster, matching format and item local dependence. The study calls for greater attention to the format of future vocabulary tests and support the use of meaning-recall formats in vocabulary testing.

Testing the Local Independence Assumption of the Rasch Model With Q 3-Based Nonparametric Model Tests

Local independence is a central assumption of commonly used item response theory models. Violations of this assumption are usually tested using test statistics based on item pairs. This study presents two quasi-exact tests based on the Q 3 statistic for testing the hypothesis of local independence in the Rasch model. The proposed tests do not require the estimation of item parameters and can also be applied to small data sets. The authors evaluate the tests with three simulation studies. Their results indicate that the quasi-exact tests hold their alpha level under the Rasch model and have higher power against different forms of local dependence than several alternative parametric and nonparametric model tests for local independence.

On a Generalization of Local Independence in Item Response Theory Based on Knowledge Space Theory

Knowledge space theory (KST) structures are introduced within item response theory (IRT) as a possible way to model local dependence between items. The aim of this paper is threefold: firstly, to generalize the usual characterization of local independence without introducing new parameters; secondly, to merge the information provided by the IRT and KST perspectives; and thirdly, to contribute to the literature that bridges continuous and discrete theories of assessment. In detail, connections are established between the KST simple learning model (SLM) and the IRT General Graded Response Model, and between the KST Basic Local Independence Model and IRT models in general. As a consequence, local independence is generalized to account for the existence of prerequisite relations between the items, IRT models become a subset of KST models, IRT likelihood functions can be generalized to broader families, and the issues of local dependence and dimensionality are partially disentangled. Models are discussed for both dichotomous and polytomous items and conclusions are drawn on their interpretation. Considerations on possible consequences in terms of model identifiability and estimation procedures are also provided.

A diagnostic procedure to detect departures from local independence in item response theory models

Item response theory (IRT) is a widely used measurement model. When considering its use in education, health outcomes, and psychology, it is likely to be one of the most impactful psychometric models in existence. IRT has many advantages over classical test theory-based measurement models. For these advantages to hold in practice, strong assumptions must be satisfied. One of these assumptions, local independence, is the focus of the work described here. Local independence is the assumption that, conditional on the latent variable(s), item responses are unrelated to one another (i.e., independent). Stated another way, local independence implies that the only thing causing items to covary is the modeled latent variable(s). Violations of this assumption, quite aptly titled local dependence, can have serious consequences for the estimated parameters. A new diagnostic is proposed, based on parameter stability in an item-level jackknife resampling procedure. We review the ideas underlying the new diagnostic and how it is computed before covering some simulated and real examples demonstrating its effectiveness. (PsycINFO Database Record

Bayes Factor Covariance Testing in Item Response Models

Two marginal one-parameter item response theory models are introduced, by integrating out the latent variable or random item parameter. It is shown that both marginal response models are multivariate (probit) models with a compound symmetry covariance structure. Several common hypotheses concerning the underlying covariance structure are evaluated using (fractional) Bayes factor tests. The support for a unidimensional factor (i.e., assumption of local independence) and differential item functioning are evaluated by testing the covariance components. The posterior distribution of common covariance components is obtained in closed form by transforming latent responses with an orthogonal (Helmert) matrix. This posterior distribution is defined as a shifted-inverse-gamma, thereby introducing a default prior and a balanced prior distribution. Based on that, an MCMC algorithm is described to estimate all model parameters and to compute (fractional) Bayes factor tests. Simulation studies are used to show that the (fractional) Bayes factor tests have good properties for testing the underlying covariance structure of binary response data. The method is illustrated with two real data studies.

Controlling response dependence in the measurement of change using the Rasch model

The advantages of using person location estimates from the Rasch model over raw scores for the measurement of change using a common test include the linearization of scores and the automatic handling of statistical properties of repeated measurements. However, the application of the model requires that the responses to the items are statistically independent in the sense that the specific responses to the items on the first time of testing do not affect the responses at a second time. This requirement implies that the responses to the items at both times of assessment are governed only by the invariant location parameters of the items at the two times of testing and the location parameters of each person each time. A specific form of dependence that is pertinent when the same items are used is when the observed response to an item at the second time of testing is affected by the response to the same item at the first time, a form of dependence which has been referred to as response dependence. This paper presents the logic of applying the Rasch model to quantify, control and remove the effect of response dependence in the measurement of change when the same items are used on two occasions. The logic is illustrated with four sets of simulation studies with dichotomous items and with a small example of real data. It is shown that the presence of response dependence can reduce the evidence of change, a reduction which may impact interpretations at the individual, research, and policy levels.

A Q3-Based Permutation Test for Assessing Local Independence

Item response theory (IRT) is a powerful statistical methodology used in the analysis of psychological and educational assessments. IRT rests on three fundamental assumptions about the data, including local independence, which means that after accounting for the latent trait(s) being measured, the item responses are independent of one another. Traditionally, this assumption is assessed using Yen's Q₃ statistic. However, Q₃ does not have a known sampling distribution, and thus, it is typically used in a descriptive fashion, such that values larger than an arbitrary cut-value (e.g., 0.2) indicate the presence of local dependence. The current study introduces a formal test of the null hypothesis that for a given item pair Q₃ is 0, based on permutation test methodology. A small simulation study carried out to assess the Type I error and power rates of the Q₃ permutation test found that this new statistic maintains good Type I error control, while also yielding power for detecting local dependence at a rate higher than that associated with the use of the 0.2 cut-value.

Comparing score tests and other local dependence diagnostics for the graded response model

Score tests for identifying locally dependent item pairs have been proposed for binary item response models. In this article, both the bifactor and the threshold shift score tests are generalized to the graded response model. For the bifactor test, the generalization is straightforward; it adds one secondary dimension associated only with one pair of items. For the threshold shift test, however, multiple generalizations are possible: in particular, conditional, uniform, and linear shift tests are discussed in this article. Simulation studies show that all of the score tests have accurate Type I error rates given large enough samples, although their small-sample behaviour is not as good as that of Pearson's Χ2 and M2 as proposed in other studies for the purpose of local dependence (LD) detection. All score tests have the highest power to detect the LD which is consistent with their parametric form, and in this case they are uniformly more powerful than Χ2 and M2 ; even wrongly specified score tests are more powerful than Χ2 and M2 in most conditions. An example using empirical data is provided for illustration.

Causality, mediation and time: a dynamic viewpoint

Time dynamics are often ignored in causal modelling. Clearly, causality must operate in time and we show how this corresponds to a mechanistic, or system, understanding of causality. The established counterfactual definitions of direct and indirect effects depend on an ability to manipulate the mediator which may not hold in practice, and we argue that a mechanistic view may be better. Graphical representations based on local independence graphs and dynamic path analysis are used to facilitate communication as well as providing an overview of the dynamic relations 'at a glance'. The relationship between causality as understood in a mechanistic and in an interventionist sense is discussed. An example using data from the Swiss HIV Cohort Study is presented.