A nonlinear mixed model framework for item response theory

Modeling item-level heterogeneous treatment effects: A tutorial with the glmer function from the lme4 package in R

Recent advancements in education scholarship have introduced Item Response Theory (IRT) models to address treatment heterogeneity at the assessment item level. These models for item-level heterogeneous treatment effects (IL-HTE) enable detailed analyses of treatments that may have varying impacts on individual items within an assessment. This article offers a comprehensive tutorial for applied researchers interested in implementing IL-HTE analysis in R, utilizing the lme4 package. Using empirical data from a second-grade reading comprehension assessment as a running example, this tutorial emphasizes model-building strategies, interpretation techniques, visualization methods, and extensions. By following this tutorial, researchers will gain practical insights into utilizing IL-HTE analysis for enhanced understanding and interpretation of treatment effects at the item level.

An Explanatory Multidimensional Random Item Effects Rating Scale Model

Random item effects item response theory (IRT) models, which treat both person and item effects as random, have received much attention for more than a decade. The random item effects approach has several advantages in many practical settings. The present study introduced an explanatory multidimensional random item effects rating scale model. The proposed model was formulated under a novel parameterization of the nominal response model (NRM), and allows for flexible inclusion of person-related and item-related covariates (e.g., person characteristics and item features) to study their impacts on the person and item latent variables. A new variant of the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm designed for latent variable models with crossed random effects was applied to obtain parameter estimates for the proposed model. A preliminary simulation study was conducted to evaluate the performance of the MH-RM algorithm for estimating the proposed model. Results indicated that the model parameters were well recovered. An empirical data set was analyzed to further illustrate the usage of the proposed model.

A Bayesian Approach Towards Missing Covariate Data in Multilevel Latent Regression Models

The measurement of latent traits and investigation of relations between these and a potentially large set of explaining variables is typical in psychology, economics, and the social sciences. Corresponding analysis often relies on surveyed data from large-scale studies involving hierarchical structures and missing values in the set of considered covariates. This paper proposes a Bayesian estimation approach based on the device of data augmentation that addresses the handling of missing values in multilevel latent regression models. Population heterogeneity is modeled via multiple groups enriched with random intercepts. Bayesian estimation is implemented in terms of a Markov chain Monte Carlo sampling approach. To handle missing values, the sampling scheme is augmented to incorporate sampling from the full conditional distributions of missing values. We suggest to model the full conditional distributions of missing values in terms of non-parametric classification and regression trees. This offers the possibility to consider information from latent quantities functioning as sufficient statistics. A simulation study reveals that this Bayesian approach provides valid inference and outperforms complete cases analysis and multiple imputation in terms of statistical efficiency and computation time involved. An empirical illustration using data on mathematical competencies demonstrates the usefulness of the suggested approach.

Bayesian explanatory additive IRT models

In this article we extend the framework of explanatory mixed IRT models to a more general class called explanatory additive IRT models. We do this by augmenting the linear predictors in terms of smooth functions. This development offers many new modeling options such as the inclusion of nonlinear covariate effects, the specification of various temporal and spatial dependency patterns, and parameter partitioning across covariates. We use integrated nested Laplace approximation (INLA) for accurate and computationally efficient estimation of the parameters. Uninformative, weakly informative, and informative prior settings for the hyperparameters are discussed. Running time experiments and Monte Carlo parameter recovery simulations are performed in order to study the accuracy and computational efficiency of INLA when applied to the proposed explanatory additive IRT model class. Using a real-life dataset, a variety of application scenarios is explored, and the results are compared with classical maximum likelihood estimation when possible. R code is included in the supplemental materials to allow readers to fully reproduce the examples computed in the paper.

A Bayesian nonparametric approach to dynamic item-response modeling: An application to the GUSTO cohort study

Statistical analysis of questionnaire data is often performed employing techniques from item-response theory. In this framework, it is possible to differentiate respondent profiles and characterize the questions (items) included in the questionnaire via interpretable parameters. These models are often crosssectional and aim at evaluating the performance of the respondents. The motivating application of this work is the analysis of psychometric questionnaires taken by a group of mothers at different time points and by their children at one later time point. The data are available through the GUSTO cohort study. To this end, we propose a Bayesian semiparametric model and extend the current literature by: (i) introducing temporal dependence among questionnaires taken at different time points; (ii) jointly modeling the responses to questionnaires taken from different, but related, groups of subjects (in our case mothers and children), introducing a further dependency structure and therefore sharing of information; (iii) allowing clustering of subjects based on their latent response profile. The proposed model is able to identify three main groups of mother/child pairs characterized by their response profiles. Furthermore, we report an interesting maternal reporting bias effect strongly affecting the clustering structure of the mother/child dyads.

Psychometric evaluation of the Southampton hand assessment procedure (SHAP) in a sample of upper limb prosthesis users

BACKGROUND

The 26-item Southampton Hand Assessment Protocol (SHAP) is a test of prosthetic hand function that generates an Index of Functionality (IOF), and prehensile pattern (PP) scores. Prior researchers identified potential issues in SHAP scoring, proposing alternative scoring methods (LIF and W-LIF).

STUDY DESIGN

Cross-sectional study.

PURPOSE

Evaluate the psychometric properties of the SHAP IOF, LIF, and W-LIF and PP scores and develop the Prosthesis Index of Functionality (P-IOF).

METHODS

We examined item completion, floor andceiling effects, concurrent, discriminant, construct and structural validity. The P-IOF used increased boundary limits and information from item completion and completion time. Calibration used a nonlinear mixed model. Scores were estimated using maximum a posteriori Bayesian estimation. Mixed integer linear programing (MILP) informed development of a shorter measure. Validity analyses were repeated using the P-IOF.

RESULTS

126 persons, mean age 57 (sd 15.8), 69% with transradial amputation were included. Floors effects were observed in 18.3%-19.1% for the IOF, LIF, and W-LIF. Ten items were not completed by >15% of participants. Boundary limits were problematic for all but 1 item. Correlations with dexterity measures were strong (r =  0.54-0.73). Scores differed by amputation level (p > .0001). Factor analysis did not support use of PP scores. The P-IOF used expanded boundary limits to decrease floor effects. MILP identified 10 items that could be dropped. The 26-item P-IOF and 16-item P-IOF had reduced floor effects (<7.5%), strong evidence of concurrent and discriminant validity, and construct validity. P-IOF reduced administrative burden by 9.5 (sd 5.6) minutes.

DISCUSSION

Floor effects limit a measure's ability to distinguish between persons with low function.

CONCLUSION

Analyses supported the validity of the SHAP IOF, LIF, and W-LIF, but identified large floor effects, as well as issues with structural validity of the PP scores. The 16-item P-IOF minimizes floor effects and reduces administrative burden.

Multi-level longitudinal learning curve regression models integrated with item difficulty metrics for deliberate practice of visual diagnosis: groundwork for adaptive learning

Visual diagnosis of radiographs, histology and electrocardiograms lends itself to deliberate practice, facilitated by large online banks of cases. Which cases to supply to which learners in which order is still to be worked out, with there being considerable potential for adapting the learning. Advances in statistical modeling, based on an accumulating learning curve, offer methods for more effectively pairing learners with cases of known calibrations. Using demonstration radiograph and electrocardiogram datasets, the advantages of moving from traditional regression to multilevel methods for modeling growth in ability or performance are demonstrated, with a final step of integrating case-level item-response information based on diagnostic grouping. This produces more precise individual-level estimates that can eventually support learner adaptive case selection. The progressive increase in model sophistication is not simply statistical but rather brings the models into alignment with core learning principles including the importance of taking into account individual differences in baseline skill and learning rate as well as the differential interaction with cases of varying diagnosis and difficulty. The developed approach can thus give researchers and educators a better basis on which to anticipate learners' pathways and individually adapt their future learning.

Assessing farmers' attitudes to, and the behavioural costs of, organic fertiliser practices in northern Ghana: An application of the behavioural cost approach

The use of organic fertiliser to improve soil health is crucial to halting the downward trend of crop yields in sub-Saharan Africa. If this goal is to be achieved, however, farmers require support to adopt organic fertiliser practices that match their attitudes and decision-making capacity. This study evaluated farmers' attitudes to a set of prevailing organic fertiliser practices and their associated behavioural costs (difficulty). The explanatory Rasch model was applied to a set of primary data from 250 farming households in north-east Ghana. The results showed that the average attitude of farmers was much less than the difficulty estimate of an average organic fertiliser practice, although the practices generally showed a moderate difficulty. On average, farmers' attitudes matched just three of sixteen practices on the scale, with most (70 %) of the farmers showing very weak attitudes towards the input. Latent regression results revealed that the weak attitude levels were strongly related to key factors in the farmers' background, including education, resource endowment and access to extension services. Participation in determining policies on organic fertiliser use enhances farmers' knowledge and skills concerning use of the input. Hence, access to such policies can replace education for the less-educated majority of farmers. Thus, training programmes are proposed that develop the average farmer's capacity to adopt these practices in this area, especially the less difficult ones. Supporting farmers with the acquisition of animal-drawn vehicles can also facilitate uptake of the more difficult organic fertiliser practices and increase use of the input.

Latent trait models for perceived risk assessment using a Covid-19 data survey

Aim of the contribution is analyzing potential events that may negatively impact individuals, assets, and/or the environment, and making judgments about the perceived personal and social riskiness of Covid-19 compared to other hazards belonging to health (AIDS, cancer, infarction), environmental (climate change), behavioral (serious car accidents), and technological (nuclear weapons) domains. The comparative risk analysis has been performed on a survey data collected during the first Italian Covid-19 lockdown. An item response theory model for polytomously scored items has been implemented for the analysis of the positioning of Covid-19 with respect to the other hazards in terms of perceived risk. Among the attributes determining the hazard's perceived risk, Covid-19 distinguishes for the knowledge of risks from the hazard, media attention, and fear caused by the hazard in the peers. Besides, through a latent regression analysis, the role of some individual characteristics on the perceived risk for Covid-19 has been examined. Our contribution allows us to disentangle among several aspects of hazards and describe the main factors affecting the perceived risk. It also contributes to determine if existing control measures are perceived as adequate and the interest for new media with related impact on a person's reaction.

Improving the Precision of Ability Estimates Using Time-On-Task Variables: Insights From the PISA 2012 Computer-Based Assessment of Mathematics

Log-file data from computer-based assessments can provide useful collateral information for estimating student abilities. In turn, this can improve traditional approaches that only consider response accuracy. Based on the amounts of time students spent on 10 mathematics items from the PISA 2012, this study evaluated the overall changes in and measurement precision of ability estimates and explored country-level heterogeneity when combining item responses and time-on-task measurements using a joint framework. Our findings suggest a notable increase in precision with the incorporation of response times and indicate differences between countries in how respondents approached items as well as in their response processes. Results also showed that additional information could be captured through differences in the modeling structure when response times were included. However, such information may not reflect the testing objective.

Racial and ethnic differences in self-reported telehealth use during the COVID-19 pandemic: a secondary analysis of a US survey of internet users from late March

Objective

Widespread technological changes, like the rapid uptake of telehealth in the US during the COVID-19 pandemic, risk creating or widening racial/ethnic disparities. We conducted a secondary analysis of a cross-sectional, nationally representative survey of internet users to evaluate whether there were racial/ethnic disparities in self-reported telehealth use early in the pandemic.

Materials and Methods

The Pew Research Center fielded the survey March 19–24, 2020. Telehealth use because of the pandemic was measured by asking whether respondents (N = 10 624) “used the internet or e-mail to connect with doctors or other medical professionals as a result of the coronavirus outbreak.” We conducted survey-weighted logistic regressions, adjusting for respondents’ socioeconomic characteristics and perceived threat of the pandemic to their own health (eg, no threat, minor, major).

Results

Approximately 17% of respondents reported using telehealth because of the pandemic, with significantly higher unadjusted odds among Blacks, Latinos, and those identified with other race compared to White respondents. The multivariable logistic regressions and sensitivity analyses show Black respondents were more likely than Whites to report using telehealth because of the pandemic, particularly when perceiving the pandemic as a minor threat to their own health.

Discussion

Black respondents are most likely to report using telehealth because of the COVID-19 pandemic, particularly when they perceive the pandemic as a minor health threat.

Conclusion

The systemic racism creating health and health care disparities has likely raised the need for telehealth among Black patients during the pandemic. Findings suggest opportunities to leverage a broadly defined set of telehealth tools to reduce health care disparities postpandemic.

The inhibition of verbally aggressive behaviour

We studied the inhibition of verbal aggression, defined as not displaying verbal aggression when one would want to. The approach we used was based on a situation–response questionnaire containing 15 anger provoking situations and three verbally aggressive responses. Two questions were asked for each combination of a situation and a response: one about wanting to react in a verbally aggressive way and one about actually displaying the reaction. This questionnaire was administered to 316 participants. Based on different theories about inhibition, several logistic mixed models were constructed and tested against each other. In the best fitting model, inhibition was conceptualized as a trait. Trait inhibition was negatively correlated with external measures of Anger Out and positively with Control of Anger Out. Copyright © 2004 John Wiley & Sons, Ltd.

Measurement of Inter-Individual Variability in Assessing the Quality of Life in Respondents with Celiac Disease

Quality of life of Celiac Disease (CD) patients is affected by constraints in their physical, social and emotional behaviour. Our objective is to assess differences in two relevant dimensions of the Celiac Quality of Life (CQoL) scale, Limitations due to the disease and Dysphoria (i.e., feelings of depression and discomfort), in relation to the perceived social support and some individual and disease-related characteristics. The paper exploits suitable unidimensional Item Response Theory (IRT) models to individually analyse the two mentioned dimensions of the CQoL and Multidimensional Latent Class IRT models for ordinal polytomous items in order to detect sub-populations of CD patients that are homogenous with respect to the perceived CQoL. The latter methods allow to address patients with similar characteristics to the same treatment, performing at the same time a more tailored overture to health promotion programmes. The analysis extracts the relevant patterns and relations among CD patients, disentangling respondents receiving CD diagnosis in adolescence or adult age rather than in childhood (the first perceive high levels of Limitations and Dysphoria), patients with high perceived social support, a factor influencing in a positive way motivation to engage in management of CD-related distress and psychological well-being, and participants who are married or cohabiting. The latter report higher latent trait levels.

Polytomous Item Explanatory Item Response Theory Models

This study investigates polytomous item explanatory item response theory models under the multivariate generalized linear mixed modeling framework, using the linear logistic test model approach. Building on the original ideas of the many-facet Rasch model and the linear partial credit model, a polytomous Rasch model is extended to the item location explanatory many-facet Rasch model and the step difficulty explanatory linear partial credit model. To demonstrate the practical differences between the two polytomous item explanatory approaches, two empirical studies examine how item properties explain and predict the overall item difficulties or the step difficulties each in the Carbon Cycle assessment data and in the Verbal Aggression data. The results suggest that the two polytomous item explanatory models are methodologically and practically different in terms of (a) the target difficulty parameters of polytomous items, which are explained by item properties; (b) the types of predictors for the item properties incorporated into the design matrix; and (c) the types of item property effects. The potentials and methodological advantages of item explanatory modeling are discussed as well.

Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning

A common challenge in the behavioral sciences is evaluating measurement invariance, or whether the measurement properties of a scale are consistent for individuals from different groups. Measurement invariance fails when differential item functioning (DIF) exists, that is, when item responses relate to the latent variable differently across groups. To identify DIF in a scale, many data-driven procedures iteratively test for DIF one item at a time while assuming other items have no DIF. The DIF-free items are used to anchor the scale of the latent variable across groups, identifying the model. A major drawback to these iterative testing procedures is that they can fail to select the correct anchor items and identify true DIF, particularly when DIF is present in many items. We propose an alternative method for selecting anchors and identifying DIF. Namely, we use regularization, a machine learning technique that imposes a penalty function during estimation to remove parameters that have little impact on the fit of the model. We focus specifically here on a lasso penalty for group differences in the item parameters within the two-parameter logistic item response theory model. We compare lasso regularization with the more commonly used likelihood ratio test method in a 2-group DIF analysis. Simulation and empirical results show that when large amounts of DIF are present and sample sizes are large, lasso regularization has far better control of Type I error than the likelihood ratio test method with little decrement in power. This provides strong evidence that lasso regularization is a promising alternative for testing DIF and selecting anchors. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Examining the relation of personality factors to substance use disorder by explanatory item response modeling of DSM-5 symptoms

This paper explores how personality factors affect substance use disorders (SUDs) using explanatory item response modeling (EIRM). A total of 606 Chinese illicit drug users participated in our study. After removing the cases with missing values on the covariate measures, a final sample of 573 participants was used for data analysis. The Diagnostic and Statistical Manual of Mental Disorders (DSM-5) was used to measure the illicit drug users’ SUD level. Four personality factors–anxiety sensitivity, impulsivity, sensation seeking and hopelessness–along with gender and alcohol use were included in EIRM as person covariates. The results indicated that gender, alcohol use, and their interaction significantly predicted the SUD level. The only personality factor that strongly predicted the SUD level was sensation seeking. In addition, the interaction between gender and hopelessness was also found to be a significant predictor of the SUD level, indicating that the negative effect of hopelessness on SUD is stronger for women than for men. The findings suggest that sensation seeking plays an important role in influencing SUDs, and thus, it should be considered when designing intervention or screening procedures for potential illicit drug users. In addition, several DSM-5 SUD symptoms were found to exhibit differential effects by gender, alcohol use, and personality factors. The possible explanations were discussed.

Modeling Multiple Item Context Effects With Generalized Linear Mixed Models

Item context effects refer to the impact of features of a test on an examinee's item responses. These effects cannot be explained by the abilities measured by the test. Investigations typically focus on only a single type of item context effects, such as item position effects, or mode effects, thereby ignoring the fact that different item context effects might operate simultaneously. In this study, two different types of context effects were modeled simultaneously drawing on data from an item calibration study of a multidimensional computerized test (N = 1,632) assessing student competencies in mathematics, science, and reading. We present a generalized linear mixed model (GLMM) parameterization of the multidimensional Rasch model including item position effects (distinguishing between within-block position effects and block position effects), domain order effects, and the interactions between them. Results show that both types of context effects played a role, and that the moderating effect of domain orders was very strong. The findings have direct consequences for planning and applying mixed domain assessment designs.

Gleaning Information for Cognitive Operations from Don't Know Responses in Cognitive and Noncognitive Assessments

The Don't Know (DK) response - taking the form of an omitted response or not-reached at the end of a cognitive test, or explicitly presented as a response option in a social survey - contains important information that is often overlooked. Direct psychometric modeling efforts for DK responses are few and far between. In this article, the linear logistic test model (LLTM) is proposed for delineating the impacts of cognitive operations for a test that contains DK responses. We assume that the DK response is a valid response. The assumption is reasonable for many situations, including low-stakes cognitive tests and attitudinal assessments. By extracting information embedded in the DK response, the method shows how DK can inform the latent construct of interest and the cognitive operations underlying the response to stimuli. Using a proven recoding scheme, the LLTM could be implemented through commonly used programs such as PROC GLIMMIX. Two simulation experiments to evaluate how well the parameters can be recovered were conducted. In addition, two real data examples, from a noncognitive test of health belief assessment and a cognitive test of knowledge in diabetes, are also presented as case studies to illustrate the LLTM for DK response.

Characterizing belief bias in syllogistic reasoning: A hierarchical Bayesian meta-analysis of ROC data

The belief-bias effect is one of the most-studied biases in reasoning. A recent study of the phenomenon using the signal detection theory (SDT) model called into question all theoretical accounts of belief bias by demonstrating that belief-based differences in the ability to discriminate between valid and invalid syllogisms may be an artifact stemming from the use of inappropriate linear measurement models such as analysis of variance (Dube et al., Psychological Review, 117(3), 831-863, 2010). The discrepancy between Dube et al.'s, Psychological Review, 117(3), 831-863 (2010) results and the previous three decades of work, together with former's methodological criticisms suggests the need to revisit earlier results, this time collecting confidence-rating responses. Using a hierarchical Bayesian meta-analysis, we reanalyzed a corpus of 22 confidence-rating studies (N = 993). The results indicated that extensive replications using confidence-rating data are unnecessary as the observed receiver operating characteristic functions are not systematically asymmetric. These results were subsequently corroborated by a novel experimental design based on SDT's generalized area theorem. Although the meta-analysis confirms that believability does not influence discriminability unconditionally, it also confirmed previous results that factors such as individual differences mediate the effect. The main point is that data from previous and future studies can be safely analyzed using appropriate hierarchical methods that do not require confidence ratings. More generally, our results set a new standard for analyzing data and evaluating theories in reasoning. Important methodological and theoretical considerations for future work on belief bias and related domains are discussed.

An item response theory based integrated model of headache, nausea, photophobia, and phonophobia in migraine patients

This study developed an integrated model of severity scores of migraine headache and the incidence of nausea, photophobia, and phonophobia to predict the natural time course of migraine symptoms, which are likely to occur by a common disease progression mechanism. Data were acquired from two phase 3 clinical trials conducted during the development of eletriptan. Only the placebo arm was used for analysis. A conventional proportional odds model was compared with an item response theory (IRT) based approach. Results suggested that the IRT based approach led to a better model fit, successfully revealing the difference in relief rates among different symptoms, which was the fastest in phonophobia and the slowest in headache. Simulation with the developed model suggested that using headache scores at 4 h post-dose attained greatest statistical power, yielding sample size of 100 per arm given drug effect of 40%, as compared to that of 200 per arm when 2 h post-dose scores were used as in the original eletriptan protocol. This work demonstrated the usefulness of an IRT based model as applied to analyzing multidimensional migraine symptoms and designing clinical trials. Our model can be similarly applied to analyzing other multiple endpoints sharing a common underlying mechanism.

A Probit Latent State IRT Model With Latent Item-Effect Variables

Abstract. We present a probit latent state model for dichotomous items with one latent state variable for each occasion of measurement and one latent item-effect variable for each item, except for a reference item. All latent variables are well defined in terms of conditional probabilities. The model offers the possibility to include explanatory variables for the latent states as well as for the latent item-effect variables. We illustrate the model by a data example with the life satisfaction scale of the Freiburg Personality Inventory (FPI-R; Fahrenberg, Hampel, & Selg, 1984 ) assessed at three time points. Allowing for item-effect variables improves model fit considerably and enhances our knowledge about the items. In some applications, this model opens new ways to investigate differential item functioning, and in others it allows to study response styles.

A Variational Maximization-Maximization Algorithm for Generalized Linear Mixed Models with Crossed Random Effects

We present a variational maximization-maximization algorithm for approximate maximum likelihood estimation of generalized linear mixed models with crossed random effects (e.g., item response models with random items, random raters, or random occasion-specific effects). The method is based on a factorized variational approximation of the latent variable distribution given observed variables, which creates a lower bound of the log marginal likelihood. The lower bound is maximized with respect to the factorized distributions as well as model parameters. With the proposed algorithm, a high-dimensional intractable integration is translated into a two-dimensional integration problem. We incorporate an adaptive Gauss-Hermite quadrature method in conjunction with the variational method in order to increase computational efficiency. Numerical studies show that under the small sample size conditions that are considered the proposed algorithm outperforms the Laplace approximation.

Morphine Pharmacodynamics in Mechanically Ventilated Preterm Neonates Undergoing Endotracheal Suctioning

To date, morphine pharmacokinetics (PKs) are well quantified in neonates, but results about its efficacy are ambiguous. This work presents an analysis of a previously published study on pain measurements in mechanically ventilated preterm neonates who received either morphine or placebo to improve comfort during invasive ventilation. The research question was whether morphine reduces the pain associated with endotracheal or nasal suctioning before, during, and after suctioning. Because these neonates cannot verbalize their pain levels, pain was assessed on the basis of several validated pain measurement instruments (i.e., COMFORT‐B, preterm infant pain profile [PIPP], Neonatal Infant Pain Scale (NIPS), and visual analogue scale (VAS)). The item response theory (IRT) was used to analyze the data in order for us to handle the data from multiple‐item pain scores. The analysis showed an intra‐individual relationship between morphine concentrations and pain reduction, as measured by COMFORT‐B and VAS. However, the small magnitude of the morphine effect was not considered clinically relevant for this intervention in preterm neonates.

Classification of Psychopathology

Many have criticized the Diagnostic and Statistical Manual of Mental Disorders ( DSM-IV), and few regard it as a vehicle of truth, yet its most serious limitation is that its frank operationism in defining manifest categories has distracted attention from theories about what is going on at the latent level. We sketch a Generalized Interpersonal Theory of Personality and Psychopathology and apply it to interpersonal aspects of depression to illustrate how structural individual differences combine with functional dynamic processes to cause interpersonal behavior and affect. Such a causal account relies on a realist ontology in which manifest diagnoses are only a means to learning about the latent distribution, whether categorical or dimensional. Comorbidity of DSM diagnoses suggests that dimensionality will be the rule, not the exception, with internalization and externalization describing common diagnoses.

First Steps in Using Multi-Voxel Pattern Analysis to Disentangle Neural Processes Underlying Generalization of Spider Fear

A core symptom of anxiety disorders is the tendency to interpret ambiguous information as threatening. Using electroencephalography and blood oxygenation level dependent magnetic resonance imaging (BOLD-MRI), several studies have begun to elucidate brain processes involved in fear-related perceptual biases, but thus far mainly found evidence for general hypervigilance in high fearful individuals. Recently, multi-voxel pattern analysis (MVPA) has become popular for decoding cognitive states from distributed patterns of neural activation. Here, we used this technique to assess whether biased fear generalization, characteristic of clinical fear, is already present during the initial perception and categorization of a stimulus, or emerges during the subsequent interpretation of a stimulus. Individuals with low spider fear (n = 20) and high spider fear (n = 18) underwent functional MRI scanning while viewing series of schematic flowers morphing to spiders. In line with previous studies, individuals with high fear of spiders were behaviorally more likely to classify ambiguous morphs as spiders than individuals with low fear of spiders. Univariate analyses of BOLD-MRI data revealed stronger activation toward spider pictures in high fearful individuals compared to low fearful individuals in numerous areas. Yet, neither average activation, nor support vector machine classification (i.e., a form of MVPA) matched the behavioral results – i.e., a biased response toward ambiguous stimuli – in any of the regions of interest. This may point to limitations of the current design, and to challenges associated with classifying emotional and neutral stimuli in groups that differ in their judgment of emotionality. Improvements for future research are suggested.

Analysis of Multiple Partially Ordered Responses to Belief Items with Don't Know Option

Understanding beliefs, values, and preferences of patients is a tenet of contemporary health sciences. This application was motivated by the analysis of multiple partially ordered set (poset) responses from an inventory on layman beliefs about diabetes. The partially ordered set arises because of two features in the data-first, the response options contain a Don't Know (DK) option, and second, there were two consecutive occasions of measurement. As predicted by the common sense model of illness, beliefs about diabetes were not necessarily stable across the two measurement occasions. Instead of analyzing the two occasions separately, we studied the joint responses across the occasions as a poset response. Few analytic methods exist for data structures other than ordered or nominal categories. Poset responses are routinely collapsed and then analyzed as either rank ordered or nominal data, leading to the loss of nuanced information that might be present within poset categories. In this paper we developed a general class of item response models for analyzing the poset data collected from the Common Sense Model of Diabetes Inventory. The inferential object of interest is the latent trait that indicates congruence of belief with the biomedical model. To apply an item response model to the poset diabetes inventory, we proved that a simple coding algorithm circumvents the requirement of writing new codes such that standard IRT software could be directly used for the purpose of item estimation and individual scoring. Simulation experiments were used to examine parameter recovery for the proposed poset model.

A Mixed-effects Location-Scale Model for Ordinal Questionnaire Data

In health studies, questionnaire items are often scored on an ordinal scale, for example on a Likert scale. For such questionnaires, item response theory (IRT) models provide a useful approach for obtaining summary scores for subjects (i.e., the model's random subject effect) and characteristics of the items (e.g., item difficulty and discrimination). In this article, we describe a model that allows the items to additionally exhibit different within-subject variance, and also includes a subject-level random effect to the within-subject variance specification. This permits subjects to be characterized in terms of their mean level, or location, and their variability, or scale, and the model allows item difficulty and discrimination in terms of both random subject effects (location and scale). We illustrate application of this location-scale mixed model using data from the Social Subscale of the Drinking Motives Questionnaire (SS-DMQ) assessed in an adolescent study. We show that the proposed model fits the data significantly better than simpler IRT models, and is able to identify items and subjects that are not well-fit by the simpler models. The proposed model has useful applications in many areas where questionnaires are often rated on an ordinal scale, and there is interest in characterizing subjects in terms of both their mean and variability.

Modeling Learning in Doubly Multilevel Binary Longitudinal Data Using Generalized Linear Mixed Models: An Application to Measuring and Explaining Word Learning

When word learning is supported by instruction in experimental studies for adolescents, word knowledge outcomes tend to be collected from complex data structure, such as multiple aspects of word knowledge, multilevel reader data, multilevel item data, longitudinal design, and multiple groups. This study illustrates how generalized linear mixed models can be used to measure and explain word learning for data having such complexity. Results from this application provide deeper understanding of word knowledge than could be attained from simpler models and show that word knowledge is multidimensional and depends on word characteristics and instructional contexts.

A Cross-Classified CFA-MTMM Model for Structurally Different and Nonindependent Interchangeable Methods

Multirater (multimethod, multisource) studies are increasingly applied in psychology. Eid and colleagues (2008) proposed a multilevel confirmatory factor model for multitrait-multimethod (MTMM) data combining structurally different and multiple independent interchangeable methods (raters). In many studies, however, different interchangeable raters (e.g., peers, subordinates) are asked to rate different targets (students, supervisors), leading to violations of the independence assumption and to cross-classified data structures. In the present work, we extend the ML-CFA-MTMM model by Eid and colleagues (2008) to cross-classified multirater designs. The new C4 model (Cross-Classified CTC[M-1] Combination of Methods) accounts for nonindependent interchangeable raters and enables researchers to explicitly model the interaction between targets and raters as a latent variable. Using a real data application, it is shown how credibility intervals of model parameters and different variance components can be obtained using Bayesian estimation techniques.

Marginalized Maximum Likelihood Estimation for the 1PL-AG IRT Model

Marginal maximum likelihood estimation based on the expectation-maximization algorithm (MML/EM) is developed for the one-parameter logistic model with ability-based guessing (1PL-AG) item response theory (IRT) model. The use of the MML/EM estimator is cross-validated with estimates from NLMIXED procedure (PROC NLMIXED) in Statistical Analysis System. Numerical data are provided for comparisons of results from MML/EM and PROC NLMIXED.

A modular approach for item response theory modeling with the R package flirt

The new R package flirt is introduced for flexible item response theory (IRT) modeling of psychological, educational, and behavior assessment data. flirt integrates a generalized linear and nonlinear mixed modeling framework with graphical model theory. The graphical model framework allows for efficient maximum likelihood estimation. The key feature of flirt is its modular approach to facilitate convenient and flexible model specifications. Researchers can construct customized IRT models by simply selecting various modeling modules, such as parametric forms, number of dimensions, item and person covariates, person groups, link functions, etc. In this paper, we describe major features of flirt and provide examples to illustrate how flirt works in practice.

Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model

A variety of statistical methods have been suggested for detecting differential item functioning (DIF) in the Rasch model. Most of these methods are designed for the comparison of pre-specified focal and reference groups, such as males and females. Latent class approaches, on the other hand, allow the detection of previously unknown groups exhibiting DIF. However, this approach provides no straightforward interpretation of the groups with respect to person characteristics. Here, we propose a new method for DIF detection based on model-based recursive partitioning that can be considered as a compromise between those two extremes. With this approach it is possible to detect groups of subjects exhibiting DIF, which are not pre-specified, but result from combinations of observed covariates. These groups are directly interpretable and can thus help generate hypotheses about the psychological sources of DIF. The statistical background and construction of the new method are introduced by means of an instructive example, and extensive simulation studies are presented to support and illustrate the statistical properties of the method, which is then applied to empirical data from a general knowledge quiz. A software implementation of the method is freely available in the R system for statistical computing.

Reliability measures in item response theory: manifest versus latent correlation functions

For item response theory (IRT) models, which belong to the class of generalized linear or non-linear mixed models, reliability at the scale of observed scores (i.e., manifest correlation) is more difficult to calculate than latent correlation based reliability, but usually of greater scientific interest. This is not least because it cannot be calculated explicitly when the logit link is used in conjunction with normal random effects. As such, approximations such as Fisher's information coefficient, Cronbach's α, or the latent correlation are calculated, allegedly because it is easy to do so. Cronbach's α has well-known and serious drawbacks, Fisher's information is not meaningful under certain circumstances, and there is an important but often overlooked difference between latent and manifest correlations. Here, manifest correlation refers to correlation between observed scores, while latent correlation refers to correlation between scores at the latent (e.g., logit or probit) scale. Thus, using one in place of the other can lead to erroneous conclusions. Taylor series based reliability measures, which are based on manifest correlation functions, are derived and a careful comparison of reliability measures based on latent correlations, Fisher's information, and exact reliability is carried out. The latent correlations are virtually always considerably higher than their manifest counterparts, Fisher's information measure shows no coherent behaviour (it is even negative in some cases), while the newly introduced Taylor series based approximations reflect the exact reliability very closely. Comparisons among the various types of correlations, for various IRT models, are made using algebraic expressions, Monte Carlo simulations, and data analysis. Given the light computational burden and the performance of Taylor series based reliability measures, their use is recommended.

Polytomous multilevel testlet models for testlet-based assessments with complex sampling designs

Applications of standard item response theory models assume local independence of items and persons. This paper presents polytomous multilevel testlet models for dual dependence due to item and person clustering in testlet-based assessments with clustered samples. Simulation and survey data were analysed with a multilevel partial credit testlet model. This model was compared with three alternative models - a testlet partial credit model (PCM), multilevel PCM, and PCM - in terms of model parameter estimation. The results indicated that the deviance information criterion was the fit index that always correctly identified the true multilevel testlet model based on the quantified evidence in model selection, while the Akaike and Bayesian information criteria could not identify the true model. In general, the estimation model and the magnitude of item and person clustering impacted the estimation accuracy of ability parameters, while only the estimation model and the magnitude of item clustering affected the item parameter estimation accuracy. Furthermore, ignoring item clustering effects produced higher total errors in item parameter estimates but did not have much impact on the accuracy of ability parameter estimates, while ignoring person clustering effects yielded higher total errors in ability parameter estimates but did not have much effect on the accuracy of item parameter estimates. When both clustering effects were ignored in the PCM, item and ability parameter estimation accuracy was reduced.

On the explaining-away phenomenon in multivariate latent variable models

Many probabilistic models for psychological and educational measurements contain latent variables. Well-known examples are factor analysis, item response theory, and latent class model families. We discuss what is referred to as the 'explaining-away' phenomenon in the context of such latent variable models. This phenomenon can occur when multiple latent variables are related to the same observed variable, and can elicit seemingly counterintuitive conditional dependencies between latent variables given observed variables. We illustrate the implications of explaining away for a number of well-known latent variable models by using both theoretical and real data examples.

Beyond Dichotomies

In this paper, the state of research on the assessment of competencies in higher education is reviewed. Fundamental conceptual and methodological issues are clarified by showing that current controversies are built on misleading dichotomies. By systematically sketching conceptual controversies, competing competence definitions are unpacked (analytic/trait vs. holistic/real-world performance) and commonplaces are identified. Disagreements are also highlighted. Similarly, competing statistical approaches to assessing competencies, namely item-response theory (latent trait) versus generalizability theory (sampling error variance), are unpacked. The resulting framework moves beyond dichotomies and shows how the different approaches complement each other. Competence is viewed along a continuum from traits that underlie perception, interpretation, and decision-making skills, which in turn give rise to observed behavior in real-world situations. Statistical approaches are also viewed along a continuum from linear to nonlinear models that serve different purposes. Item response theory (IRT) models may be used for scaling item responses and modeling structural relations, and generalizability theory (GT) models pinpoint sources of measurement error variance, thereby enabling the design of reliable measurements. The proposed framework suggests multiple new research studies and may serve as a “grand” structural model.

Item selection via Bayesian IRT models

With reference to a questionnaire that aimed to assess the quality of life for dysarthric speakers, we investigate the usefulness of a model-based procedure for reducing the number of items. We propose a mixed cumulative logit model, which is known in the psychometrics literature as the graded response model: responses to different items are modelled as a function of individual latent traits and as a function of item characteristics, such as their difficulty and their discrimination power. We jointly model the discrimination and the difficulty parameters by using a k-component mixture of normal distributions. Mixture components correspond to disjoint groups of items. Items that belong to the same groups can be considered equivalent in terms of both difficulty and discrimination power. According to decision criteria, we select a subset of items such that the reduced questionnaire is able to provide the same information that the complete questionnaire provides. The model is estimated by using a Bayesian approach, and the choice of the number of mixture components is justified according to information criteria. We illustrate the proposed approach on the basis of data that are collected for 104 dysarthric patients by local health authorities in Lecce and in Milan.

Measurement confounding affects the extent to which verbal IQ explains social gradients in mortality

BACKGROUND

IQ is thought to explain social gradients in mortality. IQ scores are based roughly equally on Verbal IQ (VIQ) and Performance IQ tests. VIQ tests, however, are suspected to confound true verbal ability with socioeconomic status (SES), raising the possibility that associations between SES and IQ scores might be overestimated. We examined, first, whether two of the most common types of VIQ tests exhibited differential item functioning (DIF) favouring persons of higher SES and/or majority race/ethnicity. Second, we assessed what impact, if any, this had on estimates of the extent to which VIQ explains social gradients in mortality.

METHODS

Data from the General Social Survey-National Death Index cohort, a US population representative dataset, was used. Item response theory models queried social-factor DIF on the Thorndike Verbal Intelligence Scale and Wechsler Adult Intelligence Scales, Revised Similarities test. Cox models examined mortality associations among SES and VIQ scores corrected and uncorrected for DIF.

RESULTS

When uncorrected for DIF, VIQ was correlated with income, education, occupational prestige and race, with correlation coefficients ranging between |0.12| and |0.43|. After correcting for DIF, correlations ranged from |0.06| to |0.16|. Uncorrected VIQ scores explained 11-40% of the Relative Index of Inequalities in mortality for social factors, while DIF-corrected scores explained 2-29%.

CONCLUSIONS

Two of the common forms of VIQ tests appear confound verbal intelligence with SES. Since these tests appear in most IQ batteries, circumspection may be warranted in estimating the amount of social inequalities in mortality attributable to IQ.

Measuring change for a multidimensional test using a generalized explanatory longitudinal item response model

Even though many educational and psychological tests are known to be multidimensional, little research has been done to address how to measure individual differences in change within an item response theory framework. In this paper, we suggest a generalized explanatory longitudinal item response model to measure individual differences in change. New longitudinal models for multidimensional tests and existing models for unidimensional tests are presented within this framework and implemented with software developed for generalized linear models. In addition to the measurement of change, the longitudinal models we present can also be used to explain individual differences in change scores for person groups (e.g., learning disabled students versus non-learning disabled students) and to model differences in item difficulties across item groups (e.g., number operation, measurement, and representation item groups in a mathematics test). An empirical example illustrates the use of the various models for measuring individual differences in change when there are person groups and multiple skill domains which lead to multidimensionality at a time point.

Robust Bayesian inference for multivariate longitudinal data by using normal/independent distributions

Many randomized clinical trials collect multivariate longitudinal measurements in different scales, for example, binary, ordinal, and continuous. Multilevel item response models are used to evaluate the global treatment effects across multiple outcomes while accounting for all sources of correlation. Continuous measurements are often assumed to be normally distributed. But the model inference is not robust when the normality assumption is violated because of heavy tails and outliers. In this article, we develop a Bayesian method for multilevel item response models replacing the normal distributions with symmetric heavy-tailed normal/independent distributions. The inference is conducted using a Bayesian framework via Markov Chain Monte Carlo simulation implemented in BUGS language. Our proposed method is evaluated by simulation studies and is applied to Earlier versus Later Levodopa Therapy in Parkinson's Disease study, a motivating clinical trial assessing the effect of Levodopa therapy on the Parkinson's disease progression rate.

Integrating prospective longitudinal data: modeling personality and health in the Terman Life Cycle and Hawaii Longitudinal Studies

The present study used a collaborative framework to integrate 2 long-term prospective studies: the Terman Life Cycle Study and the Hawaii Personality and Health Longitudinal Study. Within a 5-factor personality-trait framework, teacher assessments of child personality were rationally and empirically aligned to establish similar factor structures across samples. Comparable items related to adult self-rated health, education, and alcohol use were harmonized, and data were pooled on harmonized items. A structural model was estimated as a multigroup analysis. Harmonized child personality factors were then used to examine markers of physiological dysfunction in the Hawaii sample and mortality risk in the Terman sample. Harmonized conscientiousness predicted less physiological dysfunction in the Hawaii sample and lower mortality risk in the Terman sample. These results illustrate how collaborative, integrative work with multiple samples offers the exciting possibility that samples from different cohorts and ages can be linked together to directly test life span theories of personality and health.

Twice random, once mixed: applying mixed models to simultaneously analyze random effects of language and participants

Psychologists, psycholinguists, and other researchers using language stimuli have been struggling for more than 30 years with the problem of how to analyze experimental data that contain two crossed random effects (items and participants). The classical analysis of variance does not apply; alternatives have been proposed but have failed to catch on, and a statistically unsatisfactory procedure of using two approximations (known as F(1) and F(2)) has become the standard. A simple and elegant solution using mixed model analysis has been available for 15 years, and recent improvements in statistical software have made mixed models analysis widely available. The aim of this article is to increase the use of mixed models by giving a concise practical introduction and by giving clear directions for undertaking the analysis in the most popular statistical packages. The article also introduces the DJMIXED: add-on package for SPSS, which makes entering the models and reporting their results as straightforward as possible.