Extended Mixed-Effects Item Response Models With the MH-RM Algorithm

Person explanatory multidimensional item response theory with the instrument package in R

We present the new R package instrument to perform Bayesian estimation of person explanatory multidimensional item response theory. The package implements an exploratory multidimensional item response theory model and a higher-order multidimensional item response theory model, a type of confirmatory multidimensional item response theory. Explanation of person parameters is accomplished by fixed and random effect linear regression models. Estimation is carried out using Hamiltonian Monte Carlo in Stan. In this article, we provide a detailed description of the models; we use the instrument package to demonstrate fitting explanatory item response models with fixed and random effects (i.e., mixed modeling) of person parameters in R; and, we perform a simulation study to evaluate the performance of our implementation of the models.

An Explanatory Multidimensional Random Item Effects Rating Scale Model

Random item effects item response theory (IRT) models, which treat both person and item effects as random, have received much attention for more than a decade. The random item effects approach has several advantages in many practical settings. The present study introduced an explanatory multidimensional random item effects rating scale model. The proposed model was formulated under a novel parameterization of the nominal response model (NRM), and allows for flexible inclusion of person-related and item-related covariates (e.g., person characteristics and item features) to study their impacts on the person and item latent variables. A new variant of the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm designed for latent variable models with crossed random effects was applied to obtain parameter estimates for the proposed model. A preliminary simulation study was conducted to evaluate the performance of the MH-RM algorithm for estimating the proposed model. Results indicated that the model parameters were well recovered. An empirical data set was analyzed to further illustrate the usage of the proposed model.

Spectral quantitative and semi-quantitative EEG provide complementary information on the life-long effects of early childhood malnutrition on cognitive decline

Objective

This study compares the complementary information from semi-quantitative EEG (sqEEG) and spectral quantitative EEG (spectral-qEEG) to detect the life-long effects of early childhood malnutrition on the brain.

Methods

Resting-state EEGs (N = 202) from the Barbados Nutrition Study (BNS) were used to examine the effects of protein-energy malnutrition (PEM) on childhood and middle adulthood outcomes. sqEEG analysis was performed on Grand Total EEG (GTE) protocol, and a single latent variable, the semi-quantitative Neurophysiological State (sqNPS) was extracted. A univariate linear mixed-effects (LME) model tested the dependence of sqNPS and nutritional group. sqEEG was compared with scores on the Montreal Cognitive Assessment (MoCA). Stable sparse classifiers (SSC) also measured the predictive power of sqEEG, spectral-qEEG, and a combination of both. Multivariate LME was applied to assess each EEG modality separately and combined under longitudinal settings.

Results

The univariate LME showed highly significant differences between previously malnourished and control groups (p < 0.001); age (p = 0.01) was also significant, with no interaction between group and age detected. Childhood sqNPS (p = 0.02) and adulthood sqNPS (p = 0.003) predicted MoCA scores in adulthood. The SSC demonstrated that spectral-qEEG combined with sqEEG had the highest predictive power (mean AUC 0.92 ± 0.005). Finally, multivariate LME showed that the combined spectral-qEEG+sqEEG models had the highest log-likelihood (-479.7).

Conclusion

This research has extended our prior work with spectral-qEEG and the long-term impact of early childhood malnutrition on the brain. Our findings showed that sqNPS was significantly linked to accelerated cognitive aging at 45-51 years of age. While sqNPS and spectral-qEEG produced comparable results, our study indicated that combining sqNPS and spectral-qEEG yielded better performance than either method alone, suggesting that a multimodal approach could be advantageous for future investigations.

Significance

Based on our findings, a semi-quantitative approach utilizing GTE could be a valuable diagnostic tool for detecting the lasting impacts of childhood malnutrition. Notably, sqEEG has not been previously explored or reported as a biomarker for assessing the longitudinal effects of malnutrition. Furthermore, our observations suggest that sqEEG offers unique features and information not captured by spectral quantitative EEG analysis and could lead to its improvement.

Item Response Analysis of the Financial Capacity Instrument-Short Form

OBJECTIVE

The Financial Capacity Instrument-Short Form (FCI-SF) is a performance-based measure of everyday financial skills that takes 15 min to administer. Although the FCI-SF has demonstrated excellent psychometric properties, advanced psychometric methods such as item response theory (IRT) can provide important information on the performance of individual test items in measuring financial capacity and in distinguishing between healthy and cognitively impaired individuals.

METHOD

Participants were 272 older adults diagnosed with mild cognitive impairment (MCI) and 1,344 cognitively healthy controls recruited from the Mayo Clinic Study of Aging at the Mayo Clinic in Rochester, Minnesota and also from the Cognitive Observations in Seniors study at the University of Alabama at Birmingham. Participants in each study were administered the FCI-SF, which evaluates coin/currency calculation, financial conceptual knowledge, use of a checkbook/register, and use of a bank statement.

RESULTS

A unidimensional two-parameter logistic model best fit the 37 FCI-SF Test items, and most FCI-SF items fit the unidimensional two-parameter model well. The results indicated that all FCI-SF items robustly distinguished cognitively healthy controls from persons with MCI.

CONCLUSIONS

The study results showed that the FCI-SF performed well under IRT analysis, further highlighted the psychometric properties of the FCI-SF as a valid and reliable measure of financial capacity, and demonstrated the clinical utility of the FCI-SF in distinguishing between cognitively normal and cognitively impaired individuals.

Automatic Quality Control of the numerical accuracy of EEG Lead fields

Precise individualized EEG source localization is predicated on having accurate subject-specific Lead Fields (LFs) obtained from their Magnetic Resonance Images (MRI). LF calculation is a complex process involving several error-prone steps that start with obtaining a realistic head model from the MRI and finalizing with computationally expensive solvers such as the Boundary Element Method (BEM) or Finite Element Method (FEM). Current Big-Data applications require the calculation of batches of hundreds or thousands of LFs. LF. Quality Control is conventionally checked subjectively by experts, a procedure not feasible in practice for larger batches. To facilitate this step, we introduce the Lead Field Automatic-Quality Control Index (LF-AQI) that flags LF with potential errors. We base our LF-AQI on the assumption that LFs obtained from simpler head models, i.e., the homogeneous head model LF (HHM-LF) or spherical head model LF (SHM-LF), deviate only moderately from a "good" realistic test LF. Since these simpler LFs are easier to compute and check for errors, they may serve as "reference LF" to detect anomalous realistic test LF. We investigated this assumption by comparing correlation-based channel ρ_min(ref,test)and source τ_min(ref,test) similarity indices (SI) between "gold standards," i.e., very accurate FEM and BEM LFs, and the proposed references (HHM-LF and SHM-LF). Surprisingly we found that the most uncomplicated possible reference, HHM-LF had high SI values with the gold standards-leading us to explore further use of the channel ρ_min(HHM-LF,test)and source τ_min(HHM-LF,test)_{SI as a basis for} our LF-AQI. Indeed, these SI successfully detected five simulated scenarios of LFs artifacts. This result encouraged us to evaluate the SI on a large dataset and thus define our LF-AQI. We thus computed the SI of 1251 LFs obtained from the Child Mind Institute (CMI) MRI dataset. When ρ_min(HHM-LF,test)and source τ_min(HHM-LF,test) were plotted for all test subjects on a 2D space, most were tightly clustered around the median of a high similarity centroid (HSC), except for a smaller proportion of outliers. We define the LF-AQI for a given LF as the log Euclidean distance between its SI and the HSC median. To automatically detect outliers, the threshold is at the 90^th percentile of the CMI LF-AQIs (-0.9755). LF-AQI greater than this threshold flag individual LF to be checked. The robustness of this LF-AQI screening was checked by repeated out-of-sample validation. Strikingly, minor corrections in re-processing the flagged cases eliminated their status as outliers. Furthermore, the "doubtful" labels assigned by LF-AQI were validated by neuroscience students using a Likert scale questionnaire designed to manually check the LF's quality. Item Response Theory (IRT) analysis was applied to the questionnaire results to compute an optimized model and a latent variable θ for that model. A linear mixed model (LMM) between the θ and LF-AQI resulted in an effect with a Cohen's d value of 1.3 and a p-value <0.001, thus validating the correspondence of LF-AQI with the visual quality control. We provide an open-source pipeline to implement both LF calculation and its quality control to allow further evaluation of our index.

Validity and Reliability of Automatically Generated Propositional Reasoning Items

Abstract. This study introduces a newly developed public-domain multilingual automatic item generator that creates propositional reasoning (PR) items belonging to 15 item families by using various inference rules. Psychometric properties of the resulting written PR test were investigated in three diverse samples in English, simplified Chinese, and German, respectively. Internal consistency was good to excellent across samples. The ICAR16 short form test of cognitive abilities ( Condon & Revelle, 2014 ) was used to evaluate construct validity. Correlations of ICAR16 scores and PR scores were high. Furthermore, items within families appeared to be equivalent, with only minor differential item functioning between the Chinese- and English-speaking samples. Performance on the PR test was shown to be reasonably stable over the course of 1 week. Differences of total scores between test forms (pen and paper vs. computerized administration) were not detected. Findings suggest that the automatically generated PR test is a valuable instrument for the assessment of propositional reasoning ability.

Studying a Moving Target in Development: The Challenge and Opportunity of Heterotypic Continuity

Many psychological constructs show heterotypic continuity-their behavioral manifestations change with development but their meaning remains the same (e.g., externalizing problems). However, research has paid little attention to how to account for heterotypic continuity. Conceptual and methodological challenges of heterotypic continuity may prevent researchers from examining lengthy developmental spans. Developmental theory requires that measurement accommodate changes in manifestation of constructs. Simulation and empirical work demonstrate that failure to account for heterotypic continuity when collecting or analyzing longitudinal data results in faulty developmental inferences. Accounting for heterotypic continuity may require using different measures across time with approaches that link measures on a comparable scale. Creating a developmental scale (i.e., developmental scaling) is recommended to link measures across time and account for heterotypic continuity, which is crucial in understanding development across the lifespan. The current synthesized review defines heterotypic continuity, describes how to identify it, and presents solutions to account for it. We note challenges of addressing heterotypic continuity, and propose steps in leveraging opportunities it creates to advance empirical study of development.

Incremental criterion validity of message perceptions and effects perceptions in the context of anti-smoking messages

To select promising health messages, formative research has often relied on perceived message effectiveness (PME) scales assessing either of two related constructs, message perceptions (persuasive potential) and effects perceptions (potential for behavioral impact). We sought to examine their incremental criterion validity within a comparative framework. Participants were 703 U.S. adult smokers (ages [Formula: see text] 21) who received anti-smoking or comparable control (littering) messages on their cigarette packs for 3 weeks. Structural equation models examined both PME constructs as simultaneous correlates of outcomes from the UNC Tobacco Warnings Model. Message perceptions demonstrated incremental criterion validity with attention, an early behavioral antecedent ([Formula: see text] = 0.82, p < .001). Effects perceptions demonstrated incremental criterion validity with later behavioral antecedents (range [Formula: see text] = 0.74-0.87, all p < .01) and quitting behaviors ([Formula: see text] = 0.36-0.66, all p < .001). Formative research on anti-smoking messages may benefit from focusing on effects perceptions to characterize potential for behavior change.

Exploring the Correlation Between Multiple Latent Variables and Covariates in Hierarchical Data Based on the Multilevel Multidimensional IRT Model

In many large-scale tests, it is very common that students are nested within classes or schools and that the test designers try to measure their multidimensional latent traits (e.g., logical reasoning ability and computational ability in the mathematics test). It is particularly important to explore the influences of covariates on multiple abilities for development and improvement of educational quality monitoring mechanism. In this study, motivated by a real dataset of a large-scale English achievement test, we will address how to construct an appropriate multilevel structural models to fit the data in many of multilevel models, and what are the effects of gender and socioeconomic-status differences on English multidimensional abilities at the individual level, and how does the teachers' satisfaction and school climate affect students' English abilities at the school level. A full Gibbs sampling algorithm within the Markov chain Monte Carlo (MCMC) framework is used for model estimation. Moreover, a unique form of the deviance information criterion (DIC) is used as a model comparison index. In order to verify the accuracy of the algorithm estimation, two simulations are considered in this paper. Simulation studies show that the Gibbs sampling algorithm works well in estimating all model parameters across a broad spectrum of scenarios, which can be used to guide the real data analysis. A brief discussion and suggestions for further research are shown in the concluding remarks.

UNC Perceived Message Effectiveness: Validation of a Brief Scale

BACKGROUND

Interventionists commonly identify promising messages for health communication efforts based on audience members' ratings of perceived message effectiveness (PME).

PURPOSE

We sought to validate a new PME measure that improved on existing scales by focusing on the behavior and respondent, being brief, and having strong psychometric properties.

METHODS

Participants were a national convenience sample of 999 adults and national probability samples of 1,692 adults and 869 adolescents recruited in 2015. Smokers and nonsmokers rated up to six brief messages about the chemicals in cigarette smoke on two PME scales. The first was the new three-item University of North Carolina (UNC) PME Scale that assessed effects perceptions. The second was an established six-item PME scale that assessed message perceptions. We examined the UNC PME Scale's psychometric properties and compared both scales using item factor analysis.

RESULTS

The UNC PME Scale measured the same construct across multiple chemical messages (all factor loadings ≥ 0.86). It exhibited high reliability (>0.85) over very low to moderate levels of PME (z = -2.5 to 0.2), a range that is useful for identifying more promising messages. Samples of adults and adolescents showed a similar pattern of results. As expected, the UNC PME Scale was strongly positively correlated with message perceptions (r = .84). It also exhibited strong psychometric properties among participants regardless of education, reactance, sex, and smoking status.

DISCUSSION

The UNC PME Scale reliably and validly measured PME among adults and adolescents from diverse groups. This brief scale may be used to efficiently evaluate candidate antismoking messages and may be suitable for adaptation to other health risk behaviors.

Numerical approximation of the observed information matrix with Oakes' identity

An efficient and accurate numerical approximation methodology useful for obtaining the observed information matrix and subsequent asymptotic covariance matrix when fitting models with the EM algorithm is presented. The numerical approximation approach is compared to existing algorithms intended for the same purpose, and the computational benefits and accuracy of this new approach are highlighted. Instructive and real-world examples are included to demonstrate the methodology concretely, properties of the estimator are discussed in detail, and a Monte Carlo simulation study is included to investigate the behaviour of a multi-parameter item response theory model using three competing finite-difference algorithms.

Profile-likelihood Confidence Intervals in Item Response Theory Models

Confidence intervals (CIs) are fundamental inferential devices which quantify the sampling variability of parameter estimates. In item response theory, CIs have been primarily obtained from large-sample Wald-type approaches based on standard error estimates, derived from the observed or expected information matrix, after parameters have been estimated via maximum likelihood. An alternative approach to constructing CIs is to quantify sampling variability directly from the likelihood function with a technique known as profile-likelihood confidence intervals (PL CIs). In this article, we introduce PL CIs for item response theory models, compare PL CIs to classical large-sample Wald-type CIs, and demonstrate important distinctions among these CIs. CIs are then constructed for parameters directly estimated in the specified model and for transformed parameters which are often obtained post-estimation. Monte Carlo simulation results suggest that PL CIs perform consistently better than Wald-type CIs for both non-transformed and transformed parameters.

Examining the Performance of the Metropolis-Hastings Robbins-Monro Algorithm in the Estimation of Multilevel Multidimensional IRT Models

The purpose of this study was to examine the performance of the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm in the estimation of multilevel multidimensional item response theory (ML-MIRT) models. The accuracy and efficiency of MH-RM in recovering item parameters, latent variances and covariances, as well as ability estimates within and between clusters (e.g., schools) were investigated in a simulation study, varying the number of dimensions, the intraclass correlation coefficient, the number of clusters, and cluster size, for a total of 24 conditions. Overall, MH-RM performed well in recovering the item, person, and group-level parameters of the model. Ratios of the empirical to analytical standard errors indicated that the analytical standard errors reported in flexMIRT were somewhat overestimated for the cluster-level ability estimates, a little too large for the person-level ability estimates, and essentially accurate for the other parameters. Limitations of the study, implications for educational measurement practice, and directions for future research are offered.

Plausible-Value Imputation Statistics for Detecting Item Misfit

When tests consist of a small number of items, the use of latent trait estimates for secondary analyses is problematic. One area in particular where latent trait estimates have been problematic is when testing for item misfit. This article explores the use of plausible-value imputations to lessen the severity of the inherent measurement unreliability in shorter tests, and proposes a parametric bootstrap procedure to generate empirical sampling characteristics for null-hypothesis tests of item fit. Simulation results suggest that the proposed item-fit statistics provide conservative to nominal error detection rates. Power to detect item misfit tended to be less than Stone's χ 2 * item-fit statistic but higher than the S - X 2 statistic proposed by Orlando and Thissen, especially in tests with 20 or more dichotomously scored items.

Evaluating Testing, Profile Likelihood Confidence Interval Estimation, and Model Comparisons for Item Covariate Effects in Linear Logistic Test Models

The linear logistic test model (LLTM) has been widely applied to investigate the effects of item covariates on item difficulty. The LLTM was extended with random item residuals to account for item differences not explained by the item covariates. This extended LLTM is called the LLTM-R. In this article, statistical inference methods are investigated for these two models. Type I error rates and power are compared via Monte Carlo studies. Based on the simulation results, the use of the likelihood ratio test (LRT) is recommended over the paired-sample t test based on sum scores, the Wald z test, and information criteria, and the LRT is recommended over the profile likelihood confidence interval because of the simplicity of the LRT. In addition, it is concluded that the LLTM-R is the better general model approach. Inferences based on the LLTM while the LLTM-R is the true model appear to be largely biased in the liberal way, while inferences based on the LLTM-R while the LLTM is the true model are only biased in a very minor and conservative way. Furthermore, in the absence of residual variance, Type I error rate and power were acceptable except for power when the number of items is small (10 items) and also the number of persons is small (200 persons). In the presence of residual variance, however, the number of items needs to be large (80 items) to avoid an inflated Type I error and to reach a power level of .90 for a moderate effect.