1
|
Kleinsasser MJ, Mistry R, Hsieh HF, McCarthy WJ, Raghunathan T. Person explanatory multidimensional item response theory with the instrument package in R. Behav Res Methods 2024:10.3758/s13428-024-02490-5. [PMID: 39187738 DOI: 10.3758/s13428-024-02490-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/31/2024] [Indexed: 08/28/2024]
Abstract
We present the new R package instrument to perform Bayesian estimation of person explanatory multidimensional item response theory. The package implements an exploratory multidimensional item response theory model and a higher-order multidimensional item response theory model, a type of confirmatory multidimensional item response theory. Explanation of person parameters is accomplished by fixed and random effect linear regression models. Estimation is carried out using Hamiltonian Monte Carlo in Stan. In this article, we provide a detailed description of the models; we use the instrument package to demonstrate fitting explanatory item response models with fixed and random effects (i.e., mixed modeling) of person parameters in R; and, we perform a simulation study to evaluate the performance of our implementation of the models.
Collapse
Affiliation(s)
| | - Ritesh Mistry
- Department of Health Behavior and Health Education, University of Michigan, Ann Arbor, Michigan, USA
| | - Hsing-Fang Hsieh
- Department of Health Behavior and Health Education, University of Michigan, Ann Arbor, Michigan, USA
| | - William J McCarthy
- Department of Health Policy and Management, University of California Los Angeles, Los Angeles, California, USA
| | | |
Collapse
|
2
|
Huang S, Luo J(J, Cai L. An Explanatory Multidimensional Random Item Effects Rating Scale Model. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT 2023; 83:1229-1248. [PMID: 37974656 PMCID: PMC10638980 DOI: 10.1177/00131644221140906] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/19/2023]
Abstract
Random item effects item response theory (IRT) models, which treat both person and item effects as random, have received much attention for more than a decade. The random item effects approach has several advantages in many practical settings. The present study introduced an explanatory multidimensional random item effects rating scale model. The proposed model was formulated under a novel parameterization of the nominal response model (NRM), and allows for flexible inclusion of person-related and item-related covariates (e.g., person characteristics and item features) to study their impacts on the person and item latent variables. A new variant of the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm designed for latent variable models with crossed random effects was applied to obtain parameter estimates for the proposed model. A preliminary simulation study was conducted to evaluate the performance of the MH-RM algorithm for estimating the proposed model. Results indicated that the model parameters were well recovered. An empirical data set was analyzed to further illustrate the usage of the proposed model.
Collapse
Affiliation(s)
| | | | - Li Cai
- University of California, Los Angeles, USA
| |
Collapse
|
3
|
Razzaq FA, Calzada-Reyes A, Tang Q, Guo Y, Rabinowitz AG, Bosch-Bayard J, Galan-Garcia L, Virues-Alba T, Suarez-Murias C, Miranda I, Riaz U, Bernardo Lagomasino V, Bryce C, Anderson SG, Galler JR, Bringas-Vega ML, Valdes-Sosa PA. Spectral quantitative and semi-quantitative EEG provide complementary information on the life-long effects of early childhood malnutrition on cognitive decline. Front Neurosci 2023; 17:1149102. [PMID: 37781256 PMCID: PMC10540225 DOI: 10.3389/fnins.2023.1149102] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2023] [Accepted: 07/18/2023] [Indexed: 10/03/2023] Open
Abstract
Objective This study compares the complementary information from semi-quantitative EEG (sqEEG) and spectral quantitative EEG (spectral-qEEG) to detect the life-long effects of early childhood malnutrition on the brain. Methods Resting-state EEGs (N = 202) from the Barbados Nutrition Study (BNS) were used to examine the effects of protein-energy malnutrition (PEM) on childhood and middle adulthood outcomes. sqEEG analysis was performed on Grand Total EEG (GTE) protocol, and a single latent variable, the semi-quantitative Neurophysiological State (sqNPS) was extracted. A univariate linear mixed-effects (LME) model tested the dependence of sqNPS and nutritional group. sqEEG was compared with scores on the Montreal Cognitive Assessment (MoCA). Stable sparse classifiers (SSC) also measured the predictive power of sqEEG, spectral-qEEG, and a combination of both. Multivariate LME was applied to assess each EEG modality separately and combined under longitudinal settings. Results The univariate LME showed highly significant differences between previously malnourished and control groups (p < 0.001); age (p = 0.01) was also significant, with no interaction between group and age detected. Childhood sqNPS (p = 0.02) and adulthood sqNPS (p = 0.003) predicted MoCA scores in adulthood. The SSC demonstrated that spectral-qEEG combined with sqEEG had the highest predictive power (mean AUC 0.92 ± 0.005). Finally, multivariate LME showed that the combined spectral-qEEG+sqEEG models had the highest log-likelihood (-479.7). Conclusion This research has extended our prior work with spectral-qEEG and the long-term impact of early childhood malnutrition on the brain. Our findings showed that sqNPS was significantly linked to accelerated cognitive aging at 45-51 years of age. While sqNPS and spectral-qEEG produced comparable results, our study indicated that combining sqNPS and spectral-qEEG yielded better performance than either method alone, suggesting that a multimodal approach could be advantageous for future investigations. Significance Based on our findings, a semi-quantitative approach utilizing GTE could be a valuable diagnostic tool for detecting the lasting impacts of childhood malnutrition. Notably, sqEEG has not been previously explored or reported as a biomarker for assessing the longitudinal effects of malnutrition. Furthermore, our observations suggest that sqEEG offers unique features and information not captured by spectral quantitative EEG analysis and could lead to its improvement.
Collapse
Affiliation(s)
- Fuleah A. Razzaq
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformatics, University of Electronic Science and Technology of China, Chengdu, China
| | | | - Qin Tang
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformatics, University of Electronic Science and Technology of China, Chengdu, China
| | - Yanbo Guo
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformatics, University of Electronic Science and Technology of China, Chengdu, China
| | | | | | | | | | | | - Ileana Miranda
- National Center for Animal and Plant Health, CENSA, San José de las Lajas, Mayabeque, Cuba
| | - Usama Riaz
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformatics, University of Electronic Science and Technology of China, Chengdu, China
| | | | - Cyralene Bryce
- The George Alleyne Chronic Disease Research Centre, Caribbean Institute for Health Research, University of the West Indies, Cave Hill, Barbados
| | - Simon G. Anderson
- The George Alleyne Chronic Disease Research Centre, Caribbean Institute for Health Research, University of the West Indies, Cave Hill, Barbados
- The George Alleyne Chronic Disease Research Centre, Caribbean Institute for Health Research, University of the West Indies, Cave Hill, Barbados
| | - Janina R. Galler
- The George Alleyne Chronic Disease Research Centre, Caribbean Institute for Health Research, University of the West Indies, Cave Hill, Barbados
- Division of Pediatric Gastroenterology and Nutrition, MassGeneral Hospital for Children, Boston, MA, United States
| | - Maria L. Bringas-Vega
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformatics, University of Electronic Science and Technology of China, Chengdu, China
| | - Pedro A. Valdes-Sosa
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformatics, University of Electronic Science and Technology of China, Chengdu, China
- Cuban Neuroscience Center, La Habana, Cuba
| |
Collapse
|
4
|
Gerstenecker A, Kennedy R, Zhang Y, Martin RC, Mackin RS, Weiner MW, Howell T, Petersen RC, Roberson ED, Marson DC. Item Response Analysis of the Financial Capacity Instrument-Short Form. Arch Clin Neuropsychol 2023; 38:739-758. [PMID: 36644855 PMCID: PMC10369359 DOI: 10.1093/arclin/acac112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/29/2022] [Indexed: 06/17/2023] Open
Abstract
OBJECTIVE The Financial Capacity Instrument-Short Form (FCI-SF) is a performance-based measure of everyday financial skills that takes 15 min to administer. Although the FCI-SF has demonstrated excellent psychometric properties, advanced psychometric methods such as item response theory (IRT) can provide important information on the performance of individual test items in measuring financial capacity and in distinguishing between healthy and cognitively impaired individuals. METHOD Participants were 272 older adults diagnosed with mild cognitive impairment (MCI) and 1,344 cognitively healthy controls recruited from the Mayo Clinic Study of Aging at the Mayo Clinic in Rochester, Minnesota and also from the Cognitive Observations in Seniors study at the University of Alabama at Birmingham. Participants in each study were administered the FCI-SF, which evaluates coin/currency calculation, financial conceptual knowledge, use of a checkbook/register, and use of a bank statement. RESULTS A unidimensional two-parameter logistic model best fit the 37 FCI-SF Test items, and most FCI-SF items fit the unidimensional two-parameter model well. The results indicated that all FCI-SF items robustly distinguished cognitively healthy controls from persons with MCI. CONCLUSIONS The study results showed that the FCI-SF performed well under IRT analysis, further highlighted the psychometric properties of the FCI-SF as a valid and reliable measure of financial capacity, and demonstrated the clinical utility of the FCI-SF in distinguishing between cognitively normal and cognitively impaired individuals.
Collapse
Affiliation(s)
- Adam Gerstenecker
- Department of Neurology, University of Alabama at Birmingham, San Francisco, CA, USA
- Alzheimer’s Disease Center, University of Alabama at Birmingham, San Francisco, CA, USA
- Evelyn F. McKnight Brain Institute, University of Alabama at Birmingham, San Francisco, CA, USA
| | - Richard Kennedy
- Alzheimer’s Disease Center, University of Alabama at Birmingham, San Francisco, CA, USA
- Evelyn F. McKnight Brain Institute, University of Alabama at Birmingham, San Francisco, CA, USA
- Division of Gerontology, Geriatrics, and Palliative Care, Department of Medicine, University of Alabama at Birmingham, San Francisco, CA, USA
- Interdisciplinary Center for Aging Research, University of Alabama at Birmingham, San Francisco, CA, USA
| | - Yue Zhang
- Division of Gerontology, Geriatrics, and Palliative Care, Department of Medicine, University of Alabama at Birmingham, San Francisco, CA, USA
| | - Roy C Martin
- Department of Neurology, University of Alabama at Birmingham, San Francisco, CA, USA
- Alzheimer’s Disease Center, University of Alabama at Birmingham, San Francisco, CA, USA
- Evelyn F. McKnight Brain Institute, University of Alabama at Birmingham, San Francisco, CA, USA
| | - R Scott Mackin
- Department of Psychiatry, Weill Institute for Neurosciences, University of California San Francisco, San Francisco, CA, USA
- San Francisco Veteran's Administration Medical Center, San Francisco, CA, USA
| | - Michael W Weiner
- San Francisco Veteran's Administration Medical Center, San Francisco, CA, USA
- Department of Radiology & Biomedical Imaging, University of California, San Francisco, CA, USA
| | - Taylor Howell
- Department of Psychiatry, Weill Institute for Neurosciences, University of California San Francisco, San Francisco, CA, USA
| | | | - Erik D Roberson
- Department of Neurology, University of Alabama at Birmingham, San Francisco, CA, USA
- Alzheimer’s Disease Center, University of Alabama at Birmingham, San Francisco, CA, USA
- Evelyn F. McKnight Brain Institute, University of Alabama at Birmingham, San Francisco, CA, USA
- Interdisciplinary Center for Aging Research, University of Alabama at Birmingham, San Francisco, CA, USA
| | - Daniel C Marson
- Department of Neurology, University of Alabama at Birmingham, San Francisco, CA, USA
- Alzheimer’s Disease Center, University of Alabama at Birmingham, San Francisco, CA, USA
- Evelyn F. McKnight Brain Institute, University of Alabama at Birmingham, San Francisco, CA, USA
| |
Collapse
|
5
|
Riaz U, Razzaq FA, Areces-Gonzalez A, Piastra MC, Vega MLB, Paz-Linares D, Valdés-Sosa PA. Automatic Quality Control of the numerical accuracy of EEG Lead fields. Neuroimage 2023; 273:120091. [PMID: 37060935 DOI: 10.1016/j.neuroimage.2023.120091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2021] [Revised: 03/22/2023] [Accepted: 04/04/2023] [Indexed: 04/17/2023] Open
Abstract
Precise individualized EEG source localization is predicated on having accurate subject-specific Lead Fields (LFs) obtained from their Magnetic Resonance Images (MRI). LF calculation is a complex process involving several error-prone steps that start with obtaining a realistic head model from the MRI and finalizing with computationally expensive solvers such as the Boundary Element Method (BEM) or Finite Element Method (FEM). Current Big-Data applications require the calculation of batches of hundreds or thousands of LFs. LF. Quality Control is conventionally checked subjectively by experts, a procedure not feasible in practice for larger batches. To facilitate this step, we introduce the Lead Field Automatic-Quality Control Index (LF-AQI) that flags LF with potential errors. We base our LF-AQI on the assumption that LFs obtained from simpler head models, i.e., the homogeneous head model LF (HHM-LF) or spherical head model LF (SHM-LF), deviate only moderately from a "good" realistic test LF. Since these simpler LFs are easier to compute and check for errors, they may serve as "reference LF" to detect anomalous realistic test LF. We investigated this assumption by comparing correlation-based channel ρmin(ref,test)and source τmin(ref,test) similarity indices (SI) between "gold standards," i.e., very accurate FEM and BEM LFs, and the proposed references (HHM-LF and SHM-LF). Surprisingly we found that the most uncomplicated possible reference, HHM-LF had high SI values with the gold standards-leading us to explore further use of the channel ρmin(HHM-LF,test)and source τmin(HHM-LF,test)SI as a basis for our LF-AQI. Indeed, these SI successfully detected five simulated scenarios of LFs artifacts. This result encouraged us to evaluate the SI on a large dataset and thus define our LF-AQI. We thus computed the SI of 1251 LFs obtained from the Child Mind Institute (CMI) MRI dataset. When ρmin(HHM-LF,test)and source τmin(HHM-LF,test) were plotted for all test subjects on a 2D space, most were tightly clustered around the median of a high similarity centroid (HSC), except for a smaller proportion of outliers. We define the LF-AQI for a given LF as the log Euclidean distance between its SI and the HSC median. To automatically detect outliers, the threshold is at the 90th percentile of the CMI LF-AQIs (-0.9755). LF-AQI greater than this threshold flag individual LF to be checked. The robustness of this LF-AQI screening was checked by repeated out-of-sample validation. Strikingly, minor corrections in re-processing the flagged cases eliminated their status as outliers. Furthermore, the "doubtful" labels assigned by LF-AQI were validated by neuroscience students using a Likert scale questionnaire designed to manually check the LF's quality. Item Response Theory (IRT) analysis was applied to the questionnaire results to compute an optimized model and a latent variable θ for that model. A linear mixed model (LMM) between the θ and LF-AQI resulted in an effect with a Cohen's d value of 1.3 and a p-value <0.001, thus validating the correspondence of LF-AQI with the visual quality control. We provide an open-source pipeline to implement both LF calculation and its quality control to allow further evaluation of our index.
Collapse
Affiliation(s)
- Usama Riaz
- The Clinical Hospital of Chengdu Brain Sciences, University of Electronic Science and Technology of China, Chengdu, China
| | - Fuleah A Razzaq
- The Clinical Hospital of Chengdu Brain Sciences, University of Electronic Science and Technology of China, Chengdu, China
| | - Ariosky Areces-Gonzalez
- The Clinical Hospital of Chengdu Brain Sciences, University of Electronic Science and Technology of China, Chengdu, China; Department of Informatics, University of Pinar del Rio Hermanos Saiz Montes de Oca, Cuba
| | | | - Maria L Bringas Vega
- The Clinical Hospital of Chengdu Brain Sciences, University of Electronic Science and Technology of China, Chengdu, China; Cuban Neuroscience Center, Havana, Cuba
| | - Deirel Paz-Linares
- The Clinical Hospital of Chengdu Brain Sciences, University of Electronic Science and Technology of China, Chengdu, China
| | - Pedro A Valdés-Sosa
- The Clinical Hospital of Chengdu Brain Sciences, University of Electronic Science and Technology of China, Chengdu, China; Cuban Neuroscience Center, Havana, Cuba.
| |
Collapse
|
6
|
Gühne D, Doebler P, Condon DM, Luo F, Sun L. Validity and Reliability of Automatically Generated Propositional Reasoning Items. EUROPEAN JOURNAL OF PSYCHOLOGICAL ASSESSMENT 2021. [DOI: 10.1027/1015-5759/a000616] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Abstract. This study introduces a newly developed public-domain multilingual automatic item generator that creates propositional reasoning (PR) items belonging to 15 item families by using various inference rules. Psychometric properties of the resulting written PR test were investigated in three diverse samples in English, simplified Chinese, and German, respectively. Internal consistency was good to excellent across samples. The ICAR16 short form test of cognitive abilities ( Condon & Revelle, 2014 ) was used to evaluate construct validity. Correlations of ICAR16 scores and PR scores were high. Furthermore, items within families appeared to be equivalent, with only minor differential item functioning between the Chinese- and English-speaking samples. Performance on the PR test was shown to be reasonably stable over the course of 1 week. Differences of total scores between test forms (pen and paper vs. computerized administration) were not detected. Findings suggest that the automatically generated PR test is a valuable instrument for the assessment of propositional reasoning ability.
Collapse
Affiliation(s)
- Daniela Gühne
- Department of Statistics, TU Dortmund University, Germany
| | | | - David M. Condon
- Department of Psychology, University of Oregon, Eugene, OR, USA
| | - Fang Luo
- Faculty of Psychology, Beijing Normal University, Beijing, China
| | - Luning Sun
- The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, United Kingdom
| |
Collapse
|
7
|
Zhang J, Lu J, Xu X, Tao J. Bayesian multilevel multidimensional item response modeling approach for multiple latent variables in a hierarchical structure. COMMUN STAT-SIMUL C 2021. [DOI: 10.1080/03610918.2021.1919707] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Affiliation(s)
- Jiwei Zhang
- Key Lab of Statistical Modeling and Data Analysis of Yunnan Province, School of Mathematics and Statistics, Yunnan University, Kunming, Yunnan, China
| | - Jing Lu
- Key Laboratory of Applied Statistics of MOE, School of Mathematics and Statistics, Northeast Normal University, Changchun, Jilin, China
| | - Xin Xu
- Key Laboratory of Applied Statistics of MOE, School of Mathematics and Statistics, Northeast Normal University, Changchun, Jilin, China
| | - Jian Tao
- Key Laboratory of Applied Statistics of MOE, School of Mathematics and Statistics, Northeast Normal University, Changchun, Jilin, China
| |
Collapse
|
8
|
Cross-validation of the Utility of Test of Memory Malingering (TOMM) Cut-offs in a Large Colombian Sample. PSYCHOLOGICAL INJURY & LAW 2021. [DOI: 10.1007/s12207-021-09407-z] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
9
|
Petersen IT, Choe DE, LeBeau B. Studying a Moving Target in Development: The Challenge and Opportunity of Heterotypic Continuity. DEVELOPMENTAL REVIEW 2020; 58:100935. [PMID: 33244192 PMCID: PMC7685252 DOI: 10.1016/j.dr.2020.100935] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
Abstract
Many psychological constructs show heterotypic continuity-their behavioral manifestations change with development but their meaning remains the same (e.g., externalizing problems). However, research has paid little attention to how to account for heterotypic continuity. Conceptual and methodological challenges of heterotypic continuity may prevent researchers from examining lengthy developmental spans. Developmental theory requires that measurement accommodate changes in manifestation of constructs. Simulation and empirical work demonstrate that failure to account for heterotypic continuity when collecting or analyzing longitudinal data results in faulty developmental inferences. Accounting for heterotypic continuity may require using different measures across time with approaches that link measures on a comparable scale. Creating a developmental scale (i.e., developmental scaling) is recommended to link measures across time and account for heterotypic continuity, which is crucial in understanding development across the lifespan. The current synthesized review defines heterotypic continuity, describes how to identify it, and presents solutions to account for it. We note challenges of addressing heterotypic continuity, and propose steps in leveraging opportunities it creates to advance empirical study of development.
Collapse
|
10
|
Incremental criterion validity of message perceptions and effects perceptions in the context of anti-smoking messages. J Behav Med 2020; 44:74-83. [PMID: 32519300 DOI: 10.1007/s10865-020-00163-0] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2019] [Accepted: 05/21/2020] [Indexed: 10/24/2022]
Abstract
To select promising health messages, formative research has often relied on perceived message effectiveness (PME) scales assessing either of two related constructs, message perceptions (persuasive potential) and effects perceptions (potential for behavioral impact). We sought to examine their incremental criterion validity within a comparative framework. Participants were 703 U.S. adult smokers (ages [Formula: see text] 21) who received anti-smoking or comparable control (littering) messages on their cigarette packs for 3 weeks. Structural equation models examined both PME constructs as simultaneous correlates of outcomes from the UNC Tobacco Warnings Model. Message perceptions demonstrated incremental criterion validity with attention, an early behavioral antecedent ([Formula: see text] = 0.82, p < .001). Effects perceptions demonstrated incremental criterion validity with later behavioral antecedents (range [Formula: see text] = 0.74-0.87, all p < .01) and quitting behaviors ([Formula: see text] = 0.36-0.66, all p < .001). Formative research on anti-smoking messages may benefit from focusing on effects perceptions to characterize potential for behavior change.
Collapse
|
11
|
Zhang J, Lu J, Chen F, Tao J. Exploring the Correlation Between Multiple Latent Variables and Covariates in Hierarchical Data Based on the Multilevel Multidimensional IRT Model. Front Psychol 2019; 10:2387. [PMID: 31708833 PMCID: PMC6823212 DOI: 10.3389/fpsyg.2019.02387] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2019] [Accepted: 10/07/2019] [Indexed: 11/13/2022] Open
Abstract
In many large-scale tests, it is very common that students are nested within classes or schools and that the test designers try to measure their multidimensional latent traits (e.g., logical reasoning ability and computational ability in the mathematics test). It is particularly important to explore the influences of covariates on multiple abilities for development and improvement of educational quality monitoring mechanism. In this study, motivated by a real dataset of a large-scale English achievement test, we will address how to construct an appropriate multilevel structural models to fit the data in many of multilevel models, and what are the effects of gender and socioeconomic-status differences on English multidimensional abilities at the individual level, and how does the teachers' satisfaction and school climate affect students' English abilities at the school level. A full Gibbs sampling algorithm within the Markov chain Monte Carlo (MCMC) framework is used for model estimation. Moreover, a unique form of the deviance information criterion (DIC) is used as a model comparison index. In order to verify the accuracy of the algorithm estimation, two simulations are considered in this paper. Simulation studies show that the Gibbs sampling algorithm works well in estimating all model parameters across a broad spectrum of scenarios, which can be used to guide the real data analysis. A brief discussion and suggestions for further research are shown in the concluding remarks.
Collapse
Affiliation(s)
- Jiwei Zhang
- School of Mathematics and Statistics, Yunnan University, Kunming, China
| | - Jing Lu
- School of Mathematics and Statistics, Northeast Normal University, Changchun, China
| | - Feng Chen
- Department of East Asian Studies, The University of Arizona, Tucson, AZ, United States
| | - Jian Tao
- School of Mathematics and Statistics, Northeast Normal University, Changchun, China
| |
Collapse
|
12
|
Baig SA, Noar SM, Gottfredson NC, Boynton MH, Ribisl KM, Brewer NT. UNC Perceived Message Effectiveness: Validation of a Brief Scale. Ann Behav Med 2019; 53:732-742. [PMID: 30321252 PMCID: PMC6636889 DOI: 10.1093/abm/kay080] [Citation(s) in RCA: 67] [Impact Index Per Article: 13.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Interventionists commonly identify promising messages for health communication efforts based on audience members' ratings of perceived message effectiveness (PME). PURPOSE We sought to validate a new PME measure that improved on existing scales by focusing on the behavior and respondent, being brief, and having strong psychometric properties. METHODS Participants were a national convenience sample of 999 adults and national probability samples of 1,692 adults and 869 adolescents recruited in 2015. Smokers and nonsmokers rated up to six brief messages about the chemicals in cigarette smoke on two PME scales. The first was the new three-item University of North Carolina (UNC) PME Scale that assessed effects perceptions. The second was an established six-item PME scale that assessed message perceptions. We examined the UNC PME Scale's psychometric properties and compared both scales using item factor analysis. RESULTS The UNC PME Scale measured the same construct across multiple chemical messages (all factor loadings ≥ 0.86). It exhibited high reliability (>0.85) over very low to moderate levels of PME (z = -2.5 to 0.2), a range that is useful for identifying more promising messages. Samples of adults and adolescents showed a similar pattern of results. As expected, the UNC PME Scale was strongly positively correlated with message perceptions (r = .84). It also exhibited strong psychometric properties among participants regardless of education, reactance, sex, and smoking status. DISCUSSION The UNC PME Scale reliably and validly measured PME among adults and adolescents from diverse groups. This brief scale may be used to efficiently evaluate candidate antismoking messages and may be suitable for adaptation to other health risk behaviors.
Collapse
Affiliation(s)
- Sabeeh A Baig
- Department of Health Behavior, Gillings School of Global Public Health, University of North Carolina, Chapel Hill, NC, USA
| | - Seth M Noar
- School of Media and Journalism, University of North Carolina, Chapel Hill, NC, USA
- Lineberger Comprehensive Cancer Center, University of North Carolina, Chapel Hill, NC, USA
| | - Nisha C Gottfredson
- Department of Health Behavior, Gillings School of Global Public Health, University of North Carolina, Chapel Hill, NC, USA
| | - Marcella H Boynton
- Department of Health Behavior, Gillings School of Global Public Health, University of North Carolina, Chapel Hill, NC, USA
- Lineberger Comprehensive Cancer Center, University of North Carolina, Chapel Hill, NC, USA
| | - Kurt M Ribisl
- Department of Health Behavior, Gillings School of Global Public Health, University of North Carolina, Chapel Hill, NC, USA
- Lineberger Comprehensive Cancer Center, University of North Carolina, Chapel Hill, NC, USA
| | - Noel T Brewer
- Department of Health Behavior, Gillings School of Global Public Health, University of North Carolina, Chapel Hill, NC, USA
- Lineberger Comprehensive Cancer Center, University of North Carolina, Chapel Hill, NC, USA
| |
Collapse
|
13
|
Chalmers RP. Numerical approximation of the observed information matrix with Oakes' identity. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2018; 71:415-436. [PMID: 29315543 DOI: 10.1111/bmsp.12127] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/02/2017] [Revised: 09/15/2017] [Indexed: 06/07/2023]
Abstract
An efficient and accurate numerical approximation methodology useful for obtaining the observed information matrix and subsequent asymptotic covariance matrix when fitting models with the EM algorithm is presented. The numerical approximation approach is compared to existing algorithms intended for the same purpose, and the computational benefits and accuracy of this new approach are highlighted. Instructive and real-world examples are included to demonstrate the methodology concretely, properties of the estimator are discussed in detail, and a Monte Carlo simulation study is included to investigate the behaviour of a multi-parameter item response theory model using three competing finite-difference algorithms.
Collapse
Affiliation(s)
- R Philip Chalmers
- Department of Educational Psychology, University of Georgia, Athens, Georgia, USA
| |
Collapse
|
14
|
Chalmers RP, Pek J, Liu Y. Profile-likelihood Confidence Intervals in Item Response Theory Models. MULTIVARIATE BEHAVIORAL RESEARCH 2017; 52:533-550. [PMID: 28594582 DOI: 10.1080/00273171.2017.1329082] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
Confidence intervals (CIs) are fundamental inferential devices which quantify the sampling variability of parameter estimates. In item response theory, CIs have been primarily obtained from large-sample Wald-type approaches based on standard error estimates, derived from the observed or expected information matrix, after parameters have been estimated via maximum likelihood. An alternative approach to constructing CIs is to quantify sampling variability directly from the likelihood function with a technique known as profile-likelihood confidence intervals (PL CIs). In this article, we introduce PL CIs for item response theory models, compare PL CIs to classical large-sample Wald-type CIs, and demonstrate important distinctions among these CIs. CIs are then constructed for parameters directly estimated in the specified model and for transformed parameters which are often obtained post-estimation. Monte Carlo simulation results suggest that PL CIs perform consistently better than Wald-type CIs for both non-transformed and transformed parameters.
Collapse
Affiliation(s)
| | - Jolynn Pek
- b Department of Psychology , The Ohio State University
| | - Yang Liu
- c Department of Human Development and Quantitative Methodology , University of Maryland , College Park
| |
Collapse
|
15
|
Bashkov BM, DeMars CE. Examining the Performance of the Metropolis-Hastings Robbins-Monro Algorithm in the Estimation of Multilevel Multidimensional IRT Models. APPLIED PSYCHOLOGICAL MEASUREMENT 2017; 41:323-337. [PMID: 29881095 PMCID: PMC5978673 DOI: 10.1177/0146621616688923] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
The purpose of this study was to examine the performance of the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm in the estimation of multilevel multidimensional item response theory (ML-MIRT) models. The accuracy and efficiency of MH-RM in recovering item parameters, latent variances and covariances, as well as ability estimates within and between clusters (e.g., schools) were investigated in a simulation study, varying the number of dimensions, the intraclass correlation coefficient, the number of clusters, and cluster size, for a total of 24 conditions. Overall, MH-RM performed well in recovering the item, person, and group-level parameters of the model. Ratios of the empirical to analytical standard errors indicated that the analytical standard errors reported in flexMIRT were somewhat overestimated for the cluster-level ability estimates, a little too large for the person-level ability estimates, and essentially accurate for the other parameters. Limitations of the study, implications for educational measurement practice, and directions for future research are offered.
Collapse
|
16
|
Chalmers RP, Ng V. Plausible-Value Imputation Statistics for Detecting Item Misfit. APPLIED PSYCHOLOGICAL MEASUREMENT 2017; 41:372-387. [PMID: 29882543 PMCID: PMC5978675 DOI: 10.1177/0146621617692079] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/24/2023]
Abstract
When tests consist of a small number of items, the use of latent trait estimates for secondary analyses is problematic. One area in particular where latent trait estimates have been problematic is when testing for item misfit. This article explores the use of plausible-value imputations to lessen the severity of the inherent measurement unreliability in shorter tests, and proposes a parametric bootstrap procedure to generate empirical sampling characteristics for null-hypothesis tests of item fit. Simulation results suggest that the proposed item-fit statistics provide conservative to nominal error detection rates. Power to detect item misfit tended to be less than Stone's χ 2 * item-fit statistic but higher than the S - X 2 statistic proposed by Orlando and Thissen, especially in tests with 20 or more dichotomously scored items.
Collapse
|
17
|
Cho SJ, De Boeck P, Lee WY. Evaluating Testing, Profile Likelihood Confidence Interval Estimation, and Model Comparisons for Item Covariate Effects in Linear Logistic Test Models. APPLIED PSYCHOLOGICAL MEASUREMENT 2017; 41:353-371. [PMID: 29881097 PMCID: PMC5978674 DOI: 10.1177/0146621617692078] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
The linear logistic test model (LLTM) has been widely applied to investigate the effects of item covariates on item difficulty. The LLTM was extended with random item residuals to account for item differences not explained by the item covariates. This extended LLTM is called the LLTM-R. In this article, statistical inference methods are investigated for these two models. Type I error rates and power are compared via Monte Carlo studies. Based on the simulation results, the use of the likelihood ratio test (LRT) is recommended over the paired-sample t test based on sum scores, the Wald z test, and information criteria, and the LRT is recommended over the profile likelihood confidence interval because of the simplicity of the LRT. In addition, it is concluded that the LLTM-R is the better general model approach. Inferences based on the LLTM while the LLTM-R is the true model appear to be largely biased in the liberal way, while inferences based on the LLTM-R while the LLTM is the true model are only biased in a very minor and conservative way. Furthermore, in the absence of residual variance, Type I error rate and power were acceptable except for power when the number of items is small (10 items) and also the number of persons is small (200 persons). In the presence of residual variance, however, the number of items needs to be large (80 items) to avoid an inflated Type I error and to reach a power level of .90 for a moderate effect.
Collapse
|