Generalisability of the Barthel Index and the Functional Independence Measure: robustness of disability measures to Differential Item Functioning

PURPOSE

Differential Item Functioning (DIF), an item malfunctioning, causes Differential Test Functioning (DTF), thus biasing questionnaire measures. The current study evaluates the relationship between DIF and DTF for the Barthel Index and the Functional Independence Measure, likely the most used disability measures. The aim is to understand under which conditions DIF can be ignored as its DTF is negligible.

METHODS

A simulation study was run. Disability measures were obtained for the Barthel Index and FIM motor domain using Rasch analysis with previously published item calibrations. Several DIF scenarios have been assessed. DTF was tolerable if ≤0.50 logits.

RESULTS

Simulations showed that the larger the DIF, the larger the DTF and that, keeping the overall DIF constant, the total number of items with DIF does not affect DTF. DIF of the items with the lowest or highest calibrations is the most dangerous. The DIF of central items should be so massive to matter in DTF terms that it is unlikely to happen in practice. The FIM robustness to DIF is better than that of the Barthel Index.

CONCLUSIONS

The FIM and the Barthel Index show remarkable robustness to DIF. Thanks to this feature, sample invariant, generalisable disability measures are available.

DIF Analysis with Unknown Groups and Anchor Items

Ensuring fairness in instruments like survey questionnaires or educational tests is crucial. One way to address this is by a Differential Item Functioning (DIF) analysis, which examines if different subgroups respond differently to a particular item, controlling for their overall latent construct level. DIF analysis is typically conducted to assess measurement invariance at the item level. Traditional DIF analysis methods require knowing the comparison groups (reference and focal groups) and anchor items (a subset of DIF-free items). Such prior knowledge may not always be available, and psychometric methods have been proposed for DIF analysis when one piece of information is unknown. More specifically, when the comparison groups are unknown while anchor items are known, latent DIF analysis methods have been proposed that estimate the unknown groups by latent classes. When anchor items are unknown while comparison groups are known, methods have also been proposed, typically under a sparsity assumption - the number of DIF items is not too large. However, DIF analysis when both pieces of information are unknown has not received much attention. This paper proposes a general statistical framework under this setting. In the proposed framework, we model the unknown groups by latent classes and introduce item-specific DIF parameters to capture the DIF effects. Assuming the number of DIF items is relatively small, an L 1 -regularised estimator is proposed to simultaneously identify the latent classes and the DIF items. A computationally efficient Expectation-Maximisation (EM) algorithm is developed to solve the non-smooth optimisation problem for the regularised estimator. The performance of the proposed method is evaluated by simulation studies and an application to item response data from a real-world educational test.

Combining Item Purification and Multiple Comparison Adjustment Methods in Detection of Differential Item Functioning

Many of the differential item functioning (DIF) detection methods rely on a principle of testing for DIF item by item, while considering the rest of the items or at least some of them being DIF-free. Computational algorithms of these DIF detection methods involve the selection of DIF-free items in an iterative procedure called item purification. Another aspect is the need to correct for multiple comparisons, which can be done with a number of existing multiple comparison adjustment methods. In this article, we demonstrate that implementation of these two controlling procedures together may have an impact on which items are detected as DIF items. We propose an iterative algorithm combining item purification and adjustment for multiple comparisons. Pleasant properties of the newly proposed algorithm are shown with a simulation study. The method is demonstrated on a real data example.

Semi-automated Rasch analysis with differential item functioning

Rasch analysis is a procedure to develop and validate instruments that aim to measure a person's traits. However, manual Rasch analysis is a complex and time-consuming task, even more so when the possibility of differential item functioning (DIF) is taken into consideration. Furthermore, manual Rasch analysis by construction relies on a modeler's subjective choices. As an alternative approach, we introduce a semi-automated procedure that is based on the optimization of a new criterion, called in-plus-out-of-questionnaire log likelihood with differential item functioning (IPOQ-LL-DIF), which extends our previous criterion. We illustrate our procedure on artificially generated data as well as on several real-world datasets containing potential DIF items. On these real-world datasets, our procedure found instruments with similar clinimetric properties as those suggested by experts through manual analyses.

A Modified Menstrual Attitudes Scale: Heteronormative Attitudes, Sexism, and Attitudes Toward Menstruation in Male and Female Adults

Social shame and stigma surround menstruation, which may compromise women's health and rights in various contexts. Men's attitudes are particularly important because men often hold positions of power that influence women's experience. This study examined factors associated with menstrual attitudes, including heteronormative attitudes, sexism, and family influences. A cross-sectional Qualtrics panel survey (n = 802; aged 18-44; 50.8% female) was performed. We tested a revised menstrual attitudes scale based on items drawn from previously validated measures. Data were analyzed using a structural equation modeling framework. Factor analysis identified and confirmed a 5-factor model for menstrual attitudes. Men endorsed more negative attitudes toward menstruation than women; however, this difference was largely explained by factors other than gender in the structural equation model. After controlling for family and demographic characteristics, attitudes toward openness and secrecy surrounding menstruation were most strongly associated with gender role expectations and hostile sexism. Benevolent sexism was associated with finding menstruation debilitating, denying menstrual symptoms, and endorsing avoidance of activities during menstruation. Heteronormative and sexist attitudes were associated with more negative menstrual attitudes, while increased menstrual knowledge was associated with more positive menstrual attitudes. The difference in menstrual attitudes between males and females was explained largely by heteronormative attitudes and sexism. This suggests that attitudes toward menstruation are closely linked to social ideals about men and women.

Zumbo'nun madde tepki sürecine yönelik eleştirel bakış açısıyla Türkçeyi Ölçme ve değerlendirmede yeni yaklaşımlar

Ana dili olarak Türkçenin ölçülmesi ve değerlendirilmesi maddenin kendisi ve maddeye yönelik katılımcı tepkilerine odaklı bir yaklaşımla sürmektedir. 21. yüzyılda Türkçenin ölçülmesi ise MEB ve ÖSYM öncülüğündeki testler yoluyla gerçekleşmekte ve aynı yaklaşımla paydaşlar tarafından yürütülmektedir. Bununla beraber Dünya’da maddeye tepki sürecinin ekolojik arka planının araştırılması ehemmiyetini giderek artırmaktadır. Madde tepki sürecinde madde ile karşılaşan katılımcı, tepkisinde arka plandaki kompleks bir ekolojik örüntüden etkilenmektedir. Dolayısıyla ilgili örüntünün geçerlilik araştırmalarına dâhil edilmemesi tablonun sınırlı görülebilmesine ve ölçme sonuçlarının eksik yorumlanmasına neden olmaktadır. Madde yanlılığı araştırmalarının mühim bir parçası olan diferansiyel madde fonksiyonunu belirlemeye yönelik çalışmalar bu noktada umut vermektedir. Buradan hareketle Türkçenin ana dili olarak ölçülmesindeki kısıtlı yaklaşımlara da karşılık 3 adımda bu çalışma şekillendirilmiştir. Birinci adımda ana dili olarak Türkçenin ölçülmesi ve değerlendirilmesine yönelik bir çerçeve sunulmuştur. İkinci adımda madde tepki sürecinin ekolojisi tartışılmıştır. Üçüncü adımda kompleks arka planın keşfi için diferansiyel madde fonksiyonu araştırmalarından potansiyel bir çözüm olarak Zumbo ve diğerleri (2015) tarafından geliştirilen ekolojik model tanıtılmıştır. Son aşamada ise kurum bazında MEB ve ÖSYM başta olmak üzere madde yazarları, araştırmacılar ve politika yapıcılar için tavsiye ve teklifler sıralanmıştır. Tartışmanın çıkarımından hareketle ana dili olarak Türkçenin ölçülmesi ve değerlendirilmesinde ekolojik yaklaşımın ve ekolojik modelin geçerlilik araştırmaları için büyük bir potansiyel taşıdığı söylenebilir. Dolayısıyla Türkiye’nin ölçme ihtiyaçlarına uygun olarak ekoloji tabanlı bir perspektifin Türkçenin ana dilinde ölçülmesine adapte edilmesi gerekmektedir. Böylece Türkçe dersi öğretim programının ölçmeye ve değerlendirmeye yönelik yaklaşım ihtiyacının karşılanması mümkün olacaktır.

A New Stopping Criterion for Rasch Trees Based on the Mantel-Haenszel Effect Size Measure for Differential Item Functioning

To detect differential item functioning (DIF), Rasch trees search for optimal splitpoints in covariates and identify subgroups of respondents in a data-driven way. To determine whether and in which covariate a split should be performed, Rasch trees use statistical significance tests. Consequently, Rasch trees are more likely to label small DIF effects as significant in larger samples. This leads to larger trees, which split the sample into more subgroups. What would be more desirable is an approach that is driven more by effect size rather than sample size. In order to achieve this, we suggest to implement an additional stopping criterion: the popular Educational Testing Service (ETS) classification scheme based on the Mantel-Haenszel odds ratio. This criterion helps us to evaluate whether a split in a Rasch tree is based on a substantial or an ignorable difference in item parameters, and it allows the Rasch tree to stop growing when DIF between the identified subgroups is small. Furthermore, it supports identifying DIF items and quantifying DIF effect sizes in each split. Based on simulation results, we conclude that the Mantel-Haenszel effect size further reduces unnecessary splits in Rasch trees under the null hypothesis, or when the sample size is large but DIF effects are negligible. To make the stopping criterion easy-to-use for applied researchers, we have implemented the procedure in the statistical software R. Finally, we discuss how DIF effects between different nodes in a Rasch tree can be interpreted and emphasize the importance of purification strategies for the Mantel-Haenszel procedure on tree stopping and DIF item classification.

Semi-automated Rasch analysis using in-plus-out-of-questionnaire log likelihood

Rasch analysis is a popular statistical tool for developing and validating instruments that aim to measure human performance, attitudes and perceptions. Despite the availability of various software packages, constructing a good instrument based on Rasch analysis is still considered to be a complex, labour-intensive task, requiring human expertise and rather subjective judgements along the way. In this paper we propose a semi-automated method for Rasch analysis based on first principles that reduces the need for human input. To this end, we introduce a novel criterion, called in-plus-out-of-questionnaire log likelihood (IPOQ-LL). On artificial data sets, we confirm that optimization of IPOQ-LL leads to the desired behaviour in the case of multi-dimensional and inhomogeneous surveys. On three publicly available real-world data sets, our method leads to instruments that are, for all practical purposes, indistinguishable from those obtained by Rasch analysis experts through a manual procedure.

Dimensionality and differential item endorsement of depressive symptoms among aging Black populations in South Africa: Findings from the HAALSI study

BACKGROUND

The Center for Epidemiologic Studies-Depression (CES-D) scale is a widely used measure of depressive symptoms, but its construct validity has not been adequately assessed in sub-Saharan Africa. This study validates the CES-D among an aging Shangaan-speaking and predominantly Black African sample in rural South Africa, with a special emphasis on gender differences.

METHODS

An 8-item CES-D scale was administered in Shangaan to 5059 respondents, aged 40+ years, residing in Agincourt, South Africa. We used Cronbach's alpha and exploratory and confirmatory factor analysis to examine and confirm dimensionality of the CES-D scale. Differential endorsement of CES-D items by gender were assessed using the Mantel-Haenszel (MH) odds ratio test.

RESULTS

Reliability of the CES-D scale differed by gender with women reporting higher internal consistency on items than men. A two-factor solution was retained and confirmed representing two latent factors: (1) Negative Affect (six items) and (2) Diminished Positive Affect (two items). MH results showed that men exhibited significantly higher odds of putting an effort in everything that they did (OR: 1.33, 95% CI: 1.15-1.54) and lower odds of feeling depressed (OR: 0.71, 95% CI: 0.56-0.89) and having restless sleep (OR: 0.67, 95% CI:0.58-0.77) than women.

LIMITATIONS

Analyses were limited to a dichotomous, short form of the CES-D, a self-reported population-based measure.

CONCLUSION

Aging Black Africans differ in endorsing affective and somatic items on the CES-D scale by gender, which may lead to skewed population-level estimates of depression in key subpopulations. These findings highlight the importance of continued research disentangling cross-cultural and gendered nuances of depression measurements.

A data mining and item response mixture modeling method to retrospectively measure Diagnostic and Statistical Manual of Mental Disorders-5 attention deficit hyperactivity disorder in the 1970 British Cohort Study

OBJECTIVE

To facilitate future outcome studies, we aimed to develop a robust and replicable method for estimating a categorical and dimensional measure of Diagnostic and Statistical Manual of Mental Disorders-5 (DSM-5) attention deficit hyperactivity disorder (ADHD) in the 1970 British Cohort Study (BCS70).

METHOD

Following a data mining framework, we mapped DSM-5 ADHD symptoms to age 10 BCS70 data (N = 11,426) and derived a 16-item scale (α = 0.85). Mapping was validated by an expert panel. A categorical subgroup was derived (n = 594, 5.2%), and a zero-inflated item response theory (IRT) mixture model fitted to estimate a dimensional measure.

RESULTS

Subgroup composition was comparable with other ADHD samples. Relative risk ratios (ADHD/not ADHD) included boys = 1.38, unemployed fathers = 2.07, below average reading = 2.58, and depressed parent = 3.73. Our estimated measures correlated with two derived reference scales: Strengths and Difficulties Questionnaire hyperactivity (r = 0.74) and a Rutter/Conners-based scale (r = 0.81), supporting construct validity. IRT model items (symptoms) had moderate to high discrimination (0.90-2.81) and provided maximum information at average to moderate theta levels of ADHD (0.5-1.75).

CONCLUSION

We extended previous work to identify ADHD in BCS70, derived scales from existing data, modeled ADHD items with IRT, and adjusted for a zero-inflated distribution. Psychometric properties were promising, and this work will enable future studies of causal mechanisms in ADHD.

Effects of Discontinue Rules on Psychometric Properties of Test Scores

This paper provides results on a form of adaptive testing that is used frequently in intelligence testing. In these tests, items are presented in order of increasing difficulty. The presentation of items is adaptive in the sense that a session is discontinued once a test taker produces a certain number of incorrect responses in sequence, with subsequent (not observed) responses commonly scored as wrong. The Stanford-Binet Intelligence Scales (SB5; Riverside Publishing Company, 2003) and the Kaufman Assessment Battery for Children (KABC-II; Kaufman and Kaufman, 2004), the Kaufman Adolescent and Adult Intelligence Test (Kaufman and Kaufman 2014) and the Universal Nonverbal Intelligence Test (2nd ed.) (Bracken and McCallum 2015) are some of the many examples using this rule. He and Wolfe (Educ Psychol Meas 72(5):808-826, 2012. https://doi.org/10.1177/0013164412441937 ) compared different ability estimation methods in a simulation study for this discontinue rule adaptation of test length. However, there has been no study, to our knowledge, of the underlying distributional properties based on analytic arguments drawing on probability theory, of what these authors call stochastic censoring of responses. The study results obtained by He and Wolfe (Educ Psychol Meas 72(5):808-826, 2012. https://doi.org/10.1177/0013164412441937 ) agree with results presented by DeAyala et al. (J Educ Meas 38:213-234, 2001) as well as Rose et al. (Modeling non-ignorable missing data with item response theory (IRT; ETS RR-10-11), Educational Testing Service, Princeton, 2010) and Rose et al. (Psychometrika 82:795-819, 2017. https://doi.org/10.1007/s11336-016-9544-7 ) in that ability estimates are biased most when scoring the not observed responses as wrong. This scoring is used operationally, so more research is needed in order to improve practice in this field. The paper extends existing research on adaptivity by discontinue rules in intelligence tests in multiple ways: First, an analytical study of the distributional properties of discontinue rule scored items is presented. Second, a simulation is presented that includes additional scoring rules and uses ability estimators that may be suitable to reduce bias for discontinue rule scored intelligence tests.

Factors Influencing High School Students to Persist in Aspirations of Teaching Careers

This study longitudinally tracks 10th grade students for 4 years regarding their persistence in aspirations of teaching careers using a nationally representative sample (National Educational Longitudinal Survey of 1988). Factors contributing to persistence in aspirations of teaching careers are examined based on the social-cognitive career theory (SCCT). Results suggest that there are racial differences in persistence in aspirations to teaching careers. Students who persist perform better on academic achievement, score higher on locus of control, and come from a family that had a higher socioeconomic status and a higher parental education level than students who do not persist. Results also suggest a good fit of the social-cognitive model in prediction of persistence in teaching aspirations.