Applying Item Response Theory to Develop a Shortened Version of the Need for Cognition Scale

The Beck Hopelessness Scale's psychometric features: A new short form based on item response theory

BACKGROUND

The assessment of hopelessness plays a significant role in preventing various psychological disorders and major life events within the general population. However, the psychometric properties of the Beck Hopelessness Scale (BHS) have been a subject of controversy, primarily studied in clinical groups. The aim of the present study was to gain new psychometric insights and propose a new short version of the BHS for the general population using the Item Response Theory (IRT) approach.

METHODS

A total of 2164 Argentinean individuals completed the BHS alongside the Inventory of Suicide Orientation-30. We compared IRT models with two and three parameters for the original BHS version, exploring the removal of redundant and less informative items. Convergent and discriminant validity was also examined.

RESULTS

Our results support the 2PL model for the BHS-19. In addition, the BHS-10 short version adequately depicted the same range of the measured trait as the original version, showing reasonable measurement accuracy in the middle-high levels of the trait (marginal reliability = 0.70, Cronbach's α = 0.86). Notably, a positive correlation was found between the factorial score of BHS-10, BHS-19, and suicide orientation.

CONCLUSIONS

In conclusion, our findings support the use of a simplified version as a practical and valuable tool for both research and clinical practice in the future.

Development of a rapid tool for screening financial toxicity based on the comprehensive score for financial toxicity

OBJECTIVE

This study aimed to use item response theory (IRT) to develop and validate a short version of the Chinese COmprehensive Score for financial Toxicity (COST-S-C) in patients with cancer.

METHODS

A total of 500 patients with cancer participated in the survey. Confirmatory factor analysis (CFA) was used to assess the factor structure of the COST-S-C. A two-parameter IRT model was used to examine the item fit of the COST-S-C. Item discrimination, item characteristic curves (ICCs), item information curves (IICs), and item fit were estimated. Differential item functioning (DIF) was assessed for participants' gender and age. Reliability, convergent and discriminant validity was examined as well as estimating the optimal cut-off points of the COST-S-C.

RESULTS

The CFA results supported the bi-factor structure of the COST-S-C. The IRT analysis findings revealed that several items showed problems with the ICCs, providing little information in terms of IICs as well as poor discrimination. Item 5 showed a negligible DIF problem with age. A valid 3-item COST-S-C scale and its optimal cut-off point was developed.

CONCLUSIONS

The COST-S-C is a valid and quick screening tool used to distinguish patients with cancer who report significant financial toxicity in various medical settings.

An item response theory analysis of the matrix reasoning item bank (MaRs-IB)

Matrix reasoning tasks are among the most widely used measures of cognitive ability in the behavioral sciences, but the lack of matrix reasoning tests in the public domain complicates their use. Here, we present an extensive investigation and psychometric validation of the matrix reasoning item bank (MaRs-IB), an open-access set of matrix reasoning items. In a first study, we calibrate the psychometric functioning of the items in the MaRs-IB in a large sample of adult participants (N = 1501). Using additive multilevel item structure models, we establish that the MaRs-IB has many desirable psychometric properties: its items span a wide range of difficulty, possess medium-to-large levels of discrimination, and exhibit robust associations between item complexity and difficulty. However, we also find that item clones are not always psychometrically equivalent and cannot be assumed to be exchangeable. In a second study, we demonstrate how experimenters can use the estimated item parameters to design new matrix reasoning tests using optimal item assembly. Specifically, we design and validate two new sets of test forms in an independent sample of adults (N = 600). We find these new tests possess good reliability and convergent validity with an established measure of matrix reasoning. We hope that the materials and results made available here will encourage experimenters to use the MaRs-IB in their research.

A Scale of Parent-to-Child Emotions (SPCE): Development and validation of a short form

Aim

Parents' emotions towards a child are extremely important. The Scale of Parent-to-Child Emotions (SPCE) consists of five basic and four self-conscious emotion domains for assessment of parental emotional states. Abridgement of the SPCE is needed for research and clinical settings.

Methods

Our previous investigational data for SPCE development were used in this study. The sample of 2336 fathers and 2264 mothers, whose eldest child's age was up to 12 years old, was analyzed. Total information for each pair (form) of items corresponding to a latent trait (θ) was calculated. The form with the greatest amount of total information was selected as the best for each domain. In addition, relative efficiency for each form and correlations of raw sum scores in classical test theory (CTT) for short forms with factor scores in item response theory (IRT) were calculated.

Results

The SPCE was shortened to 18 items by selecting two items each for nine domains. Correlations of raw sum scores in CTT for short forms with factor scores in IRT were correlated strongly and significantly.

Conclusion

This abridged form of the scale, the SPCE-18, may be applicable in a busy clinical setting or research works to investigate the trajectory of parent-to-child emotions across a long span of time.

Measurement Efficiency of a Teacher Rating Scale to Screen for Students at Risk for Social, Emotional, and Behavioral Problems

Teacher rating scales are broadly used for psycho-educational assessment in schools. In particular, they play an important role in screening students for social, emotional, and behavioral problems. In order to optimize the efficiency of these measures, it is important to minimize the number of items comprising them while maintaining sound psychometric characteristics. This study examines the measurement efficiency of a teacher rating scale for student social, emotional, and behavioral risk. The goal was to shorten an existing behavior screening tool. A total of 139 classroom teachers and 2566 students from Grades 1-6 (M_age = 8.96 years, SD = 1.61) participated in the study. In sum, 35 items assessing internalizing and externalizing behavior problems were analyzed applying the item response theory (generalized partial credit model). The results show that social, emotional, and behavioral risks can be captured with a total of 12 items. This reduction of almost 66% of the initial item pool would take teachers about 90 s to fill out for one student. Thus, the rating scale can be used by teachers in an efficient yet psychometrically sound manner.

A psychometric evaluation of Chinese chronic hepatitis B virus infection-related stigma scale using classical test theory and item response theory

Purpose

To validate the hepatitis B virus infection-related stigma scale (HBVISS) using Classical Test Theory and Item Response Theory in a sample of Chinese chronic HBV carriers.

Methods

Feasibility, internal consistency reliability, split-half reliability and construct validity were evaluated using a cross-sectional validation study (n = 1,058) in Classical Test Theory. Content validity was assessed by COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) criteria. The Item Response Theory (IRT) model parameters were estimated using Samejima's graded response model, after which item response category characteristic curves were drawn. Item information, test information, and IRT-based marginal reliability were calculated. Measurement invariance was assessed using differential item functioning (DIF). SPSS and R software were used for the analysis.

Results

The response rate reached 96.4% and the scale was completed in an average time of 5 min. Content validity of HBVISS was sufficient (+) and the quality of the evidence was high according to COSMIN criteria. Confirmatory factor analysis showed acceptable goodness-of-fit (χ ²/df = 5.40, standardized root mean square residual = 0.057, root mean square error of approximation = 0.064, goodness-of-fit index = 0.902, comparative fit index = 0.925, incremental fit index = 0.926, and Tucker-Lewis index = 0.912). Cronbach's α fell in the range of 0.79-0.89 for each dimension and 0.93 for the total scale. Split-half reliability was 0.96. IRT discrimination parameters were estimated to range between 0.959 and 2.333, and the threshold parameters were in the range-3.767 to 3.894. The average score for test information was 12.75 (information >10) when the theta level reached between-4 and + 4. The IRT-based marginal reliability was 0.95 for the total scale and fell in the range of 0.83-0.91 for each dimension. No measurement invariance was detected (d-R ² < 0.02).

Conclusion

HBVISS exhibited good feasibility, reliability, validity, and item quality, making it suitable for assessing chronic Hepatitis B virus infection-related stigma.

Reliability and validity of a perinatal depression screening instrument in rural Mali

Background

In order to reduce the burden of perinatal depression in low- and middle-income countries, health systems must be able to identify and treat women suffering from depression. The objective of our study was to develop a locally valid and reliable screening instrument for use in identifying pregnant women and mothers of young children with a local depression syndrome, dusukasi, in rural Mali.

Methods

We administered a locally adapted screening instrument containing items from the Edinburgh Postpartum Depression Scale (EPDS) and Hopkins Symptom Checklist (HSCL-25) to 180 pregnant women and mothers of children under age 2 in Sélingué, Mali to assess the instrument's psychometric properties and validity. Item Response Theory was used to develop an abbreviated version of the measure and the validity and psychometric properties of this shortened version were compared with the full-length scale.

Results

The full 28-item scale exhibited a single factor structure with good internal consistency (Cronbach's alpha = 0.92). Women who self-identified as suffering from dusukasi (n = 87) in a known groups analysis to assess construct validity had significantly higher depression and anxiety symptom scores (p < 0.0001) and functional impairment scores (p < 0.0001) compared to women not reporting dusukasi (n = 93). The shortened 16-item scale performed as well as the full scale in identifying women with dusukasi.

Conclusions

Construct validity of our adapted screening instrument was supported for identifying dusukasi in rural Malian women. Our methodology can be applied in other settings to develop similarly valid screening instruments for perinatal depression.

Developing a shortened version of the dementia knowledge assessment scale (DKAS-TC) with a sample in Taiwan: an item response theory approach

BACKGROUND

The 25-item Dementia Knowledge Assessment Scale (DKAS2) is a widely used tool for measuring knowledge of dementia. To increase the applicability of the Chinese-language version of the tool (DKAS-TC) for the general public, this study aimed to develop a shortened version using the item response theory (IRT) approach.

METHODS

A total of 401 participants voluntarily completed a Chinese-language version of the DKAS2 questionnaire (DKAS-TC) at the start of dementia awareness training courses in 2020 and 2021. The four Rasch family models were used to analyze the dimensionality of the shortened scale (the DKAS-s) and to confirm its accuracy in measuring dementia knowledge.

RESULTS

The results justified supported the use of a dichotomous response scale for responding to the DKAS-s and demonstrated good fit of the data to a Rasch model with the four dimensions of "Causes and Characteristics", "Communication and Engagement", "Care Needs", and "Risks and Health Promotion". Moreover, we shortened the DKAS-TC by selecting items that had both above-average discriminative ability and above-average information. The DKAS-s retained 64.13% of the information contained in the DKAS-TC, resulting in a 16-item scale which retained four items in each of the original four dimensions. The DKAS-s also correlated highly (≥0.95) with the DKAS-TC and exhibited a sizeable range of difficulty of dementia knowledge.

CONCLUSIONS

The DKAS-s is expected to be more efficient in field settings while retaining an acceptable level of psychometric properties when used as a survey instrument to measure the general public's knowledge of dementia.

Infographics and the Elaboration Likelihood Model (ELM): Differences between Visual and Textual Health Messages

Effective health communication is vital to improving health outcomes. One increasingly popular form of health communication is the use of infographics. Despite its popularity, there is a lack of empirical investigation regarding its information processing mechanism. We applied the Elaboration Likelihood Model (ELM) to study the conditions that lead to elaboration. Participants were randomly assigned to two groups: a text-only health message and a health infographic. Results showed that the infographic had better image appeal rating than the text-only message, and the group that viewed the infographic reported greater elaboration. In addition, for both groups, prior knowledge, ease of understanding, and image appeal had effects on elaboration. Presentation format (text-only vs infographic) interacted with prior knowledge to impact elaboration. Our findings demonstrate that visual design cannot be sidelined as a peripheral interest, but should be positioned as one of the focal points in health communication research.

Development and Diagnostic Accuracy of a Shortened Dutch Naming Test for People with Aphasia Using Item Response Theory

Objective

The first objective was to assess the psychometric properties of the 92-item Dutch Naming Test (DNT-92), developed to assess word finding difficulties in people with aphasia, using Item Response Theory (IRT). The second objective was to select suitable items for a short version with a discriminative purpose.

Method

This study has a retrospective, psychometric research design, in which 510 DNT-92-forms of people with aphasia and 192 DNT-forms of healthy participants were used for analyses. An IRT analysis was performed and information on the item- and person parameters was obtained. Item selection for the short version was based on a combination of the discriminative ability of the items and their estimated theta or difficulty. Items with the highest information load, and a difficulty parameter in the range of overlap between the sample of people with aphasia and healthy participants were selected.

Results

A 2-PL IRT analysis showed best fit to the data. Assumptions of unidimensionality, local independence, and monotonicity were met. Items were removed incrementally, whilst checking sensitivity and specificity of the remaining short form. A selection of six items proved optimal in terms of sensitivity and specificity, with an area under the curve value of 0.85. Differences were found between participants younger than 70 and older.

Conclusions

The IRT assumptions for the DNT-92 were met, indicating that the test has good psychometric properties. A reduction of items to just six items proved possible, leading to a reliable six item short form with a discriminatory purpose.

Need for Cognition Among Users of Self-Monitoring Systems for Physical Activity: Survey Study

Background

Need for cognition (NFC) is among the most studied personality traits in psychology. Despite its apparent relevance for engaging with technology and the use of information, it has not been studied in the context of self-monitoring systems and wearables for health. This study is the first to explore the relationship between NFC and commercial self-monitoring systems among healthy users.

Objective

This study aims to explore the effect of NFC levels on the selection of self-monitoring systems and evaluation of system features of self-monitoring and feedback, as well as perceived credibility and perceived persuasiveness. We also assessed perceived behavior change in the form of self-reported activity after adopting the system.

Methods

Survey data were collected in October 2019 among university students and personnel. The invitation to respond to the questionnaire was addressed to those who had used a digital system to monitor their physical activity for at least two months. The web-based questionnaire comprised the following 3 parts: details of system use, partially randomly ordered theoretical measurement items, and user demographics. The data were analyzed using structural equation modeling. The effect of NFC was assessed both as 3 groups (low, moderate, and high) and as a continuous moderator variable.

Results

In all, 238 valid responses to the questionnaire were obtained. Individuals with high NFC reported all tested system features with statistically significantly higher scores. The NFC also had some effect on system selection. Hypothesized relationships with perceived credibility gained support in a different way for individuals with low and high NFC; for those with low NFC, credibility increased the persuasiveness of the system, but this effect was absent among individuals with high NFC. For users with high NFC, credibility was related to feedback and self-monitoring and perhaps continuously evaluated during prolonged use instead of being a static system property. Furthermore, the relationship between perceived persuasiveness and self-reported activity after adopting the system had a large effect size (Cohen f²=0.355) for individuals with high NFC, a small effect size for individuals with moderate NFC (Cohen f²=0.107), and a nonsignificant path (P=.16) for those with low NFC. We also detected a moderating effect of NFC in two paths on perceived persuasiveness but only among women. Our research model explained 59.2%, 63.9%, and 47.3% of the variance in perceived persuasiveness of the system among individuals with low, moderate, and high NFC, respectively.

Conclusions

The system choices of individuals seem to reflect their intrinsic motivations to engage with rich data, and commercial systems might themselves be a tailoring strategy. Important characteristics of the system, such as perceived credibility, have different roles depending on the NFC levels. Our data demonstrate that NFC as a trait that differentiates information processing has several implications for the selection, design, and tailoring of self-monitoring systems.

A Short Form of the Chinese Version of the Weinstein Noise Sensitivity Scale through Optimal Test Assembly

This study developed a short form of the traditional Chinese version of the Weinstein Noise Sensitivity Scale (WNSS) through optimal test assembly (OTA). A total of 1069 Chinese adults (64.8% female) completed the territory-wide cross-sectional study. We first removed Items 12 and 5 which had negative factor loading and gender-related differential item functioning (DIF), respectively. The optimal length was then determined as the minimal one that reasonably resembled the reliability and validity of the scale without DIF items. OTA identified an 8-item WNSS (WNSS-8) which retained 67.2% of the test information of the original 21-item scale and had a Cronbach's alpha of 0.83. It also showed significant correlations of 0.272 and -0.115 with the neuroticism and extraversion scales of Chinese NEO-Five Factor Inventory, respectively. Adequate model fit of the WNSS-8 was demonstrated by the confirmatory factor analysis. The Chinese WNSS-8 can be used to assess noise sensitivity without compromising reliability and validity.

Nonrestorative sleep scale: a reliable and valid short form of the traditional Chinese version

Purpose

Previous research has suggested the essential unidimensionality of the 12-item traditional Chinese version of the Nonrestorative Sleep Scale (NRSS). This study aimed to develop a short form of the traditional Chinese version of the NRSS without compromising its reliability and validity.

Methods

Data were collected from 2 cross-sectional studies with identical target groups of adults residing in Hong Kong. An iterative Wald test was used to assess differential item functioning by gender. Based on the generalized partial credit model, we first obtained a shortened version such that further shortening would result in substantial sacrifice of test information and standard error of measurement. Another shortened version was obtained by the optimal test assembly (OTA). The two shortened versions were compared for test information, Cronbach’s alpha, and convergent validity.

Results

Data from a total of 404 Chinese adults (60.0% female) who had completed the Chinese NRSS were gathered. All items were invariant by gender. A 6-item version was obtained beyond which the test performance substantially deteriorated, and a 9-item version was obtained by OTA. The 9-item version performed better than the 6-item version in test information and convergent validity. It had discrimination and difficulty indices ranging from 0.44 to 2.23 and − 7.58 to 2.13, respectively, and retained 92% of the test information of the original 12-item version.

Conclusion

The 9-item Chinese NRSS is a reliable and valid tool to measure nonrestorative sleep for epidemiological studies.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02523-4) contains supplementary material, which is available to authorized users.