Differential item functioning was negligible in an adaptive test of functional status for patients with knee impairments who spoke English or Hebrew

Development and psychometric evaluation of item banks for memory and attention - supplements to the EORTC CAT Core instrument

BACKGROUND

Cancer patients may experience a decrease in cognitive functioning before, during and after cancer treatment. So far, the Quality of Life Group of the European Organisation for Research and Treatment of Cancer (EORTC QLG) developed an item bank to assess self-reported memory and attention within a single, cognitive functioning scale (CF) using computerized adaptive testing (EORTC CAT Core CF item bank). However, the distinction between different cognitive functions might be important to assess the patients' functional status appropriately and to determine treatment impact. To allow for such assessment, the aim of this study was to develop and psychometrically evaluate separate item banks for memory and attention based on the EORTC CAT Core CF item bank.

METHODS

In a multistep process including an expert-based content analysis, we assigned 44 items from the EORTC CAT Core CF item bank to the memory or attention domain. Then, we conducted psychometric analyses based on a sample used within the development of the EORTC CAT Core CF item bank. The sample consisted of 1030 cancer patients from Denmark, France, Poland, and the United Kingdom. We evaluated measurement properties of the newly developed item banks using confirmatory factor analysis (CFA) and item response theory model calibration.

RESULTS

Item assignment resulted in 31 memory and 13 attention items. Conducted CFAs suggested good fit to a 1-factor model for each domain and no violations of monotonicity or indications of differential item functioning. Evaluation of CATs for both memory and attention confirmed well-functioning item banks with increased power/reduced sample size requirements (for CATs ≥ 4 items and up to 40% reduction in sample size requirements in comparison to non-CAT format).

CONCLUSION

Two well-functioning and psychometrically robust item banks for memory and attention were formed from the existing EORTC CAT Core CF item bank. These findings could support further research on self-reported cognitive functioning in cancer patients in clinical trials as well as for real-word-evidence. A more precise assessment of attention and memory deficits in cancer patients will strengthen the evidence on the effects of cancer treatment for different cancer entities, and therefore contribute to shared and informed clinical decision-making.

Psychometric evaluation of an item bank for computerized adaptive testing of the EORTC QLQ-C30 cognitive functioning dimension in cancer patients

BACKGROUND

The European Organisation of Research and Treatment of Cancer (EORTC) Quality of Life Group is developing computerized adaptive testing (CAT) versions of all EORTC Quality of Life Questionnaire (QLQ-C30) scales with the aim to enhance measurement precision. Here we present the results on the field-testing and psychometric evaluation of the item bank for cognitive functioning (CF).

METHODS

In previous phases (I-III), 44 candidate items were developed measuring CF in cancer patients. In phase IV, these items were psychometrically evaluated in a large sample of international cancer patients. This evaluation included an assessment of dimensionality, fit to the item response theory (IRT) model, differential item functioning (DIF), and measurement properties.

RESULTS

A total of 1030 cancer patients completed the 44 candidate items on CF. Of these, 34 items could be included in a unidimensional IRT model, showing an acceptable fit. Although several items showed DIF, these had a negligible impact on CF estimation. Measurement precision of the item bank was much higher than the two original QLQ-C30 CF items alone, across the whole continuum. Moreover, CAT measurement may on average reduce study sample sizes with about 35-40% compared to the original QLQ-C30 CF scale, without loss of power.

CONCLUSION

A CF item bank for CAT measurement consisting of 34 items was established, applicable to various cancer patients across countries. This CAT measurement system will facilitate precise and efficient assessment of HRQOL of cancer patients, without loss of comparability of results.

Development of an item bank for the EORTC Role Functioning Computer Adaptive Test (EORTC RF-CAT)

BACKGROUND

Role functioning (RF) as a core construct of health-related quality of life (HRQOL) comprises aspects of occupational and social roles relevant for patients in all treatment phases as well as for survivors. The objective of the current study was to improve its assessment by developing a computer-adaptive test (CAT) for RF. This was part of a larger project whose objective is to develop a CAT version of the EORTC QLQ-C30 which is one of the most widely used HRQOL instruments in oncology.

METHODS

In accordance with EORTC guidelines, the development of the RF-CAT comprised four phases. Phase I involved the conceptualization of RF. In Phase II, a provisional list of items was defined and revised by experts in the field. In phase III, feedback was obtained from cancer patients in various countries. Phase IV comprised field testing in an international sample, calibration of the item bank, and evaluation of the psychometric performance of the RF-CAT.

RESULTS

Phases I-III yielded a list of 12 items eligible for phase IV field-testing. The field-testing sample included 1,023 patients from Austria, Denmark, Italy, and the UK. Psychometric evaluation and item response theory analyses yielded 10 items with good psychometric properties. The resulting item bank exhibits excellent reliability (mean reliability = 0.85, median = 0.95). Using the RF-CAT may allow sample size savings from 11 % up to 50 % compared to using the QLQ-C30 RF scale.

CONCLUSIONS

The RF-CAT item bank improves the precision and efficiency with which RF can be assessed, promoting its integration into oncology research and clinical practice.

An emotional functioning item bank of 24 items for computerized adaptive testing (CAT) was established

OBJECTIVE

To improve measurement precision, the European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Group is developing an item bank for computerized adaptive testing (CAT) of emotional functioning (EF). The item bank will be within the conceptual framework of the widely used EORTC Quality of Life questionnaire (QLQ-C30).

STUDY DESIGN AND SETTING

On the basis of literature search and evaluations by international samples of experts and cancer patients, 38 candidate items were developed. The psychometric properties of the items were evaluated in a large international sample of cancer patients. This included evaluations of dimensionality, item response theory (IRT) model fit, differential item functioning (DIF), and of measurement precision/statistical power.

RESULTS

Responses were obtained from 1,023 cancer patients from four countries. The evaluations showed that 24 items could be included in a unidimensional IRT model. DIF did not seem to have any significant impact on the estimation of EF. Evaluations indicated that the CAT measure may reduce sample size requirements by up to 50% compared to the QLQ-C30 EF scale without reducing power.

CONCLUSION

On the basis of thorough psychometric evaluations, we have established an EF item bank of 24 items. This will allow for more precise and flexible measurement of EF, while maintaining backward compatibility with the QLQ-C30 EF scale.

The self-report fecal incontinence and constipation questionnaire in patients with pelvic-floor dysfunction seeking outpatient rehabilitation

BACKGROUND

Fecal incontinence and constipation affect men and women of all ages.

OBJECTIVE

The purpose of this study was to psychometrically analyze the Fecal Incontinence and Constipation Questionnaire (FICQ) in patients seeking outpatient rehabilitation services due to pelvic-floor dysfunction (PFD).

DESIGN

This was a retrospective analysis of cross-sectional data from 644 patients (mean age=52 years, SD=16, range=18-91) being treated for PFD in 64 outpatient rehabilitation clinics in 20 states (United States).

METHODS

We assessed the 20-item FICQ for unidimensionality and local independence, differential item functioning (DIF), item fit, item hierarchical structure, and test precision using an item response theory model.

RESULTS

Factor analyses supported the 2-factor subscales as originally defined; items related to severity of leakage or constipation. Removal of 2 leakage items improved unidimensionality and local independence of the leakage scale. Among the remaining items, 2 items were suggestive of adjustment for DIF by age group and by number of PFD comorbid conditions. Item difficulties were suitable for patients with PFD with no ceiling or floor effect. Mean item difficulty parameters for leakage and constipation subscales ranged from 38.8 to 62.3 and 28.1 to 63.3 (0-100 scale), respectively. Endorsed leakage items representing highest difficulty levels were related to delay defecation and confidence to control bowel leakage. Endorsed constipation items representing highest difficulty levels were related to the need to strain during a bowel movement and the frequency of bowel movements.

LIMITATIONS

A limitation of this study was the lack of medical diagnostic criteria to classify patients.

CONCLUSIONS

After removing 2 items and adjusting for DIF, the results supported sound psychometric properties of the FICQ items and its initial use for patients with PFD in outpatient rehabilitation services.

Statistical considerations in the psychometric validation of outcome measures

The evaluation of the outcomes of total knee arthroplasty requires measurement tools that are valid, reliable, and responsive to change. However, the accuracy of any outcome measurement is determined by the validity and reliability of the instrument used. To ensure this accuracy, it is imperative that each instrument used in orthopaedics is free of biases leading to inaccurate estimates of treatment effects. WHERE ARE WE NOW?: Many patient-derived outcome instruments have been developed and tested through the application of the standard assessments that form the basis of classical test theory: validity, reliability, and responsiveness. These assessments determine if the instrument reliably measures what it is intended to measure, and if it captures differences among groups of patients or changes over time. WHERE DO WE NEED TO GO?: Thorough evaluation of the outcome instruments used in orthopaedics is a critical prerequisite for the continued improvement of effective patient care. Additional steps of psychometric testing that are sometimes overlooked include testing for differential item functioning (DIF) and the effects of the mode of administration of the outcome instrument. The use of suitable approaches to test for these potential sources of bias would facilitate the development of more robust outcome assessment in research and clinical practice. HOW DO WE GET THERE?: Testing for DIF, including the effects of mode of administration, may be performed using several analytical approaches. This will allow optimal application of each outcome instrument with respect to patient characteristics, time and mode of the administration, and modification, as necessary.

Deep massage to posterior calf muscles in combination with neural mobilization exercises as a treatment for heel pain: a pilot randomized clinical trial

BACKGROUND

Plantar heel pain syndrome (PHPS) is a common foot disorder; however, there is limited clinical evidence on which to base treatment. Repeated clinical observations indicating heel pain during heel rise and minisquat on the affected leg, involving activation of posterior calf muscles, formed the basis of this study.

OBJECTIVE

To compare deep massage therapy to posterior calf muscles and neural mobilization with a self-stretch exercise program (DMS) to a common treatment protocol of ultrasound therapy to the painful heel area with the same self-stretch exercises (USS).

METHODS

Patients with PHPS were assigned to a program of 8 treatments over a period of 4-6 weeks in a single-blind randomized clinical trial. Functional status (FS) at admission and discharge from therapy as measured by the Foot & Ankle Computerized Adaptive Test was the main outcome measure.

RESULTS

Sixty-nine patients were included in the trial (mean age 53, standard deviation (SD) 13, range 25-86, 57% women), 36 received DMS treatment and 33 with USS. The overall group-by-time interaction for the mixed-model analysis of variance (ANOVA) was found statistically significant (p=0.034), with a change of (mean (confidence interval, CI)) 15 (9-21) and 6 (1-11) FS points for the DMS and USS groups, respectively.

CONCLUSIONS

Data indicated that both treatment protocols resulted in an overall short-term improvement, however, DMS treatment was significantly more effective in treating PHPS than USS treatment.

Psychometric properties and practicability of the self-report urinary incontinence questionnaire in patients with pelvic-floor dysfunction seeking outpatient rehabilitation

BACKGROUND

Pelvic-floor dysfunction (PFD) affects a substantial proportion of individuals, mostly women. In responding to the demands in measuring PFD outcomes in outpatient rehabilitation, the Urinary Incontinence Questionnaire (UIQ) was developed by FOTO in collaboration with an experienced physical therapist who has a specialty in treating patients with PFD.

OBJECTIVE

The purpose of this study was to evaluate psychometric properties and practicability of the 21-item UIQ in patients seeking outpatient physical therapy services due to PFD.

DESIGN

This was a retrospective analysis of cross-sectional data from 1,628 patients (mean age=53 years, SD=16, range=18-91) being treated for their PFD in 91 outpatient physical therapy clinics in 24 states (United States).

METHODS

Using a 2-parameter logistic item response theory (IRT) procedure and the graded response model, the UIQ was assessed for unidimensionality and local independence, differential item functioning (DIF), discriminating ability, item hierarchical structure, and test precision.

RESULTS

Four items were dropped to improve unidimensionality and discriminating ability. Remaining UIQ items met IRT assumptions of unidimensionality and local independence. One item was adjusted for DIF by age group. Item difficulties were suitable for patients with PFD with no ceiling or floor effect. Item difficulty parameters ranged from -2.20 to 0.39 logits. Endorsed items representing highest difficulty levels were related to control urine flow, impact of leaking urine on life, and confidence to control the urine leakage problem. Item discrimination parameters ranged from 0.48 to 1.18. Items with higher discriminating abilities were those related to impact on life of leaking urine, confidence to control the urine leakage problem, and the number of protective garments for urine leakage.

LIMITATIONS

Because this study was a secondary analysis of prospectively collected data, missing data might have influenced our results.

CONCLUSIONS

Preliminary analyses supported sound psychometric properties of the UIQ items and their initial use for patients with PFD in outpatient physical therapy services.

Differential Item Functioning in a Computerized Adaptive Test of Functional Status for People with Shoulder Impairments is Negligible across Pain Intensity, Gender, and Age Groups

People with shoulder impairments ( N = 3,767) reported upper extremity function using a 37-item shoulder-specific computerized adaptive test (shoulder CAT). The authors determined whether items of the shoulder CAT have differential item functioning (DIF) by pain intensity (low and high), gender (men and women), and age groups (young-adult, middle-aged and old-adult). They assessed whether items have uniform and/or non-uniform DIF using an ordinal logistic regression and item response theory approaches and applied large and small DIF criteria to assess the magnitude of DIF. The analyses revealed that uniform DIF was absent in all 37 items. Only six items exhibited non-uniform DIF using the large DIF criterion. Adjusting the person-ability measures for DIF had minimal practical impact on the overall measure of shoulder function estimated using the shoulder CAT. The shoulder CAT provided a precise measurement of function without discriminating for pain intensity, gender, and age among patients referred to rehabilitation with shoulder impairment.

Examining multiple sources of differential item functioning on the Clinician & Group CAHPS® survey

OBJECTIVE

To evaluate psychometric properties of a widely used patient experience survey.

DATA SOURCES

English-language responses to the Clinician & Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS®) survey (n = 12,244) from a 2008 quality improvement initiative involving eight southern California medical groups.

METHODS

We used an iterative hybrid ordinal logistic regression/item response theory differential item functioning (DIF) algorithm to identify items with DIF related to patient sociodemographic characteristics, duration of the physician-patient relationship, number of physician visits, and self-rated physical and mental health. We accounted for all sources of DIF and determined its cumulative impact.

PRINCIPAL FINDINGS

The upper end of the CG-CAHPS® performance range is measured with low precision. With sensitive settings, some items were found to have DIF. However, overall DIF impact was negligible, as 0.14 percent of participants had salient DIF impact. Latinos who spoke predominantly English at home had the highest prevalence of salient DIF impact at 0.26 percent.

CONCLUSIONS

The CG-CAHPS® functions similarly across commercially insured respondents from diverse backgrounds. Consequently, previously documented racial and ethnic group differences likely reflect true differences rather than measurement bias. The impact of low precision at the upper end of the scale should be clarified.

Construct validation of a knee-specific functional status measure: a comparative study between the United States and Israel

BACKGROUND

Comparative effectiveness research (CER) requires valid outcome measures that discriminate patients by risk factors in similar ways across settings. Standardized functional status (FS) measures in physical therapy are used routinely in multiple countries, creating the potential for CER among countries.

OBJECTIVE

The purpose of this study was to assess known-groups construct validity of a knee-specific FS measure within and between 2 countries for patients receiving outpatient physical therapy due to knee impairments.

DESIGN

This was a longitudinal, observational cohort study.

METHODS

The participants were 4,972 and 2,964 adult (age ≥18 years) patients with knee impairments from Israel and the United States, respectively. Differences in patient characteristics between the 2 countries were assessed using chi-square statistics and 2-sample t tests, as appropriate. Known-groups validity within and between the countries was assessed using 2-way analysis of covariance predicting FS at discharge, with sex, age, symptom acuity, surgical and exercise history, intake medication use, and country as risk-adjustment factors. Intake FS was the covariate. To compare how FS discriminated patient groups between countries, each factor was tested separately with models including an interaction term between the factor and country.

RESULTS

Patients were different between countries but had similar discharge FS trends, including: higher outcomes in patients who were male, were younger, had acute conditions, had one surgical procedure related to their knee impairment, were more physically active, and did not use related medication at admission. Interactions were not significant for sex, symptom acuity, and exercise history but were significant for age, surgical history, and medication use. Limitations Although strict patient selection criteria were set, some patient selection bias still might have existed.

CONCLUSIONS

The results demonstrated the knee FS measures would be valid for use in CER between Hebrew-speaking patients (Israel) and English-speaking patients (United States).

Association between directional preference and centralization in patients with low back pain

STUDY DESIGN

Prospective, longitudinal, observational cohort.

OBJECTIVES

Primary aims were to determine (1) baseline prevalence of directional preference (DP) or no directional preference (no-DP) observed for patients with low back pain whose symptoms centralized (CEN), did not centralize (non-CEN), or could not be classified (NC), and (2) to determine if classifying patients at intake by DP or no-DP combined with CEN, non-CEN, or NC predicted functional status and pain intensity at discharge from rehabilitation.

BACKGROUND

Although evidence suggests that patient response classification criteria DP or CEN improve outcomes, previous studies did not delineate relations between DP and CEN findings and outcomes.

METHODS

Eight therapists classified patients using standardized definitions for DP and CEN. Prevalence rates for DP and no-DP and CEN,non-CEN, and NC were calculated. Ordinary least-squares multivariate regression models assessed whether multilevel classification combining DP and CEN (DP/CEN, DP/non-CEN, DP/NC, no-DP/non-CEN, and no-DP/NC categories) predicted discharge functional status (scale range, 0 to 100, with higher values representing better function) or pain intensity (scale range, 0 to 10, with higher values representing more pain).

RESULTS

Overall prevalence of DP and CEN was 60% and 41%, respectively. For those with DP, prevalence rates for DP/CEN, DP/non-CEN, and DP/NC were 65%, 27%, and 8%, respectively. The amount of variance explained (R2 values) for function and pain models was 0.50 and 0.39, respectively. Compared to patients classified as DP/CEN, patients classified as DP/non-CEN or no-DP/non-CEN reported 7.7 and 11.6 functional status units less at discharge (P<.001), respectively, and patients classified as no-DP/non-CEN reported 1.7 pain units more at discharge (P<.001).

CONCLUSIONS

Findings suggest that classification by pain pattern and DP can improve a therapist's ability to provide a short-term prognosis for function and pain outcomes.

LEVEL OF EVIDENCE

Prognosis, level 1b-.

Cross-cultural differences in knee functional status outcomes in a polyglot society represented true disparities not biased by differential item functioning

BACKGROUND

Comparative effectiveness research across cultures requires unbiased measures that accurately detect clinical differences between patient groups.

OBJECTIVE

The purpose of this study was to assess the presence and impact of differential item functioning (DIF) in knee functional status (FS) items administered using computerized adaptive testing (CAT) as a possible cause for observed differences in outcomes between 2 cultural patient groups in a polyglot society.

DESIGN

This study was a secondary analysis of prospectively collected data.

METHODS

We evaluated data from 9,134 patients with knee impairments from outpatient physical therapy clinics in Israel. Items were analyzed for DIF related to sex, age, symptom acuity, surgical history, exercise history, and language used to complete the functional survey (Hebrew versus Russian).

RESULTS

Several items exhibited DIF, but unadjusted FS estimates and FS estimates that accounted for DIF were essentially equal (intraclass correlation coefficient [2,1]>.999). No individual patient had a difference between unadjusted and adjusted FS estimates as large as the median standard error of the unadjusted estimates. Differences between groups defined by any of the covariates considered were essentially unchanged when using adjusted instead of unadjusted FS estimates. The greatest group-level impact was <0.3% of 1 standard deviation of the unadjusted FS estimates.

LIMITATIONS

Complete data where patients answered all items in the scale would have been preferred for DIF analysis, but only CAT data were available.

CONCLUSIONS

Differences in FS outcomes between groups of patients with knee impairments who answered the knee CAT in Hebrew or Russian in Israel most likely reflected true differences that may reflect societal disparities in this health outcome.

Differential item functioning (DIF) analyses of health-related quality of life instruments using logistic regression

BACKGROUND

Differential item functioning (DIF) methods can be used to determine whether different subgroups respond differently to particular items within a health-related quality of life (HRQoL) subscale, after allowing for overall subgroup differences in that scale. This article reviews issues that arise when testing for DIF in HRQoL instruments. We focus on logistic regression methods, which are often used because of their efficiency, simplicity and ease of application.

METHODS

A review of logistic regression DIF analyses in HRQoL was undertaken. Methodological articles from other fields and using other DIF methods were also included if considered relevant.

RESULTS

There are many competing approaches for the conduct of DIF analyses and many criteria for determining what constitutes significant DIF. DIF in short scales, as commonly found in HRQL instruments, may be more difficult to interpret. Qualitative methods may aid interpretation of such DIF analyses.

CONCLUSIONS

A number of methodological choices must be made when applying logistic regression for DIF analyses, and many of these affect the results. We provide recommendations based on reviewing the current evidence. Although the focus is on logistic regression, many of our results should be applicable to DIF analyses in general. There is a need for more empirical and theoretical work in this area.