Computer adaptive testing improved accuracy and precision of scores over random item selection in a physical functioning item bank

Computerized-adaptive testing versus short forms for pediatric inflammatory bowel disease patient-reported outcome assessment

Introduction

Computerized-adaptive testing (CAT) may increase reliability or reduce respondent burden for assessing patient-reported outcomes compared with static short forms (SFs). We compared CAT versus SF administration of the Patient-Reported Outcomes Measurement Information System® (PROMIS®) Pediatric measures in pediatric inflammatory bowel disease (IBD).

Methods

Participants completed 4-item CAT, 5- or 6-item CAT, and 4-item SF versions of the PROMIS Pediatric measures. We compared average T-scores, intra-class correlations (ICCs), floor and ceiling effects, and standard error of measurement (SEM) across forms, along with mean effect sizes between active versus quiescent IBD disease activity groups.

Results

Average PROMIS T-scores across forms were <3 points (minimally important difference) of each other. All forms correlated highly with each other (ICCs ≥0.90) and had similar ceiling effects, but the CAT-5/6 had lower floor effects. The CAT-5/6 had lower SEM than the CAT-4 and SF-4, and the CAT-4 had a lower SEM than the SF-4. Mean effect sizes were similar across forms when contrasting disease activity groups.

Conclusions

The CAT and SF forms produced similar score results, but the CAT had better precision and lower floor effects. Researchers should consider PROMIS pediatric CAT if they anticipate that their sample will skew toward symptom extremes.

A High-Sensitivity International Knee Documentation Committee Survey Index From the PROMIS System: The Next-Generation Patient-Reported Outcome for a Knee Injury Population

BACKGROUND

Patient-reported outcomes (PROs) measure progression and quality of care. While legacy PROs such as the International Knee Documentation Committee (IKDC) survey are well-validated, a lengthy PRO creates a time burden on patients, decreasing adherence. In recent years, PROs such as the Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function and Pain Interference surveys were developed as computer adaptive tests, reducing time to completion. Previous studies have examined correlation between legacy PROs and PROMIS; however, no studies have developed effective prediction models utilizing PROMIS to create an IKDC index. While the IKDC is the standard knee PRO, computer adaptive PROs offer numerous practical advantages.

PURPOSE

To develop a nonlinear predictive model utilizing PROMIS Physical Function and Pain Interference to estimate IKDC survey scores and examine algorithm sensitivity and validity.

STUDY DESIGN

Cohort study (diagnosis); Level of evidence, 3.

METHODS

The MOTION (Military Orthopaedics Tracking Injuries and Outcomes Network) database is a prospectively collected repository of PROs and intraoperative variables. Patients undergoing knee surgery completed the IKDC and PROMIS surveys at varying time points. Nonlinear multivariable predictive models using Gaussian and beta distributions were created to establish an IKDC index score, which was then validated using leave-one-out techniques and minimal clinically important difference analysis.

RESULTS

A total of 1011 patients completed the IKDC and PROMIS Physical Function and Pain Interference, providing 1618 complete observations. The algorithms for the Gaussian and beta distribution were validated to predict the IKDC (Pearson = 0.84-0.86; R² = 0.71-0.74; root mean square error = 9.3-10.0).

CONCLUSION

The publicly available predictive models can approximate the IKDC score. The results can be used to compare PROMIS Physical Function and Pain Interference against historical IKDC scores by creating an IKDC index score. Serial use of the IKDC index allows for a lower minimal clinically important difference than the conventional IKDC. PROMIS can be substituted to reduce patient burden, increase completion rates, and produce orthopaedic-specific survey analogs.

Linking AM-PAC Cognition to PROMIS Cognitive Function

OBJECTIVE

To link the Activity Measure for Post-Acute Care (AM-PAC) Applied Cognition to the Patient-Reported Outcomes Measurement Information System (PROMIS) Cognitive Function, allowing for a common metric across scales.

DESIGN

Cross-sectional survey study.

SETTING

Outpatient rehabilitation clinics.

PARTICIPANTS

Consecutive sample of 500 participants (N=500) aged ≥18 years presenting for outpatient therapy (physical, occupation, speech).

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

AM-PAC Medicare and Generic Cognition short forms and PROMIS Cognitive Function items representing the PROMIS Cognitive Function item bank.

RESULTS

The calibration of 25 AM-PAC cognition items with 11 fixed PROMIS cognitive function item parameters using item-response theory indicated that items were measuring the same underlying construct (cognition). Both scales measured a wide range of functioning. The AM-PAC Generic Cognitive assessment showed more reliability with lower levels of cognition, whereas the PROMIS Cognitive Function full-item bank was more reliable across a larger distribution of scores. Data were appropriate for a fixed-anchor item response theory-based crosswalk and AM-PAC Cognition raw scores were mapped onto the PROMIS metric.

CONCLUSIONS

The crosswalk developed in this study allows for converting scores from the AM-PAC Applied Cognition to the PROMIS Cognitive Function scale.

Improving the Delivery of Function-Directed Care During Acute Hospitalizations: Methods to Develop and Validate the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT)

OBJECTIVE

To (1) develop a patient-reported, multidomain functional assessment tool focused on medically ill patients in acute care settings; (2) characterize the measure's psychometric performance; and (3) establish clinically actionable score strata that link to easily implemented mobility preservation plans.

DESIGN

This article describes the approach that our team pursued to develop and characterize this tool, the Functional Assessment in Acute Care Multidimensional Computer Adaptive Test (FAMCAT). Development involved a multistep process that included (1) expanding and refining existing item banks to optimize their salience for hospitalized patients; (2) administering candidate items to a calibration cohort; (3) estimating multidimensional item response theory models; (4) calibrating the item banks; (5) evaluating potential multidimensional computerized adaptive testing (MCAT) enhancements; (6) parameterizing the MCAT; (7) administering it to patients in a validation cohort; and (8) estimating its predictive and psychometric characteristics.

SETTING

A large (2000-bed) Midwestern Medical Center.

PARTICIPANTS

The overall sample included 4495 adults (2341 in a calibration cohort, 2154 in a validation cohort) who were admitted either to medical services with at least 1 chronic condition or to surgical/medical services if they required readmission after a hospitalization for surgery (N=4495).

INTERVENTION

Not applicable.

MAIN OUTCOME MEASURES

Not applicable.

RESULTS

The FAMCAT is an instrument designed to permit the efficient, precise, low-burden, multidomain functional assessment of hospitalized patients. We tried to optimize the FAMCAT's efficiency and precision, as well as its ability to perform multiple assessments during a hospital stay, by applying cutting edge methods such as the adaptive measure of change (AMC), differential item functioning computerized adaptive testing, and integration of collateral test-taking information, particularly item response times. Evaluation of these candidate methods suggested that all may enhance MCAT performance, but none were integrated into initial MCAT parameterization.

CONCLUSIONS

The FAMCAT has the potential to address a longstanding need for structured, frequent, and accurate functional assessment among patients hospitalized with medical diagnoses and complications of surgery.

Linking AM-PAC Mobility and Daily Activity to the PROMIS Physical Function Metric

OBJECTIVE

The purpose of this study was to link Activity Measure for Post-Acute Care (AM-PAC) Mobility and Daily Activity scales to the PROMIS Physical Function (PF) allowing for a common metric across scales.

METHODS

Cross-sectional study of patients 18 years and older presenting to 1 of 8 outpatient rehabilitation clinics in southwestern Pennsylvania. Patients completed one survey with questions from the AM-PAC Daily Activity and Mobility short forms, and the PROMIS PF item bank. Using item response theory, 2 rounds of fixed-parameter calibration were performed. In the first, the AM-PAC Daily Activity and Mobility items were calibrated with 27 fixed item parameters from the PROMIS PF. Second, the AM-PAC Daily Activity items were calibrated with 11 PROMIS Upper Extremity fixed item parameters. This process uses the construct of physical function and equates AM-PAC items on the same underlying measurement scale for the PROMIS PF.

RESULTS

Both scales measured a wide range of functioning and demonstrated good calibration. Data were appropriate for a fixed-parameter item response theory-based crosswalk. AM-PAC Daily Activity and Mobility raw scores were mapped onto the PROMIS PF metric. AM-PAC Daily Activity scores were also mapped onto the PROMIS PF Upper Extremity metric.

CONCLUSION

Question items from the AM-PAC Daily Activity, AM-PAC Mobility, and PROMIS PF similarly measure the construct of physical function. This consistency allows for a crosswalk of AM-PAC scores onto the PROMIS PF metric.

IMPACT

Crosswalk tables developed in this study allow for converting scores from the AM-PAC Daily Activity and Mobility scales to the PROMIS PF metric. This will facilitate monitoring of longitudinal change in function over time and across settings.

Generating the American Shoulder and Elbow Surgeons Score Using Multivariable Predictive Models and Computer Adaptive Testing to Reduce Survey Burden

BACKGROUND

The preferred patient-reported outcome measure for the assessment of shoulder conditions continues to evolve. Previous studies correlating the Patient-Reported Outcomes Measurement Information System (PROMIS) computer adaptive tests (CATs) to the American Shoulder and Elbow Surgeons (ASES) score have focused on a singular domain (pain or physical function) but have not evaluated the combined domains of pain and physical function that compose the ASES score. Additionally, previous studies have not provided a multivariable prediction tool to convert PROMIS scores to more familiar legacy scores.

PURPOSE

To establish a valid predictive model of ASES scores using a nonlinear combination of PROMIS domains for physical function and pain.

STUDY DESIGN

Cohort study (Diagnosis); Level of evidence, 3.

METHODS

The Military Orthopaedics Tracking Injuries and Outcomes Network (MOTION) database is a prospectively collected repository of patient-reported outcomes and intraoperative variables. Patients in MOTION research who underwent shoulder surgery and completed the ASES, PROMIS Physical Function, and PROMIS Pain Interference at varying time points were included in the present analysis. Nonlinear multivariable predictive models were created to establish an ASES index score and then validated using "leave 1 out" techniques and minimal clinically important difference /substantial clinical benefit (MCID/SCB) analysis.

RESULTS

A total of 909 patients completed the ASES, PROMIS Physical Function, and PROMIS Pain Interference at presurgery, 6 weeks, 6 months, and 1 year after surgery, providing 1502 complete observations. The PROMIS CAT predictive model was strongly validated to predict the ASES (Pearson coefficient = 0.76-0.78; R² = 0.57-0.62; root mean square error = 13.3-14.1). The MCID/SCB for the ASES was 21.7, and the best ASES index MCID/SCB was 19.4, suggesting that the derived ASES index is effective and can reliably re-create ASES scores.

CONCLUSION

The PROMIS CAT predictive models are able to approximate the ASES score within 13 to 14 points, which is 7 points more accurate than the ASES MCID/SCB derived from the sample. Our ASES index algorithm, which is freely available online (https://osf.io/ctmnd/), has a lower MCID/SCB than the ASES itself. This algorithm can be used to decrease patient survey burden by 11 questions and provide a reliable ASES analog to clinicians.

Development of an Item Bank to Measure Medication Adherence: Systematic Review

Background

Medication adherence is important in managing the progression of chronic diseases. A promising approach to reduce cognitive burden when measuring medication adherence lies in the use of computer‐adaptive tests (CATs) or in the development of shorter patient-reported outcome measures (PROMs). However, the lack of an item bank currently hampers this progress.

Objective

We aim to develop an item bank to measure general medication adherence.

Methods

Using the preferred reporting items for systematic review and meta-analysis (PRISMA), articles published before October 2019 were retrieved from PubMed, Embase, CINAHL, the Cochrane Library, and Web of Science. Items from existing PROMs were classified and selected (“binned” and “winnowed”) according to standards published by the Patient-Reported Outcomes Measurement Information System (PROMIS) Cooperative Group.

Results

A total of 126 unique PROMs were identified from 213 studies in 48 countries. Items from the literature review (47 PROMs with 579 items for which permission has been obtained) underwent binning and winnowing. This resulted in 421 candidate items (77 extent of adherence and 344 reasons for adherence).

Conclusions

We developed an item bank for measuring general medication adherence using items from validated PROMs. This will allow researchers to create new PROMs from selected items and provide the foundation to develop CATs.

Performance of the Patient-Reported Outcome Measurement Information System in Patients With Patellofemoral Instability

Background:

The Patient-Reported Outcome Measurement Information System (PROMIS) was developed to improve patient-reported outcome measures (PROMs) and administration through a computer adaptive test (CAT). The PROMIS physical function-CAT (PF-CAT) has not been investigated in patients with patellofemoral instability (PFI).

Purpose/Hypothesis:

The purpose of this study was to evaluate the construct validity of the PROMIS PF-CAT with previously validated tools for measuring PROMs in patients with a diagnosis of PFI. We hypothesized that the PF-CAT will have the strongest correlations with other PROMs that evaluated PF as well as moderate correlations with PROMs that measured other health domains.

Study Design:

Cohort study (diagnosis); Level of evidence, 2.

Methods:

Patients enrolled in this study who underwent operative intervention for PFI completed the following evaluations preoperatively: PROMIS PF-CAT, 36-Item Short Form Health Survey (SF-36), Knee injury and Osteoarthritis Outcome Score (KOOS), EuroQol-5 dimensions (EQ-5D), and Kujala Anterior Knee Pain Scale (AKPS). Correlation coefficients and the percentage of patients achieving the highest and lowest possible outcome score of each instrument were calculated to assess floor and ceiling effects. Statistical significance was defined as P < .05.

Results:

In total, 91 participants (63.7% females; mean age, 20.1 ± 7.2 years) completed the questionnaires. PF-CAT had the lowest number of questions (4.3 ± 1.1). The strongest correlations were between the PF-CAT and SF-36 PF subscale (r = 0.78; P < .01), AKPS (r = 0.68; P < .01), and KOOS Activities of Daily Living subscale (r = 0.68; P < .01). Correlation was moderate between the PF-CAT and the KOOS subscales of Sports/Recreation (r = 0.58; P < .01), Quality of Life (r = 0.53; P < .01), and Symptoms (r = 0.47; P < .01). The PROMIS PF-CAT demonstrated no floor or ceiling effects.

Conclusion:

In patients with PFI, construct validity of the PROMIS PF-CAT was supported by strong correlations demonstrated between the PF-CAT and PROMs evaluating PF and moderate correlations with those assessing other health domains. Our results demonstrated a low respondent burden and no floor or ceiling effects associated with the PROMIS PF-CAT. The PROMIS PF-CAT may be considered a beneficial alternative to previously established PF PROMs for preoperative evaluation of patients with PFI.

Psychometric properties of the Dutch-Flemish Patient-Reported Outcomes Measurement Information System Pain Behavior item bank in patients with musculoskeletal complaints

We studied the psychometric properties of the 39-item v1.1 Dutch-Flemish Patient-Reported Outcomes Measurement Information System Pain Behavior item bank in a sample of 1,602 patients with musculoskeletal complaints. We evaluated the assumptions of the underlying item response theory (IRT) model (unidimensionality and local dependency with confirmatory factor analyses), and monotonicity with scalability coefficients). We studied the IRT model fit of all items and estimated the item parameters of the IRT model. Differential item functioning (DIF) was studied for age and gender, and DIF for language was studied as a measure of cross-cultural validity. Confirmatory factor analyses showed suboptimal fit of a unidimensional model, but a bifactor model showed low risk of bias when a unidimensional model was assumed (Omega H = .92, explained common variance of .70). Fifteen item pairs (2%) were locally dependent. Five items showed poor scalability. All items fitted the IRT model; slope parameters ranged from .60 to 2.00, and threshold parameters from -2.05 to 6.80. One item showed DIF for age, 1 item DIF for gender, and 5 items showed DIF for language, but the impact on total scores was low. Our study shows limitations of the Dutch-Flemish Patient-Reported Outcomes Measurement Information System Pain Behavior item bank when used in a primary care population with musculoskeletal complaints. PERSPECTIVE: We studied the psychometric properties of the Dutch-Flemish Patient-Reported Outcomes Measurement Information System Pain Behavior item bank in a large primary care population of patients with musculoskeletal complaints. It showed that the Pain Behavior item bank has limitations when used in this population.

Validation of the Dutch-Flemish PROMIS Pain Interference Item Bank in Patients With Musculoskeletal Complaints

STUDY DESIGN

A cross-sectional study.

OBJECTIVE

The aim of this study was to validate the Dutch-Flemish PROMIS Pain Interference item bank in patients with musculoskeletal complaints.

SUMMARY OF BACKGROUND DATA

PROMIS item banks have been developed and validated in the US. They need to be further validated in various patient populations and in different languages.

METHODS

One thousand six hundred seventy-seven patients answered the full item bank. A Graded Response Model (GRM) was used to study dimensionality with confirmatory factor analyses and by assessing local independency. Monotonicity was evaluated with Mokken scaling. An Item Response Theory (IRT) model was used to study item fit and to estimate slope and threshold parameters. Differential item functioning (DIF) for language, age, and gender was assessed using ordinal logistic regression analyses. DIF for language was evaluated by comparing our data with a similar US sample. Hypotheses concerning construct validity were tested by correlating item bank-scores with scores on several legacy instruments.

RESULTS

The GRM showed suboptimal evidence of unidimensionality in confirmatory factor analysis [Comparative Fit Index (CFI): 0.903, Tucker-Lewis Index (TLI): 0.897, Root Mean Square Error of Approximation (RMSEA): 0.144], and 99 item pairs with local dependence. A bifactor model showed good fit (CFI: 0.964, TLI: 0.961, RMSEA: 0.089), with a high Omega-H (0.97), a high explained common variance (ECV: 0.81), and no local dependence. Sufficient monotonicity was shown for all items (Mokken H(i): 0.367-0.686). The unidimensional IRT model showed good fit (only two items with S-X < 0.001), with slope parameters ranging from 1.00 to 4.27, and threshold parameters ranging from -1.77 to 3.66. None of the items showed DIF for age or gender. One item showed DIF for language. Correlations with legacy instruments were high (Pearson R: 0.53-0.75), supporting construct validity.

CONCLUSION

The high omega-H and the high ECV indicate that the item bank could be considered essentially unidimensional. The item bank showed good item fit, good coverage of the pain interference trait, and good construct validity.

LEVEL OF EVIDENCE

N/A.

Quality of life outcomes in patients presenting for evaluation of CNS tumors

Background

We describe patient-reported outcomes (PROs) in adults with CNS tumors and evaluate their correlation with physician-reported functional status.

Methods

We completed a retrospective cohort study of patients managed at a high-volume CNS tumor institute between September 2013 and September 2014. PROs were measured using 6 domains from the PROs Measurement Information System (PROMIS): anxiety, physical function, pain interference, sleep disturbance, fatigue, and satisfaction with social roles. Physician-reported outcomes were measured using the Eastern Cooperative Oncology Group Scale of Performance Status (ECOG). We compared differences in PROMIS scores across tumor types using analysis of variance and measured the correlation between PROMIS scores and ECOG scores using spearman correlations. Finally, we compared the range of PROMIS physical function scores within each ECOG level.

Results

In a cohort of 2,828 patients, 1,284 (45.4%) completed all 6 PROMIS domains. There were significant differences in PROMIS scores across tumor types for all domains except anxiety. The strength of the correlation between PROMIS and ECOG scores was weak to moderate for all PROMIS domains (all p < 0.001). The correlation was the strongest between the physical function domain and ECOG score (ρ = -0.54), although there was a broad distribution of physical function scores within ECOG level, with scores spanning nearly 5 SDs within most ECOG levels.

Conclusions

Symptom burden was associated with tumor type. There were only weak to moderate correlations between PROMIS and ECOG scores, underscoring the importance of integrating PROs into clinical practice for patients with CNS tumors.

A comparison of computerized adaptive testing and fixed-length short forms for the Prosthetic Limb Users Survey of Mobility (PLUS-MTM)

BACKGROUND

New health status instruments can be administered by computerized adaptive test or short forms. The Prosthetic Limb Users Survey of Mobility (PLUS-MTM) is a self-report measure of mobility for prosthesis users with lower limb loss. This study used the PLUS-M to examine advantages and disadvantages of computerized adaptive test and short forms.

OBJECTIVES

To compare scores obtained from computerized adaptive test to scores obtained from fixed-length short forms (7-item and 12-item) in order to provide guidance to researchers and clinicians on how to select the best form of administration for different uses.

STUDY DESIGN

Cross-sectional, observational study.

METHODS

Individuals with lower limb loss completed the PLUS-M by computerized adaptive test and short forms. Administration time, correlations between the scores, and standard errors were compared.

RESULTS

Scores and standard errors from the computerized adaptive test, 7-item short form, and 12-item short form were highly correlated and all forms of administration were efficient. Computerized adaptive test required less time to administer than either paper or electronic short forms; however, time savings were minimal compared to the 7-item short form.

CONCLUSION

Results indicate that the PLUS-M computerized adaptive test is most efficient, and differences in scores between administration methods are minimal. The main advantage of the computerized adaptive test was more reliable scores at higher levels of mobility compared to short forms. Clinical relevance Health-related item banks, like the Prosthetic Limb Users Survey of Mobility (PLUS-MTM), can be administered by computerized adaptive testing (CAT) or as fixed-length short forms (SFs). Results of this study will help clinicians and researchers decide whether they should invest in a CAT administration system or whether SFs are more appropriate.

Measuring pain phenomena after spinal cord injury: Development and psychometric properties of the SCI-QOL Pain Interference and Pain Behavior assessment tools

OBJECTIVE

To develop modern patient-reported outcome measures that assess pain interference and pain behavior after spinal cord injury (SCI).

DESIGN

Grounded-theory based qualitative item development; large-scale item calibration field-testing; confirmatory factor analyses; graded response model item response theory analyses; statistical linking techniques to transform scores to the Patient Reported Outcome Measurement Information System (PROMIS) metric.

SETTING

Five SCI Model Systems centers and one Department of Veterans Affairs medical center in the United States.

PARTICIPANTS

Adults with traumatic SCI.

INTERVENTIONS

N/A.

OUTCOME MEASURES

Spinal Cord Injury - Quality of Life (SCI-QOL) Pain Interference item bank, SCI-QOL Pain Interference short form, and SCI-QOL Pain Behavior scale.

RESULTS

Seven hundred fifty-seven individuals with traumatic SCI completed 58 items addressing various aspects of pain. Items were then separated by whether they assessed pain interference or pain behavior, and poorly functioning items were removed. Confirmatory factor analyses confirmed that each set of items was unidimensional, and item response theory analyses were used to estimate slopes and thresholds for the items. Ultimately, 7 items (4 from PROMIS) comprised the Pain Behavior scale and 25 items (18 from PROMIS) comprised the Pain Interference item bank. Ten of these 25 items were selected to form the Pain Interference short form.

CONCLUSIONS

The SCI-QOL Pain Interference item bank and the SCI-QOL Pain Behavior scale demonstrated robust psychometric properties. The Pain Interference item bank is available as a computer adaptive test or short form for research and clinical applications, and scores are transformed to the PROMIS metric.

Cultural and semantic equivalence of the activity measure post-acute care (AM-PAC) after its translation into Mandarin Chinese

Purpose: The purpose of this study was to investigate the cultural and semantic equivalence of the Activity Measure Post-Acute Care (AM-PAC) outpatient short forms after they were translated from American English to Mandarin Chinese. Method: This study was conducted with a composite sample of a group of 483 American adults (mean age 63.1 years, 42.4% males) and 553 Taiwanese adults (mean age 60.6 years, 47.2% males) who were undergoing home care or outpatient rehabilitation services. A confirmatory factor analysis tested the cultural equivalence of the AM-PAC dimensions between the US and Taiwanese samples. Semantic equivalence was tested through an item response theory-based differential item functioning (DFI) analysis. Results: Results of the confirmatory factor analysis demonstrated good model-data fit of the AM-PAC in both the US and Taiwanese samples. Significant DFI was found for six Basic Mobility items, two Daily Activity items, and four Applied Cognition items. A DFI adjusted conversion table was generated to transform the raw scores of the measure for comparisons between the two countries. Conclusions: This study provides evidence to support the conceptual equivalence of the AM-PAC when used in rehabilitation patients between American and Taiwanese cultural contexts. Some DFI items between the two groups were found, suggesting that some differences in semantic understanding of these items between cultures require cross-cultural adjustments. Implications for Rehabilitation Cultural equivalence needs to be tested before applying a clinical measure to another context. The findings of this study supported the three activity domains that the Activity Measure Post-Acute Care measures: Basic Mobility, Daily Activity, and Applied Cognition, which are all important rehabilitation outcomes that need to be assessed and monitored across rehabilitation settings. Differential item functioning was observed between the English and Chinese versions of the Activity Measure Post-Acute Care, indicating that linguistic and cultural differences across countries need to be adjusted for before using the translated measure in clinical practice.

Association Between 2 Measures of Cognitive Instrumental Activities of Daily Living and Their Relation to the Montreal Cognitive Assessment in Persons With Stroke

OBJECTIVES

To explore the relation between a computer adaptive functional cognitive questionnaire and a performance-based measure of cognitive instrumental activities of daily living (C-IADL) and to determine whether the Montreal Cognitive Assessment (MoCA) at admission can identify those with C-IADL difficulties at discharge.

DESIGN

Prospective cohort study.

SETTING

Acute inpatient rehabilitation unit of an academic medical center.

PARTICIPANTS

Inpatients (N=148) with a diagnosis of stroke (mean age, 68y; median, 13d poststroke) who had mild cognitive and neurological deficits.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Admission cognitive status was assessed by the MoCA. C-IADL at discharge was assessed by the Executive Function Performance Test (EFPT) bill paying task and Activity Measure of Post-Acute Care (AM-PAC) Applied Cognition scale.

RESULTS

Greater cognitive impairment on the MoCA was associated with more assistance on the EFPT bill paying task (ρ=-.63; P<.01) and AM-PAC Applied Cognition scale (ρ=-.43; P<.01). This relation was nonsignificant for higher MoCA scores and EFPT bill paying task scores. The AM-PAC Applied Cognition scale and the EFPT bill paying task had low agreement in classifying functional performance (Cohen's κ=.20). A receiver operating characteristic curve identified optimal MoCA cutoff scores of 20 and 21 for classifying EFPT bill paying task status and AM-PAC Applied Cognition scale status, respectively. For values above 20 and 21, sensitivity increased whereas specificity decreased for classifying functional deficits. Approximately one third of the participants demonstrated C-IADL deficits on at least 1 C-IADL measure at discharge despite having a MoCA score of ≥26 at admission.

CONCLUSIONS

Questionnaire and performance-based methods of assessment appear to yield different estimates of C-IADL. Low MoCA scores (<20) are more likely to identify those with C-IADL deficits on the EFPT bill paying task. The results suggest that C-IADL should be assessed in those who have mild or no cognitive difficulties at admission.

Development of an IRT-Based Short Form to Assess Applied Cognitive Function in Outpatient Rehabilitation

OBJECTIVE

Item response theory and computerized adaptive testing methods allow the development of a large calibrated item bank from which different subsets of questions can be selected for administration and scored on a common scale. The objective of this study was to develop an outpatient rehabilitation self-report short form for the Activity Measure for Post-Acute Care Applied Cognition item bank.

DESIGN

Using data from a convenience sample of 235 rehabilitation outpatients, item content and item response theory-based test information function parameters were used in item selection. Internal consistency reliability, intraclass correlation coefficient (ICC), and percentage at the lowest (floor) and highest (ceiling) scores were evaluated for the short form and full item bank.

RESULTS

A 15-item short form was developed. The internal consistency of the short form was 0.86. The ICC3,1 for the short form and item bank was 0.97 (95% confidence interval, 0.94-0.98). No floor effects were noted, and ceiling effects were 27.66% (short form) and 26.38% (full item bank).

CONCLUSIONS

The Applied Cognition outpatient short form demonstrated acceptable psychometric properties and provides a bridge to item response theory-based measurement for settings where point-of-care computing is not available.

Interpreting physical and behavioral health scores from new work disability instruments

OBJECTIVE

To develop a system to guide interpretation of scores generated from 2 new instruments measuring work-related physical and behavioral health functioning (Work Disability - Physical Function (WD-PF) and WD - Behavioral Function (WD-BH)).

DESIGN

Cross-sectional, secondary data from 3 independent samples to develop and validate the functional levels for physical and behavioral health functioning.

SUBJECTS

Physical group: 999 general adult subjects, 1,017 disability applicants and 497 work-disabled subjects. Behavioral health group: 1,000 general adult subjects, 1,015 disability applicants and 476 work-disabled subjects.

METHODS

Three-phase analytic approach including item mapping, a modified-Delphi technique, and known-groups validation analysis were used to develop and validate cut-points for functional levels within each of the WD-PF and WD-BH instrument's scales.

RESULTS

Four and 5 functional levels were developed for each of the scales in the WD-PF and WD-BH instruments. Distribution of the comparative samples was in the expected direction: the general adult samples consistently demonstrated scores at higher functional levels compared with the claimant and work-disabled samples.

CONCLUSION

Using an item-response theory-based methodology paired with a qualitative process appears to be a feasible and valid approach for translating the WD-BH and WD-PF scores into meaningful levels useful for interpreting a person's work-related physical and behavioral health functioning.

The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) and how to select an outcome measurement instrument

BACKGROUND

COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) is an initiative of an international multidisciplinary team of researchers who aim to improve the selection of outcome measurement instruments both in research and in clinical practice by developing tools for selecting the most appropriate available instrument.

METHOD

In this paper these tools are described, i.e. the COSMIN taxonomy and definition of measurement properties; the COSMIN checklist to evaluate the methodological quality of studies on measurement properties; a search filter for finding studies on measurement properties; a protocol for systematic reviews of outcome measurement instruments; a database of systematic reviews of outcome measurement instruments; and a guideline for selecting outcome measurement instruments for Core Outcome Sets in clinical trials. Currently, we are updating the COSMIN checklist, particularly the standards for content validity studies. Also new standards for studies using Item Response Theory methods will be developed. Additionally, in the future we want to develop standards for studies on the quality of non-patient reported outcome measures, such as clinician-reported outcomes and performance-based outcomes.

CONCLUSIONS

In summary, we plea for more standardization in the use of outcome measurement instruments, for conducting high quality systematic reviews on measurement instruments in which the best available outcome measurement instrument is recommended, and for stopping the use of poor outcome measurement instruments.

Age, sex, and symptom intensity influence test taking parameters on functional patient-reported outcomes

OBJECTIVE

Item response theory-based patient-reported outcomes such as the Activity Measure for Post Acute Care Computerized Adaptive Test are gaining use because of their flexibility and ease of administration. Their psychometric properties are being explored, but little is known about how respondent characteristics may impact precision. The goal of this study was, therefore, to assess the effects of age, sex, and symptom intensity on respondents' test taking behaviors and scores.

DESIGN

Three hundred eleven adults with late-stage lung cancer were consecutively enrolled between April 2008 and April 2009. Demographics and comorbidities were abstracted from their electronic medical records. The participants were followed on a 3- to 4-wk basis by telephonic interviews that involved administration of the Activity Measure for Post Acute Care Computerized Adaptive Test, followed by numerical rating scales scoring of their pain, fatigue, and dyspnea.

RESULTS

In more than 2538 computerized adaptive test (CAT) sessions, three findings were prominent. First, the women and the older patients took longer to complete CAT sessions, were more likely to skip items, and produced scores with larger standard errors. Second, the respondents with higher levels of dyspnea and fatigue, but not pain, completed their CAT sessions more rapidly and were less likely to skip items. Third, fatigue and dyspnea interact with age but not sex to influence CAT duration and skip count.

CONCLUSIONS

The findings of this study suggest that certain common clinical populations, for example, women, geriatric patients, and patients with intense symptoms, differ systematically in the time they are willing to devote to testing and the precision of their responses. The latter finding, unstable precision, is unlikely to be CAT specific and has implications for the interpretation of the scores of the Activity Measure for Post Acute Care Computerized Adaptive Test and other patient-reported outcomes.

Relative performance of commonly used physical function questionnaires in rheumatoid arthritis and a patient-reported outcomes measurement information system computerized adaptive test

OBJECTIVE

To evaluate and compare the measurement precision and sensitivity to change of the Health Assessment Questionnaire disability index (HAQ DI), the Short Form 36 physical functioning scale (PF-10), and simulated Patient-Reported Outcomes Measurement Information System (PROMIS) physical function computer adaptive tests (CATs) with 5, 10, and 15 items, using item response theory-based simulation studies.

METHODS

The measurement precision of the various physical function instruments was evaluated by calculating root mean square errors (RMSEs) between true physical function levels (latent physical function score) and estimated physical function levels. Measurement precision was evaluated at 9 levels of physical function, with 5,000 simulated response patterns per level. Sensitivity to change was evaluated by the ability of a simple statistical test to detect simulated change scores of small to moderate magnitude (standardized effect sizes 0.20, 0.35, and 0.50).

RESULTS

RMSEs were smaller for the PROMIS physical function 15-item CAT (CAT-15) and CAT-10 than for the HAQ DI and PF-10 across all levels of the latent physical function scale. Only marginal improvement in performance was observed for the CAT-15 compared with the CAT-10, and the CAT-5 performed quite similarly to the HAQ DI and PF-10 across most levels of the latent physical function scale. Substantially improved sensitivity to change was observed for the CAT-10 compared with the HAQ DI and PF-10, particularly in detecting moderate effect sizes.

CONCLUSION

Clearly higher measurement precision was observed for the PROMIS CAT compared with the HAQ DI and PF-10. Higher reliability also translated into lower sample size requirements for detecting changes in clinical status.

Current perspectives and emerging issues on cancer rehabilitation

Cancer rehabilitation is a rapidly emerging and evolving medical field in both Europe and the United States, in large part because of increases in the number of cancer survivors. Although few argue with the need to restore function and quality of life to patients affected by cancer and its treatments, differences exist between European countries with regard to the funding, accessibility, and even the definition of cancer rehabilitation services. In the United States, there is tremendous variability in the provision of rehabilitation services resulting from a variety of factors, including a lack of highly trained cancer rehabilitation physicians and therapists as well as a lack of comprehensive cancer rehabilitation programs, even at the majority of top cancer centers. Although studies evaluating the effectiveness of rehabilitation programs in the cancer setting, particularly exercise, have influenced clinical decision-making in both Europe and the United States for some time, this emerging evidence base also is now starting to influence guideline and policy making. Coordinated research efforts are essential to establish a robust framework to support future investigation and establish shared initiatives. Determining the best way forward for cancer survivors will require investment in large-scale prospective cohort studies that sufficiently describe their rehabilitation needs through the continuum of the survivorship experience.

Development and calibration of an item bank for the assessment of activities of daily living in cardiovascular patients using Rasch analysis

Background

To develop and calibrate the activities of daily living item bank (ADLib-cardio) as a prerequisite for a Computer-adaptive test (CAT) for the assessment of ADL in patients with cardiovascular diseases (CVD).

Methods

After pre-testing for relevance and comprehension a pool of 181 ADL items were answered on a five-point Likert scale by 720 CVD patients, who were recruited in fourteen German cardiac rehabilitation centers. To verify that the relationship between the items is due to one factor, a confirmatory factor analysis (CFA) was conducted. A Mokken analysis was computed to examine the double monotonicity (i.e. every item generates an equivalent order of person traits, and every person generates an equivalent order of item difficulties). Finally, a Rasch analysis based on the partial credit model was conducted to test for unidimensionality and to calibrate the item bank.

Results

Results of CFA and Mokken analysis confirmed a one factor structure and double monotonicity. In Rasch analysis, merging response categories and removing items with misfit, differential item functioning or local response dependency reduced the ADLib-cardio to 33 items. The ADLib-cardio fitted to the Rasch model with a nonsignificant item-trait interaction (chi-square=105.42, df=99; p=0.31). Person-separation reliability was 0.81 and unidimensionality could be verified.

Conclusions

The ADLib-cardio is the first calibrated, unidimensional item bank that allows for the assessment of ADL in rehabilitation patients with CVD. As such, it provides the basis for the development of a CAT for the assessment of ADL in patients with cardiovascular diseases. Calibrating the ADLib-cardio in other than rehabilitation cardiovascular patient settings would further increase its generalizability.

Development of an instrument to measure behavioral health function for work disability: item pool construction and factor analysis

OBJECTIVES

To develop a broad set of claimant-reported items to assess behavioral health functioning relevant to the Social Security disability determination processes, and to evaluate the underlying structure of behavioral health functioning for use in development of a new functional assessment instrument.

DESIGN

Cross-sectional.

SETTING

Community.

PARTICIPANTS

Item pools of behavioral health functioning were developed, refined, and field tested in a sample of persons applying for Social Security disability benefits (N=1015) who reported difficulties working because of mental or both mental and physical conditions.

INTERVENTIONS

None.

MAIN OUTCOME MEASURE

Social Security Administration Behavioral Health (SSA-BH) measurement instrument.

RESULTS

Confirmatory factor analysis (CFA) specified that a 4-factor model (self-efficacy, mood and emotions, behavioral control, social interactions) had the optimal fit with the data and was also consistent with our hypothesized conceptual framework for characterizing behavioral health functioning. When the items within each of the 4 scales were tested in CFA, the fit statistics indicated adequate support for characterizing behavioral health as a unidimensional construct along these 4 distinct scales of function.

CONCLUSIONS

This work represents a significant advance both conceptually and psychometrically in assessment methodologies for work-related behavioral health. The measurement of behavioral health functioning relevant to the context of work requires the assessment of multiple dimensions of behavioral health functioning. Specifically, we identified a 4-factor model solution that represented key domains of work-related behavioral health functioning. These results guided the development and scale formation of a new SSA-BH instrument.

Computerized adaptive testing--ready for ambulatory monitoring?

BACKGROUND

Computerized adaptive tests (CATs) have abundant theoretical advantages over established static instruments, which could improve ambulatory monitoring of patient-reported outcomes (PROs). However, an empirical demonstration of their practical benefits is warranted.

METHODS

We reviewed the literature and evaluated existing data to discuss the potential of CATs for use in ambulatory monitoring outside clinical facilities.

RESULTS

Computerized adaptive tests are not being used for ambulatory monitoring, but initial results from their use in health care research allow for discussion of some issues relevant to ambulatory care. Evidence shows that CATs can capture the most relevant health outcomes as well as established static tools, with substantially decreased respondent burden. They can be more precise than static tools of similar length and can reduce floor and ceiling effects. Computerized adaptive tests can reliably measure a construct over time with different items, which yields the potential of introducing item exposure control in ambulatory monitoring. Studies have shown that CATs can be at least as valid as well-designed static tools in group comparisons, but further investigation is needed to determine whether psychometric advantages lead to increased responsiveness of CATs.

CONCLUSIONS

Ambulatory monitoring of PROs demands short, yet very precise measurements, which can be repeated up to many times a day. Computerized adaptive tests may address several present shortcomings in ambulatory monitoring of PROs efficiently. However, most CAT developments have primarily focused on psychometric improvements. To use the full potential of CATs for ambulatory monitoring purposes, content must also be carefully considered.

Computerized adaptive testing: the capitalization on chance problem

This paper describes several simulation studies that examine the effects of capitalization on chance in the selection of items and the ability estimation in CAT, employing the 3-parameter logistic model. In order to generate different estimation errors for the item parameters, the calibration sample size was manipulated (N = 500, 1000 and 2000 subjects) as was the ratio of item bank size to test length (banks of 197 and 788 items, test lengths of 20 and 40 items), both in a CAT and in a random test. Results show that capitalization on chance is particularly serious in CAT, as revealed by the large positive bias found in the small sample calibration conditions. For broad ranges of theta, the overestimation of the precision (asymptotic Se) reaches levels of 40%, something that does not occur with the RMSE (theta). The problem is greater as the item bank size to test length ratio increases. Potential solutions were tested in a second study, where two exposure control methods were incorporated into the item selection algorithm. Some alternative solutions are discussed.

Accuracy and precision of the Pediatric Evaluation of Disability Inventory computer-adaptive tests (PEDI-CAT)

AIM

The aims of the study were to: (1) build new item banks for a revised version of the Pediatric Evaluation of Disability Inventory (PEDI) with four content domains: daily activities, mobility, social/cognitive, and responsibility; and (2) use post-hoc simulations based on the combined normative and disability calibration samples to assess the accuracy and precision of the PEDI computer-adaptive tests (PEDI-CAT) compared with the administration of all items.

METHOD

Parents of typically developing children (n = 2205) and parents of children and adolescents with disabilities (n = 703) between the ages of 0 and 21 years, stratified by age and sex, participated by responding to PEDI-CAT surveys through an existing Internet opt-in survey panel in the USA and by computer tablets in clinical sites.

RESULTS

Confirmatory factor analyses supported four unidimensional content domains. Scores using the real data post hoc demonstrated excellent accuracy (intraclass correlation coefficients ≥ 0.95) with the full item banks. Simulations using item parameter estimates demonstrated relatively small bias in the 10-item and 15-item CAT versions; error was generally higher at the scale extremes.

INTERPRETATION

These results suggest the PEDI-CAT can be an accurate and precise assessment of children's daily performance at all functional levels.

Linking the activity measure for post acute care and the quality of life outcomes in neurological disorders

OBJECTIVE

To use item response theory (IRT) methods to link physical functioning items in the Activity Measure for Post Acute Care (AM-PAC) and the Quality of Life Outcomes in Neurological Disorders (Neuro-QOL).

DESIGN

Secondary data analysis of the physical functioning items of AM-PAC and Neuro-QOL. We used a nonequivalent group design with 36 core items common to both instruments and a test characteristic curve transformation method for linking AM-PAC and Neuro-QOL scores. Linking was conducted so that both raw and scaled AM-PAC and Neuro-QOL scores (mean ± SD converted-logit scores, 50 ± 10) could be compared.

SETTING

AM-PAC items were administered to rehabilitation patients in post-acute care (PAC) settings. Neuro-QOL items were administered to a community sample of adults through the Internet.

PARTICIPANTS

PAC patients (N=1041) for the AM-PAC sample and community-dwelling adults (N=549) for the Neuro-QOL sample.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Mobility (N=25) and activity of daily living (ADL) items (N=11) common to both instruments were included in analysis.

RESULTS

Neuro-QOL items were linked to the AM-PAC scale by using the generalized partial credit model. Mobility and ADL subscale scores from the 2 instruments were calibrated to the AM-PAC metric.

CONCLUSIONS

An IRT-based linking method placed AM-PAC and Neuro-QOL mobility and ADL scores on a common metric. This linking allowed estimation of AM-PAC mobility and ADL subscale scores based on Neuro-QOL mobility and ADL subscale scores and vice versa. The accuracy of these results should be validated in a future sample in which participants respond to both instruments.

The impact of cataract surgery questionnaire: re-evaluation using Rasch analysis

PURPOSE

The Impact of Cataract Surgery (ICS) questionnaire was designed to assess cataract surgery outcomes. The aim of this study was to describe the psychometric properties of the ICS questionnaire using the Rasch model in a cataract population.

METHODS

Ninety-one patients waiting to undergo cataract surgery in the first or second eye at the Flinders Medical Centre, Adelaide, South Australia self-administered the four-item ICS questionnaire. Rasch analysis was performed to assess behaviour of response categories, ability to differentiate between participants' visual abilities (person separation; minimum acceptable 2.0), if items measure a single underlying construct [i.e. unidimensionality assessed by fit statistics and further by principal components analysis (PCA)] and matching of item difficulty to participant ability (targeting; ideal < 0.5 logits). Adequate person separation was defined as basic requirement for a measure, failing which further assessment such as PCA was not performed.

RESULTS

The four-item ICS questionnaire did not meet the required measurement properties (person separation zero). Response categories did not behave as intended, requiring the collapsing of categories for one item (read ordinary newspaper-size print). One item misfit (estimating distance) indicating that it was not measuring the same construct as other items. However, person separation failed to improve following the deletion of this item. Targeting was -0.46 logits, indicating that the item difficulty was well suited to the visual abilities of the participants.

CONCLUSION

In its present form, the ICS is unsuitable for visual disability assessment in patients awaiting cataract surgery. Other, better visual function questionnaires are available and preferred.

Using video animation to assess mobility in older adults

BACKGROUND

Although the importance of the context of task performance in the assessment of mobility in older adults is generally understood, there is little empirical evidence that demonstrates how sensitive older adults are to subtle changes in task demands. Thus, we developed a novel approach to examine this issue.

METHODS

We collected item response data to 81 animated video clips, where various mobility-related tasks were modified in a systematic fashion to manipulate task difficulty.

RESULTS

The participants (N = 234), 166 women and 68 men, had an average age of 81.9 years and a variety of comorbidities. Histograms of item responses revealed dramatic and systematic effects on older adults' self-reported ability when varying walking speed, use of a handrail during ascent and descent of stairs, walking at different speeds outdoors over uneven terrain, and carrying an object. For example, there was almost a threefold increase in reporting the inability to walk at the fast speed compared with the slow speed for a minute or less, and twice as many participants reported the inability to walk at the fast speed outdoors over uneven terrain compared with indoors.

CONCLUSIONS

The data provide clear evidence that varying the contextual features and demands of a simple task such as stair climbing has a significant impact on older adults' self-reporting of ability related to mobility. More work is needed on the psychometric properties of such assessments and to determine if this methodology has conceptual and clinical relevance in studying mobility disability.

Assessment of function and disability in longitudinal studies

Over the past 2 decades, there has been considerable progress in the assessment of function and disability in older persons. Tests of physical performance are now routinely included in longitudinal studies to measure functional limitations, which are considered the building blocks of functioning. In addition, new strategies have been developed to assess the presence and onset of disability and to expand the scope of disability assessments beyond traditional indicators of difficulty and dependence. Contemporary measurement technologies, such as item response theory and computer adaptive testing, show great promise in the assessment of functional status and disability, but prospective studies are needed to demonstrate their true value, particularly to identify the circumstances in which their use will improve the assessment of functional outcomes in older persons. Another high priority for future research is to validate and further refine strategies to more completely and accurately ascertain the occurrence of disability in older persons.

Is Geriatric Depression Scale-15 a suitable instrument for measuring depression in Brazil? Results of a Rasch analysis

Depressive symptoms are the most prevalent mental health condition in older adults. Since it cannot be measured directly, the use of instruments is mandatory. The 15-item Geriatric Depression Scale (GDS) is one of the most widely used scales to measure depression in the elderly. It is recognized that the cultural context is a major determinant of the instrument's psychometric performance. Up to the present, this scale has mainly been investigated through classical psychometric approaches. The present study aims to explore whether the 15-item GDS is a suitable instrument in a Brazilian sample. In addition, it explores the potential improvement in the psychometric performance by item refinement. Four hundred twenty-four elderly adults selected through convenience sampling completed the 15-item GDS. Data were analyzed by the Rasch Measurement Model. The Rasch analysis is a powerful modern approach to explore psychometric performance of instruments in health sciences. It examines both the scale and the individual item performance in depth. The 15-item GDS proved not to be suitable in a Brazilian sample. Item misfit and differential item functioning were responsible for considerable misperformance. Scale reduction led to a 10-item structure. This refined format presented adequate psychometric performance and no differential item functioning. The present study offers an alternative and more adequate version of the GDS to be applied in Brazilian subjects. It is also in line with the need for shorter, valid scales in clinical settings. Further investigations are needed to develop a set of cultural-invariant items, which could then be applied in transcultural investigations free of bias.

Development of symptom assessments utilising item response theory and computer-adaptive testing--a practical method based on a systematic review

Assessment of individual patients' distress is a cornerstone of clinical care for advanced cancer. Patients' ability to fill out lengthy questionnaires is compromised by many factors. Computer-adaptive tests (CAT) offer a promising approach to developing tailored instruments, that administer only items relevant to the individual patient. A systematic review of the literature about CATs in medical databases was conducted. Based on the results, a method for developing a CAT was designed that requires nine steps: (1) build an item pool; (2) administer the items to a predefined sample in a calibration study; (3) eliminate inappropriate items; (4) examine whether all items are influenced by a single dominant trait; (5) calibrate the items to the best-fitting item response theory (IRT) model; (6) evaluate items' parameter equivalence across subgroups; (7) build an item bank with the calibrated items; (8) develop the CAT; and (9) pilot test the developed CAT. CAT offers the chance to extend the usefulness of patient-reported outcome (PRO) measurements from clinical studies to daily clinical practice.

Developing a contemporary functional outcome measure for spinal cord injury research

STUDY DESIGN

This study used qualitative analysis of focus group discussions.

OBJECTIVE

The primary objective was to select functional activities to include in an item pool, which is the first step in developing a spinal cord injury computer adaptive test (SCI-CAT).

SETTING

This multisite study was conducted at six US National Spinal Cord Injury Model Systems Programs.

METHODS

Focus group discussions, which included persons with tetraplegia and paraplegia and clinicians, were conducted. Transcripts were analyzed using a grounded theory approach. Functional activities were identified, binned, winnowed, written as functional items, and cognitively tested.

RESULTS

Focus group discussion analysis identified 326 functional activity items that fit into categories outlined in the International Classification of Functioning, Disability and Health (ICF) framework: Mobility (193 items), including assessment of functioning in a manual (44 items) and power wheelchair (19 items); self-care (109 items); and communication (19 items). Items related to sexual function were also identified (5 items).

CONCLUSION

The SCI-CAT item pool includes items that assess functional activities important to persons with SCI. Items cover a wide range of functional ability and reflect most ICF categories. The SCI-CAT pool is currently being field tested to develop a calibrated item bank. Further development will yield a CAT of functional activities appropriate for SCI research.

A computer-adaptive disability instrument for lower extremity osteoarthritis research demonstrated promising breadth, precision, and reliability

OBJECTIVE

To develop and evaluate a prototype measure (OA-DISABILITY-CAT) for osteoarthritis research using item response theory (IRT) and computer-adaptive test (CAT) methodologies.

STUDY DESIGN AND SETTING

We constructed an item bank consisting of 33 activities commonly affected by lower extremity (LE) osteoarthritis. A sample of 323 adults with LE osteoarthritis reported their degree of limitation in performing everyday activities, and completed the Health Assessment Questionnaire-II (HAQ-II). We used confirmatory factor analyses to assess scale unidimensionality and IRT methods to calibrate the items and examine the fit of the data. Using CAT simulation analyses, we examined the performance of OA-DISABILITY-CATs of different lengths compared with the full-item bank and the HAQ-II.

RESULTS

One distinct disability domain was identified. The 10-item OA-DISABILITY-CAT demonstrated a high degree of accuracy compared with the full-item bank (r=0.99). The item bank and the HAQ-II scales covered a similar estimated scoring range. In terms of reliability, 95% of OA-DISABILITY reliability estimates were over 0.83 vs. 0.60 for the HAQ-II. Except at the highest scores, the 10-item OA-DISABILITY-CAT demonstrated superior precision to the HAQ-II.

CONCLUSION

The prototype OA-DISABILITY-CAT demonstrated promising measurement properties compared with the HAQ-II, and is recommended for use in LE osteoarthritis research.

A functional difficulty and functional pain instrument for hip and knee osteoarthritis

INTRODUCTION

The objectives of this study were to develop a functional outcome instrument for hip and knee osteoarthritis research (OA-FUNCTION-CAT) using item response theory (IRT) and computer adaptive test (CAT) methods and to assess its psychometric performance compared to the current standard in the field.

METHODS

We conducted an extensive literature review, focus groups, and cognitive testing to guide the construction of an item bank consisting of 125 functional activities commonly affected by hip and knee osteoarthritis. We recruited a convenience sample of 328 adults with confirmed hip and/or knee osteoarthritis. Subjects reported their degree of functional difficulty and functional pain in performing each activity in the item bank and completed the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC). Confirmatory factor analyses were conducted to assess scale uni-dimensionality, and IRT methods were used to calibrate the items and examine the fit of the data. We assessed the performance of OA-FUNCTION-CATs of different lengths relative to the full item bank and WOMAC using CAT simulation analyses.

RESULTS

Confirmatory factor analyses revealed distinct functional difficulty and functional pain domains. Descriptive statistics for scores from 5-, 10-, and 15-item CATs were similar to those for the full item bank. The 10-item OA-FUNCTION-CAT scales demonstrated a high degree of accuracy compared with the item bank (r = 0.96 and 0.89, respectively). Compared to the WOMAC, both scales covered a broader score range and demonstrated a higher degree of precision at the ceiling and reliability across the range of scores.

CONCLUSIONS

The OA-FUNCTION-CAT provided superior reliability throughout the score range and improved breadth and precision at the ceiling compared with the WOMAC. Further research is needed to assess whether these improvements carry over into superior ability to measure change.

Blending activity and participation sub-domains of the ICF

OBJECTIVE

The objective of the study was to test the supposition that distinct Activity and Participation sub-domains of the International Classification of Functioning, Disability, and Health (ICF) could be identified using physical function questionnaire items drawn from the Activity and Participation Measures for Post Acute Care.

DESIGN

A cross-sectional survey design was employed. Subjects. The sample consisted of 272 patients who had received post acute care across inpatient and community care settings during the previous year.

METHODS

Exploratory factor analysis with oblique rotation was used to identify interpretable dimensions underlying 83 physical functioning questionnaire items.

RESULTS

Factor analysis findings revealed five distinct ICF conceptual sub-domains that explained 61% of the total variance of the data, with root mean square residual equal to 0.089. These domains were labeled: (i) Daily activity, (ii) Applied cognitive, (iii) Role participation, (iv) Mobility, and (v) Social participation. These five factors were correlated to varying degrees and achieved acceptable levels of internal consistency with coefficient alphas from 0.59-0.93.

CONCLUSION

These analyses suggest that clear distinctions between Activity and Participation sub-domains of the ICF could not be identified in a sample of adults who had recently received post acute care. We believe this is important evidence of the ability to measure blended Activity and Participation sub-domains that cut across distinct and interpretable areas of life functioning.

Evaluation of an item bank for a computerized adaptive test of activity in children with cerebral palsy

BACKGROUND

Contemporary clinical assessments of activity are needed across the age span for children with cerebral palsy (CP). Computerized adaptive testing (CAT) has the potential to efficiently administer items for children across wide age spans and functional levels.

OBJECTIVE

The objective of this study was to examine the psychometric properties of a new item bank and simulated computerized adaptive test to assess activity level abilities in children with CP.

DESIGN

This was a cross-sectional item calibration study.

METHODS

The convenience sample consisted of 308 children and youth with CP, aged 2 to 20 years (X=10.7, SD=4.0), recruited from 4 pediatric hospitals. We collected parent-report data on an initial set of 45 activity items. Using an Item Response Theory (IRT) approach, we compared estimated scores from the activity item bank with concurrent instruments, examined discriminate validity, and developed computer simulations of a CAT algorithm with multiple stop rules to evaluate scale coverage, score agreement with CAT algorithms, and discriminant and concurrent validity.

RESULTS

Confirmatory factor analysis supported scale unidimensionality, local item dependence, and invariance. Scores from the computer simulations of the prototype CATs with varying stop rules were consistent with scores from the full item bank (r=.93-.98). The activity summary scores discriminated across levels of upper-extremity and gross motor severity and were correlated with the Pediatric

OUTCOMES

Data Collection Instrument (PODCI) physical function and sports subscale (r=.86), the Functional Independence Measure for Children (Wee-FIM) (r=.79), and the Pediatric Quality of Life Inventory-Cerebral Palsy version (r=.74).

LIMITATIONS

The sample size was small for such IRT item banks and CAT development studies. Another limitation was oversampling of children with CP at higher functioning levels.

CONCLUSIONS

The new activity item bank appears to have promise for use in a CAT application for the assessment of activity abilities in children with CP across a wide age range and different levels of motor severity.

Adaptive cognitive testing in dementia

The objective of this study was to investigate whether the Cambridge Cognitive Examination (CAMCOG), a widely used screening test for dementia, can be tailored to the individual patient with Computerized Adaptive Testing (CAT). CAT accomplishes this by only using items that are appropriate for the level of ability of the patient under investigation. Potential advantages of CAT for clinical practice and research are efficient cognitive testing and a reduction of the test burden in elderly patients and consequently less measurement error during testing. In a two step method with previously collected CAMCOG data (n = 797) (1) patient abilities and CAMCOG item difficulties were estimated with the One Parameter Logistic Model (OPLM), a Rasch type of model. CAT was then used (2) to re-estimate the patient abilities. Despite an average test reduction of 60%, CAT estimates were in excellent agreement (intra-class correlation > 0.98) with the results based on the entire CAMCOG and they also had similar accuracy for the diagnosis of dementia (area under the curve 0.91) as the original CAMCOG. These results were replicated in an independent sample (n = 170). We conclude that tailored testing with CAT enables much more efficient screening for dementia than testing with an extensive instrument.

Methodological challenges confronting researchers of wheeled mobility aids and other assistive technologies

PURPOSE

To identify gaps in assistive technology (AT) research and solutions, with a focus on wheeled mobility.

METHODS

Literature review.

RESULTS

AT researchers have identified a need to move beyond determining whether a device works well in the laboratory or is perceived favorably by users. The vital next step in AT research is to determine the effects of AT when used for day-to-day activities by typical consumers. Four challenges affect AT research on everyday mobility: (1) the heterogeneity of the population using AT, the environments in which AT is used, and the devices themselves; (2) the dependency of empirical research on objective data for valid causal inference; (3) the need for detailed information to capture the interaction between the person, the device, and the environment in which it is used; and (4) the extent to which success or failure of AT is dependent on the personal perspectives of the individual using the device. These challenges are being addressed by AT researchers through use of new measures, novel data collection methods, and by linking quantitative with qualitative data.

CONCLUSIONS

AT researchers are adapting traditional research designs and analytic methods to examine that effects of AT on everyday life.

Replenishing a computerized adaptive test of patient-reported daily activity functioning

PURPOSE

Computerized adaptive testing (CAT) item banks may need to be updated, but before new items can be added, they must be linked to the previous CAT. The purpose of this study was to evaluate 41 pretest items prior to including them into an operational CAT.

METHODS

We recruited 6,882 patients with spine, lower extremity, upper extremity, and nonorthopedic impairments who received outpatient rehabilitation in one of 147 clinics across 13 states of the USA. Forty-one new Daily Activity (DA) items were administered along with the Activity Measure for Post-Acute Care Daily Activity CAT (DA-CAT-1) in five separate waves. We compared the scoring consistency with the full item bank, test information function (TIF), person standard errors (SEs), and content range of the DA-CAT-1 to the new CAT (DA-CAT-2) with the pretest items by real data simulations.

RESULTS

We retained 29 of the 41 pretest items. Scores from the DA-CAT-2 were more consistent (ICC = 0.90 versus 0.96) than DA-CAT-1 when compared with the full item bank. TIF and person SEs were improved for persons with higher levels of DA functioning, and ceiling effects were reduced from 16.1% to 6.1%.

CONCLUSIONS

Item response theory and online calibration methods were valuable in improving the DA-CAT.

A computerized adaptive test for patients with hip impairments produced valid and responsive measures of function

OBJECTIVES

To describe the use of a computerized adaptive test (CAT) in routine clinical practice and evaluate content coverage and construct validity, sensitivity to change, and responsiveness of hip CAT functional status (FS) measures.

DESIGN

Longitudinal, prospective observational cohort study.

SETTING

Two hundred fifty-seven outpatient rehabilitation clinics in 31 states (United States).

PARTICIPANTS

Two samples were examined: intake and discharge rehabilitation FS data from patients (N=8714) treated for hip impairments between January 2005 and June 2007 and data from patients (N=444) used to develop the hip CAT were examined for comparison (2002-2004).

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Hip functional status and global rating of change.

RESULTS

The CAT used on average 7 items to produce precise estimates of FS that adequately covered the content range with negligible floor and slight ceiling effects. Test information functions and SEs supported FS measure precision. FS measures discriminated patients in clinically logical ways. Sixty-one percent of patients obtained discharge FS measures greater than or equal to minimal detectable change (95% confidence intervals). Change of 6 FS units (scale: 0-100) represented minimal clinically important improvement, which 64% of patients obtained.

CONCLUSIONS

The hip CAT was efficient; produced valid, responsive measures of FS for patients receiving therapy for hip impairments; and functioned well in routine clinical application but would benefit from more difficult items.

Creating a computer adaptive test version of the late-life function and disability instrument

BACKGROUND

This study applied item response theory (IRT) and computer adaptive testing (CAT) methodologies to develop a prototype function and disability assessment instrument for use in aging research. Herein, we report on the development of the CAT version of the Late-Life Function and Disability Instrument (Late-Life FDI) and evaluate its psychometric properties.

METHODS

We used confirmatory factor analysis, IRT methods, validation, and computer simulation analyses of data collected from 671 older adults residing in residential care facilities. We compared accuracy, precision, and sensitivity to change of scores from CAT versions of two Late-Life FDI scales with scores from the fixed-form instrument. Score estimates from the prototype CAT versus the original instrument were compared in a sample of 40 older adults.

RESULTS

Distinct function and disability domains were identified within the Late-Life FDI item bank and used to construct two prototype CAT scales. Using retrospective data, scores from computer simulations of the prototype CAT scales were highly correlated with scores from the original instrument. The results of computer simulation, accuracy, precision, and sensitivity to change of the CATs closely approximated those of the fixed-form scales, especially for the 10- or 15-item CAT versions. In the prospective study, each CAT was administered in <3 minutes and CAT scores were highly correlated with scores generated from the original instrument.

CONCLUSIONS

CAT scores of the Late-Life FDI were highly comparable to those obtained from the full-length instrument with a small loss in accuracy, precision, and sensitivity to change.

Adaptive short forms for outpatient rehabilitation outcome assessment

OBJECTIVE

To develop outpatient Adaptive Short Forms for the Activity Measure for Post-Acute Care item bank for use in outpatient therapy settings.

DESIGN

A convenience sample of 11,809 adults with spine, lower limb, upper limb, and miscellaneous orthopedic impairments who received outpatient rehabilitation in 1 of 127 outpatient rehabilitation clinics in the United States. We identified optimal items for use in developing outpatient Adaptive Short Forms based on the Basic Mobility and Daily Activities domains of the Activity Measure for Post-Acute Care item bank. Patient scores were derived from the Activity Measure for Post-Acute Care computerized adaptive testing program. Items were selected for inclusion on the Adaptive Short Forms based on functional content, range of item coverage, measurement precision, item exposure rate, and data collection burden.

RESULTS

Two outpatient Adaptive Short Forms were developed: (1) an 18-item Basic Mobility Adaptive Short Form and (2) a 15-item Daily Activities Adaptive Short Form, derived from the same item bank used to develop the Activity Measure for Post-Acute Care computerized adaptive testing program. Both Adaptive Short Forms achieved acceptable psychometric properties.

CONCLUSIONS

In outpatient postacute care settings where computerized adaptive testing outcome applications are currently not feasible, item response theory-derived Adaptive Short Forms provide the efficient capability to monitor patients' functional outcomes. The development of Adaptive Short Form functional outcome instruments linked by a common, calibrated item bank has the potential to create a bridge to outcome monitoring across postacute care settings and can facilitate the eventual transformation from Adaptive Short Forms to computerized adaptive testing applications easier and more acceptable to the rehabilitation community.

Computerized adaptive test for patients with foot or ankle impairments produced valid and responsive measures of function

OBJECTIVE

We tested the item response theory (IRT) model assumptions of the original item bank, and evaluated the practical and psychometric adequacy, of a computerized adaptive test (CAT) for patients with foot or ankle impairments seeking rehabilitation in outpatient therapy clinics.

METHODS

Data from 10,287 patients with foot or ankle impairments receiving outpatient physical therapy were analyzed. We first examined the unidimensionality, fit, and invariance IRT assumptions of the CAT item bank. Then we evaluated the efficiency of the CAT administration and construct validity and sensitivity of change of the foot/ankle CAT measure of lower-extremity functional status (FS).

RESULTS

Results supported unidimensionality, model fit, and invariance of item parameters and patient ability estimates. On average, the CAT used seven items to produce precise estimates of FS that adequately covered the content range with negligible floor and ceiling effects. Patients who were older, had more chronic symptoms, had more surgeries, had more comorbidities, and did not exercise prior to receiving rehabilitation reported worse discharge FS. Seventy-one percent of patients obtained statistically significant change at follow-up. Change of 8 FS units (scale 0-100) represented minimal clinically important improvement.

CONCLUSIONS

We concluded that the foot/ankle item bank met IRT assumptions and that the CAT FS measure was precise, valid, and responsive, supporting its use in routine clinical application.

Computerized adaptive test for patients with knee impairments produced valid and responsive measures of function

OBJECTIVE

Assess practicality of using a computerized adaptive test (CAT) in routine clinical practice, perform a psychometric evaluation of content range coverage and test precision, and assess known group construct validity, sensitivity to change and responsiveness of knee CAT functional status (FS) measures.

STUDY DESIGN AND SETTING

Secondary analysis of retrospective intake and discharge rehabilitation FS data collected in a prospective cohort study. Data represented a convenience sample of 21,896 patients with knee impairments receiving outpatient physical therapy in 291 clinics in 30 U.S. states (2005-2007).

RESULTS

The CAT used an average of seven items to produce precise estimates of FS that adequately covered the content range with negligible floor and ceiling effects. Test information functions and standard errors supported FS measure precision. FS measures discriminated patients by age, symptom acuity, surgical history, condition complexity, and prior exercise history in clinically logical ways. Seventy-two percent of patients obtained discharge FS measures > or = minimal detectable change (95% confidence interval). Change of 9 FS units (0-100 scale) represented minimal clinically important improvement, which 67% of patients obtained.

CONCLUSION

The knee CAT was efficient and produced precise, valid, and responsive measures of FS for patients receiving therapy for knee impairments and functioned well in routine clinical application.

Letting the CAT out of the bag: comparing computer adaptive tests and an 11-item short form of the Roland-Morris Disability Questionnaire

STUDY DESIGN

A post hoc simulation of a computer adaptive administration of the items of a modified version of the Roland-Morris Disability Questionnaire.

OBJECTIVE

To evaluate the effectiveness of adaptive administration of back pain-related disability items compared with a fixed 11-item short form.

SUMMARY OF BACKGROUND DATA

Short form versions of the Roland-Morris Disability Questionnaire have been developed. An alternative to paper-and-pencil short forms is to administer items adaptively so that items are presented based on a person's responses to previous items. Theoretically, this allows precise estimation of back pain disability with administration of only a few items.

MATERIALS AND METHODS

Data were gathered from 2 previously conducted studies of persons with back pain. An item response theory model was used to calibrate scores based on all items, items of a paper-and-pencil short form, and several computer adaptive tests (CATs).

RESULTS

Correlations between each CAT condition and scores based on a 23-item version of the Roland-Morris Disability Questionnaire ranged from 0.93 to 0.98. Compared with an 11-item short form, an 11-item CAT produced scores that were significantly more highly correlated with scores based on the 23-item scale. CATs with even fewer items also produced scores that were highly correlated with scores based on all items. For example, scores from a 5-item CAT had a correlation of 0.93 with full scale scores. Seven- and 9-item CATs correlated at 0.95 and 0.97, respectively. A CAT with a standard-error-based stopping rule produced scores that correlated at 0.95 with full scale scores.

CONCLUSION

A CAT-based back pain-related disability measure may be a valuable tool for use in clinical and research contexts. Use of CAT for other common measures in back pain research, such as other functional scales or measures of psychological distress, may offer similar advantages.

Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II. Participation outcomes

OBJECTIVES

To measure participation outcomes with a computerized adaptive test (CAT) and compare CAT and traditional fixed-length surveys in terms of score agreement, respondent burden, discriminant validity, and responsiveness.

DESIGN

Longitudinal, prospective cohort study of patients interviewed approximately 2 weeks after discharge from inpatient rehabilitation and 3 months later.

SETTING

Follow-up interviews conducted in patient's home setting.

PARTICIPANTS

Adults (N=94) with diagnoses of neurologic, orthopedic, or medically complex conditions.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Participation domains of mobility, domestic life, and community, social, & civic life, measured using a CAT version of the Participation Measure for Postacute Care (PM-PAC-CAT) and a 53-item fixed-length survey (PM-PAC-53).

RESULTS

The PM-PAC-CAT showed substantial agreement with PM-PAC-53 scores (intraclass correlation coefficient, model 3,1, .71-.81). On average, the PM-PAC-CAT was completed in 42% of the time and with only 48% of the items as compared with the PM-PAC-53. Both formats discriminated across functional severity groups. The PM-PAC-CAT had modest reductions in sensitivity and responsiveness to patient-reported change over a 3-month interval as compared with the PM-PAC-53.

CONCLUSIONS

Although continued evaluation is warranted, accurate estimates of participation status and responsiveness to change for group-level analyses can be obtained from CAT administrations, with a sizeable reduction in respondent burden.