A discussion of item response theory and its applications in health status assessment

Informing the development of an outcome set and banks of items to measure mobility among individuals with acquired brain injury using natural language processing

BACKGROUND

The sheer number of measures evaluating mobility and inconsistencies in terminology make it challenging to extract potential core domains and items. Automating a portion of the data synthesis would allow us to cover a much larger volume of studies and databases in a smaller fraction of the time compared to the usual process. Thus, the objective of this study was to identify a comprehensive outcome set and develop preliminary banks of items of mobility among individuals with acquired brain injury (ABI) using Natural Language Processing (NLP).

METHODS

An umbrella review of 47 reviews evaluating the content of mobility measures among individuals with ABI was conducted. A search was performed on 5 databases between 2000 and 2020. Two independent reviewers retrieved copies of the measures and extracted mobility domains and items. A pre-trained BERT model (state-of-the-art model for NLP) provided vector representations for each sentence. Using the International Classification of Functioning, Disability, and Health Framework (ICF) ontology as a guide for clustering, a k-means algorithm was used to retrieve clusters of similar sentences from their embeddings. The resulting embedding clusters were evaluated using the Silhouette score and fine-tuned according to expert input.

RESULTS

The study identified 246 mobility measures, including 474 domains and 2109 items. Encoding the clusters using the ICF ontology and expert knowledge helped in regrouping the items in a way that is more closely related to mobility terminology. Our best results identified banks of items that were used to create a 24 comprehensive outcome sets of mobility, including Upper Extremity Mobility, Emotional Function, Balance, Motor Control, Self-care, Social Life and Relationships, Cognition, Walking, Postural Transition, Recreation, and Leisure Activities, Activities of Daily Living, Physical Functioning, Communication, Work/Study, Climbing, Sensory Functions, General Health, Fatigue, Functional Independence, Pain, Alcohol and Drugs Use, Transportation, Sleeping, and Finances.

CONCLUSION

The banks of items of mobility domains represent a first step toward establishing a comprehensive outcome set and a common language of mobility to develop the ontology. It enables researchers and healthcare professionals to begin exposing the content of mobility measures as a way to assess mobility comprehensively.

Development of an Item Bank for a health-related quality of life Measure in Spondyloarthritis

OBJECTIVE

Health-related quality of life (HRQOL) is an important aspect in the management of chronic diseases such as spondyloarthritis (SpA). A promising approach to reduce respondent burden when measuring HRQOL is the use of shorter patient-reported outcome measures (PROMs) delivered using computer-adaptive tests (CATs). However, the lack of an item bank which covers the entire continuum of the HRQOL domain impedes the development of CAT to measure HRQOL among patients with SpA. We aim to develop an item bank for a HRQOL measure among patients with SpA based on the items from existing validated PROMs.

METHODS

This study is guided by the preferred reporting items for systematic review and metaanalysis (PRISMA) and Patient Reported Outcomes Measurement Information System (PROMIS) Cooperative Group guidelines. Relevant articles were retrieved from PubMed®, Embase®, and PsycINFO® (Ovid) databases. Items from existing PROMs were "binned" and "winnowed" according to the facets of HRQOL in the World Health Organisation (WHO) QOL framework.

RESULTS

We identified 147 relevant articles. 1039 items from 31 PROMs of which permission has been obtained for inclusion into the item bank underwent binning and winnowing. This resulted in 968 items covering 23 domains of HRQOL in the WHO framework, with the number of items within each domain ranging from 1 to 453.

CONCLUSION

We created an item bank to measure HRQOL among patients with SpA using items from validated PROMs. This set can provide the foundation for the development of CATs to measure HRQOL among patients with SpA.

Spinal Cord Injury-Functional Index/Capacity: Responsiveness to Change Over Time

OBJECTIVE

To establish responsiveness of 3 Spinal Cord Injury-Functional Index/Capacity (SCI-FI/C) item banks in the first year after spinal cord injury (SCI).

DESIGN

Longitudinal patient-reported outcomes assessment replicated through secondary analysis of an independent data set.

SETTING

A total of 8 SCI Model Systems rehabilitation hospitals in the United States.

PARTICIPANTS

Study 1 participants included 184 adults with recent (≤4 months) traumatic SCI and 221 community-dwelling adults (>1 year post injury) (N=405). Study 2 participants were 418 individuals with recent SCI (≤4 months) (N=418).

INTERVENTIONS

In study 1, SCI-FI/C computer adaptive tests were presented in a standardized interview format either in person or by phone call at baseline and 6-month follow-up. Responsiveness was examined by comparing 6-month changes in SCI-FI scores within and across samples (recently injured vs community-dwelling) because only the recent injury sample was expected to exhibit change over time. Effect sizes were also computed. In study 2, the study 1 results were cross-validated in a second sample with recent SCI 1 year after baseline measurement. Study 2 also compared the SCI-FI/C measures' responsiveness to that of the Self-reported Functional Measure (SRFM) and stratified results by injury diagnosis and completeness.

MAIN OUTCOME MEASURES

The SCI-FI Basic Mobility/C, Self-care/C and Fine Motor/C item banks (study 1 and study 2); Self-reported Functional Measure SRFM (study 2 only).

RESULTS

In study 1, changes in SCI-FI/C scores between baseline and 6-month follow-up were statistically significant (P<.01) for recently injured individuals. SCI-FI Basic Mobility/C, Self-care/C, and Fine Motor/C item banks demonstrated small to medium effect sizes in the recently injured sample. In the community-dwelling sample, all SCI-FI/C effects were negligible (ie, effect size<0.08). Study 2 results were similar to study 1. As expected, SCI-FI Basic Mobility/C and Self-care/C were responsive to change for all individuals in study 2, whereas the SCI-FI Fine Motor/C was responsive only for individuals with tetraplegia and incomplete paraplegia. The SRFM demonstrated a medium effect size for responsiveness (effect size=0.65).

CONCLUSIONS

The SCI-FI Basic Mobility/C and Self-care/C banks demonstrate adequate sensitivity to change at 6 months and 1 year for all individuals with SCI, while the SCI-FI/C Fine Motor item bank is sensitive to change in individuals with tetraplegia or incomplete paraplegia. All SCI-FI/C banks demonstrate stability in a sample not expected to change. Results provide support for the use of these measures for research or clinical use.

Correlation of Patient Reported Outcome Measurement Information System (PROMIS) with American Shoulder and Elbow Surgeon (ASES), and Constant (CS) scores in idiopathic adhesive capsulitis

PURPOSE

To correlate the Patient Reported Outcomes Measurement Information System Upper Extremity (PROMIS UE) score with pre-existing validated outcome scores, American Shoulder and Elbow Surgeons score (ASES), and Constant score (CS) in patients with idiopathic adhesive capsulitis (AC).

METHODS

Patients with a clinical diagnosis of idiopathic AC ("freezing" or "frozen" phases) who agreed to complete the ASES, CS, and PROMIS UE scores during their office visit were included in this study. Trained researchers performed the objective clinical assessments on the included patients. Responses to the 3 outcome scores were statistically analyzed and compared using Pearson correlation coefficients. Floor and ceiling effects were calculated.

RESULTS

The final cohort included 100 patients with AC, of whom there were 72% female and 87% right hand dominant, with a mean age of 55 years. The PROMIS UE required fewer question responses (5.02 ± 1.84) compared with the fixed question burden with ASES (12) and CS (9). The mean outcome scores were 34.6 ± 2.5 (PROMIS UE), 55 ± 22 (ASES), and 51 ± 16 (CS). The PROMIS UE displayed an excellent correlation with both the ASES (r = 0.80, 95% confidence interval [0.72, 0.86], P < .001) and CS (r = 0.76, 95% confidence interval [0.67, 0.83], P < .001). Neither ceiling nor floor effects were present.

CONCLUSION

The PROMIS UE displayed comparable efficacy to commonly used legacy outcome scores (ASES and CS) in AC. A lower question burden with the PROMIS UE carries potential for wider acceptability with the researchers and patients with shoulder pathology.

Development and Psychometric Evaluation of the PROMIS Pediatric Pain Intensity Measure in Children and Adolescents with Chronic Pain

The goal of the current study was to enhance the measurement of the pediatric chronic pain experience through a methodologically rigorous approach. This paper outlines the development and initial validation of a pain intensity measure for pediatric patients with chronic pain using Patient-Reported Outcomes Measurement Information System methodology. Measure development incorporated feedback from children with painful conditions. Based on input from pediatric participants and content experts, 4 candidate items assessing pain intensity were included for large scale testing. Children completed self-report items pertaining to their pain experience that were developed as part of a larger pool of new candidate Patient-Reported Outcomes Measurement Information System pediatric pain domain items as well as measures of pain interference, depressive symptoms, fatigue, pain behavior, pain intensity, and pain catastrophizing. The final sample for the large scale testing included N = 442 pediatric patients between the ages 8 to 18 years (Mean age = 13.54, Standard Deviation = 2.78; 71.27% female) experiencing chronic pain. Psychometric analysis resulted in a final measure that included 3 items with evidence of reliability (Cronbach alpha = .82) and convergent validity. The Likert format of the response options may be preferable to the traditional numeric rating scale for use in pediatric populations who experience chronic pain based on patients' feedback, which was directly utilized in designing the scale. Further, the inclusion of fewer and clinically meaningful response options should reduce ambiguity for young respondents. PERSPECTIVE: We have developed and evaluated a clinically sensitive and psychometrically precise 3-item pain intensity measure with Likert-type responses for self-report use among children and adolescents ages 8 to 18 years with chronic pain. Development of the item content and response options included input from children and adolescents with chronic pain. The development of pain intensity items with pediatric appropriate language, and labeled, fewer response options to yield maximal clinically meaningful information improves the precision of pain intensity measurement in children.

Validation of Yonsei-Bilateral Activity Test (Y-BAT)-Bilateral Upper Extremity Inventory Using Rasch Analysis

There is little assessment to provide specific information about quality of bilateral upper extremity movement specifically. This study examined the psychometric properties of the Yonsei-Bilateral Activity Test (Y-BAT). An observational cross-sectional design was used with 100 stroke survivors. Unidimensionality was examined using factor analyses, and Rasch analysis was used to test rating scale structure, fit statistics, and precision of the Y-BAT. The Y-BAT demonstrated a unidimensional measurement construct, and 19 of the 27 items fit the Rasch model. The instrument demonstrated good precision, including person reliability and person strata. The Y-BAT estimated person measures within a wide range of theta values with a reliability of .9 with good precision. Our results indicate that a revised, 19-item version of the Y-BAT demonstrates sound internal validity and may be a useful instrument for clinicians to measure upper extremity function with good precision following stroke.

The Manitoba Personalized Lifestyle Research (TMPLR) study protocol: a multicentre bidirectional observational cohort study with administrative health record linkage investigating the interactions between lifestyle and health in Manitoba, Canada

INTRODUCTION

Lifestyle factors, such as diet, physical activity and sleep, are associated with the development of many chronic diseases. The objective of The Manitoba Personalized Lifestyle Research study is to understand how these lifestyle factors interact with each other and with other factors, such as an individual's genetics and gut microbiome, to influence health.

METHODS

An observational study of adults, with extensive phenotyping by objective health and lifestyle assessments, and retrospective assessment of early life experiences, with retrospective and prospective utilisation of secondary data from administrative health records.

STUDY POPULATION

A planned non-random convenience sample of 840 Manitobans aged 30-46 recruited from the general population, stratified by sex (equal men and women), body mass index (BMI; 60% of participants with a BMI>25 kg/m²) and geography (25% from rural areas). These stratifications were selected based on Manitoba demographics.

MEASUREMENTS

Lifestyle factors assessed will include dietary pattern, physical activity, cardiovascular fitness, and sleep. Factors such as medical history, socioeconomic status, alcohol and tobacco consumption, cognition, stress, anxiety, and early life experiences will also be documented. A maternal survey will be performed. Body composition and bone density will be measured by dual energy X-ray absorptiometry. Blood pressure, pulse wave velocity, and augmentation index will be measured on two consecutive days. Chronic disease risk biomarkers will be measured in blood and urine samples. DNA will be extracted for genetic analysis. A faecal sample will be collected for microbiome analysis. Participants may provide their Manitoba personal health information number to link their study data with administrative health records.

ETHICS AND DISSEMINATION

Ethics approval has been obtained from the University of Manitoba Health Research Ethics Board (protocol # HS18951; 05/01/2016). Data analysis, release of results and publication of manuscripts are scheduled to start in early 2019. Additional information at www.TMPLR.ca.

TRIAL REGISTRATION NUMBER

NCT03674957; Pre-results.

Patient-reported outcome and experience measures for diabetes: development of scale models, differences between patient groups and relationships with cardiovascular and diabetes complication risk factors, in a combined registry and survey study in Sweden

PURPOSE

The Swedish National Diabetes Register (NDR) has developed a diabetes-specific questionnaire to collect information on individuals' management of their diabetes, collaboration with healthcare providers and the disease's impact on daily life. Our main objective was to develop measures of well-being, abilities to manage diabetes and judgements of diabetes care, and to detect and quantify differences using the NDR questionnaire.

DESIGN, SETTING AND PARTICIPANTS

The questionnaire was analysed with using responses from 3689 participants with type 1 and 2 diabetes, randomly sampled from the NDR population, combined with register data on patient characteristics and cardiovascular and diabetes complication risk factors.

METHODS

We used item response theory to develop scales for measuring well-being, abilities to manage diabetes and judgements of diabetes care (scores). Test-retest reliability on the scale level was analysed with intraclass correlation. Associations between scores and risk factor levels were investigated with subgroup analyses and correlations.

RESULTS

We obtained scales with satisfactory measurement properties, covering patient reported outcome measures such as general well-being and being free of worries, and patient reported experience measure, for example, access and continuity in diabetes care. All scales had acceptable test-retest reliability and could detect differences between diabetes types, age, gender and treatment subgroups. In several aspects, for example, freedom of worries, type 1 patients report lower than type 2, and younger patients lower than older. Associations were found between some scores and glycated haemoglobin, but none with systolic blood pressure or low-density lipoprotein cholesterol. Clinicians report positive experience of using scores, visually presented, in the patient dialogue.

CONCLUSIONS

The questionnaire measures and detects differences in patient well-being, abilities and judgements of diabetes care, and identifies areas for improvement. To further improve diabetes care, we conclude that patient-reported measures are important supplements to cardiovascular and diabetes complication risk factors, reflecting patient experiences of living with diabetes and diabetes care.

The relationship between therapist-rated function and patient-reported outcome measures

STUDY DESIGN

Prospective cohort study.

INTRODUCTION

Some third-party payers require hand therapists to rate patient's functional disability based on patient self-rating using patient-reported outcome measures (PROMs), objective measurements of impairment, and observation of functional tasks-hand therapist-rated function (HTRF).

PURPOSE OF THE STUDY

To test the correlation between HTRF and PROMs (upper limb functional index [ULFI] and Patient-Reported Outcomes Measurement Information System upper extremity [PROMIS UE]) and its association with psychological factors.

METHODS

In 2014, 100 new patients with upper extremity illness presenting to hand therapists were asked to participate in an observational cross-sectional study. Demographic-, condition-related, and psychological factors were obtained in addition to PROMs and HTRF.

RESULTS

HTRF correlated moderately with PROMIS UE (r = -0.49, P < .001) and ULFI (r = -0.56, P < .001). Correlation between PROMIS UE and ULFI was strong (r = 0.78, P < .001). Psychological factors explained most of the variations in both HTRF and PROMs.

CONCLUSIONS

Hand therapists' ratings of patient function correlate less strongly with PROMs than PROMs correlate with one other. The discrepancy between HTRF and PROMs may offer an opportunity to address stress, distress, or ineffective coping strategies that can interfere with recovery-an opportunity for therapists and patients to collaborate and develop goals and for future research to develop effective and feasible strategies for hand therapists.

LEVEL OF EVIDENCE

Level II, diagnostic study.

Measuring pain phenomena after spinal cord injury: Development and psychometric properties of the SCI-QOL Pain Interference and Pain Behavior assessment tools

OBJECTIVE

To develop modern patient-reported outcome measures that assess pain interference and pain behavior after spinal cord injury (SCI).

DESIGN

Grounded-theory based qualitative item development; large-scale item calibration field-testing; confirmatory factor analyses; graded response model item response theory analyses; statistical linking techniques to transform scores to the Patient Reported Outcome Measurement Information System (PROMIS) metric.

SETTING

Five SCI Model Systems centers and one Department of Veterans Affairs medical center in the United States.

PARTICIPANTS

Adults with traumatic SCI.

INTERVENTIONS

N/A.

OUTCOME MEASURES

Spinal Cord Injury - Quality of Life (SCI-QOL) Pain Interference item bank, SCI-QOL Pain Interference short form, and SCI-QOL Pain Behavior scale.

RESULTS

Seven hundred fifty-seven individuals with traumatic SCI completed 58 items addressing various aspects of pain. Items were then separated by whether they assessed pain interference or pain behavior, and poorly functioning items were removed. Confirmatory factor analyses confirmed that each set of items was unidimensional, and item response theory analyses were used to estimate slopes and thresholds for the items. Ultimately, 7 items (4 from PROMIS) comprised the Pain Behavior scale and 25 items (18 from PROMIS) comprised the Pain Interference item bank. Ten of these 25 items were selected to form the Pain Interference short form.

CONCLUSIONS

The SCI-QOL Pain Interference item bank and the SCI-QOL Pain Behavior scale demonstrated robust psychometric properties. The Pain Interference item bank is available as a computer adaptive test or short form for research and clinical applications, and scores are transformed to the PROMIS metric.

Development and validation of the self-reported PROMIS pediatric pain behavior item bank and short form scale

Pain behaviors are important indicators of functioning in chronic pain; however, no self-reported pain behavior instrument has been developed for pediatric populations. The purpose of this study was to create a brief pediatric measure of patient-reported pain behaviors as part of the Patient-Reported Outcome Measurement Information System (PROMIS). A pool of 47 candidate items for this measure had been previously developed through qualitative research. In this study, youth with chronic pain associated with juvenile fibromyalgia, juvenile idiopathic arthritis, or sickle cell disease (ages 8-18 years) from 3 pediatric centers completed all 47 candidate items for development of the pain behavior item bank along with established measures of pain interference, depressive symptoms, fatigue, average pain intensity, and pain catastrophizing. Caregivers reported on sociodemographic information and health history. Psychometric properties of the pain behavior items were examined using an item response theory framework with confirmatory factor analysis and examination of differential item functioning, internal consistency, and test information curves. Results were used along with expert consensus and alignment with the adult PROMIS pain behavior items to arrive at an 8-item pediatric pain behavior short form, and all 47 items were retained in a calibrated item bank. Confirmatory factor analysis and correlations with validated measures of pain, pain interference, and psychosocial functioning provided support for the short form's reliability and validity. The new PROMIS pediatric pain behavior scale provides a reliable, precise, and valid measure for future research on pain behavior in school-aged children with chronic pain.

Does Mindfulness Correlate With Physical Function and Pain Intensity in Patients With Upper Extremity Illness?

BACKGROUND

Mindfulness skills training interventions seem efficacious in increasing physical function and decreasing pain intensity in patients with chronic pain. The relationship of mindfulness and upper extremity complaints in patients presenting to orthopedic surgical practices is not known. The aim of this study was to assess if mindfulness has a relationship to physical function and pain intensity in patients with upper extremity illness.

METHODS

In this cross-sectional study, a total of 126 patients with a nontraumatic upper extremity condition were asked to fill out questionnaires assessing the 5 facets of mindfulness, pain intensity, and upper extremity physical function, along with clinical and demographic variables prior to their visit with the surgeon.

RESULTS

Nonreactivity to inner experiences was the only facet of mindfulness that was correlated with upper extremity physical function and pain intensity. The overall mindfulness score was correlated with pain intensity only. In multivariable analyses, mindfulness was not associated with either physical function or with pain intensity. Pain interference was the most important predictor of both pain intensity and physical function.

CONCLUSIONS

Greater overall mindfulness was associated with lower pain intensity, and greater ability to be nonreactive to inner experiences was associated with both pain intensity and upper extremity physical function in bivariate but not multivariable analyses. Pain interference was the most important predictor of both pain intensity and upper extremity physical function. Psychosocial interventions focused on improving physical function and decreasing pain intensity in this population should focus primarily on reducing pain interference, and secondarily on teaching patients mindfulness skills.

Standardized Patient Reported Outcomes Do Not Capture Functional Deficits of Patients Following Contemporary Total Knee Replacement: Descriptive Study

BACKGROUND

The physical function subscale of the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC-PF) is widely used and endorsed by professional organizations for patients with knee osteoarthritis. Its use post total knee replacement (TKR) has been challenged as it may not represent the high level of functional performance that is expected by patients who undergo contemporary TKR with more advanced techniques and care pathways.

OBJECTIVE

To assess whether the items of the WOMAC-PF reflect the activity limitations identified by patients following TKR.

DESIGN

Data for this descriptive study were obtained from baseline assessments of a randomized clinical trial comparing exercise interventions following TKR.

METHODS

Participants completed the WOMAC-PF and identified activity limitations in the Canadian Occupational Performance Measure (COPM) in the same day. The responses to both questionnaires were compared.

RESULTS

This investigation included 50 participants (36 women, mean age 63.8±6.7). The WOMAC-PF failed to capture 50% of the activity limitations identified by participants in the COPM. These activities included kneeling, squatting, carrying/lifting items, strength/endurance exercise, floor transfer, lower extremity exercise, walking up/down hills, yard work, climbing a ladder, driving, managing the environment, carrying objects up/down stairs, gait initiation, balance, and going up/down curbs. Only one activity on the WOMAC-PF (going shopping) was not identified by participant responses on the COPM.

LIMITATIONS

Participants were included if they had TKR between 3 and 6 months prior, which may limit generalizability to those immediately after TKR, and the study sample was relatively small.

CONCLUSIONS

In individuals following TKR, the WOMAC-PF failed to represent a subset of higher level, more physically demanding activities that were identified as important by patients following TKR.

First report of factors associated with satisfaction in patients with neurofibromatosis

Patient satisfaction is an integral part of quality health care. We assessed whether health literacy and psychosocial factors are associated with patient satisfaction among adults with neurofibromatosis. Eighty adults (mean age = 44 years; 55% female, 87% white) with NF (50% NF1, 41% NF2, and 9% schwannomatosis) completed an adapted Functional, Communicative, and Critical Health Literacy Questionnaire (FCCHL), the Health Literacy Assessment, a series of Patient Reported Outcome Measures Information System (PROMIS) psychosocial tests, and demographics before the medical visit. After, participants completed two measures of satisfaction: the Medical Interview Satisfaction Scale (MISS) to assess satisfaction with the medical visit, and an adapted version of the Consumer Assessment of Healthcare Providers and Systems Health Literacy Item Set (CAHPS-HL) to assess satisfaction with communication with the provider. Although higher FCCHL health literacy (r = 0.319, P = 0.002), male gender (t = 2.045, P = 0.044) and better psychosocial functioning (r = -0.257 to 0.409, P < 0.05) were associated with higher satisfaction with the medical visit in bivariate correlations, only male gender and higher health literacy remained as significant predictors in multivariable analyses. Higher FCCHL health literacy, less pain interference, fewer pain behaviors, and higher satisfaction with social roles and social discretionary activities (r = -0.231 to 0.331, P < 0.05) were associated with higher satisfaction with the communication with the provider in bivariate analyses. Results support the use of psychosocial and health literacy measures in clinical practice. Referrals to psychosocial treatments in addition to brief interventions focused on increasing health literacy may also be beneficial. © 2017 Wiley Periodicals, Inc.

Evolution of Patient-Reported Outcomes and Their Role in Multiple Sclerosis Clinical Trials

Patient-reported outcomes (PROs) are playing an increasing role in multiple sclerosis (MS) research and practice, and are essential for understanding the effects that MS and MS treatments have on patients' lives. PROs are captured directly from patients and include symptoms, function, health status, and health-related quality of life. In this article, we review different categories (e.g., generic, targeted, preference-based) of PRO measures and considerations in selecting a measure. The PROs included in MS clinical research have evolved over time, as have the measures used to assess them. We describe findings from recent MS clinical trials that included PROs when evaluating Food and Drug Administration-approved disease-modifying therapies (e.g., daclizumab, teriflunomide). Variation in the measures used in these trials makes it difficult to draw any conclusions from the data. We therefore suggest a standardized approach to PRO assessment in MS research and describe 2 generic, National Institutes of Health-supported measurement systems [Neuro-QoL and the Patient-Reported Outcomes Measurement Information System (PROMIS)] that would facilitate such an approach. The use of PROs in MS care and research is expanding beyond clinical trials, as is demonstrated by examples from comparative effectiveness and other patient-centered research. The importance of PRO assessment is expected to continue to grow in the future.

Patient Reported Outcomes Measurement Information System (PROMIS) in the upper extremity: the future of outcomes reporting?

Patient reported outcomes (PROs) serve an integral role in clinical research by helping to determine the impact of clinical care as experienced by the patient. With recent initiatives in health care policy and pay for performance, outcome reporting is now recognized as a policy-driven requirement in addition to a clinical research tool. For outcome measures to satisfy these regulatory requirements and provide value in understanding disease outcomes, they must be responsive and efficient. Recent research has uncovered certain concerns regarding traditional PROs in patients with upper extremity disability and injury. These include lack of consensus regarding selection of PROs for a given diagnoses, inconsistent techniques of administration of the same PROs, and the administrative burden to patients and providers of completing these forms. To address these limitations, emphasis has been placed on streamlining the outcomes reporting process, and, as a result, the National Institutes of Health (NIH) created the Patient Reported Outcomes Measurement Information System (PROMIS). PROMIS forms were created to comprehensively and efficiently measure outcomes across multiple disease states, including orthopedics. These tools exist in computer adaptive testing and short forms with the intention of more efficiently measuring outcomes compared with legacy PROs. The goals of this review are to highlight the main components of PROMIS reporting tools and identify recent use of the scores in the upper extremity literature. The review will also highlight the research and health policy potentials and limitations of implementing PROMIS into everyday orthopedic practice.

Cross-Cultural Evaluation of Health Status Using Item Response Theory

To make meaningful cross-cultural comparisons of health-related quality of life (HRQOL) or to pool international research data, it is essential to create culturally unbiased measures that detect clinically important differences between patients. We evaluated the measurement properties of the Functional Assessment of Cancer Therapy-Breast (FACT-B) in 111 Austrian and 144 U.S. patients with breast cancer using item response theory (IRT) methods. A small number of items were identified as displaying statistically significant differential item functioning (DIF), suggesting possible measurement bias. The majority of the items functioned similarly between the two cultural groups. U.S. patients reported lower (worse) physical function and well-being compared with Austrian patients, higher (better) social/family well-being and similar emotional well-being, before and after adjustment for DIF. IRT and related measurement models provide useful methods for assessing cross-cultural equivalence and determining which items can be pooled across languages before analyzing HRQOL data. Determination of clinically significant cross-cultural differences will require additional investigation.

Development of an IRT-Based Short Form to Assess Applied Cognitive Function in Outpatient Rehabilitation

OBJECTIVE

Item response theory and computerized adaptive testing methods allow the development of a large calibrated item bank from which different subsets of questions can be selected for administration and scored on a common scale. The objective of this study was to develop an outpatient rehabilitation self-report short form for the Activity Measure for Post-Acute Care Applied Cognition item bank.

DESIGN

Using data from a convenience sample of 235 rehabilitation outpatients, item content and item response theory-based test information function parameters were used in item selection. Internal consistency reliability, intraclass correlation coefficient (ICC), and percentage at the lowest (floor) and highest (ceiling) scores were evaluated for the short form and full item bank.

RESULTS

A 15-item short form was developed. The internal consistency of the short form was 0.86. The ICC3,1 for the short form and item bank was 0.97 (95% confidence interval, 0.94-0.98). No floor effects were noted, and ceiling effects were 27.66% (short form) and 26.38% (full item bank).

CONCLUSIONS

The Applied Cognition outpatient short form demonstrated acceptable psychometric properties and provides a bridge to item response theory-based measurement for settings where point-of-care computing is not available.

Self-Reported Changes in Sun-Protection Behaviors at Different Latitudes in Australia

Sun exposure is the most important source of vitamin D, but is also a risk factor for skin cancer. This study investigated attitudes toward vitamin D, and changes in sun-exposure behavior due to concern about adequate vitamin D. Participants (n = 1002) were recruited from four regions of Australia and completed self- and interviewer-administered surveys. Chi-square tests were used to assess associations between participants' latitude of residence, vitamin D-related attitudes and changes in sun-exposure behaviors during the last summer. Multivariate logistic regression analyses were used to model the association between attitudes and behaviors. Overall, people who worried about their vitamin D status were more likely to have altered sun protection and spent more time in the sun people not concerned about vitamin D. Concern about vitamin D was also more common with increasing latitude. Use of novel item response theory analysis highlighted the potential impact of self-reported behavior change on skin cancer predisposition due concern to vitamin. This cross-sectional study shows that the strongest determinants of self-reported sun-protection behavior changes due to concerns about vitamin D were attitudes and location, with people at higher latitudes worrying more.

Validation and important differences for the Sarcoidosis Assessment Tool. A new patient-reported outcome measure

RATIONALE

Patient-reported outcome (PRO) measures have been developed to measure symptoms and other aspects of health-related quality of life.

OBJECTIVES

The Sarcoidosis Assessment Tool (SAT), a sarcoidosis-specific PRO, was administered in a lung and skin sarcoidosis treatment trial. We explored SAT performance characteristics and correlation with standard clinical measurements to validate it as a useful clinical sarcoidosis-specific PRO.

METHODS

The SAT analyses focused on baseline and Week 16 assessments. Besides the SAT, participants underwent clinical and physician assessments plus additional PROs that were used as anchor variables and were compared with the SAT. Reliability was evaluated by using Cronbach α coefficient. Spearman correlation coefficients were used to evaluate the association between SAT scores with clinical and other PRO measures. Changes between assessments in the clinical and PRO "anchor" variables were classified as improved, stable, or worsened. Mean differences between adjacent categories of the known groups and mean changes from the ability to detect change analyses were reviewed for appropriate clinically important difference estimates.

MEASUREMENTS AND MAIN RESULTS

Results from 173 patients were analyzed. Each SAT module reflected appropriate anchor variables at baseline and in terms of change. The Cronbach α for each of these modules was at least 0.87. In addition, we successfully established a clinically important difference range for each SAT module.

CONCLUSIONS

We demonstrated that the SAT is a reliable and consistent sarcoidosis-specific PRO. It has excellent internal consistency and reliability. A range of clinically important differences has been established for the SAT modules. Clinical trial registered with www.clinicaltrials.gov (NCT 00955279).

Kalpakjian C, W. Choi S, Jette AM, Forchheimer M, Cella D. Overview of the Spinal Cord Injury--Quality of Life (SCI-QOL) measurement system

CONTEXT/OBJECTIVE

The Spinal Cord Injury--Quality of Life (SCI-QOL) measurement system was developed to address the shortage of relevant and psychometrically sound patient reported outcome (PRO) measures available for clinical care and research in spinal cord injury (SCI) rehabilitation. Using a computer adaptive testing (CAT) approach, the SCI-QOL builds on the Patient Reported Outcomes Measurement Information System (PROMIS) and the Quality of Life in Neurological Disorders (Neuro-QOL) initiative. This initial manuscript introduces the background and development of the SCI-QOL measurement system. Greater detail is presented in the additional manuscripts of this special issue.

DESIGN

Classical and contemporary test development methodologies were employed. Qualitative input was obtained from individuals with SCI and clinicians through interviews, focus groups, and cognitive debriefing. Item pools were field tested in a multi-site sample (n=877) and calibrated using item response theory methods. Initial reliability and validity testing was performed in a new sample of individuals with traumatic SCI (n=245).

SETTING

Five Model SCI System centers and one Department of Veterans Affairs Medical Center across the United States.

PARTICIPANTS

Adults with traumatic SCI.

INTERVENTIONS

n/a

OUTCOME MEASURES

n/a

RESULTS

The SCI-QOL consists of 19 item banks, including the SCI-Functional Index banks, and 3 fixed-length scales measuring physical, emotional, and social aspects of health-related QOL (HRQOL).

CONCLUSION

The SCI-QOL measurement system consists of psychometrically sound measures for individuals with SCI. The manuscripts in this special issue provide evidence of the reliability and initial validity of this measurement system. The SCI-QOL also links to other measures designed for a general medical population.

Assessment of patient-reported symptoms of anxiety

Patient self-reported symptoms are of crucial importance to identify anxiety disorders, as well as to monitor their treatment in clinical practice and research. Thus, for evidence-based medicine, a precise, reliable, and valid (ie, “objective”) assessment of the patient's reported “subjective” symptoms is warranted. There is a plethora of instruments available, which can provide psychometrically sound assessments of anxiety, but there are several limitations of current tools that need to be carefully considered for their successful use. Nevertheless, the empirical assessment of mental health status is not as accepted in medicine as is the assessment of biomarkers. One reason for this may be that different instruments assessing the same psychological construct use different scales. In this paper we present some new developments that promise to provide one common metric for the assessment of anxiety, to facilitate the general acceptance of mental health assessments in the future.

Development of self-report measures of social attitudes that act as environmental barriers and facilitators for people with disabilities

OBJECTIVE

To describe the development of new self-report measures of social attitudes that act as environmental facilitators or barriers to the participation of people with disabilities in society.

DESIGN

A mixed-methods approach included a literature review; item classification, selection, and writing; cognitive interviews and field testing of participants with spinal cord injury (SCI), traumatic brain injury (TBI), or stroke; and rating scale analysis to evaluate initial psychometric properties.

SETTING

General community.

PARTICIPANTS

Individuals with SCI, TBI, or stroke participated in cognitive interviews (n=9); community residents with those same conditions participated in field testing (n=305).

INTERVENTIONS

None.

MAIN OUTCOME MEASURE

Self-report item pool of social attitudes that act as facilitators or barriers to people with disabilities participating in society.

RESULTS

An interdisciplinary team of experts classified 710 existing social environment items into content areas and wrote 32 new items. Additional qualitative item review included item refinement and winnowing of the pool prior to cognitive interviews and field testing of 82 items. Field test data indicated that the pool satisfies a 1-parameter item response theory measurement model and would be appropriate for development into a calibrated item bank.

CONCLUSIONS

Our qualitative item review process supported a social environment conceptual framework that includes both social support and social attitudes. We developed a new social attitudes self-report item pool. Calibration testing of that pool is underway with a larger sample to develop a social attitudes item bank for persons with disabilities.

The PROMIS physical function correlates with the QuickDASH in patients with upper extremity illness

BACKGROUND

To assess disability more efficiently with less burden on the patient, the National Institutes of Health has developed the Patient Reported Outcomes Measurement Information System (PROMIS) Physical Function-an instrument based on item response theory and using computer adaptive testing (CAT). Initially, upper and lower extremity disabilities were not separated and we were curious if the PROMIS Physical Function CAT could measure upper extremity disability and the Quick Disability of Arm, Shoulder and Hand (QuickDASH).

QUESTIONS/PURPOSES

We aimed to find correlation between the PROMIS Physical Function and the QuickDASH questionnaires in patients with upper extremity illness. Secondarily, we addressed whether the PROMIS Physical Function and QuickDASH correlate with the PROMIS Depression CAT and PROMIS Pain Interference CAT instruments. Finally, we assessed factors associated with QuickDASH and PROMIS Physical Function in multivariable analysis.

METHODS

A cohort of 93 outpatients with upper extremity illnesses completed the QuickDASH and three PROMIS CAT questionnaires: Physical Function, Pain Interference, and Depression. Pain intensity was measured with an 11-point ordinal measure (0-10 numeric rating scale). Correlation between PROMIS Physical Function and the QuickDASH was assessed. Factors that correlated with the PROMIS Physical Function and QuickDASH were assessed in multivariable regression analysis after initial bivariate analysis.

RESULTS

There was a moderate correlation between the PROMIS Physical Function and the QuickDASH questionnaire (r=-0.55, p<0.001). Greater disability as measured with the PROMIS and QuickDASH correlated most strongly with PROMIS Depression (r=-0.35, p<0.001 and r=0.34, p<0.001 respectively) and Pain Interference (r=-0.51, p<0.001 and r=0.74, p<0.001 respectively). The factors accounting for the variability in PROMIS scores are comparable to those for the QuickDASH except that the PROMIS Physical Function is influenced by other pain conditions while the QuickDASH is not.

CONCLUSIONS

The PROMIS Physical Function instrument may be used as an upper extremity disability measure, as it correlates with the QuickDASH questionnaire, and both instruments are influenced most strongly by the degree to which pain interferes with achieving goals.

LEVEL OF EVIDENCE

Level III, diagnostic study. See the Instructions for Authors for a complete description of levels of evidence.

Patient-reported outcome measures and risk factors in a quality registry: a basis for more patient-centered diabetes care in Sweden

Diabetes is one of the chronic diseases that constitute the greatest disease burden in the world. The Swedish National Diabetes Register is an essential part of the diabetes care system. Currently it mainly records clinical outcomes, but here we describe how it has started to collect patient-reported outcome measures, complementing the standard registry data on clinical outcomes as a basis for evaluating diabetes care. Our aims were to develop a questionnaire to measure patient abilities and judgments of their experience of diabetes care, to describe a Swedish diabetes patient sample in terms of their abilities, judgments, and risk factors, and to characterize groups of patients with a need for improvement. Patient abilities and judgments were estimated using item response theory. Analyzing them together with standard risk factors for diabetes comorbidities showed that the different types of data describe different aspects of a patient's situation. These aspects occasionally overlap, but not in any particularly useful way. They both provide important information to decision makers, and neither is necessarily more relevant than the other. Both should therefore be considered, to achieve a more complete evaluation of diabetes care and to promote person-centered care.

Extending the floor and the ceiling for assessment of physical function

OBJECTIVE

To improve the assessment of physical function by enhancing precision of physical function assessment as it pertains to subjects at extreme ends of the health continuum (i.e., subjects with extremely poor function ["floor"] or extremely good health ["ceiling"]).

METHODS

Under the Patient-Reported Outcomes Measurement Information System (PROMIS) (a National Institutes of Health initiative), we developed new items to assess floor and ceiling physical function in order to supplement the existing item bank. Using item response theory and standard PROMIS methodology, we developed 31 floor items and 31 ceiling items and administered the items during a 12-month prospective, observational study of 737 subjects whose health status was at either extreme. Effect size was calculated and change over time was compared across anchor instruments and across items. Using the observed changes in scores, we back-calculated sample size requirements for the new and comparison measures.

RESULTS

We studied 444 subjects who had been diagnosed as having a chronic illness and/or were of old age and 293 generally fit subjects (including athletes in training). Item response theory analyses confirmed that the new floor and ceiling items outperformed reference items (P < 0.001). The estimated post hoc sample size requirements were reduced by a factor of 2-4 for the floor population and a factor of 2 for the ceiling population.

CONCLUSION

Extending the range of items by which physical function is measured can substantially improve measurement quality, reduce sample size requirements, and improve research efficiency. The paradigm shift from assessing disability to assessing physical function focuses assessment on the entire spectrum of physical function, signals improvement in the conceptual base of outcome assessment, and may be transformative as medical goals more closely approach societal goals for health.

The stigma scale for chronic illnesses 8-item version (SSCI-8): development, validation and use across neurological conditions

BACKGROUND

Although the impact of stigma has been highlighted for epileptic populations, the experiences of people living with other neurological conditions have been less studied.

PURPOSE

In order to promote research on stigma among people with neurological conditions, we sought to develop and psychometrically validate an eight-item questionnaire measuring internalised and enacted stigma experienced by people with epilepsy, multiple sclerosis (MS), Parkinson's disease (PD), stroke and amyotrophic lateral sclerosis (ALS).

METHOD

We used item response theory methodologies to select items and field tested our items with 587 participants from eight academic medical centres across the USA.

RESULTS

We conducted exploratory and confirmatory factor analysis as well as examined the scale's reliability and validity. In addition, we conducted an analysis of variance test to examine mean total score differences across the five neurological conditions. Data from people across conditions revealed that the shortened instrument conformed to an essentially unidimensional model of multifaceted stigma as a one-factor questionnaire with correlated residuals on a pair of items that distinctly measured internalised stigma.

CONCLUSION

Preliminary evidence suggests that the Stigma Scale for Chronic Illness 8-item version fits a unidimensional model, which assesses enacted and internalised stigma, and has adequate internal consistency/reliability and validity in relation to psychological distress and patient performance. Our results suggest fairly low stigma for neurological populations. In addition, our results suggest that stigma may be more severe for patients with ALS relative to those with MS and PD. In the future, the SSCI-8 scale could be used practically in clinic settings to examine stigma without the patient burden associated with lengthier scales.

Development and assessment of floor and ceiling items for the PROMIS physical function item bank

Introduction

Disability and Physical Function (PF) outcome assessment has had limited ability to measure functional status at the floor (very poor functional abilities) or the ceiling (very high functional abilities). We sought to identify, develop and evaluate new floor and ceiling items to enable broader and more precise assessment of PF outcomes for the NIH Patient-Reported-Outcomes Measurement Information System (PROMIS).

Methods

We conducted two cross-sectional studies using NIH PROMIS item improvement protocols with expert review, participant survey and focus group methods. In Study 1, respondents with low PF abilities evaluated new floor items, and those with high PF abilities evaluated new ceiling items for clarity, importance and relevance. In Study 2, we compared difficulty ratings of new floor items by low functioning respondents and ceiling items by high functioning respondents to reference PROMIS PF-10 items. We used frequencies, percentages, means and standard deviations to analyze the data.

Results

In Study 1, low (n = 84) and high (n = 90) functioning respondents were mostly White, women, 70 years old, with some college, and disability scores of 0.62 and 0.30. More than 90% of the 31 new floor and 31 new ceiling items were rated as clear, important and relevant, leaving 26 ceiling and 30 floor items for Study 2. Low (n = 246) and high (n = 637) functioning Study 2 respondents were mostly White, women, 70 years old, with some college, and Health Assessment Questionnaire (HAQ) scores of 1.62 and 0.003. Compared to difficulty ratings of reference items, ceiling items were rated to be 10% more to greater than 40% more difficult to do, and floor items were rated to be about 12% to nearly 90% less difficult to do.

Conclusions

These new floor and ceiling items considerably extend the measurable range of physical function at either extreme. They will help improve instrument performance in populations with broad functional ranges and those concentrated at one or the other extreme ends of functioning. Optimal use of these new items will be assisted by computerized adaptive testing (CAT), reducing questionnaire burden and insuring item administration to appropriate individuals.

Development and validation of a new self-report measure of pain behaviors

Pain behaviors that are maintained beyond the acute stage after injury can contribute to subsequent psychosocial and physical disability. Critical to the study of pain behaviors is the availability of psychometrically sound pain behavior measures. In this study we developed a self-report measure of pain behaviors, the Pain Behaviors Self Report (PaB-SR). PaB-SR scores were developed using item response theory and evaluated using a rigorous, multiple-witness approach to validity testing. Participants included 661 survey participants with chronic pain and with multiple sclerosis, back pain, or arthritis; 618 survey participants who were significant others of a chronic pain participant; and 86 participants in a videotaped pain behavior observation protocol. Scores on the PaB-SR were found to be measurement invariant with respect to clinical condition. PaB-SR scores, observer reports, and the videotaped protocol yielded distinct, but convergent views of pain behavior, supporting the validity of the new measure. The PaB-SR is expected to be of substantial utility to researchers wishing to explore the relationship between pain behaviors and constructs such as pain intensity, pain interference, and disability.

Nesting Monte Carlo EM for high-dimensional item factor analysis

The item factor analysis model for investigating multidimensional latent spaces has proved to be useful. Parameter estimation in this model requires computationally demanding high-dimensional integrations. While several approaches to approximate such integrations have been proposed, they suffer various computational difficulties. This paper proposes a Nesting Monte Carlo Expectation-Maximization (MCEM) algorithm for item factor analysis with binary data. Simulation studies and a real data example suggest that the Nesting MCEM approach can significantly improve computational efficiency while also enjoying the good properties of stable convergence and easy implementation.

Designing and implementing a system for tracking functional status after stroke: a feasibility study

OBJECTIVE

To determine the feasibility of tracking stroke patients' functional outcomes in an integrated health system across a care continuum using the computer version of the Activity Measure of Post-Acute Care (AM-PAC).

SETTING

A large integrated health care system in northern California.

PARTICIPANTS

A total of 222 stroke patients (aged ≥18 years) who were hospitalized after an acute cerebrovascular accident.

METHODS

An AM-PAC assessment was made at discharge from sites of care, including acute hospital, inpatient rehabilitation hospital, skilled nursing facility, home during home care, and outpatient settings. Assessments also were completed in the patient's home at 6 months. Data from the AM-PAC program were integrated with the health care system's databases.

MAIN OUTCOME MEASUREMENTS

(1) AM-PAC administration time at the various sites of care; (2) assessment of a floor or a ceiling effect; and (3) administrative burden of tracking participants.

RESULTS

AM-PAC assessment sessions averaged 7.9 minutes for data acquisition in 3 domains: Basic Mobility, Activities of Daily Living, and Applied Cognition. Participants answered, on average, 27 AM-PAC questions per session. A small ceiling effect was observed at 6 months, and there was a larger ceiling effect when the instrument was administered in an institution, ie, when the AM-PAC institutional item bank was used rather than the community item bank. It was feasible to track patients and to assess their function using the AM-PAC instrument from institutional to community settings. Implementation of the AM-PAC in clinical environments, and the success of the project, were influenced by instrumental, technological, operational, resource, and cultural factors.

CONCLUSIONS

This study demonstrates the feasibility of implementing a single functional outcome instrument in clinical and community settings to measure rehabilitation functional outcomes of stroke patients. Integrating the AM-PAC measurement system into clinical workflows and the electronic medical record could provide assistance to clinicians for medical decision making, functional prognostication, and discharge planning.

Psychometric evaluation of the Taiwan Chinese version of the EORTC QLQ-PR25 for HRQOL assessment in prostate cancer patients

Objective

To evaluate the psychometric properties of the Taiwan Chinese Version of the EORTC QLQ-PR25 health-related quality of life (HRQOL) questionnaire for patients with prostate cancer.

Methods

135 prostate cancer patients were recruited in the urology outpatient clinic of a university teaching hospital. Each patient completed the EORTC QLQ-PR25 at every clinic visit between 2004 and 2008, totaling 633 assessments. Confirmatory factor analysis and Rasch analysis were used to evaluate the domain- and item-level psychometric properties.

Results

The results supported the unidimensionality of each of the four EORTC QLQ-PR25 domains (urinary, bowel, and hormonal-treatment-related symptoms, and sexual functioning). Item calibrations for each domain were found invariant across the three assessment time periods. The item-person maps showed 71.3% of item coverage for the urinary symptoms domain and 13–42.7% for the other three domains.

Conclusions

The Taiwan Chinese Version of the EORTC QLQ-PR25 questionnaire is reliable and can be used to measure HRQOL over time. Adding new items to each domain may improve its clinical content coverage and measurement precision.

Computerized adaptive testing--ready for ambulatory monitoring?

BACKGROUND

Computerized adaptive tests (CATs) have abundant theoretical advantages over established static instruments, which could improve ambulatory monitoring of patient-reported outcomes (PROs). However, an empirical demonstration of their practical benefits is warranted.

METHODS

We reviewed the literature and evaluated existing data to discuss the potential of CATs for use in ambulatory monitoring outside clinical facilities.

RESULTS

Computerized adaptive tests are not being used for ambulatory monitoring, but initial results from their use in health care research allow for discussion of some issues relevant to ambulatory care. Evidence shows that CATs can capture the most relevant health outcomes as well as established static tools, with substantially decreased respondent burden. They can be more precise than static tools of similar length and can reduce floor and ceiling effects. Computerized adaptive tests can reliably measure a construct over time with different items, which yields the potential of introducing item exposure control in ambulatory monitoring. Studies have shown that CATs can be at least as valid as well-designed static tools in group comparisons, but further investigation is needed to determine whether psychometric advantages lead to increased responsiveness of CATs.

CONCLUSIONS

Ambulatory monitoring of PROs demands short, yet very precise measurements, which can be repeated up to many times a day. Computerized adaptive tests may address several present shortcomings in ambulatory monitoring of PROs efficiently. However, most CAT developments have primarily focused on psychometric improvements. To use the full potential of CATs for ambulatory monitoring purposes, content must also be carefully considered.

Neurobehavioral functions evaluated in naturalistic contexts: Rasch analysis of the A-ONE Neurobehavioral Impact Scale

OBJECTIVE

The fact that different neurobehavioral impairments affect daily life task performance of clients with different neurological diagnoses currently restricts between-group comparisons in rehabilitation. The purpose of this study was to determine whether a single neurobehavioral impact scale could be constructed for use with different diagnostic groups.

METHODS AND RESULTS

Rasch analysis of 422 clients (diagnosed with CVA and dementia) demonstrated that 29 of 55 items from the A-ONE Neurobehavioral Scale could be used to construct a short-form, Common Scale.

CONCLUSIONS

While the use of different and longer diagnostic-specific scales versions may be more useful clinically, the short-form, Common Scale has the potential to be used in research focusing on comparison of groups. Further research will be needed to validate the common, short version.

Use of non-parametric item response theory to develop a shortened version of the Positive and Negative Syndrome Scale (PANSS)

BACKGROUND

Nonparametric item response theory (IRT) was used to examine (a) the performance of the 30 Positive and Negative Syndrome Scale (PANSS) items and their options ((levels of severity), (b) the effectiveness of various subscales to discriminate among differences in symptom severity, and (c) the development of an abbreviated PANSS (Mini-PANSS) based on IRT and a method to link scores to the original PANSS.

METHODS

Baseline PANSS scores from 7,187 patients with Schizophrenia or Schizoaffective disorder who were enrolled between 1995 and 2005 in psychopharmacology trials were obtained. Option characteristic curves (OCCs) and Item Characteristic Curves (ICCs) were constructed to examine the probability of rating each of seven options within each of 30 PANSS items as a function of subscale severity, and summed-score linking was applied to items selected for the Mini-PANSS.

RESULTS

The majority of items forming the Positive and Negative subscales (i.e. 19 items) performed very well and discriminate better along symptom severity compared to the General Psychopathology subscale. Six of the seven Positive Symptom items, six of the seven Negative Symptom items, and seven out of the 16 General Psychopathology items were retained for inclusion in the Mini-PANSS. Summed score linking and linear interpolation was able to produce a translation table for comparing total subscale scores of the Mini-PANSS to total subscale scores on the original PANSS. Results show scores on the subscales of the Mini-PANSS can be linked to scores on the original PANSS subscales, with very little bias.

CONCLUSIONS

The study demonstrated the utility of non-parametric IRT in examining the item properties of the PANSS and to allow selection of items for an abbreviated PANSS scale. The comparisons between the 30-item PANSS and the Mini-PANSS revealed that the shorter version is comparable to the 30-item PANSS, but when applying IRT, the Mini-PANSS is also a good indicator of illness severity.

The psychosocial impact of cancer: evidence in support of independent general positive and negative components

PURPOSE

Considerable research has demonstrated the negative psychosocial impact of cancer. Recent work has highlighted positive psychosocial outcomes. Research is now needed to evaluate the relationship between negative and positive impacts. This paper reports the development and validation of a measurement model capturing positive and negative psychosocial illness impacts.

METHODS

The sample included 754 cancer patients on- or post-treatment. Item development was informed by literature review, expert input patient interviews and the results of a pilot study of 205 cancer patients, resulting in 43 positive and 46 negative items. Factor analyses were used to evaluate the dimensionality of the item pools. Analysis of variance (ANOVA) was used to examine relationships between psychosocial illness impact and other variables.

RESULTS

Unidimensionality was demonstrated within but not across negative and positive impact items. ANOVA results showed differential relationships between negative and positive impacts, respectively, and patient sociodemographic and clinical variables.

CONCLUSION

Positive and negative psychosocial illness impacts are best conceptualized and measured as two independent factors. Computerized adaptive tests and short-form measures developed from this comprehensive psychosocial illness impact item bank may benefit future research and clinical applications.

Development of a conceptual framework and calibrated item banks to measure patient-reported dyspnea severity and related functional limitations

OBJECTIVES

Chronic obstructive pulmonary disease is a major global health problem. Although several patient-reported outcome (PRO) measures of chronic obstructive pulmonary disease exist, none were developed using patient-driven concept development. We developed an item bank for dyspnea severity and related functional limitations on the basis of a PRO conceptual framework derived from patient input.

METHODS

We identified a large pool of existing items based on a conceptual framework and literature review. Using patient and expert review panels and an item refinement/modification process, we developed an item bank aligned with the conceptual framework, which subsequently underwent psychometric testing via an online Internet panel of dyspnea patients (N = 608).

RESULTS

Exploratory factor analysis suggested a dominant first factor accounting for about 78% of the total variance. Confirmatory factor analysis supported a unidimensional model. Item response theory analysis demonstrated good model fit, and differential item functioning analyses indicated that the 33-item scale showed potential for measurement equivalence across sex. A 10-item short form produced comparable scores (r = 0.98) and a computerized adaptive-testing simulation indicated efficient measurement with fewer items (mean 4.65 items).

CONCLUSIONS

An efficient patient-reported measure of dyspnea severity and related functional limitations, based on a patient-driven PRO conceptual framework, is now available for further validation and use.

Health literacy assessment using talking touchscreen technology (Health LiTT): a new item response theory-based measure of health literacy

The importance of health literacy has grown considerably among researchers, clinicians, patients, and policymakers. Better instruments and measurement strategies are needed. Our objective was to develop a new health literacy instrument using novel health information technology and modern psychometrics. We designed Health LiTT as a self-administered multimedia touchscreen test based on item response theory (IRT) principles. We enrolled a diverse group of 619 English-speaking, primary care patients in clinics for underserved patients. We tested three item types (prose, document, quantitative) that worked well together to reliably measure a single dimension of health literacy. The Health LiTT score meets psychometric standards (reliability of 0.90 or higher) for measurement of individual respondents in the low to middle range. Mean Health LiTT scores were associated with age, race/ethnicity, education, income, and prior computer use (p < .05). We created an IRT-calibrated item bank of 82 items. Standard setting needs to be performed to classify and map items onto the construct and to identify measurement gaps. We are incorporating Health LiTT into an existing online research management tool. This will enable administration of Health LiTT on the same touchscreen used for other patient-reported outcomes, as well as real-time scoring and reporting of health literacy scores.

Development and validation of participation and positive psychologic function measures for stroke survivors

OBJECTIVE

To evaluate the reliability and validity of Neurologic Quality of Life (NeuroQOL) item banks that assess quality-of-life (QOL) domains not typically included in poststroke measures.

DESIGN

Secondary analysis of item responses to selected NeuroQOL domains.

SETTING

Community.

PARTICIPANTS

Community-dwelling stroke survivors (n=111) who were at least 12 months poststroke.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Five measures developed for 3 NeuroQoL domains: ability to participate in social activities, satisfaction with participation in social activities, and positive psychologic function.

RESULTS

A single bank was developed for the positive psychologic function domain, but 2 banks each were developed for the ability-to-participate and satisfaction-with-participation domains. The resulting item banks showed good psychometric properties and external construct validity with correlations with the legacy instruments, ranging from .53 to .71. Using these measures, stroke survivors in this sample reported an overall high level of QOL.

CONCLUSIONS

The NeuroQoL-derived measures are promising and valid methods for assessing aspects of QOL not typically measured in this population.

Measuring social health in the patient-reported outcomes measurement information system (PROMIS): item bank development and testing

PURPOSE

To develop a social health measurement framework, to test items in diverse populations and to develop item response theory (IRT) item banks.

METHODS

A literature review guided framework development of Social Function and Social Relationships sub-domains. Items were revised based on patient feedback, and Social Function items were field-tested. Analyses included exploratory factor analysis (EFA), confirmatory factor analysis (CFA), two-parameter IRT modeling and evaluation of differential item functioning (DIF).

RESULTS

The analytic sample included 956 general population respondents who answered 56 Ability to Participate and 56 Satisfaction with Participation items. EFA and CFA identified three Ability to Participate sub-domains. However, because of positive and negative wording, and content redundancy, many items did not fit the IRT model, so item banks do not yet exist. EFA, CFA and IRT identified two preliminary Satisfaction item banks. One item exhibited trivial age DIF.

CONCLUSION

After extensive item preparation and review, EFA-, CFA- and IRT-guided item banks help provide increased measurement precision and flexibility. Two Satisfaction short forms are available for use in research and clinical practice. This initial validation study resulted in revised item pools that are currently undergoing testing in new clinical samples and populations.

Item response theory and health-related quality of life in cancer

Identifying health-related quality of life concerns is a priority when caring for people with cancer. Specific problem areas such as pain, fatigue, emotional distress, disease- and treatment-related symptoms, as well as physical functioning can be routinely assessed using applications that draw upon item response theory. Item response theory measurement models can improve on the classical approach to health-related quality of life assessment with advantages that include comparison of patients across diverse instruments, flexibility in degree of precision desired, availability of multiple short forms, interval measurement and capability for individual assessment (real-time clinical monitoring) using computerized adaptive testing. This review describes a model of health-related quality of life in oncology and the contribution of item response theory to assessment using that model.

Literacy-fair measurement of health-related quality of life will facilitate comparative effectiveness research in Spanish-speaking cancer outpatients

BACKGROUND

Health-related quality of life (HRQL) assessment is frequently used in comparative effectiveness research, but low-literacy patients are often excluded. Appropriately translated and user-friendly HRQL measures are essential to ensure inclusion of low-literate and non-English-speaking patients in comparative effectiveness research.

OBJECTIVES

To compare HRQL responses across literacy levels in Spanish-speaking patients with cancer using a multimedia touch screen program.

SUBJECTS

A total of 414 adult patients with cancer (213 with low literacy and 201 with high literacy).

RESEARCH DESIGN

The touch screen system administered 3 questionnaires: The Functional Assessment of Cancer Therapy-General, the Short Form-36 Health Survey, and the Standard Gamble Utility Questionnaire. Measurement bias was evaluated using item response theory. Effects of literacy on HRQL were evaluated using regression models.

RESULTS

Patients rated the touch screen easy to use and commented favorably on the multimedia approach. There was statistically significant item response theory measurement bias in 6 of 10 HRQL subscales; however, only 3 showed meaningful bias. Low-literacy patients had significantly lower mean scores on 3 of 4 Functional Assessment of Cancer Therapy-General subscales, before and after adjustment for patient characteristics. Low-literacy patients also had significantly lower mean scores on 5 of 6 Short Form-36 subscales; adjustment for patient characteristics attenuated or eliminated differences. Similar proportions of low- and high-literacy patients valued their current health as equivalent to perfect health.

CONCLUSIONS

This study demonstrates the feasibility of this multimedia touch screen program for low-literacy patients. The program will provide opportunities to evaluate the effectiveness of interventions in more diverse patient populations.

Development and validation of a computer adaptive test for measuring dyspnea in heart failure

BACKGROUND

Dyspnea is a common symptom among patients with heart failure. Currently, there is no standardized, rapid, precise method to assess dyspnea.

METHODS AND RESULTS

From a review of the literature, we pooled questions from various questionnaires assessing dyspnea. A total of 201 patients with heart failure completed all questions in the preliminary item bank. Each item asks how much shortness of breath the patient had when doing an activity. Medical charts were reviewed for hospitalization within 1 or 3 months of completing the questions. We created a dyspnea item bank of 44 items. Computer adaptive tests (CAT) generated from this item bank can assess dyspnea by administering on average 10 questions. Simulation CAT scores were generated to compare with the item bank scores. The CAT scores had a correlation of 0.98 with item bank scores. Logistic regression models predicting the probability of being hospitalized from the dyspnea score were statistically significant (P < .05). A 5-point score increase was associated with a 32% increased odds of hospitalization in 1 month and a 20% increased odds of hospitalization in 3 months.

CONCLUSIONS

This computer-based tool for dyspnea assessment obtains similar precision to that of answering the entire dyspnea item bank with less patient burden.

Better assessment of physical function: item improvement is neglected but essential

Introduction

Physical function is a key component of patient-reported outcome (PRO) assessment in rheumatology. Modern psychometric methods, such as Item Response Theory (IRT) and Computerized Adaptive Testing, can materially improve measurement precision at the item level. We present the qualitative and quantitative item-evaluation process for developing the Patient Reported Outcomes Measurement Information System (PROMIS) Physical Function item bank.

Methods

The process was stepwise: we searched extensively to identify extant Physical Function items and then classified and selectively reduced the item pool. We evaluated retained items for content, clarity, relevance and comprehension, reading level, and translation ease by experts and patient surveys, focus groups, and cognitive interviews. We then assessed items by using classic test theory and IRT, used confirmatory factor analyses to estimate item parameters, and graded response modeling for parameter estimation. We retained the 20 Legacy (original) Health Assessment Questionnaire Disability Index (HAQ-DI) and the 10 SF-36's PF-10 items for comparison. Subjects were from rheumatoid arthritis, osteoarthritis, and healthy aging cohorts (n = 1,100) and a national Internet sample of 21,133 subjects.

Results

We identified 1,860 items. After qualitative and quantitative evaluation, 124 newly developed PROMIS items composed the PROMIS item bank, which included revised Legacy items with good fit that met IRT model assumptions. Results showed that the clearest and best-understood items were simple, in the present tense, and straightforward. Basic tasks (like dressing) were more relevant and important versus complex ones (like dancing). Revised HAQ-DI and PF-10 items with five response options had higher item-information content than did comparable original Legacy items with fewer response options. IRT analyses showed that the Physical Function domain satisfied general criteria for unidimensionality with one-, two-, three-, and four-factor models having comparable model fits. Correlations between factors in the test data sets were > 0.90.

Conclusions

Item improvement must underlie attempts to improve outcome assessment. The clear, personally important and relevant, ability-framed items in the PROMIS Physical Function item bank perform well in PRO assessment. They will benefit from further study and application in a wider variety of rheumatic diseases in diverse clinical groups, including those at the extremes of physical functioning, and in different administration modes.

Development of items designed to evaluate activity performance and participation in children and adolescents with spinal cord injury

Background/Objective. Outcomes-based data, whether used clinically or for research, are difficult to collect in the pediatric spinal cord injury (SCI) population due to a lack of appropriate assessment measures. The purpose of this paper is twofold: to describe the process by which two item pools were developed to evaluate activity performance and participation among children with SCI and to introduce the resultant items specific to pediatric SCI. Methods. The process of item development, including construct development, review of related assessment tools, chart review, item writing and refinement using focus groups, cognitive interviews, and further refinement, was used to create the items pools for activity and participation for children and adolescents with SCI. Results. A total of 347 items were written for the activity performance construct and 61 items were written for the participation construct. Several domains were established within each construct and items were written for both child and parent respondents. Conclusion. The process of detailed item development is the first step in the process of developing an outcomes instrument for children and adolescents with SCI to assess activity performance and participation. The items are representative of pediatric SCI because they address areas specific to children and adolescents with SCI such as wheeled mobility, upper extremity function with adaptive equipment, role performance, and socialization. After testing these items in calibration studies, we will determine if these items can be developed into effective computer-adaptive testing applications.

The development of an ICF-oriented, adaptive physician assessment instrument of mobility, self-care, and domestic life

The objective of this study was development and psychometric testing of an adaptive, International Classification of Functioning, Disability, and Health (ICF)-oriented questionnaire to be processed by the rehabilitation physician that aids in assessing mobility, self-care, and domestic life (Moses-Physician). The intent is to develop a physician version, analogous in content to the existing patient questionnaire 'Moses-Patient'. The 58 items of Moses-Patient were converted to an external assessment format without altering the content. The data were compiled for 549 patients with musculoskeletal diseases, 212 patients with cardiac diseases, and 259 neurology rehabilitation patients. Analyses were carried out on the basis of the one-parameter item response theory (Rasch model). Effect sizes and the reliable change index were calculated to test responsiveness. Differential item functioning (DIF) was tested using DIF contrasts, equivalent to Mantel-Haenszel DIF sizes. After the item response theory analysis, 47 of 58 items remained, distributed over 12 scales. The scales are more homogeneous in content than in the patient version because of the omission of 11 items and thus do not cover the ICF categories as broadly. Model fit indices (infit and outfit mean square statistics) were in an acceptable range for all items. Cronbach's alpha was between 0.73 and 0.95. Moreover, there is clear evidence of unidimensionality and sensitivity to change of the scales of Moses-Physician. The item parameters of Moses-Physician are invariant with respect to sex and age for all scales. However, there are clear differences regarding disease groups. The Moses-Physician questionnaire is an adaptive, Rasch-scaled assessment instrument that, to a great extent, covers the contents of the ICF chapters 'mobility', 'self-care', and 'domestic life'.

Function and disability in late life: comparison of the Late-Life Function and Disability Instrument to the Short-Form-36 and the London Handicap Scale

PURPOSE

We evaluated the Late-Life Function and Disability Instrument's (LLFDI) concurrent validity, comprehensiveness and precision by comparing it with the Short-Form-36 physical functioning (PF-10) and the London Handicap Scale (LHS).

METHODS

We administered the LLFDI, PF-10 and LHS to 75 community-dwelling adults (> 60 years of age). We used Pearson correlation coefficients to examine concurrent validity and Rasch analysis to compare the item hierarchies, content ranges and precision of the PF-10 and LLFDI function domains, and the LHS and the LLFDI disability domains.

RESULTS

LLFDI Function (lower extremity scales) and PF-10 scores were highly correlated (r = 0.74 - 0.86, p > 0.001); moderate correlations were found between the LHS and the LLFDI Disability limitation (r = 0.66, p < 0.0001) and Disability frequency (r = 0.47, p < 0.001) scores. The LLFDI had a wider range of content coverage, less ceiling effects and better relative precision across the spectrum of function and disability than the PF-10 and the LHS. The LHS had slightly more content range and precision in the lower end of the disability scale than the LLFDI.

CONCLUSIONS

The LLFDI is a more comprehensive and precise instrument compared to the PF-10 and LHS for assessing function and disability in community-dwelling older adults.