Validation of a computer-adaptive test to evaluate generic health-related quality of life

Psychometric properties of computerized adaptive testing for chronic obstructive pulmonary disease patient-reported outcome measurement

BACKGROUND

Computerized adaptive testing (CAT) is an effective way to reduce time, repetitious redundancy, and respond burden, and has been used to measure outcomes in many diseases. This study aimed to develop and validate a comprehensive disease-specific CAT for chronic obstructive pulmonary disease (COPD) patient-reported outcome measurement.

METHODS

The discrimination and difficulty of the items from the modified patient-reported outcome scale for COPD (mCOPD-PRO) were analyzed using item response theory. Then the initial item, item selection method, ability estimation method, and stopping criteria were further set based on Concerto platform to form the CAT. Finally, the reliability and validity were validated.

RESULTS

The item discrimination ranged from 1.05 to 2.71, and the item difficulty ranged from - 3.08 to 3.65. The measurement reliability of the CAT ranged from 0.910 to 0.922 using random method, while that ranged from 0.910 to 0.924 using maximum Fisher information (MFI) method. The content validity was good. The correlation coefficient between theta of the CAT and COPD assessment test and modified Medical Research Council dyspnea scale scores using random method was 0.628 and 0.540 (P < 0.001; P < 0.001) respectively, while that using MFI method was 0.347 and 0.328 (P = 0.007; P = 0.010) respectively. About 11 items (reducing by 59.3%) on average were tested using random method, while about seven items (reducing by 74.1%) on average using MFI method. The correlation coefficient between theta of the CAT and mCOPD-PRO total scores using random method was 0.919 (P < 0.001), while that using MFI method was 0.760 (P < 0.001).

CONCLUSIONS

The comprehensive disease-specific CAT for COPD patient-reported outcome measurement is well developed with good psychometric properties, which can provide an efficient, accurate, and user-friendly measurement for patient-reported outcome of COPD.

Applying multidimensional computerized adaptive testing to the MSQOL-54: a simulation study

BACKGROUND

The Multiple Sclerosis Quality of Life-54 (MSQOL-54) is one of the most commonly-used MS-specific health-related quality of life (HRQOL) measures. It is a multidimensional, MS-specific HRQOL inventory, which includes the generic SF-36 core items, supplemented with 18 MS-targeted items. Availability of an adaptive short version providing immediate item scoring may improve instrument usability and validity. However, multidimensional computerized adaptive testing (MCAT) has not been previously applied to MSQOL-54 items. We thus aimed to apply MCAT to the MSQOL-54 and assess its performance.

METHODS

Responses from a large international sample of 3669 MS patients were assessed. We calibrated 52 (of the 54) items using bifactor graded response model (10 group factors and one general HRQOL factor). Then, eight simulations were run with different termination criteria: standard errors (SE) for the general factor and group factors set to different values, and change in factor estimates from one item to the next set at < 0.01 for both the general and the group factors. Performance of the MCAT was assessed by the number of administered items, root mean square difference (RMSD), and correlation.

RESULTS

Eight items were removed due to local dependency. The simulation with SE set to 0.32 (general factor), and no SE thresholds (group factors) provided satisfactory performance: the median number of administered items was 24, RMSD was 0.32, and correlation was 0.94.

CONCLUSIONS

Compared to the full-length MSQOL-54, the simulated MCAT required fewer items without losing precision for the general HRQOL factor. Further work is needed to add/integrate/revise MSQOL-54 items in order to make the calibration and MCAT performance efficient also on group factors, so that the MCAT version may be used in clinical practice and research.

Health-Related Quality of Life (HRQOL) Instruments and Mobility: A Systematic Review

Physical function is one of the most important constructs assessed in health-related quality of life (HRQOL), and it could be very useful to assess movement ability from the perspective of the patient. The objective of this study was to compare the content of the domains related to mobility covered by the HRQOL questionnaires based on the International Classification of Functioning, Disability and Health (ICF) and to evaluate their quality according to the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) guidance. For this, a systematic review was carried out in the databases Scopus, Web of Science and Science Direct. The inclusion criteria were development and/or validation studies about generic HRQOL measures, and the instruments had to include items related to mobility and studies written in English or Spanish. The comparison of content was performed using the ICF coding system. A total of 3614 articles were found, 20 generic HRQOL instruments were identified and 120 (22.4%) mobility-related items were found. Walking was the most represented category. Low-quality evidence on some measurement properties of the generic HRQOL instruments was revealed. The CAT-Health is a useful questionnaire to be used in rehabilitation due to its psychometric properties and its content.

Patient-Reported Outcome Measurement Information System (PROMIS) in Orthopaedic Trauma Research

This review describes the development, advantages and disadvantages, and applications of the Patient-Reported Outcome Measurement Information System (PROMIS) in orthopaedic trauma. PROMIS is a useful tool for quantifying outcomes in orthopedic trauma. It allows measurement of outcomes across multiple domains while minimizing administration time. PROMIS also reliably identifies clinical, social, and psychological risk factors for poor outcomes across a variety of orthopaedic injuries and disease states. However, PROMIS lacks specificity for certain anatomic regions and validation for mental health outcomes. It also is limited by ceiling effects in certain active patient populations. Orthopaedic traumatologists should be familiar with PROMIS, as its use is increasing and it is a valuable tool that can aid in clinical decision making.

Patient Reported Outcomes Measurement Information System (PROMIS) in the upper extremity: the future of outcomes reporting?

Patient reported outcomes (PROs) serve an integral role in clinical research by helping to determine the impact of clinical care as experienced by the patient. With recent initiatives in health care policy and pay for performance, outcome reporting is now recognized as a policy-driven requirement in addition to a clinical research tool. For outcome measures to satisfy these regulatory requirements and provide value in understanding disease outcomes, they must be responsive and efficient. Recent research has uncovered certain concerns regarding traditional PROs in patients with upper extremity disability and injury. These include lack of consensus regarding selection of PROs for a given diagnoses, inconsistent techniques of administration of the same PROs, and the administrative burden to patients and providers of completing these forms. To address these limitations, emphasis has been placed on streamlining the outcomes reporting process, and, as a result, the National Institutes of Health (NIH) created the Patient Reported Outcomes Measurement Information System (PROMIS). PROMIS forms were created to comprehensively and efficiently measure outcomes across multiple disease states, including orthopedics. These tools exist in computer adaptive testing and short forms with the intention of more efficiently measuring outcomes compared with legacy PROs. The goals of this review are to highlight the main components of PROMIS reporting tools and identify recent use of the scores in the upper extremity literature. The review will also highlight the research and health policy potentials and limitations of implementing PROMIS into everyday orthopedic practice.

Evaluation of Computerized Adaptive Tests (CATs) for longitudinal monitoring of depression, anxiety, and stress reactions

BACKGROUND

Computerized adaptive testing (CAT) based on Item Response Theory, (IRT) offers an efficient way for accurate measurement of patient reported outcomes. The efficiency lies within a minimal response burden and a high measurement precision over a broad measurement range. The objective of the study was to evaluate and compare the responsiveness of CATs measuring anxiety, depression, and stress reaction to standard static self-assessment tools.

METHODS

Longitudinal data of n=595 psychosomatic inpatients were analyzed for evaluating retest-reliability and sensitivity to change of the CATs compared to static measures (GAD-7, PHQ-9, and PSQ) using correlational and ANOVA statistics. The study hypothesized that CATs are at least as retest-reliable and as sensitive to change as static tools.

RESULTS

The three CATs show a low burden for patients, administering on average 5-7 (±2-6SD) items with similar retest-reliability compared to the static tools applied (A-CAT: r=.78 vs. GAD-7: r=.75, D-CAT: r=.71 vs. PHQ-9: r=.75, S-CAT: r=.80 vs. PSQworries scale: r=.80). The CATs were overall as sensitive to change as the static tools (Cohen׳s d ranged between .19 and .69).

LIMITATIONS

This is a monocenter, observational, longitudinal study without external clinical criteria; thus generalization to other settings may be limited.

CONCLUSIONS

The tested CATs belong to the first generation of CATs being used in daily routine for more than a decade. They are as retest reliable and sensitive to change as static tools. Newer CATs may provide further practical advantages.

Adaptive Measurement of Well-Being: Maximizing Efficiency and Optimizing User Experience during Individual Assessment

Well-being is linked to important societal factors such as health care costs and productivity and has experienced a surge in development activity of both theories and measurement. This study builds on validation of the Well-Being 5 survey and for the first time applies Item Response Theory, a modern and flexible measurement paradigm, to form the basis of adaptive population well-being measurement. Adaptive testing allows survey questions to be administered selectively, thereby reducing the number of questions required of the participant. After the graded response model was fit to a sample of size N = 12,035, theta scores were estimated based on both the full-item bank and a simulation of Computerized Adaptive Testing (CAT). Comparisons of these 2 sets of score estimates with each other and of their correlations with external outcomes of job performance, absenteeism, and hospital admissions demonstrate that the CAT well-being scores maintain accuracy and validity. The simulation indicates that the average survey taker can expect a reduction in number of items administered during the CAT process of almost 50%. An increase in efficiency of this extent is of considerable value because of the time savings during the administration of the survey and the potential improvement of user experience, which in turn can help secure the success of a total population-based well-being improvement program. (Population Health Management 2016;19:284–290)

Development of a computerized adaptive test for Schizotypy assessment

BACKGROUND

Schizotypal traits in adolescents from the general population represent the behavioral expression of liability for psychotic disorders. Schizotypy assessment in this sector of population has advanced considerably in the last few years; however, it is necessary to incorporate recent advances in psychological and educational measurement.

OBJECTIVE

The main goal of this study was to develop a Computerized Adaptive Test (CAT) to evaluate schizotypy through "The Oviedo Questionnaire for Schizotypy Assessment" (ESQUIZO-Q), in non-clinical adolescents.

METHODS

The final sample consisted of 3,056 participants, 1,469 males, with a mean age of 15.9 years (SD=1.2).

RESULTS

The results indicated that the ESQUIZO-Q scores presented adequate psychometric properties under both Classical Test Theory and Item Response Theory. The Information Function estimated using the Gradual Response Model indicated that the item pool effectively assesses schizotypy at the high end of the latent trait. The correlation between the CAT total scores and the paper-and-pencil test was 0.92. The mean number of presented items in the CAT with the standard error fixed at ≤ 0.30 was of 34 items.

CONCLUSION

The CAT showed adequate psychometric properties for schizotypy assessment in the general adolescent population. The ESQUIZO-Q adaptive version could be used as a screening method for the detection of adolescents at risk for psychosis in both educational and mental health settings.

Validity of computer adaptive tests of daily routines for youth with spinal cord injury

OBJECTIVE

To evaluate the accuracy of computer adaptive tests (CATs) of daily routines for child- and parent-reported outcomes following pediatric spinal cord injury (SCI) and to evaluate the validity of the scales.

METHODS

One hundred ninety-six daily routine items were administered to 381 youths and 322 parents. Pearson correlations, intraclass correlation coefficients (ICC), and 95% confidence intervals (CI) were calculated to evaluate the accuracy of simulated 5-item, 10-item, and 15-item CATs against the full-item banks and to evaluate concurrent validity. Independent samples t tests and analysis of variance were used to evaluate the ability of the daily routine scales to discriminate between children with tetraplegia and paraplegia and among 5 motor groups.

RESULTS

ICC and 95% CI demonstrated that simulated 5-, 10-, and 15-item CATs accurately represented the full-item banks for both child- and parent-report scales. The daily routine scales demonstrated discriminative validity, except between 2 motor groups of children with paraplegia. Concurrent validity of the daily routine scales was demonstrated through significant relationships with the FIM scores.

CONCLUSION

Child- and parent-reported outcomes of daily routines can be obtained using CATs with the same relative precision of a full-item bank. Five-item, 10-item, and 15-item CATs have discriminative and concurrent validity.

Computerized adaptive testing--ready for ambulatory monitoring?

BACKGROUND

Computerized adaptive tests (CATs) have abundant theoretical advantages over established static instruments, which could improve ambulatory monitoring of patient-reported outcomes (PROs). However, an empirical demonstration of their practical benefits is warranted.

METHODS

We reviewed the literature and evaluated existing data to discuss the potential of CATs for use in ambulatory monitoring outside clinical facilities.

RESULTS

Computerized adaptive tests are not being used for ambulatory monitoring, but initial results from their use in health care research allow for discussion of some issues relevant to ambulatory care. Evidence shows that CATs can capture the most relevant health outcomes as well as established static tools, with substantially decreased respondent burden. They can be more precise than static tools of similar length and can reduce floor and ceiling effects. Computerized adaptive tests can reliably measure a construct over time with different items, which yields the potential of introducing item exposure control in ambulatory monitoring. Studies have shown that CATs can be at least as valid as well-designed static tools in group comparisons, but further investigation is needed to determine whether psychometric advantages lead to increased responsiveness of CATs.

CONCLUSIONS

Ambulatory monitoring of PROs demands short, yet very precise measurements, which can be repeated up to many times a day. Computerized adaptive tests may address several present shortcomings in ambulatory monitoring of PROs efficiently. However, most CAT developments have primarily focused on psychometric improvements. To use the full potential of CATs for ambulatory monitoring purposes, content must also be carefully considered.