Item banking and computer-adaptive testing in clinical trials: Standing in sight of the PROMISed land

Remote Symptom Monitoring With Ecological Momentary Computerized Adaptive Testing: Pilot Cohort Study of a Platform for Frequent, Low-Burden, and Personalized Patient-Reported Outcome Measures

BACKGROUND

Remote patient-reported outcome measure (PROM) data capture can provide useful insights into research and clinical practice and deeper insights can be gained by administering assessments more frequently, for example, in ecological momentary assessment. However, frequent data collection can be limited by the burden of multiple, lengthy questionnaires. This burden can be reduced with computerized adaptive testing (CAT) algorithms that select only the most relevant items from a PROM for an individual respondent. In this paper, we propose "ecological momentary computerized adaptive testing" (EMCAT): the use of CAT algorithms to reduce PROM response burden and facilitate high-frequency data capture via a smartphone app. We develop and pilot a smartphone app for performing EMCAT using a popular hand surgery PROM.

OBJECTIVE

The aim of this study is to determine the feasibility of EMCAT as a system for remote PROM administration.

METHODS

We built the EMCAT web app using Concerto, an open-source CAT platform maintained by the Psychometrics Centre, University of Cambridge, and hosted it on an Amazon Web Service cloud server. The platform is compatible with any questionnaire that has been parameterized with item response theory or Rasch measurement theory. For this study, the PROM we chose was the patient evaluation measure, which is commonly used in hand surgery. CAT algorithms were built using item response theory models derived from UK Hand Registry data. In the pilot study, we enrolled 40 patients with hand trauma or thumb-base arthritis, across 2 sites, between July 13, 2022, and September 14, 2022. We monitored their symptoms with the patient evaluation measure, via EMCAT, over a 12-week period. Patients were assessed thrice weekly, once daily, or thrice daily. We additionally administered full-length PROM assessments at 0, 6, and 12 weeks, and the User Engagement Scale at 12 weeks.

RESULTS

The use of EMCAT significantly reduced the length of the PROM (median 2 vs 11 items) and the time taken to complete it (median 8.8 seconds vs 1 minute 14 seconds). Very similar scores were obtained when EMCAT was administered concurrently with the full-length PROM, with a mean error of <0.01 on a logit (z score) scale. The median response rate in the daily assessment group was 93%. The median perceived usability score of the User Engagement Scale was 4.0 (maximum possible score 5.0).

CONCLUSIONS

EMCAT reduces the burden of PROM assessments, enabling acceptable high-frequency, remote PROM data capture. This has potential applications in both research and clinical practice. In research, EMCAT could be used to study temporal variations in symptom severity, for example, recovery trajectories after surgery. In clinical practice, EMCAT could be used to monitor patients remotely, prompting early intervention if a patient's symptom trajectory causes clinical concern.

TRIAL REGISTRATION

ISRCTN 19841416; https://www.isrctn.com/ISRCTN19841416.

Key considerations to reduce or address respondent burden in patient-reported outcome (PRO) data collection

Patient-reported outcomes (PROs) are used in clinical trials to provide evidence of the benefits and risks of interventions from a patient perspective and to inform regulatory decisions and health policy. The collection of PROs in routine practice can facilitate monitoring of patient symptoms; identification of unmet needs; prioritisation and/or tailoring of treatment to the needs of individual patients and inform value-based healthcare initiatives. However, respondent burden needs to be carefully considered and addressed to avoid high rates of missing data and poor reporting of PRO results, which may lead to poor quality data for regulatory decision making and/or clinical care.

The collection of patient-reported outcomes (PROs) may capture patients’ assessments of their health status. Here authors highlight PRO-specific issues that should be considered to minimise respondent burden in clinical trials and routine care.

The Role of Patient-Reported Outcomes in Sleep Measurements

Several questionnaires aka patient-reported outcome measures (PROMs) have been developed for specific use in sleep medicine. Some PROMS are "disease-specific," that is, related to a specific sleep disorder, whereas others are generic. These PROMS constitute a valuable add-on to the conventional history taking. They can be used in the areas of research, clinical practice, and quality of health care appraisal. Still, these instruments have inherent limitations, requiring proficient application in the various areas of interest. Disease-specificity includes a risk for nosologic bias that may confound diagnostic and therapeutic results. Future research should provide solutions for shortcomings of presently available questionnaires.

Self-care research: How to grow the evidence base? (reprint)

BACKGROUND AND OBJECTIVE

The number of studies in the area of self-care is growing and international researchers are increasingly developing self-care interventions to improve outcomes of individual patients and communities. However, growth of the evidence is still slow due to challenges with designing and testing self-care interventions. In this article we address major methodological challenges with regard to the definition of self-care, use of theory, and research design, intended to provide guidance to researchers in this field.

METHOD

During the inaugural conference of the International Center for Self-Care Research held in Rome, Italy in June 2019 we identified important issues in existing self-care research. Discussion and literature review lead to eight recommendation for future self-care research.

RESULTS

In preparation, begin with a theoretically sound definition of self-care. In planning the intervention, build on and extend previous studies. Use theory to develop self-care interventions and consider translational models to guide development, evaluation and implementation of complex self-care interventions. Employ a study design that fits the current phase and objectives of the research and measure self-care and related factors carefully. In reporting, describe the sample and setting sufficiently so that others can draw conclusions about generalizability and applicability to their practice and patient population. In interpretation, describe how the intervention is assumed to work (causal assumptions) and its key components.

CONCLUSION

Our review of existing self-care research clearly illustrates that the recommendations we provide are needed if we are to substantially grow the evidence base supporting self-care. Embracing a core set of principles will allow us to build on each other's work. Tweetable abstract: A core set of methodological principles is needed to substantially grow the evidence base supporting self-care.

An analysis of missing items in real-world electronic patient reported outcomes data: implications for clinical care

PURPOSE

Utilization of electronic patient-reported outcomes (ePROs) in the clinic can improve quality of life and prolong survival in cancer care. However, there remain unanswered questions regarding trends in missing data and the potential effect on real-time patient care.

METHODS

This study utilized a prospectively collected dataset of ePROs from oncology clinics that administered the Patient Care Monitor 2.0 (PCM), a validated symptoms survey assessing 78 items for men, and 86 for women. We tabulated the frequency of missing items, by item and domain (emotional, functional and physical symptom-related), and examined these by age, gender, education, race and marital status.

RESULTS

Within 20,986 encounters, there were responses to at least 1 PCM item from 6933 unique patients. The highest frequency of missing answers occurred for: "attend a paid job" (10.7%), "reduced sexual enjoyment" (3.8%), and "run" (3.7%). By domain, 12.3% of functional, 8.4% of physical symptom-related, and 1.6% of emotional constructs contained at least one missing item. For functional and physical symptom-related items, missingness was most common in patients >60 years old.

CONCLUSION

The frequency of missingness was highest for functional items, like attending a paid job, suggesting that some respondents (e.g., retirees without a paid job) skipped questions that were less applicable to them. More universal issues for cancer patients, such as emotional well-being, had much lower frequencies of missingness. This suggests differential item completion that warrants further study to understand the inherent drivers. Identifying causes of missingness could improve the clinical utility of ePROs and highlight opportunities to personalize care.

Applying Item Response Theory to the OPD Structure Questionnaire: Identification of a Unidimensional Core Construct and Feasibility of Computer Adaptive Testing

Recent developments in the dimensional assessment of personality functioning have made the implementation of latent measurement models increasingly attractive. In this study, we applied item response theory (IRT) to a well-established personality functioning instrument (the OPD Structure Questionnaire) to identify a unidimensional latent trait and to evaluate the feasibility of computer adaptive testing (CAT). We hypothesized that the use of IRT could reduce the test burden - compared to a fixed short form - while maintaining high precision over a wide range of the latent trait. The OPD-SQ was collected from 1235 patients in a psychosomatic clinic. IRT assumptions were fulfilled. A 9-factor model yielded sufficient fit and unidimensionality in exploratory factor analysis with bifactor rotation. Items were iteratively reduced, and a graded-response IRT model was fitted to the data. Simulations showed that a CAT with approximately 7 items was able to capture an OPD-SQ global severity score with an accuracy similar to that of a fixed 12-item short form. The final item bank and CAT yielded satisfactory content validity. Strong correlations with depression and anxiety replicated previous results on the OPD-SQ. We concluded that IRT applications could be useful to reduce the test burden of personality functioning instruments.

Accuracy and Validity of Computer Adaptive Testing for Outcome Assessment in Patients Undergoing Total Knee Arthroplasty

BACKGROUND

Computerized adaptive test (CAT) questionnaires may allow standardization of patient-reported outcome measures and reductions in questionnaire burden. We evaluated the validity, accuracy, and efficacy of a CAT system in patients with end-stage osteoarthritis undergoing total knee arthroplasty.

METHODS

CAT Knee Osteoarthritis Outcome Scores (KOOS) and CAT KOOS-JR questionnaires were applied to 1871 standard form KOOS and 1493 KOOS-JR patient responses, respectively. Mean, standard deviations, Pearson's correlation coefficients, interclass correlation coefficients (ICCs), frequency distribution plots, and Bland-Altman plots were used to compare the precision, validity, and accuracy between CAT scores and full-form scores.

RESULTS

There was a mean reduction of 14 questions (33%) in the CAT KOOS and 1.4 questions (20%) with the CAT KOOS-JR version, compared with the standard KOOS and KOOS-JR surveys, respectively. There were no significant differences between KOOS and CAT KOOS scores with respect to pain (P = .66), symptoms (P = .43), quality of life (P = .99), activities of daily living (P = .68), and sports (P = .84). Similarly, there were no significant differences between the standard form KOOS-JR and CAT KOOS-JR scores (P = .94). There were strong correlations with minimal variability between the CAT KOOS and standard KOOS questionnaires for pain (r = 0.98, ICC: 0.98), symptoms (r = 0.97, ICC: 0.97), quality of life scores (r = 0.99, ICC: 0.99), activities of daily living scores (r = 0.99, ICC: 0.99), and sports scores (r = 0.99, ICC: 0.99). Similarly, there were strong correlations between the KOOS-JR and the CAT KOOS-JR scores (r = 0.99, ICC: 0.99).

CONCLUSION

CAT KOOS and the CAT KOOS-JR versions are accurate and reduce questionnaire burden up to one-third compared with standard surveys. CAT versions may improve patient compliance and decrease fatigue.

The Concerns About Pain (CAP) Scale: A Patient-Reported Outcome Measure of Pain Catastrophizing

Pain catastrophizing has been recognized as an important and consistent psychosocial predictor of nearly every key pain-related outcome. The purpose of this study was to develop a new measure of pain catastrophizing using modern psychometric methodology. People with chronic pain (N = 795) responded to thirty items. Data were analyzed using item response theory, including assessment of differential item functioning and reliability. Sensitivity to change and validity were examined using data collected from patients undergoing spinal fusion surgery (n = 184) and participating in an ongoing longitudinal aging with a disability survey study (n = 1,388). The final 24-item bank had no items with significant local dependence, misfit, or differential item functioning. Results provided strong evidence of reliability and validity. Six- and 2-item short forms were developed for use when computer adaptive testing is not feasible or desirable. The item bank was named the University of Washington Concerns About Pain scale because the term "catastrophizing" was considered stigmatizing by people with chronic pain. Guidance for score interpretation was developed with extensive feedback from individuals with chronic pain. The Concerns About Pain item bank, short forms, and user manuals are free and publicly available to all users and can be accessed online at https://uwcorr.washington.edu/measures/. PERSPECTIVE: This article presents the development of the University of Washington Concerns About Pain scale, the first item response theory-based item bank of pain catastrophizing. The measure is intended for clinicians interested in improving outcomes of patients with chronic pain and for researchers who study impact of and treatment interventions aimed at reducing pain catastrophizing.

A visual analog scale for patient-reported voice outcomes: The VAS voice

OBJECTIVES

Although patient-reported outcome measures (PROMs) can be useful for assessing quality of life, they can be complex and cognitively burdensome. In this study, we prospectively evaluated a simple patient-reported voice assessment measure on a visual analog scale (VAS voice) and compared it with the Voice Handicap Index (VHI-10).

STUDY DESIGN

Prospective survey.

METHODS

An abbreviated voice measure was designed by a team of otolaryngologists, speech pathologists, and patients that consisted of four VAS questions related to (a) a global question of voice disturbance, (b) physical function of voice, (c) functional issues, and (d) emotional handicap. All English-speaking patients presenting to an academic laryngology clinic for a voice complaint were included. Internal consistency and validity were assessed with comparison to the VHI-10.

RESULTS

A total of 209 patients were enrolled. Ninety-two percent of patients reported understanding the survey. The four-item VAS survey was highly correlated with VHI-10 score (Pearson correlation .81, P < .0001), and the Cronbach's alpha between all four VAS questions was .94. Age, gender, and diagnosis were not associated with either the global VAS or VHI-10 tool.

CONCLUSION

Reducing the complexity of instruments assessing voice-related quality of life is feasible, and the VAS voice correlated with existing measures. Simplified assessments may offer advantages compared to more cumbersome PROMs.

LEVEL OF EVIDENCE

2c.