Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report

Use of SMS and tablet computer improves the electronic collection of elective orthopaedic patient reported outcome measures

INTRODUCTION

Electronic patient reported outcome measures (PROMs) enable real time reporting back to the patient and medical team, comparison between similar patient cohorts and long-term cost effective outcome measurement. The primary objective of this three-phase pilot study was to measure uptake using a web-based PROM system following the introduction of two separate process improvements.

METHODS

Eighty consecutive new elective orthopaedic patients in a single surgeon's practice were recruited for the study. Patients in Group 1 (n=26) received only a letter reminding them to complete a symptom score. Those in Group 2 (n=31) also received a reminder SMS (short message service) message via their mobile or home telephone and those in Group 3 (n=23) also had access to a tablet computer in clinic.

RESULTS

The mean patient age in Group 1 was 55 years (range: 24-80 years), in Group 2 it was 60 years (range: 23-85 years) and in Group 3 it was 58 years (range: 37-78 years) (p>0.05). Overall, 79% of patients had internet access, and 35% of Group 1, 55% of Group 2 and 74% of Group 3 recorded an electronic PROM score (p=0.02). In Group 3, 94% of patients listed for an operation completed an electronic PROM score (p=0.006).

CONCLUSIONS

Collecting PROM data effectively in everyday clinical practice is challenging. Electronic collection should meet that challenge and improve healthcare delivery but it is in its infancy. This pilot study shows that the combination of an SMS reminder and access to a Wi-Fi enabled tablet computer in the clinic setting enabled 94% of patients listed for an operation to complete a score on a web-based clinical outcomes system. Additional staff training and telephone call reminders may further improve uptake.

Brief communication: patient satisfaction with the use of tablet computers: a pilot study in two outpatient home dialysis clinics

Background

Electronic capture of patients’ reports of their health is significant in clinical nephrology research because health-related quality of life (HRQOL) for patients with end-stage renal disease is compromised and assessment by patients of their HRQOL in practice is relatively uncommon.

Objective

The purpose of this study was to evaluate patient satisfaction with and time involved in administering HRQOL and symptom assessment measures using tablet computers in two outpatient home dialysis clinics.

Design

A cross-sectional observational study design was employed.

Setting

The study was conducted in two home dialysis clinics.

Patients

Fifty-six patients participated in the study; 35 males (63%) and 21 females (37%) with a mean age of 66 ± 12 (36-90 years old) were included. Forty-nine participants were on peritoneal dialysis (87%), 6 on home hemodialysis (11%), and 1 on nocturnal home hemodialysis (2%).

Measurements

Measures included the Kidney Disease Quality of Life-36 (KDQOL-36), the Edmonton Symptom Assessment Scale (ESAS) and Participant’s Level of Satisfaction in Using a Tablet Computer.

Methods

Using a tablet computer, participants completed the three measures. Descriptive statistics and bivariate correlations were calculated.

Results

Participants’ satisfaction with use of the tablet computer was high; 66% were “very satisfied”, 7% “satisfied”, 2% “slightly satisfied”, and 18% “neutral”. On the 7-point Likert-type scale, the mean satisfaction score was 5.11 (SD = 1.6). Mean time to complete the measures was: Level of Satisfaction 1.15 minutes (SD = 0.41), ESAS 2.55 minutes (SD = 1.04), and KDQOL 9.56 minutes (SD = 2.03); the mean time to complete all three instruments was 13.19 minutes (SD = 2.42). There were no significant correlations between level of satisfaction and age, gender, HRQOL, time taken to complete surveys, computer experience, or comfort with technology. Comfort with technology and computer experience were highly correlated, r = .7, p (one-tailed) < 0.01.

Limitations

Limitations include lack of generalizability because of a small self-selected sample of relatively healthy patients and a lack of psychometric testing on the measure of satisfaction.

Conclusions

Participants were satisfied with the platform and the time involved for completion of instruments was modest. Routine use of HRQOL measures for clinical purposes may be facilitated through use of tablet computers.

PRO data collection in clinical trials using mixed modes: report of the ISPOR PRO mixed modes good research practices task force

The objective of this report was to address the use and mixing of data collection modes within and between trials in which patient-reported outcome (PRO) end points are intended to be used to support medical product labeling. The report first addresses the factors that should be considered when selecting a mode or modes of PRO data collection in a clinical trial, which is often when mixing is first considered. Next, a summary of how to "faithfully" migrate instruments is presented followed by a section on qualitative and quantitative study designs used to evaluate measurement equivalence of the new and original modes of data collection. Finally, the report discusses a number of issues that must be taken into account when mixing modes is deemed necessary or unavoidable within or between trials, including considerations of the risk of mixing at different levels within a clinical trial program and mixing between different types of platforms. In the absence of documented evidence of measurement equivalence, it is strongly recommended that a quantitative equivalence study be conducted before mixing modes in a trial to ensure that sufficient equivalence can be demonstrated to have confidence in pooling PRO data collected by the different modes. However, we also strongly discourage the mixing of paper and electronic field-based instruments and suggest that mixing of electronic modes be considered for clinical trials and only after equivalence has been established. If proceeding with mixing modes, it is important to implement data collection carefully in the trial itself in a planned manner at the country level or higher and minimize ad hoc mixing by sites or individual subjects. Finally, when mixing occurs, it must be addressed in the statistical analysis plan for the trial and the ability to pool the data must be evaluated to then evaluate treatment effects with mixed modes data. A successful mixed modes trial requires a "faithful migration," measurement equivalence established between modes, and carefully planned implementation to minimize the risk of increased measurement error impacting the power of the trial to detect a treatment effect.

Measuring respiratory symptoms in clinical trials of COPD: reliability and validity of a daily diary

BACKGROUND

Although respiratory symptoms are characteristic features of COPD, there is no standardised method for quantifying their severity in stable disease.

OBJECTIVE

To evaluate the EXACT-Respiratory Symptom (E-RS) measure, a daily diary comprising 11 of the 14 items in the Exacerbations of Chronic Pulmonary Disease Tool (EXACT).

METHODS

Qualitative: patient focus group and interviews to address content validity. Quantitative: secondary data analyses to test reliability and validity.

RESULTS

Qualitative: n=84; mean (SD) age 65 (10) years, FEV1 1.2(0.4) L; 44% male. Subject descriptions of their respiratory symptoms were consistent with E-RS content and structure. Quantitative: n=188; mean (SD) age 66 (10) years, FEV1 1.2(0.5) L; 50% male. Factor analysis (FA) showed 3 subscales: RS-Breathlessness, RS-Cough & Sputum, and RS-Chest Symptoms; second-order FA supported a general factor and total score. Reliability (total and subscales): 0.88, 0.86, 0.73, 0.81; 2-day test-retest ICC: 0.90, 0.86, 0.87, 0.82, respectively.

VALIDITY

Total scores correlated significantly (p < 0.0001) with SGRQ Total (r=0.75), Symptoms (r=0.66), Activity (r=0.57), Impact (r=0.70) scores; subscale correlations were also significant (r=0.26, p < 0.05 (RS-Chest Symptoms with Activity) to r=0.69, p < 0.0001 (RS-Cough & Sputum with Symptoms). RS-Breathlessness correlated with rescue medication use (r=0.32, p < 0.0001), clinician-reported mMRC (r=0.33, p < 0.0001), and FEV1% predicted (r=-0.17, p < 0.05). E-RS scores differentiated groups based on chronic bronchitis diagnosis (p < 0.01-0.001), smoking status (p < 0.05-0.001), and rescue medication use (p < 0.05-0.0001).

CONCLUSIONS

Results suggest the RS-Total is a reliable and valid instrument for evaluating respiratory symptom severity in stable COPD. Further study of sensitivity to change is warranted.

Electronic versus paper-based assessment of health-related quality of life specific to HIV disease: reliability study of the PROQOL-HIV questionnaire

BACKGROUND

Electronic patient-reported outcomes (PRO) provide quick and usually reliable assessments of patients' health-related quality of life (HRQL).

OBJECTIVE

An electronic version of the Patient-Reported Outcomes Quality of Life-human immunodeficiency virus (PROQOL-HIV) questionnaire was developed, and its face validity and reliability were assessed using standard psychometric methods.

METHODS

A sample of 80 French outpatients (66% male, 52/79; mean age 46.7 years, SD 10.9) were recruited. Paper-based and electronic questionnaires were completed in a randomized crossover design (2-7 day interval). Biomedical data were collected. Questionnaire version and order effects were tested on full-scale scores in a 2-way ANOVA with patients as random effects. Test-retest reliability was evaluated using Pearson and intraclass correlation coefficients (ICC, with 95% confidence interval) for each dimension. Usability testing was carried out from patients' survey reports, specifically, general satisfaction, ease of completion, quality and clarity of user interface, and motivation to participate in follow-up PROQOL-HIV electronic assessments.

RESULTS

Questionnaire version and administration order effects (N=59 complete cases) were not significant at the 5% level, and no interaction was found between these 2 factors (P=.94). Reliability indexes were acceptable, with Pearson correlations greater than .7 and ICCs ranging from .708 to .939; scores were not statistically different between the two versions. A total of 63 (79%) complete patients' survey reports were available, and 55% of patients (30/55) reported being satisfied and interested in electronic assessment of their HRQL in clinical follow-up. Individual ratings of PROQOL-HIV user interface (85%-100% of positive responses) confirmed user interface clarity and usability.

CONCLUSIONS

The electronic PROQOL-HIV introduces minor modifications to the original paper-based version, following International Society for Pharmacoeconomics and Outcomes Research (ISPOR) ePRO Task Force guidelines, and shows good reliability and face validity. Patients can complete the computerized PROQOL-HIV questionnaire and the scores from the paper or electronic versions share comparable accuracy and interpretation.

Quantitative challenges facing patient-centered outcomes research

Patient-centered outcomes research collects and analyzes data from patients and other stakeholders to improve health care delivery and outcomes and guide health care decisions. However, there are a number of challenges in conducting quantitative analyses of patient-centered data. This article provides an overview of the analytical challenges and describes approaches to consider to overcome the challenges, as well as directions for future development.

Method of administration of PROMIS scales did not significantly impact score level, reliability, or validity

OBJECTIVES

To test the impact of the method of administration (MOA) on score level, reliability, and validity of scales developed in the Patient Reported Outcomes Measurement Information System (PROMIS).

STUDY DESIGN AND SETTING

Two nonoverlapping parallel forms each containing eight items from each of three PROMIS item banks (Physical Function, Fatigue, and Depression) were completed by 923 adults with chronic obstructive pulmonary disease, depression, or rheumatoid arthritis. In a randomized crossover design, subjects answered one form by interactive voice response (IVR) technology, paper questionnaire (PQ), personal digital assistant (PDA), or personal computer (PC) and a second form by PC, in the same administration. Method equivalence was evaluated through analyses of difference scores, intraclass correlations (ICCs), and convergent/discriminant validity.

RESULTS

In difference score analyses, no significant mode differences were found and all confidence intervals were within the prespecified minimal important difference of 0.2 standard deviation. Parallel-forms reliabilities were very high (ICC = 0.85-0.93). Only one across-mode ICC was significantly lower than the same-mode ICC. Tests of validity showed no differential effect by MOA. Participants preferred screen interface over PQ and IVR.

CONCLUSION

We found no statistically or clinically significant differences in score levels or psychometric properties of IVR, PQ, or PDA administration compared with PC.

Measurement equivalence and feasibility of the EORTC QLQ-PR25: paper-and-pencil versus touch-screen administration

OBJECTIVE

We assessed the measurement equivalence and feasibility of the paper-and-pencil and touch-screen modes of administration of the Taiwan Chinese version of the EORTC QLQ-PR25, a commonly used questionnaire to evaluate the health-related quality of life (HRQOL) in patients with prostate cancer in Taiwan.

METHODS

A cross-over design study was conducted in 99 prostate cancer patients at an urology outpatient clinic. Descriptive exact and global agreement percentages, intraclass correlation, and equivalence test based on minimal clinically important difference (MCID) approach were used to examine the equity of HRQOL scores between these two modes of administration. We also evaluated the feasibility of computerized assessment based on patients' acceptability and preference. Additionally, we used Rasch rating scale model to assess differential item functioning (DIF) between the two modes of administration.

RESULTS

The percentages of global agreement in all domains were greater than 85% in the EORTC QLQ-PR25. All results from equivalence tests were significant, except for Sexual functioning, indicating good equivalence. Only one item exhibited DIF between the two modes. Although nearly 80% of the study patients had no prior computer-use experience, the overall proportion of acceptance and preference for the touch-screen mode were quite high and there was no significant difference across age groups or between computer-use experience groups.

CONCLUSIONS

The study results showed that the data obtained from the modes of administration were equivalent. The touch-screen mode of administration can be a feasible and suitable alternative to the paper-and-pencil mode for assessment of patient-reported outcomes in patients with prostate cancer.

Use of Patient-Reported Outcome (PRO) Measures at Group and Patient Levels: Experiences From the Generic Integrated PRO System, WestChronic

Background

Patient-reported outcome (PRO) measures may be used at a group level for research and quality improvement and at the individual patient level to support clinical decision making and ensure efficient use of resources. The challenges involved in implementing PRO measures are mostly the same regardless of aims and diagnostic groups and include logistic feasibility, high response rates, robustness, and ability to adapt to the needs of patient groups and settings. If generic PRO systems can adapt to specific needs, advanced technology can be shared between medical specialties and for different aims.

Objective

We describe methodological, organizational, and practical experiences with a generic PRO system, WestChronic, which is in use among a range of diagnostic groups and for a range of purposes.

Methods

The WestChronic system supports PRO data collection, with integration of Web and paper PRO questionnaires (mixed-mode) and automated procedures that enable adherence to implementation-specific schedules for the collection of PRO. For analysis, we divided functionalities into four elements: basic PRO data collection and logistics, PRO-based clinical decision support, PRO-based automated decision algorithms, and other forms of communication. While the first element is ubiquitous, the others are optional and only applicable at a patient level. Methodological and organizational experiences were described according to each element.

Results

WestChronic has, to date, been implemented in 22 PRO projects within 18 diagnostic groups, including cardiology, neurology, rheumatology, nephrology, orthopedic surgery, gynecology, oncology, and psychiatry. The aims of the individual projects included epidemiological research, quality improvement, hospital evaluation, clinical decision support, efficient use of outpatient clinic resources, and screening for side effects and comorbidity. In total 30,174 patients have been included, and 59,232 PRO assessments have been collected using 92 different PRO questionnaires. Response rates of up to 93% were achieved for first-round questionnaires and up to 99% during follow-up. For 6 diagnostic groups, PRO data were displayed graphically to the clinician to facilitate flagging of important symptoms and decision support, and in 5 diagnostic groups PRO data were used for automatic algorithm-based decisions.

Conclusions

WestChronic has allowed the implementation of all proposed protocol for data collection and processing. The system has achieved high response rates, and longitudinal attrition is limited. The relevance of the questions, the mixed-mode principle, and automated procedures has contributed to the high response rates. Furthermore, development and implementation of a number of approaches and methods for clinical use of PRO has been possible without challenging the generic property. Generic multipurpose PRO systems may enable sharing of automated and efficient logistics, optimal response rates, and other advanced options for PRO data collection and processing, while still allowing adaptation to specific aims and patient groups.

An investigation of symptom burden and quality of life in Chinese chemo-naïve advanced lung cancer patients by using the Instrument-Cloud QOL System

OBJECTIVES

This study was designed to assess the disease-related symptom burden and quality of life (QOL) in Chinese chemo-naïve advanced lung cancer patients.

MATERIALS AND METHODS

Chemo-naïve patients with stage III/IV lung cancer were enrolled. 43 centers from 16 provinces of China participated in the study. Functional Assessment of Cancer Therapy-Lung (FACT-L) scale and Cloud QOL System were applied in the study.

RESULTS

376 eligible patients were analyzed. The three most common and severe symptoms were appetite loss (84.3%, scored 2.46), breathing difficulty (79.0%, scored 2.56), and cough (75.5%, scored 2.81). Significant correlation was found between QOL and symptoms. Regression analysis of QOL indicated that almost every symptom item (except shortness of breath) was the negative indicator of QOL. Moreover, pulmonary diseases, pleural metastases and brain metastases had significant negative impact on both symptoms and QOL. Relatively poor performance status affected the QOL only, while cardiovascular diseases merely affected the symptoms. And patients with EGFR mutations had less symptom burden than those with wild-type EGFR.

CONCLUSION

QOL evaluation by using the Cloud QOL System was feasible. Appetite loss, breathing difficulty and cough were the three most common and severe symptoms seen in Chinese chemo-naïve advanced lung cancer patients. Almost all symptoms had negative impact on QOL. And some clinical characteristics could predict the symptoms and QOL.

Cognitive interviewing of the US National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE)

PURPOSE

The National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) is a library of question items that enables patient reporting of adverse events (AEs) in clinical trials. This study contributes content validity evidence of the PRO-CTCAE by incorporating cancer patient input of the relevance and comprehensiveness of the item library.

METHODS

Cognitive interviews were conducted among patients undergoing chemotherapy or radiation therapy at multiple sites to evaluate comprehension, memory retrieval, judgment, and response mapping related to AE terms (e.g., nausea), attribute terms (regarding frequency, severity, or interference), response options, and recall period. Three interview rounds were conducted with ≥20 patients completing each item per round. Items were modified and retested if ≥3 patients exhibited cognitive difficulties or if experienced by ≤25% patients.

RESULTS

One hundred and twenty-seven patients participated (35% ≤high school, 28% non-white, and 59% female). Most AE terms (63/80) generated no cognitive difficulties. The remaining 17 were modified without further difficulties by Round 3. Terms were comprehended regardless of education level. Attribute terms and response options required no modifications. Patient adherence to recall period (7 days) was improved when the reference period was incorporated.

CONCLUSIONS

This study provides evidence confirming comprehension of the US English language versions of items in the PRO-CTCAE library for measuring symptomatic AEs from the patient perspective within the context of cancer treatment. Several minor changes were made to the items to improve item clarity, comprehension, and ease of response judgment. This study helps to establish the content validity of PRO-CTCAE items for patient reporting of AEs during cancer treatment.

mHealth Series: mHealth project in Zhao County, rural China - Description of objectives, field site and methods

Background

We set up a collaboration between researchers in China and the UK that aimed to explore the use of mHealth in China. This is the first paper in a series of papers on a large mHealth project part of this collaboration. This paper included the aims and objectives of the mHealth project, our field site, and the detailed methods of two studies.

Field site

The field site for this mHealth project was Zhao County, which lies 280 km south of Beijing in Hebei Province, China.

Methods

We described the methodology of two studies: (i) a mixed methods study exploring factors influencing sample size calculations for mHealth–based health surveys and (ii) a cross–over study determining validity of an mHealth text messaging data collection tool. The first study used mixed methods, both quantitative and qualitative, including: (i) two surveys with caregivers of young children, (ii) interviews with caregivers, village doctors and participants of the cross–over study, and (iii) researchers’ views. We combined data from caregivers, village doctors and researchers to provide an in–depth understanding of factors influencing sample size calculations for mHealth–based health surveys. The second study, a cross–over study, used a randomised cross–over study design to compare the traditional face–to–face survey method to the new text messaging survey method. We assessed data equivalence (intrarater agreement), the amount of information in responses, reasons for giving different responses, the response rate, characteristics of non–responders, and the error rate.

Conclusions

This paper described the objectives, field site and methods of a large mHealth project part of a collaboration between researchers in China and the UK. The mixed methods study evaluating factors that influence sample size calculations could help future studies with estimating reliable sample sizes. The cross–over study comparing face–to–face and text message survey data collection could help future studies with developing their mHealth tools.

Demonstrating measurement equivalence of the electronic and paper formats of the Urticaria Patient Daily Diary in patients with chronic idiopathic urticaria

BACKGROUND

The Urticaria Patient Daily Diary (UPDD), originally developed on paper, is a measure of key symptoms of chronic idiopathic urticaria (CIU). The development of the electronic version (eUPDD) involved moderate modifications to the appearance of the paper version.

OBJECTIVE

This study assessed the measurement equivalence of the electronic and paper versions of the UPDD in a sample of patients with CIU.

METHODS

This was a cross-over study of patients with moderate-severe CIU refractory to H1 antihistamines. Patients were randomized to either the eUPDD followed by the paper UPDD or vice versa. The UPDD includes morning and evening questions; both sets were administered together in this study. An hour-long filler task was given between paper and electronic administrations. Patients with stable symptoms between the two assessments were included in the analyses. Cohen's kappa coefficients and intraclass correlation coefficients (ICC) were computed as applicable to assess equivalence.

RESULTS

A total of 91 patients participated (mean age 43 years, 79.1 % female). Symptoms were stable between assessments for 67-74 (74-81 %) patients (varied by symptom). Kappa coefficients ranged from 0.82 to 1.00 for the individual UPDD items. For the Urticaria Activity Score (the sum of the 'itch severity' and 'number of hives' item scores) the ICC was 0.90 for the morning (Wilcoxon p = 0.331) and 0.95 for the evening (Wilcoxon p = 0.836).

CONCLUSIONS

All test-retest statistics in this study were well above the accepted threshold, indicating excellent agreement between the two administration methods. Findings support the measurement equivalence of the electronic and paper versions of the UPDD to measure CIU symptoms.

Text messaging data collection for monitoring an infant feeding intervention program in rural China: feasibility study

Background

An effective data collection method is crucial for high quality monitoring of health interventions. The traditional face-to-face data collection method is labor intensive, expensive, and time consuming. With the rapid increase of mobile phone subscribers, text messaging has the potential to be used for evaluation of population health interventions in rural China.

Objective

The objective of this study was to explore the feasibility of using text messaging as a data collection tool to monitor an infant feeding intervention program.

Methods

Participants were caregivers of children aged 0 to 23 months in rural China who participated in an infant feeding health education program. We used the test-retest method. First, we collected data with a text messaging survey and then with a face-to-face survey for 2 periods of 3 days. We compared the response rate, data agreement, costs, and participants’ acceptability of the two methods. Also, we interviewed participants to explore their reasons for not responding to the text messages and the reasons for disagreement in the two methods. In addition, we evaluated the most appropriate time during the day for sending text messages.

Results

We included 258 participants; 99 (38.4%) participated in the text messaging survey and 177 (68.6%) in the face-to-face survey. Compared with the face-to-face survey, the text messaging survey had much lower response rates to at least one question (38.4% vs 68.6%) and to all 7 questions (27.9% vs 67.4%) with moderate data agreement (most kappa values between .5 and .75, the intraclass correlation coefficients between .53 to .72). Participants who took part in both surveys gave the same acceptability rating for both methods (median 4.0 for both on a 5-point scale, 1=disliked very much and 5=liked very much). The costs per questionnaire for the text messaging method were much lower than the costs for the face-to-face method: ¥19.7 (US $3.13) versus ¥33.9 (US $5.39) for all questionnaires, and ¥27.1 (US $4.31) versus ¥34.4 (US $5.47) for completed questionnaires. The main reasons for not replying were that participants did not receive text messages, they were too busy to reply, or they did not see text messages in time. The main reasons for disagreement in responses were that participants forgot their answers in the text messaging survey and that they changed their minds. We found that participants were more likely to reply to text messages immediately during 2 time periods: 8 AM to 3 PM and 8 PM to 9 PM.

Conclusions

The text messaging method had reasonable data agreement and low cost, but a low response rate. Further research is needed to evaluate effectiveness of measures that can increase the response rate, especially in collecting longitudinal data by text messaging.

Review of electronic patient-reported outcomes systems used in cancer clinical care

PURPOSE

The use of electronic patient-reported outcomes (PRO) systems is increasing in cancer clinical care settings. This review comprehensively identifies existing PRO systems and explores how systems differ in the administration of PRO assessments, the integration of information into the clinic workflow and electronic health record (EHR) systems, and the reporting of PRO information.

METHODS

Electronic PRO (e-PRO) systems were identified through a semistructured review of published studies, gray literature, and expert identification. System developers were contacted to provide detailed e-PRO system characteristics and clinical implementation information using a structured review form.

RESULTS

A total of 33 unique systems implemented in cancer clinical practice were identified. Of these, 81% provided detailed information about system characteristics. Two system classifications were established: treatment-centered systems designed for patient monitoring during active cancer treatment (n = 8) and patient-centered systems following patients across treatment and survivorship periods (n = 19). There was little consensus on administration, integration, or result reporting between these system types. Patient-centered systems were more likely to provide user-friendly features such as at-home assessments, integration into larger electronic system networks (eg, EHRs), and more robust score reporting options. Well-established systems were more likely to have features that increased assessment flexibility (eg, location, automated reminders) and better clinical integration.

CONCLUSION

The number of e-PRO systems has increased. Systems can be programmed to have numerous features that facilitate integration of PRO assessment and routine monitoring into clinical care. Important barriers to system usability and widespread adoption include assessment flexibility, clinical integration, and high-quality data collection and reporting.

Measurement equivalence of touch-screen computerized and paper-based diabetes-specific quality-of-life questionnaires

Current advances in technology have enabled the development of a computer-based questionnaire that provides advantages over the paper-based mode of administration, such as automatic data entry, storage and calculations. However, before implementing a computer-based questionnaire, its equivalence with the original paper-based questionnaire must first be demonstrated. The purpose of this study was to evaluate the measurement equivalence of the computerized Diabetes-Specific Quality-of-Life questionnaire (cD-QOL) with its original paper-based counterpart. A two-period crossover design was used in this study. The measurement equivalence was evaluated using quadratic weighted kappa coefficients, intraclass correlations and Cronbach's alpha comparisons. The cD-QOL was equivalent to its original paper-based counterpart. Participants preferred the cD-QOL over the paper-based questionnaire and reported that it was easy to use.

Equivalence and measurement properties of an electronic version of the Psoriasis Symptom Inventory

PURPOSE

To evaluate the equivalence of electronic and paper versions of the Psoriasis Symptom Inventory and to examine measurement properties of the electronic version.

METHODS

In a prospective, randomized, crossover, non-interventional study in adult subjects (age ≥18 years) with plaque psoriasis conducted over a period of 15 days, subjects were randomized to two groups, completing either the paper or electronic Psoriasis Symptom Inventory daily for 7 consecutive days followed by the alternate version. Equivalence was assessed by the intraclass correlation coefficient (ICC) between both administration modes. Differences in scores were also tested using paired Student's t test. Measurement properties included internal consistency reliability, test-retest reliability, and convergent and discriminant validity between the Psoriasis Symptom Inventory and (1) disease-specific (Dermatology Life Quality Index) and (2) general health (SF-36v2) status.

RESULTS

Eighty subjects [74 % (59/80) moderate-to-severe psoriasis; 26 % (21/80) mild psoriasis receiving systemic treatment] were enrolled from 8 sites in the USA. The two modes were highly concordant for both total (ICC = 0.97) and individual item scores (ICC range = 0.93-0.97). Response bias testing showed no differences based on completion order with all ICC values >0.91. All mean score differences, except for one item ("flaking"), were non-significant (P > 0.05). Minimum values for reliability (>0.70) and validity (convergent, r ≥ 0.40) were exceeded for the electronic Psoriasis Symptom Inventory.

CONCLUSIONS

Equivalence between paper and electronic versions of the Psoriasis Symptom Inventory and strong measurement properties of the electronic mode indicated a successful migration from paper to electronic format of the Psoriasis Symptom Inventory.

Interactive voice response technology for symptom monitoring and as an adjunct to the treatment of chronic pain

Chronic pain is a medical condition that severely decreases the quality of life for those who struggle to cope with it. Interactive voice response (IVR) technology has the ability to track symptoms and disease progression, to investigate the relationships between symptom patterns and clinical outcomes, to assess the efficacy of ongoing treatments, and to directly serve as an adjunct to therapeutic treatment for chronic pain. While many approaches exist toward the management of chronic pain, all have their pitfalls and none work universally. Cognitive behavioral therapy (CBT) is one approach that has been shown to be fairly effective, and therapeutic interactive voice response technology provides a convenient and easy-to-use means of extending the therapeutic gains of CBT long after patients have discontinued clinical visitations. This review summarizes the advantages and disadvantages of IVR technology, provides evidence for the efficacy of the method in monitoring and managing chronic pain, and addresses potential future directions that the technology may take as a therapeutic intervention in its own right.

Reasons for non-completion of health related quality of life evaluations in pediatric acute myeloid leukemia: a report from the Children's Oncology Group

BACKGROUND

Health related quality of life (HRQL) assessments during therapy for pediatric cancer are important. The objective of this study was to describe reasons for failure to provide HRQL assessments during a pediatric acute myeloid leukemia (AML) clinical trial.

METHODS

We focused on HRQL assessments embedded in a multicenter pediatric AML clinical trial. The PedsQL 4.0 Generic Core Scales, PedsQL 3.0 Acute Cancer Module, PedsQL Multidimensional Fatigue Scale, and Pediatric Inventory for Parents were obtained from parent/guardian respondents at a maximum of six time points. Children provided self-report optionally. A central study coordinator contacted sites with delinquent HRQL data. Reasons for failure to submit the HRQL assessments were evaluated by three pediatric oncologists and themes were generated using thematic analysis.

RESULTS

There were 906 completed and 1091 potential assessments included in this analysis (83%). The median age of included children was 12.9 years (range 2.0 to 18.9). The five themes for non-completion were: patient too ill; passive or active refusal by respondent; developmental delay; logistical challenges; and poor knowledge of study processes from both the respondent and institutional perspective.

CONCLUSIONS

We identified reasons for non-completion of HRQL assessments during active therapy. This information will facilitate recommendations to improve study processes and future HRQL study designs to maximize response rates.

Measurement equivalence using a mixed-mode approach to administer health-related quality of life instruments

PURPOSE

To evaluate the effects of mode, order of administration, and the interaction of mode and order on health-related quality of life scales when self-administered by mixed mode (paper-mode and web-mode) for measurement equivalence.

METHODS

Health-related quality of life data was analyzed from the Cancer of the Prostate Strategic Urologic Research Endeavor using the Medical Outcomes Study (MOS) Short Form-36 (SF-36) and the University of California Los Angeles Prostate Cancer Index (UCLA-PCI). A randomized crossover design assigned participants to two groups with a preferred 2-5-day washout period. Cognitive debriefing evaluated participants' mode preference.

RESULTS

Of the 245 men enrolled, 85 % completed both modes. The majority were White (97 %), college educated (66 %), reported an annual income >$75,000 (46 %), and a median age of 69 years. Intraclass correlation coefficients were high for each item on both instruments (r = .54-.97). Exact percentage agreement for yes/no items was high (≥.88). For the SF-36, significant differences were observed for order of administration (physical component and physical function scores) and for the interaction between mode and order (mental component, role emotional, social function, vitality, and mental health scores). For the UCLA-PCI, the largest difference was 12.8 points lower for sexual bother for order of administration by web-mode first (p = .03). Seventy percent preferred the web-mode, 21 % had no preference, and 9 % preferred the paper-mode.

CONCLUSION

Web-mode and paper-mode administrations of the SF-36 and UCLA-PCI are equivalent in men with prostate cancer, implying that mixed-mode survey administration is warranted.

Difference in method of administration did not significantly impact item response: an IRT-based analysis from the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative

PURPOSE

To test the impact of method of administration (MOA) on the measurement characteristics of items developed in the Patient-Reported Outcomes Measurement Information System (PROMIS).

METHODS

Two non-overlapping parallel 8-item forms from each of three PROMIS domains (physical function, fatigue, and depression) were completed by 923 adults (age 18-89) with chronic obstructive pulmonary disease, depression, or rheumatoid arthritis. In a randomized cross-over design, subjects answered one form by interactive voice response (IVR) technology, paper questionnaire (PQ), personal digital assistant (PDA), or personal computer (PC) on the Internet, and a second form by PC, in the same administration. Structural invariance, equivalence of item responses, and measurement precision were evaluated using confirmatory factor analysis and item response theory methods.

RESULTS

Multigroup confirmatory factor analysis supported equivalence of factor structure across MOA. Analyses by item response theory found no differences in item location parameters and strongly supported the equivalence of scores across MOA.

CONCLUSIONS

We found no statistically or clinically significant differences in score levels in IVR, PQ, or PDA administration as compared to PC. Availability of large item response theory-calibrated PROMIS item banks allowed for innovations in study design and analysis.

Strengthening the assessment of factorial invariance across population subgroups: a commentary on Varni et al. (2013)

OBJECTIVES

This article provides a commentary in response to "Varni et al. (Qual Life Res. doi: 10.1007/s11136-013-0370-4 , 2013)."

METHODS AND RESULTS

The commentary argues that the approximate model fit indexes commonly used in maximum-likelihood confirmatory factor analysis and factorial invariance testing are seriously flawed, as they overlook potentially serious model misspecifications that could bias parameter estimates and compromise inference.

CONCLUSIONS

Flexible and convenient Bayesian estimation approaches are presented that can substantially aid in: (1) resolving commonly encountered specification errors in confirmatory factor models and (2) locating specific measurement parameters that are non-invariant across population subgroups. It is recommended that these methods should be more widely adopted for evaluating the factorial invariance of patient-reported outcome measures and other types of instruments.

Role Preferences of People with Multiple Sclerosis: Image-Revised, Computerized Self-Administered Version of the Control Preference Scale

Background

The Control Preference Scale (CPS) is the most frequently used measure of patients’ preferred roles in treatment decisions. We revised the original CPS and developed a new computerized patient self-administered version (eCPS). We used the eCPS to assess role preferences, and their determinants, in Italian and German people with multiple sclerosis (MS).

Methods

New cartoons were produced, based on MS health professional and patient input/feedback and previous findings, and pilot tested on 26 Italian and German MS patients. eCPS acceptability and reliability (weighted kappa statistic, wK) in comparison to the original tool, was determined in 92 MS patients who received both CPS versions in random order.

Results

The new cartoons were well accepted and easily interpreted by patients, who reported they based their choices mainly on the text and considered the images of secondary importance. eCPS reliability was moderate (wK 0.53, 95% confidence interval [CI] 0.40–0.65) and similar to the test-retest reliability of face-to-face administration assessed in a previous publication (wK 0.65, 95% CI 0.45–0.81). Higher education (odds ratio [OR] 3.74, 95% CI 1.00–14.05) and German nationality (OR 10.30, 95% CI 3.10–34.15) were associated with preference for an active role in the logistic model.

Conclusions

The newly devised eCPS was well received and considered easy to use by MS patients. Reliability was in line with that of the original version. Role preference appears affected by cultural characteristics and (borderline statistical significance) education.

Validation of electronic systems to collect patient-reported outcome (PRO) data-recommendations for clinical trial teams: report of the ISPOR ePRO systems validation good research practices task force

Outcomes research literature has many examples of high-quality, reliable patient-reported outcome (PRO) data entered directly by electronic means, ePRO, compared to data entered from original results on paper. Clinical trial managers are increasingly using ePRO data collection for PRO-based end points. Regulatory review dictates the rules to follow with ePRO data collection for medical label claims. A critical component for regulatory compliance is evidence of the validation of these electronic data collection systems. Validation of electronic systems is a process versus a focused activity that finishes at a single point in time. Eight steps need to be described and undertaken to qualify the validation of the data collection software in its target environment: requirements definition, design, coding, testing, tracing, user acceptance testing, installation and configuration, and decommissioning. These elements are consistent with recent regulatory guidance for systems validation. This report was written to explain how the validation process works for sponsors, trial teams, and other users of electronic data collection devices responsible for verifying the quality of the data entered into relational databases from such devices. It is a guide on the requirements and documentation needed from a data collection systems provider to demonstrate systems validation. It is a practical source of information for study teams to ensure that ePRO providers are using system validation and implementation processes that will ensure the systems and services: operate reliably when in practical use; produce accurate and complete data and data files; support management control and comply with any existing regulations. Furthermore, this short report will increase user understanding of the requirements for a technology review leading to more informed and balanced recommendations or decisions on electronic data collection methods.

Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPOR PRO good research practices for the assessment of children and adolescents task force

BACKGROUND

Patient-reported outcome (PRO) instruments for children and adolescents are often included in clinical trials with the intention of collecting data to support claims in a medical product label.

OBJECTIVE

The purpose of the current task force report is to recommend good practices for pediatric PRO research that is conducted to inform regulatory decision making and support claims made in medical product labeling. The recommendations are based on the consensus of an interdisciplinary group of researchers who were assembled for a task force associated with the International Society for Pharmacoeconomics and Outcomes Research (ISPOR). In those areas in which supporting evidence is limited or in which general principles may not apply to every situation, this task force report identifies factors to consider when making decisions about the design and use of pediatric PRO instruments, while highlighting issues that require further research.

GOOD RESEARCH PRACTICES

Five good research practices are discussed: 1) Consider developmental differences and determine age-based criteria for PRO administration: Four age groups are discussed on the basis of previous research (<5 years old, 5-7 years, 8-11 years, and 12-18 years). These age groups are recommended as a starting point when making decisions, but they will not fit all PRO instruments or the developmental stage of every child. Specific age ranges should be determined individually for each population and PRO instrument. 2) Establish content validity of pediatric PRO instruments: This section discusses the advantages of using children as content experts, as well as strategies for concept elicitation and cognitive interviews with children. 3) Determine whether an informant-reported outcome instrument is necessary: The distinction between two types of informant-reported measures (proxy vs. observational) is discussed, and recommendations are provided. 4) Ensure that the instrument is designed and formatted appropriately for the target age group. Factors to consider include health-related vocabulary, reading level, response scales, recall period, length of instrument, pictorial representations, formatting details, administration approaches, and electronic data collection (ePRO). 5) Consider cross-cultural issues.

CONCLUSIONS

Additional research is needed to provide methodological guidance for future studies, especially for studies involving young children and parents' observational reports. As PRO data are increasingly used to support pediatric labeling claims, there will be more information regarding the standards by which these instruments will be judged. The use of PRO instruments in clinical trials and regulatory submissions will help ensure that children's experience of disease and treatment are accurately represented and considered in regulatory decisions.

ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research

PURPOSE

An essential aspect of patient-centered outcomes research (PCOR) and comparative effectiveness research (CER) is the integration of patient perspectives and experiences with clinical data to evaluate interventions. Thus, PCOR and CER require capturing patient-reported outcome (PRO) data appropriately to inform research, healthcare delivery, and policy. This initiative's goal was to identify minimum standards for the design and selection of a PRO measure for use in PCOR and CER.

METHODS

We performed a literature review to find existing guidelines for the selection of PRO measures. We also conducted an online survey of the International Society for Quality of Life Research (ISOQOL) membership to solicit input on PRO standards. A standard was designated as "recommended" when >50 % respondents endorsed it as "required as a minimum standard."

RESULTS

The literature review identified 387 articles. Survey response rate was 120 of 506 ISOQOL members. The respondents had an average of 15 years experience in PRO research, and 89 % felt competent or very competent providing feedback. Final recommendations for PRO measure standards included: documentation of the conceptual and measurement model; evidence for reliability, validity (content validity, construct validity, responsiveness); interpretability of scores; quality translation, and acceptable patient and investigator burden.

CONCLUSION

The development of these minimum measurement standards is intended to promote the appropriate use of PRO measures to inform PCOR and CER, which in turn can improve the effectiveness and efficiency of healthcare delivery. A next step is to expand these minimum standards to identify best practices for selecting decision-relevant PRO measures.

Comparing Web-based with Mail Survey Administration of the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician and Group Survey

CONTEXT

The CAHPS^® survey instruments are widely used to assess patient experiences with care but there is limited information about web-based data collection with them.

OBJECTIVE

To compare web-based data collection with standard mail survey mode of collection of CAHPS^® Clinician and Group survey data.

DESIGN SETTING AND PATIENTS

We randomized mode of data collection (web versus mail) of the CAHPS^® Clinician and Group Survey to patients who had visited one of six clinics over a four-month period in Minnesota. A total of 410 patients responded to the web-based survey (14% response rate) and 982 patients responded to the mail survey (33% response rate).

MAIN OUTCOME MEASURES

Responses to CAHPS^® survey dimensions and individual question responses, response rates, and participant characteristics.

RESULTS

There were no significant differences in CAHPS^® survey composites and individual question responses by mode, except for those addressing access. Those responding via the web reported less positive experiences with access to an appointment for urgent care as soon as needed, getting an appointment for routine care as soon as needed, getting answers to medical questions as soon as needed, and follow-up on test results (t's=-3.64, -7.15, -2.58, -2.23; p's=0.0003, <0.0001, 0.01, 0.03, respectively). Web respondents had more positive experiences about office wait time for the most recent visit (t = 2.32, p=0.021). Those who participated in the study tended to be older than those that did not (ƛ²=247.51, df=8, p<0.0001 for mail; ƛ²= 4.56, df=8, p<0.0001 for the web). Females were significantly more likely than males to respond to the survey overall (24% vs. 18%, ƛ²=6.45, 1 df, p=0.011) and relatively more likely than males to respond to web (15% vs. 13%, ƛ²=1.32, 1 df, p=0.25) than mail (34% vs. 30%, ƛ²=5.42, 1 df, p=0.02). Mail respondents were more likely than web respondents to be male (28% versus 18%, ƛ²=16.27, 1 df, p<0.0001) and older (27% of the mail respondents and 19% of the web respondents were 65 or older, ƛ²=10.88, 1 df, p=0.001). Costs of web-based surveys were less than mailed surveys and were returned more quickly than mailed surveys. The correlations between reports and ratings of clinicians and clinics by mode were unreliable because of the relatively small number of web responses.

CONCLUSION

Web-based surveys yielded comparable results to mail (except for questions addressing access) more quickly at lower costs. The low response rates in this study are a concern although this was not intended as a test of increasing response rates. Strategies to increase response rates will be a key element of web-based data collection. The differences in costs will be an incentive for organizations to continue to pursue web-based surveying. Further studies are needed to evaluate the generalizability of the results of this one.

Core domains for a person-focused outcome measurement system in cancer (PROMS-Cancer Core) for routine care: a scoping review and Canadian Delphi Consensus

OBJECTIVES

The objectives of this scoping review study were 1) to identify core domains and dimensions for inclusion in a person-focused and self-reported outcome measurement system for cancer and 2) to reach consensus among key stakeholders including cancer survivors on the relevance, acceptability, and feasibility of a core outcome set for collection in routine clinical care.

METHODS

Following a scoping review of the literature, a Rand Delphi consensus method was used to engage key interdisciplinary decision makers, clinicians, and cancer survivors in reaching consensus on a core patient-reported outcome domain taxonomy and outcome measures.

RESULTS

Of the 21,900 citations identified in the scoping review, 1,503 citations were included in the full article review (380 conceptual articles, 461 psychometric evaluation articles, and 662 intervention studies) and subjected to data abstraction and mapping. Final consensus was reached on 20 domains, related subdimensions, and 45 self-report measures considered relevant and feasible for routine collection in cancer by the Delphi panel (PROMS-Cancer Core).

CONCLUSIONS

Standardization of patient-reported outcome data collection is key to assessing the impact of cancer and treatment on the person for population comparison and monitoring the quality of clinical care. The PROMS-Cancer Core taxonomy of domains and outcome measures can be used to guide the development of a patient-reported outcome information system for cancer.

The use of a tablet computer to complete the DASH questionnaire

PURPOSE

To determine whether electronic self-administration of the Disabilities of the Arm, Shoulder, and Hand (DASH) questionnaire using a tablet computer increased completion rate compared with paper self-administration.

METHODS

We gave the DASH in self-administered paper form to 222 new patients in a single hand surgeon's practice. After a washout period of 5 weeks, we gave the DASH in self-administered tablet computer form to 264 new patients. A maximum of 3 questions could be omitted before the questionnaire was considered unscorable. We reviewed the submitted surveys to determine the number of scorable questionnaires and the number of omitted questions in each survey. We completed univariate analysis and regression modeling to determine the influence of survey administration type on respondent error while controlling for patient age and sex.

RESULTS

Of the 486 total surveys, 60 (12%) were not scorable. A significantly higher proportion of the paper surveys (24%) were unscorable compared with electronic surveys (2%), with significantly more questions omitted in each paper survey (2.6 ± 4.4 questions) than in each electronic survey (0.1 ± 0.8 questions). Logistic regression analysis revealed survey administration mode to be significantly associated with DASH scorability while controlling for age and sex, with electronic survey administration being 14 times more likely than paper administration to yield a scorable DASH.

CONCLUSIONS

In our retrospective series, electronic self-administration of the DASH decreased the number of omitted questions and yielded a higher number of scorable questionnaires. Prospective, randomized evaluation is needed to better delineate the effect of survey administration on respondent error.

CLINICAL RELEVANCE

Administration of the DASH with a tablet computer may be beneficial for both clinical and research endeavors to increase completion rate and to gain other benefits from electronic data capture.

Systematic review: symptom assessment using patient-reported outcomes in gastroesophageal reflux disease and dyspepsia

OBJECTIVE

The importance of symptom assessment using patient-reported outcomes (PROs) is becoming increasingly recognized in the management of upper gastrointestinal (GI) disease. The authors aimed to review systematically the methodological aspects of PRO instrument development and use in the GERD or dyspepsia literature, and to assess these instruments' properties in light of the Food and Drug Administration's (FDA) guidance.

MATERIAL AND METHODS

Systematic PubMed and Embase searches (using terms based on the FDA guidance) identified studies that reported methodological aspects in developing or using PRO instruments for GERD or dyspepsia symptom measurement.

RESULTS

Ten studies were identified (six systematically and four from citation lists). Studies reported the development or use of a relevant PRO instrument, with a focus on methodological aspects that the FDA guidance describes as important for patient understanding. Studies demonstrated heterogeneity of recall periods, symptoms and response options. Two studies demonstrated that a lack of consistent vocabulary may contribute to discrepancy in symptom reporting between investigators and patients. Two studies indicated that symptoms must be described in a manner that is relevant to patients. One study described the development of a PRO instrument separately in two languages, acknowledging linguistic and cultural differences between populations. One study demonstrated changes in symptom severity based on the recall period.

CONCLUSIONS

There is considerable heterogeneity in the methodology used to develop PRO instruments for upper GI disease. Adherence to best practices in PRO development and validation may improve the quality and utility of these measures, leading to improved communication in clinical practice.

Development of the Shortness of Breath with Daily Activities questionnaire (SOBDA)

OBJECTIVES

Based on qualitative research of patients with chronic obstructive pulmonary disease (COPD), the Shortness of Breath (SOB) with Daily Activities (SOBDA) questionnaire was developed as a patient-reported outcome instrument to evaluate the impact of therapy on SOB and assess how SOB affects daily activities.

METHODS

Development of the SOBDA questionnaire consisted of three components. First, focus groups of patients with COPD were asked to describe their experiences of SOB with daily activities. A pool of items was drafted on the basis of information from the focus groups and literature reviews, and then discussed among instrument development and clinical experts. Cognitive debriefing interviews of patients were conducted to assess the draft item pool, and their feedback was used to develop newer versions of the questionnaire. Input was also sought from the Food and Drug Administration, patients, and clinicians.

RESULTS

Forty patients participated in seven focus groups. The terms most often used to describe SOB were "short of breath" or "difficulty breathing." Patients were clearly able to distinguish SOB from chest congestion and wheezing, other common symptoms associated with COPD. The resulting item pool contained 37 items to assess SOB associated with everyday activities, and concept saturation was reached. Thirty-seven patients participated in the subsequent cognitive debriefing interviews. Patients found the items clear and easy to understand with relevance to their everyday experiences, and easy to use in an electronic format.

CONCLUSIONS

Instructions and response options to the SOBDA questionnaire were well understood by patients with COPD, and item relevance was confirmed. Prospective validation and item reduction studies are highly anticipated.

The Computer-based Health Evaluation Software (CHES): a software for electronic patient-reported outcome monitoring

BACKGROUND

Patient-reported Outcomes (PROs) capturing e.g., quality of life, fatigue, depression, medication side-effects or disease symptoms, have become important outcome parameters in medical research and daily clinical practice. Electronic PRO data capture (ePRO) with software packages to administer questionnaires, storing data, and presenting results has facilitated PRO assessment in hospital settings. Compared to conventional paper-pencil versions of PRO instruments, ePRO is more economical with regard to staff resources and time, and allows immediate presentation of results to the medical staff.The objective of our project was to develop software (CHES - Computer-based Health Evaluation System) for ePRO in hospital settings and at home with a special focus on the presentation of individual patient's results.

METHODS

Following the Extreme Programming development approach architecture was not fixed up-front, but was done in close, continuous collaboration with software end users (medical staff, researchers and patients) to meet their specific demands. Developed features include sophisticated, longitudinal charts linking patients' PRO data to clinical characteristics and to PRO scores from reference populations, a web-interface for questionnaire administration, and a tool for convenient creating and editing of questionnaires.

RESULTS

By 2012 CHES has been implemented at various institutions in Austria, Germany, Switzerland, and the UK and about 5000 patients participated in ePRO (with around 15000 assessments in total). Data entry is done by the patients themselves via tablet PCs with a study nurse or an intern approaching patients and supervising questionnaire completion.

DISCUSSION

During the last decade several software packages for ePRO have emerged for different purposes. Whereas commercial products are available primarily for ePRO in clinical trials, academic projects have focused on data collection and presentation in daily clinical practice and on extending cancer registries with PRO data. CHES includes several features facilitating the use of PRO data for individualized medical decision making. With its web-interface it allows ePRO also when patients are home. Thus, it provides complete monitoring of patients'physical and psychosocial symptom burden.

Validation of online administration of the sexual health inventory for men

PURPOSE

Previously we reported the development of a novel, inexpensive, online method to collect health related quality of life information to facilitate responses among patients and decrease loss to followup. We validated the practice by comparing responses to the SHIM (Sexual Health Inventory for Men), a representative validated instrument, when administered on line and in the traditional paper form.

MATERIALS AND METHODS

Consented patients were administered validated health related quality of life instruments, including the SHIM, in office and via e-mail. Responses to the SHIM were compared between the administration formats. Paired sample testing was done to analyze test-retest reliability, concordance was assessed by intraclass analysis and a Bland-Altman plot, and the Cronbach α was used to examine internal reliability. Criterion validity was measured using SHIM defined erectile function categories and a dichotomized potency definition (SHIM 17 or greater).

RESULTS

Of the 508 men who consented to participate 359 (71%) completed the SHIM in person, 277 (55%) completed the online form (p <0.001) and 116 (23%) contemporaneously completed each instrument. Comparison of scores revealed little variation and strong correlation (r(2) = 0.83, p <0.001). Intraclass and Bland-Altman analysis revealed strong agreement between the media. The Cronbach α was excellent (0.97) for the online tool. Erectile function classification was identical in 73% of patients with only 7% differing by more than 1 class. Dichotomized potency was consistently defined in 94% of patients.

CONCLUSIONS

The online administered SHIM maintains validity and provides consistent responses. Online administration can capture patients who do not complete paper questionnaires and may serve as a reliable adjunct to paper administration for validated outcomes research.

Recommendations for incorporating patient-reported outcomes into clinical comparative effectiveness research in adult oncology

Examining the patient's subjective experience in prospective clinical comparative effectiveness research (CER) of oncology treatments or process interventions is essential for informing decision making. Patient-reported outcome (PRO) measures are the standard tools for directly eliciting the patient experience. There are currently no widely accepted standards for developing or implementing PRO measures in CER. Recommendations for the design and implementation of PRO measures in CER were developed via a standardized process including multistakeholder interviews, a technical working group, and public comments. Key recommendations are to include assessment of patient-reported symptoms as well as health-related quality of life in all prospective clinical CER studies in adult oncology; to identify symptoms relevant to a particular study population and context based on literature review and/or qualitative and quantitative methods; to assure that PRO measures used are valid, reliable, and sensitive in a comparable population (measures particularly recommended include EORTC QLQ-C30, FACT, MDASI, PRO-CTCAE, and PROMIS); to collect PRO data electronically whenever possible; to employ methods that minimize missing patient reports and include a plan for analyzing and reporting missing PRO data; to report the proportion of responders and cumulative distribution of responses in addition to mean changes in scores; and to publish results of PRO analyses simultaneously with other clinical outcomes. Twelve core symptoms are recommended for consideration in studies in advanced or metastatic cancers. Adherence to methodologic standards for the selection, implementation, and analysis/reporting of PRO measures will lead to an understanding of the patient experience that informs better decisions by patients, providers, regulators, and payers.

An electronic diary is shown to be more reliable than a paper diary: results from a randomized crossover study in patients with persistent asthma

OBJECTIVES

Test-retest reliability of an asthma paper diary versus an electronic diary (e-diary) with an integrated peak flow meter was investigated. The equivalence of the two modes was also evaluated.

METHODS

Prospective, randomized crossover study design in adolescents (12-17 years) and adults (≥18 years). Key inclusion criteria were persistent asthma, Asthma Control Test (ACT) scores ≥16, use of inhaled corticosteroid with or without long-acting beta-agonist for ≥12 weeks, nocturnal awakenings <2 times in the past week, and activity limitations <1 per week. Participants were randomized to either paper then e-diary or e-diary then paper, to be completed for 14 days each.

RESULTS

Forty-seven participants completed all study visits. Weekly percentage of symptom-free days (SFDs) and rescue-free days (RFDs) were calculated. Intraclass correlation coefficients (ICCs) of Week 1 mean SFD and RFD (test) and Week 2 mean SFD and RFD (retest), respectively, were estimated in three groups defined as stable: (i) minimal changes in asthma symptoms, as measured by the global patient reported symptom change question, (ii) less than 15% change (absolute value) in 1 second FEV(1) at adjacent study visits, and (iii) changes in ACT scores less than three points for each mode. SFD demonstrated acceptable ICC (≥0.70) using all three definitions of asthma stability for both modes.

CONCLUSION

Acceptable reproducibility of the percentage of RFD (ICC = 0.78) was only observed for the e-diary using the FEV(1) stability criterion. The ICCs for SFD and RFD were acceptable, 0.84 and 0.70, respectively, suggesting better reliability for the e-diary.

A framework for understanding modifications to measures for diverse populations

OBJECTIVE

Research on health disparities and determinants of health disparities among ethnic minorities and vulnerable older populations necessitates use of self-report measures. Most established instruments were developed on mainstream populations and may need adaptation for research with diverse populations. Although information is increasingly available on various problems using these measures in diverse groups, there is little guidance on how to modify the measures. A framework of issues to consider when modifying measures for diverse populations is presented.

METHODS

The authors describe reasons for considering modifications, the types of information that can be used as a basis for making modifications, and the types of modifications researchers have made. Recommendations are made to test modified measures to assure their appropriateness, and suggestions are provided on reporting modifications in publications using the measures.

DISCUSSION

The issues open a dialogue about what appropriate guidelines would be for researchers adapting measures in studies of ethnically diverse populations.

Electronic patient-reported outcome systems in oncology clinical practice

Patient-reported outcome (PRO) questionnaires assess topics a patient can report about his or her own health. This includes symptoms (eg, nausea, fatigue, diarrhea, pain, or frequent urination), physical functioning (eg, difficulty climbing stairs or difficulty fastening buttons), and mental health (eg, anxiety, fear, or worry). Electronic PRO (ePRO) systems are used in oncology clinical care because of 1) their ability to enhance clinical care by flagging important symptoms and saving clinicians time; 2) the availability of standardized methods for creating and implementing PROs in clinics; and 3) the existence of user-friendly platforms for patient self-reporting like tablet computers and automated telephone surveys. Many ePRO systems can provide actionable links to clinical care such as summary reports in a patient's electronic medical record and real-time e-mail alerts to providers when patients report acute needs. This review presents 5 examples of ePRO systems currently in use in oncology practice. These systems support multiple clinical activities, including assessment of symptoms and toxicities related to chemotherapy and radiation, postoperative surveillance, and symptom management during palliative care and hospice. Patient self-reporting is possible both at clinical visits and between visits over the Internet or by telephone. The implementation of an ePRO system requires significant resources and expertise, as well as user training. ePRO systems enable regular monitoring of patient symptoms, function, and needs, and can enhance the efficiency and quality of care as well as communication with patients.

Single-item electronic administration of the SF-36v2 Health Survey

PURPOSE

To evaluate the psychometric properties and factor structure of a computerized electronic version of the SF-36v2 Health Survey (SF-36v2) with items administered one-per-page versus the traditional grid format used in the paper-and-pencil version in a sample of physician-diagnosed headache patients.

METHODS

Patients (N = 180) completed the SF-36v2 administered as part of a broader study of health outcomes. Scaling assumptions, reliability, factor structure, and the tool's ability to discriminate between headache pain severity groups were examined.

RESULTS

Frequency distributions showed notable ceiling effects for the role emotional, social functioning, physical functioning, and role physical scales, but negligible (<1.2%) floor effects for any of the scales. Internal consistency reliability coefficients ranged from 0.81 to 0.95 for the eight health domains. Items passed tests of internal consistency and discriminant validity. Principal components' analyses confirmed the 2-factor structure; the pattern of correlations across scales was consistent with expectations for the physical and mental health components. As expected, patients with severe headache pain had lower mean SF-36v2 scores than those with mild or moderate pain. No significant score differences were observed between mild and moderate pain severity groups.

CONCLUSIONS

Single-item electronic administration of the SF-36v2 is reliable and valid for use with headache patients.

[The benefits of using patient-reported outcomes in cancer treatment: an overview]

The patient's perspective evaluated by patient-reported outcomes (PROs) gains more and more importance, since treatment efficacy is no longer solely linked to clinical outcomes like cure and overall survival. Ailments like pain, fatigue and social isolation can only be assessed by patients' direct expression without any interpretation made by medical staff. PROs facilitate the disclosure of quality of life issues and patients feel a stronger support due to improved communication. PROs offer many further advantages like saving of time, cost and staff, targeted intervention and sensitizing of clinicians. Also, internationally validated questionnaires are available and the development of electronic PROs eases data-collection, calculation and storage. PROs collected within clinical routine are versatile concerning their applicability: They can be used for scientific analyses, quality assurance, and health technology assessment.

Test-retest reliability of an interactive voice response version of the EQ-5D in a sample of cancer survivors

BACKGROUND

Electronic data capture technologies, such as interactive voice response (IVR) systems, are emerging as important alternatives for collecting patient-reported outcome data.

OBJECTIVE

The objective of this study was to assess the test-retest reliability of an IVR version of the EQ-5D.

METHODS

Outpatient cancer clinic patients (n = 127) were asked to complete the IVR-based EQ-5D twice, 2 days apart. The analyses tested for mean differences (paired t-test) and test-retest reliability (intraclass correlation coefficient [ICC]) to assess measurement stability over time. Equivalence of the means was established if the 95% confidence interval (CI) was within the minimally important difference (MID) interval; namely -0.035 to 0.035 for the EQ-5D index and -3.0 to 3.0 for the visual analog scale (i.e. EQ VAS). Adequacy of the ICC was established by testing whether it differed from a value of 0.70.

RESULTS

Both administrations were completed per protocol by 114 subjects (EQ-5D index) and 110 subjects (EQ VAS). For the EQ-5D index, the means (SD) of the first and second administrations were 0.871 (0.14) and 0.871 (0.15), respectively. The 95% CI of the mean difference was -0.013, 0.013, within the equivalence interval. The ICC was 0.876 (95% lower bound of 0.826) and was significantly different from 0.70. The EQ VAS means (SD) were 81.3 (17.5) and 80.8 (17.5), respectively. The 95% CI of the mean difference was -0.598, 1.617, within the equivalence interval. The EQ VAS ICC was 0.944 (95% lower bound of 0.919) and was significantly greater than 0.70.

CONCLUSION

This analysis provides substantial evidence that the scores obtained from the IVR version of the EQ-5D are reliable upon repeated administrations.

Recommendations for Clinical Trials of Off-Label Drugs Used to Treat Advanced-Stage Cancer

Purpose

To provide recommendations to trialists and sponsors that guide the design and implementation of prospective postapproval clinical trials for oncology drugs used outside US Food and Drug Administration–labeled indications for treatment of late-stage cancers.

Methods

A meeting was hosted by the Center for Medical Technology Policy in Baltimore, MD, on November 12, 2009. Discussions during the meeting and key informant interviews were conducted before and after this stakeholder meeting. Peer review by multidisciplinary stakeholders was followed by a public comment period. Input was received from patient advocacy groups, medical oncologists, pharmaceutical companies, the US Food and Drug Administration, Centers for Medicare and Medicaid Services, the National Cancer Institute, foreign government agencies involved in health technology assessment, public and private payers, drug compendia, clinical research entities, statisticians, academics, and the American Society of Clinical Oncology.

Results

To address the needs of patients and their clinical providers, compendia, payers, and policy makers, recommendations are proposed to guide the design of future prospective trials for off-label use of oncology drugs across four areas: trial design and data analysis, patient and site recruitment, comparators, and outcomes.

Conclusion

The US Food and Drug Administration provides guidance to the pharmaceutical industry and others designing randomized clinical trials for regulatory approval. However, a gap exists for postregulatory decision makers, including patients, prescribers, and payers, because regulatory trials do not answer the questions most relevant to them. Therefore, guidance is needed for trials performed in the postapproval environment for these postapproval decision makers.

Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: part 1--eliciting concepts for a new PRO instrument

The importance of content validity in developing patient reported outcomes (PRO) instruments is stressed by both the US Food and Drug Administration and the European Medicines Agency. Content validity is the extent to which an instrument measures the important aspects of concepts that developers or users purport it to assess. A PRO instrument measures the concepts most significant and relevant to a patient's condition and its treatment. For PRO instruments, items and domains as reflected in the scores of an instrument should be important to the target population and comprehensive with respect to patient concerns. Documentation of target population input in item generation, as well as evaluation of patient understanding through cognitive interviewing, can provide the evidence for content validity. Developing content for, and assessing respondent understanding of, newly developed PRO instruments for medical product evaluation will be discussed in this two-part ISPOR PRO Good Research Practices Task Force Report. Topics include the methods for generating items, documenting item development, coding of qualitative data from item generation, cognitive interviewing, and tracking item development through the various stages of research and preparing this tracking for submission to regulatory agencies. Part 1 covers elicitation of key concepts using qualitative focus groups and/or interviews to inform content and structure of a new PRO instrument. Part 2 covers the instrument development process, the assessment of patient understanding of the draft instrument using cognitive interviews and steps for instrument revision. The two parts are meant to be read together. They are intended to offer suggestions for good practices in planning, executing, and documenting qualitative studies that are used to support the content validity of PRO instruments to be used in medical product evaluation.

Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2--assessing respondent understanding

The importance of content validity in developing patient reported outcomes (PRO) instruments is stressed by both the US Food and Drug Administration and the European Medicines Agency. Content validity is the extent to which an instrument measures the important aspects of concepts developers or users purport it to assess. A PRO instrument measures the concepts most relevant and important to a patient's condition and its treatment. For PRO instruments, items and domains as reflected in the scores of an instrument should be important to the target population and comprehensive with respect to patient concerns. Documentation of target population input in item generation, as well as evaluation of patient understanding through cognitive interviewing, can provide the evidence for content validity. Part 1 of this task force report covers elicitation of key concepts using qualitative focus groups and/or interviews to inform content and structure of a new PRO instrument. Building on qualitative interviews and focus groups used to elicit concepts, cognitive interviews help developers craft items that can be understood by respondents in the target population and can ultimately confirm that the final instrument is appropriate, comprehensive, and understandable in the target population. Part 2 details: 1) the methods for conducting cognitive interviews that address patient understanding of items, instructions, and response options; and 2) the methods for tracking item development through the various stages of research and preparing this tracking for submission to regulatory agencies. The task force report's two parts are meant to be read together. They are intended to offer suggestions for good practice in planning, executing, and documenting qualitative studies that are used to support the content validity of PRO instruments to be used in medical product evaluation.