Choosing and Using Patient-Reported Outcome Measures in Clinical Practice

The gap between statistical and clinical significance: time to pay attention to clinical relevance in patient-reported outcome measures of insomnia

BACKGROUND

Appropriately defining and using the minimal important change (MIC) and the minimal clinically important difference (MCID) are crucial for determining whether the results are clinically significant. The aim of this study is to survey the status of randomized controlled trials (RCTs) for insomnia interventions to assess the inclusion and interpretation of MIC/MCID values.

METHODS

We conducted a cross-sectional study to survey the status of RCTs for insomnia interventions to assess the inclusion and appropriate interpretation of MIC/MCID values. A literature search was conducted by searching the main sleep medicine journals indexed in PubMed, the Excerpta Medica Database (EMBASE), and the Cochrane Central Register of Controlled Trials (CENTRAL) to identify a broad range of search terms. We included RCTs with no restriction on the intervention. The included studies used the Insomnia Severity Index (ISI) or the Pittsburgh Sleep Quality Index (PSQI) questionnaire as the outcome measures.

RESULTS

81 eligible studies were identified, and more than one-third of the included studies used MIC/MCID (n = 31, 38.3%). Among them, 21 studies with ISI as the outcome used MIC defined as a relative decrease ranging from 3 to 8 points. The most frequently used MIC value was a 6-point decrease (n = 7), followed by 8-point (n = 6) and 7-point decrease (n = 4), a 4 to 5-points decrease (n = 3), and a 30% reduction from baseline; 6 studies used MCID values, ranging from 2.8 to 4 points. The most frequently used MCID value was a 4-point decrease in the ISI (n = 4). 4 studies with PSQI as the outcome used a 3-point change as the MIC (n = 2) and a 2.5 to 2.7-point difference as MCID (n = 2). 4 non-inferiority design studies considered interval estimation when drawing clinically significant conclusions in their MCID usage.

CONCLUSIONS

The lack of consistent MIC/MCID interpretation and usage in outcome measures for insomnia highlights the urgent need for further efforts to address this issue and improve reporting practices.

The development of an archive of patient-reported outcome measures (PROMs) in oncology: The Italian PRO4All project

BACKGROUND

Choosing the most adequate measure of patient-reported outcomes (PROs) for a specific medical condition is not straightforward. This study aimed to develop a comprehensive archive of patient-reported outcome measures (PROMs), observer-reported outcome measures (ObsROMs) and caregiver-reported outcome measures (CROMs) in oncology and identify their main characteristics and target outcome domains.

MATERIALS AND METHODS

As part of the Italian PRO4All Project, we retrieved questionnaires through an extensive search of online databases. We developed a data extraction form to collect information on cancer type, questionnaire variant(s), recall period, and scoring system. We performed a content analysis of the questionnaires to assign each item a specific outcome domain according to a predefined 38-item taxonomy.

RESULTS

A total of 386 PROMs (n = 356), ObsROMs (n = 13) and CROMs (n = 17) were identified and described; of these, 358 were also analyzed in their content. 47.3 % of the instruments were cancer type-specific, 45.1 % were generic for cancer and 7.9 % were developed for the general population but also recommended in oncology. The great majority (92.2 %) were patient-reported. In 50.3 % the recall period was "last week". The mean number of items per questionnaire was 22.0 (range: 1-130). 7794 items were assigned an outcome domain, the most frequent being emotional functioning/wellbeing (22 %), physical functioning (15.7 %), general outcomes (10.1 %) and delivery of care (8.9 %).

CONCLUSIONS

There are a variety of patient and caregiver-reported measures in oncology. This archive can guide researchers and practitioners in selecting the most suitable measures and fostering a patient-centered approach in clinical trials, clinical practice, and regulatory activities.

Understanding factors impacting patient-reported outcome measures integration in routine clinical practice: an umbrella review

PURPOSE

Patient-report outcome measures (PROMs) have gained widespread support as a mechanism to improve healthcare quality. We aimed to map out key enablers and barriers influencing PROMs implementation strategies in routine clinical practice.

METHODS

An umbrella review was conducted to identify reviews exploring enablers and barriers related to the integration of PROMs in routine clinical practice from January 2000 to June 2023. Information on key enablers and barriers was extracted and summarised thematically according to the Theoretical Domains Framework.

RESULTS

34 reviews met our criteria for inclusion. Identified reviews highlighted barriers such as limited PROMs awareness among clinicians and patients, perceived low value by clinicians and patients, PROMs that were too complex or difficult for patients to complete, poor usability of PROMs systems, delayed feedback of PROMs data, clinician concerns related to use of PROMs as a performance management tool, patient concerns regarding privacy and security, and resource constraints. Enablers encompassed phased implementation, professional training, stakeholder engagement prior to implementation, clear strategies and goals, 'change champions' to support PROMs implementation, systems to respond to issues raised by PROMs, and integration into patient pathways. No consensus favoured paper or electronic PROMs, yet offering both options to mitigate digital literacy bias and integrating PROMs into electronic health records emerged as important facilitators.

CONCLUSIONS

The sustainable implementation of PROMs is a complex process that requires multicomponent organisational strategies covering training and guidance, necessary time and resources, roles and responsibilities, and consultation with patients and clinicians.

Developing a shortened spine functional index (SFI-10) for patients with sub-acute/chronic spinal disorders: a cross-sectional study

BACKGROUND

Brief whole-spine patient-reported outcome measures (PROMs) provide regional solutions and future directions for quantifying functional status, evidence, and effective interventions. The whole-spine regional Spine Functional Index (SFI-25) is used internationally in clinical and scientific contexts to assess general sub-acute/chronic spine populations. However, to improve structural validity and practicality a shortened version is recommended. This study developed a shortened-SFI from the determined optimal number of item questions that: correlated with criteria PROMs being highly with whole-spine, moderately with regional-spine, condition-specific and patient-specific, and moderately-low with general-health and pain; retained one-dimensional structural validity and high internal consistency; and improved practicality to reduce administrative burden.

METHODS

A cross-sectional study (n = 505, age = 18-87 yrs., average = 40.3 ± 10.1 yrs) of sub-acute/chronic spine physiotherapy outpatients from an international sample of convenience. Three shortened versions of the original SFI-25 were developed using 1) qualitative 'content-retention' methodology, 2) quantitative 'factorial' methodology, and 3) quantitative 'Rasch' methodology, with a fourth 'random' version produced as a comparative control. The clinimetric properties were established for structural validity with exploratory (EFA) and confirmatory (CFA) factorial analysis, and Rasch analysis. Criterion validity used the: whole-spine SFI-25 and Functional Rating Index (FRI); regional-spine Neck Disability Index (NDI), Oswestry Disability Index (ODI), and Roland Morris Questionnaire (RMQ), condition-specific Whiplash Disability Questionnaire (WDQ); and patient-specific functional scale (PSFS); and determined floor/ceiling effect. A post-hoc pooled international sub-acute/chronic spine sample (n = 1433, age = 18-91 yrs., average = 42.0 ± 15.7 yrs) clarified the findings and employed the general-health EuroQuol-Index (EQ-5D), and 11-point Pain Numerical Rating Scale (P-NRS) criteria.

RESULTS

A 10-item SFI retained structural validity with optimal practicality requiring no computational aid. The SFI-10 concept-retention-version demonstrated preferred criterion validity with whole-spine criteria (SFI-25 = 0.967, FRI = 0.810) and exceeded cut-off minimums with regional-spine, condition-specific, and patient-specific measures. An unequivocal one-dimensional structure was determined. Internal consistency was satisfactory (α = 0.80) with no floor/ceiling effect. Post-hoc analysis of the international sample confirmed these findings.

CONCLUSION

The SFI-10 qualitative concept-retention version was preferred to quantitative factorial and Rasch versions, demonstrated structural and criterion validity, and preferred correlation with criteria measures. Further longitudinal research is required for reliability, error, and responsiveness, plus an examination of the practical characteristics of readability and administrative burden.

Bipolar Disorder Symptom Monitoring Measures: A Mixed-Methods Study of Patient Preferences

OBJECTIVE

To determine the perceived acceptability and helpfulness of bipolar disorder symptom measures and rank the measures in order of preference among individuals with bipolar disorder.

METHODS

We recruited 20 participants 18 years of age or older with any type of bipolar disorder from a primary care clinical site and a national advocacy organization. We used a simultaneous complementary mixed-method design involving completion of symptom measures, a semistructured interview, and numerical ranking of measures. Participants completed three symptom measures or combination of measures: 1) Affective Self-Rating Scale; 2) combination Patient Mania Questionnaire-9 (PMQ-9) and Patient Health Questionnaire-9 (PHQ-9); and 3) combination Altman Self-Rating Mania Rating Scale and PHQ-9. A semistructured interview was conducted, and participants ranked their preferences for measures. Interviews focused on participants' rationale for measuring preferences. Interviews were analyzed by two psychiatrist-investigators using content analysis, and themes were determined. Average rank of each measure was determined.

RESULTS

The average rank for each measure was 1.48 for the combination PMQ-9 and PHQ-9, 1.68 for the Affective Self-Rating Scale, and 2.85 for the combination Altman Self-Rating Mania Rating Scale and PHQ-9, indicating that the combination PMQ-9 and PHQ-9 (top-ranked measure by 55% of participants) was the most preferred among the three measures. Major themes that emerged from the data were: 1) measure format; 2) patient experience; 3) clinical practice; and 4) therapeutic effects.

CONCLUSIONS

Individuals with bipolar disorder preferred the combination PMQ-9 and PHQ-9 for use in monitoring treatment due to perceived strengths such as format, ease of completion and interpretation, accurate description of experiences, and feasibility of use in practice.

Validation of the Occupational Depression Inventory in Sweden

BACKGROUND

The Occupational Depression Inventory (ODI) was recently devised to assess depressive symptoms that individuals specifically attribute to their work. One purpose of the ODI is to overcome limitations in current assessments of job-related distress. This study aimed to validate the Swedish version of the ODI.

METHODS

The study involved 365 individuals employed in Sweden. In addition to the ODI, the study included the Satisfaction with Life Scale, the Effort-Reward Imbalance Questionnaire, the Demand-Control-Support Questionnaire, the GAD-2, and the PHQ-9. We inquired into the factorial validity, dimensionality, scalability, test-score reliability, criterion validity, convergent validity, discriminant validity, and measurement invariance of the ODI.

RESULTS

Exploratory structural equation modeling bifactor analysis indicated that the ODI's Swedish version meets the requirements for essential unidimensionality (e.g., explained common variance = 0.872). Measurement invariance held across sexes, age groups, and occupational categories. The instrument exhibited strong scalability (e.g., H = 0.662). The observed total scores thus accurately ranked respondents on the latent continuum underlying the scale. The ODI's total-score reliability was high (e.g., McDonald's ω = 0.929). Speaking to the instrument's criterion validity, we found occupational depression to correlate, in the expected direction, with various work (e.g., job support) and nonwork (e.g., general anxiety) variables. Occupational depression showed large correlations with effort-reward imbalance (r = 0.613) and demand-control imbalance (r = 0.566) at work. Multiple regression analyses supported these associations further. As expected, we observed both a degree of convergent validity and a degree of discriminant validity when examining the ODI against the PHQ-9, an attribution-free measure of depression.

DISCUSSION

This study indicates that the ODI performs well within the Swedish context, consistent with the findings obtained in other linguistic and geographic contexts. The ODI promises to help researchers, practitioners, and public health decision-makers address job-related distress more effectively.

Patient-reported outcomes for people with diabetes: what and how to measure? A narrative review

Patient-reported outcomes (PROs) are valuable for shared decision making and research. Patient-reported outcome measures (PROMs) are questionnaires used to measure PROs, such as health-related quality of life (HRQL). Although core outcome sets for trials and clinical practice have been developed separately, they, as well as other initiatives, recommend different PROs and PROMs. In research and clinical practice, different PROMs are used (some generic, some disease-specific), which measure many different things. This is a threat to the validity of research and clinical findings in the field of diabetes. In this narrative review, we aim to provide recommendations for the selection of relevant PROs and psychometrically sound PROMs for people with diabetes for use in clinical practice and research. Based on a general conceptual framework of PROs, we suggest that relevant PROs to measure in people with diabetes are: disease-specific symptoms (e.g. worries about hypoglycaemia and diabetes distress), general symptoms (e.g. fatigue and depression), functional status, general health perceptions and overall quality of life. Generic PROMs such as the 36-Item Short Form Health Survey (SF-36), WHO Disability Assessment Schedule (WHODAS 2.0), or Patient-Reported Outcomes Measurement Information System (PROMIS) measures could be considered to measure commonly relevant PROs, supplemented with disease-specific PROMs where needed. However, none of the existing diabetes-specific PROM scales has been sufficiently validated, although the Diabetes Symptom Self-Care Inventory (DSSCI) for measuring diabetes-specific symptoms and the Diabetes Distress Scale (DDS) and Problem Areas in Diabetes (PAID) for measuring distress showed sufficient content validity. Standardisation and use of relevant PROs and psychometrically sound PROMs can help inform people with diabetes about the expected course of disease and treatment, for shared decision making, to monitor outcomes and to improve healthcare. We recommend further validation studies of diabetes-specific PROMs that have sufficient content validity for measuring disease-specific symptoms and consider generic item banks developed based on item response theory for measuring commonly relevant PROs.

System-wide integration of patient reported outcome measure collection through an electronic medical record system: A state-wide retrospective study

BACKGROUND AND PURPOSE

In spine neurosurgery practice, patient-reported outcome measures (PROMs) are tools used to convey information about a patient's health experience and are an integral component of a clinician's decision-making process as they help guide treatment strategies to improve outcomes and minimize pain. Currently, there is limited research showing effective integration strategies of PROMs into electronic medical records. This study aims to provide a framework for other healthcare systems by outlining the process from start to finish in seven Hartford Healthcare Neurosurgery outpatient spine clinics throughout the state of Connecticut.

METHODS

On March 1, 2021, a pilot implementation program began in one clinic and on July 1, 2021, all outpatient clinics were implementing the revised clinical workflow that included the electronic collection of PROMs within the electronic health record (EHR). A retrospective chart analysis studied all adult (18+) new patient visits in seven outpatient clinics by comparing the rates of PROMs collection in Half 1 (March 1, 2021-August 31, 2022) and in Half 2 (September 1, 2022-February 28, 2022) across all sites. Additionally, patient characteristics were studied to identify any variables that may lead to higher rates of collection.

RESULTS

During the study period, 3528 new patient visits were analyzed. There was a significant change in rates of PROMs collection across all departments between H1 and H2 (p < 0.05). Additional significant predictors for PROMs collection were the sex and ethnicity of the patient as well as the provider type for the visit (p < 0.05).

CONCLUSIONS

This study proved that implementing the electronic collection of PROMs into an already existing clinical workflow reduces previously identified collection barriers and enables PROMs collection rates that meet or exceed current benchmarks. Our results provide a successful step-by-step framework for other spine neurosurgery clinics to implement a similar approach.

Prevalence, Severity, and Co-Occurrence of SPPADE Symptoms in 31,866 Patients With Cancer

OBJECTIVES

To examine the prevalence, severity, and co-occurrence of SPPADE symptoms as well as their association with cancer type and patient characteristics.

BACKGROUND

The SPPADE symptoms (sleep disturbance, pain, physical function impairment, anxiety, depression, and low energy /fatigue) are prevalent, co-occurring, and undertreated in oncology and other clinical populations.

METHODS

Baseline SPPADE symptom data were analyzed from the E2C2 study, a stepped wedge pragmatic, population-level, cluster randomized clinical trial designed to evaluate a guideline-informed symptom management model targeting the six SPPADE symptoms. Symptom prevalence and severity were measured with a 0-10 numeric rating (NRS) scale for each of the six symptoms. Prevalence of severe (NRS ≥ 7) and potential clinically relevant (NRS ≥ 5) symptoms as well as co-occurrence of clinical symptoms were determined. Distribution-based methods were used to estimate the minimally important difference (MID). Associations of cancer type and patient characteristics with a SPPADE composite score were analyzed.

RESULTS

A total of 31,886 patients were assessed for SPPADE symptoms prior to, during, or soon after an outpatient medical oncology encounter. The proportion of patients with a potential clinically relevant symptom ranged from 17.5% for depression to 33.4% for fatigue. Co-occurrence of symptoms was high, with the proportion of patients with three or more additional clinically relevant symptoms ranging from 45.2% for fatigue to 68.6% for depression. The summed SPPADE composite score demonstrated good internal reliability (Cronbach's alpha of 0.86), with preliminary MID estimates of 4.1-4.3. Symptom burden differed across several types of cancer but was generally similar across most sociodemographic characteristics.

CONCLUSION

The high prevalence and co-occurrence of SPPADE symptoms in patients with all types of cancer warrants clinical approaches that optimize detection and management.

Digital self-report instruments for repeated measurement of mental health in the general adult population: a protocol for a systematic review

INTRODUCTION

Digital technologies present tremendous opportunities for enabling long-term measurement of mental health in the general population. Emerging studies have established preliminary efficacy of collecting self-report data digitally. However, a key challenge when developing a new self-report instrument is navigating the abundance of existing instruments to select relevant constructs for measurements. This review is a precursor to developing a novel future integrated digital instrument for repeated measurements. We interrogate the literature as the first step towards optimal measurement of the multifaceted mental health concept, in the context of digital repeated measurement. This review aims to identify (1) digital self-report instruments administered repeatedly to measure the mental health of the general adult population; (2) their structure and format; (3) their psychometric properties; (4) their usage in empirical studies; and (5) the constructs these instruments were designed to measure (as characterised in the original publication), and the constructs the instruments have been used to measure in the identified empirical studies.

METHODS AND ANALYSIS

Five major electronic databases will be searched. Studies administering mental health instruments (in English) repeatedly to community dwellers in the general adult population are eligible. A reviewer will preliminarily screen for eligible studies. Then, two reviewers will independently screen the full text of the eligible articles and extract data. Both reviewers will resolve any disagreement through discussion or with a third reviewer. After the data extraction, a reviewer will manually search for the structure, format, psychometric properties and the original constructs these instruments were developed to measure. This review will synthesise the results in a narrative approach. The reporting in this review will be guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist.

ETHICS AND DISSEMINATION

Ethical approval is not required as no data will be collected. Findings of the systematic review will be disseminated through peer-reviewed publications and conference presentations.

PROSPERO REGISTRATION NUMBER

CRD42022306547.

Are Ecological Momentary Assessments of Pain Valid and Reliable? A Systematic Review and Meta-analysis

OBJECTIVES

This systematic review and meta-analysis aimed to determine the level of evidence for the psychometric properties of Ecological Momentary Assessment (EMA) in populations with persistent pain.

MATERIALS AND METHODS

Five databases were searched from 1980 to December 2021. Two reviewers independently screened the titles, abstracts, and full text, extracted data, and assessed adherence to reporting standards and methodological rigor before evaluating the quality of evidence. A meta-analysis, including the pooling of correlations for the relevant EMA pain outcomes, was completed.

RESULTS

Overall, 3270 studies were identified, with 14 studies meeting inclusion criteria. Meta-analyses confirmed good to excellent relationships demonstrated between EMA and recalled pain intensity and interference across different timeframes. Many of the included studies did not fully adhere to recommended reporting standards, and the quality of included studies was either doubtful or inadequate due to methodological flaws. The level of evidence for measurement properties of pain outcomes was low for the criterion validity of pain intensity and interference and very low for reliability and construct validity of pain intensity and interference.

DISCUSSION

Ecological momentary assessment of pain experience appears both valid and reliable. Although the levels of evidence were low or very low, these findings provide preliminary support for the use of EMA in clinical practice and research settings. Potential strengths of EMA include providing measures with greater ecological validity while also reducing recall bias, both pertinent in pain outcome measurement. More research, including higher-quality studies, is needed to demonstrate further support for EMA, including the need for establishing other types of validity.

Score equivalence of paper-, tablet-, and interactive voice response system-based versions of PROMIS, PRO-CTCAE, and numerical rating scales among cancer patients

Background

The study tests the effects of data collection modes on patient responses associated with the multi-item measures such as Patient-Reported Outcomes Measurement System (PROMIS^®), and single-item measures such as Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE), and Numerical Rating Scale (NRS) measures.

Methods

Adult cancer patients were recruited from five cancer centers and administered measures of anxiety, depression, fatigue, sleep disturbance, pain intensity, pain interference, ability to participate in social roles and activities, global mental and physical health, and physical function. Patients were randomized to complete the measures on paper (595), interactive voice response (IVR, 596) system, or tablet computer (589). We evaluated differential item functioning (DIF) by method of data collection using the R software package, lordif. For constructs that showed no DIF, we concluded equivalence across modes if the equivalence margin, defined as ± 0.20 × pooled SD, completely surrounds 95% confidence intervals (CI's) for difference in mean score. If the 95% CI fell totally outside the equivalence margin, we concluded systematic score difference by modes. If the 95% CI partly overlaps the equivalence margin, we concluded neither equivalence nor difference.

Results

For all constructs, no DIF of any kind was found for the three modes. The scores on paper and tablet were more comparable than between IVR and other modes but none of the 95% CI’s were completely outside the equivalence margins, in which we established neither equivalence nor difference. Percentages of missing values were comparable for paper and tablet modes. Percentages of missing values were higher for IVR (2.3% to 6.5% depending on measures) compared to paper and tablet modes (0.7% to 3.3% depending on measures and modes), which was attributed to random technical difficulties experienced in some centers.

Conclusion

Across all mode comparisons, there were some measures with CI’s not completely contained within the margin of small effect. Two visual modes agreed more than visual-auditory pairs. IVR may induce differences in scores unrelated to constructs being measured in comparison with paper and tablet. The users of the surveys should consider using IVR only when paper and computer administration is not feasible.

Validity arguments for patient-reported outcomes: justifying the intended interpretation and use of data

BACKGROUND

Contrary to common usage in the health sciences, the term "valid" refers not to the properties of a measurement instrument but to the extent to which data-derived inferences are appropriate, meaningful, and useful for intended decision making. The aim of this study was to determine how validity testing theory (the Standards for Educational and Psychological Testing) and methodology (Kane's argument-based approach to validation) from education and psychology can be applied to validation practices for patient-reported outcomes that are measured by instruments that assess theoretical constructs in health.

METHODS

The Health Literacy Questionnaire (HLQ) was used as an example of a theory-based self-report assessment for the purposes of this study. Kane's five inferences (scoring, generalisation, extrapolation, theory-based interpretation, and implications) for theoretical constructs were applied to the general interpretive argument for the HLQ. Existing validity evidence for the HLQ was identified and collated (as per the Standards recommendation) through a literature review and mapped to the five inferences. Evaluation of the evidence was not within the scope of this study.

RESULTS

The general HLQ interpretive argument was built to demonstrate Kane's five inferences (and associated warrants and assumptions) for theoretical constructs, and which connect raw data to the intended interpretation and use of the data. The literature review identified 11 HLQ articles from which 57 sources of validity evidence were extracted and mapped to the general interpretive argument.

CONCLUSIONS

Kane's five inferences and associated warrants and assumptions were demonstrated in relation to the HLQ. However, the process developed in this study is likely to be suitable for validation planning for other measurement instruments. Systematic and transparent validation planning and the generation (or, as in this study, collation) of relevant validity evidence supports developers and users of PRO instruments to determine the extent to which inferences about data are appropriate, meaningful and useful (i.e., valid) for intended decisions about the health and care of individuals, groups and populations.