Poor quality patient reported outcome measures bias effect estimates in orthopaedic randomized studies

Psychometric properties of patient-reported outcome measures assessing recovery from hand fractures: a systematic review

PURPOSE

To extensively review Patient Reported Outcome Measures (PROMs) used to assess outcomes in persons with hand fractures in terms of their psychometric properties.

METHODS

A COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) methodological review was conducted. Six electronic databases were searched (Medline, Embase, Scopus, Web of Science, PsycINFO, CINAHL) for studies evaluating the psychometric properties of PROMs assessing recovery from hand fracture. Titles and abstracts, full text review, quality assessment and data extraction were performed by two reviewers. Any disagreements were resolved after review by a third, expert reviewer. Quality assessment of included studies was performed using the COSMIN checklist.

RESULTS

This COSMIN review found that there were only 4 studies that fulfilled the inclusion criteria in terms of assessing measurement properties of PROMs in hand fractures. Only the construct validity of the Disabilities of the Arm, Shoulder, and Hand (DASH), the Quick DASH (QDASH) and the Duruoz Hand Index (DHI), and the responsiveness of the DASH, the Patient Specific Functional Scale (PSFS), the Patient-Rated Wrist and Hand Evaluation (PRWHE) and the Michigan Hand Outcomes Questionnaire (MHQ) were assessed in these studies. The overall rating of the studies was assessed as insufficient or indeterminate and quality of evidence was assessed as moderate, low, or very low by our team.

CONCLUSIONS

This study identified that there is a lack of evidence in the medical literature with regards to the measurement properties of PROMs in patients with hand fractures. The 4 included studies do not provide good quality data to support the use of these PROMS in patients with hand fractures. There is a need for more studies for more PROMs. This has important consequences for how outcomes will be measured in clinical studies in hand research and in clinical practice.

Single-center pilot study of remote therapeutic monitoring in patients with operative spinal pathologies

OBJECTIVES

Spine pathology affects a significant portion of the population, leading to neck and back pain, impacting quality of life, and potentially requiring surgical intervention. Current pre- and postoperative monitoring methods rely on patient reported outcome (PRO) measures and lack continuous objective data on patients' recoveries. Remote therapeutic monitoring (RTM) using wearable devices offers a promising solution to bridge this gap, providing real-time physical function data. This study aims to assess the feasibility and correlation between changes in physical function and daily activity levels using RTM for individuals with operative spinal pathologies.

METHODS

A single-center pilot study involving 21 participants with operative spinal pathologies was conducted at an academic hospital. Participants were provided Bluetooth-enabled Fitbit Inspire 2 activity trackers and asked to wear them daily for 100 days. The Healthcare Recovery Solutions (HRS) mobile application facilitated remote administration of the PROMIS - Physical Function Short Form 6b PROs questionnaire at days 1, 30, and 90. Linear regression, Students' paired T tests, and one-way ANOVA were used to analyze collected data.

RESULTS

Average compliance with RTM was found to be 82.4% compared to only 48% for PROMs. Changes in daily steps were moderately positively correlated with changes in PROs at both 30 and 90 days. Participant satisfaction with RTM was high, and responses indicated greater satisfaction with RTM compared to PROMs.

CONCLUSIONS

RTM offers continuous and objective data collection, presenting a potential solution to the limitations of intermittent clinical assessments and self-reported outcomes. The study demonstrated a moderate correlation between changes in activity levels and changes in PROs, suggesting that RTM data could serve as a surrogate for PROs. Participants' high compliance and satisfaction with RTM underscore its feasibility and potential clinical utility. This study lays the groundwork for larger future investigations into the clinical benefits and broader application of RTM in spine care.

A Comprehensive Approach to PROMs in Elective Orthopedic Surgery: Comparing Effect Sizes across Patient Subgroups

Background: There is limited knowledge regarding the comparative patient-reported outcomes (PROMs) and effect sizes (ESs) across orthopedic elective surgery. Methods: All patient data between January 2020 and December 2022 were collected, and treatment outcomes assessed as a PROM difference between baseline and one-year follow-up. The cohort was divided into subgroups (hand, elbow, shoulder, spine, hip, knee, and foot/ankle). The PROM ESs were calculated for each patient separately, and patients with ES > 0.5 were considered responders. Results: In total, 7695 patients were operated on. The mean ES across all patient groups was 1.81 (SD 1.41), and the largest ES was observed in shoulder patients and the smallest in hand patients. Overall, shoulder, hip, and knee patients had a larger ES compared to hand, spine, and foot/ankle patients (p < 0.0001). The proportion of positive responders ranged between 91-94% in the knee, shoulder, and hip, and 69-70% in the hand, spine, and foot/ankle subgroups. Conclusions: The ESs are generally high throughout elective orthopedic surgery. However, based on our institutional observations, shoulder, hip, and knee patients experience larger treatment effects compared to hand, spine, and foot/ankle patients, among whom there are also more non-responders. The expected treatment outcomes should be clearly communicated to patients when considering elective surgery. Because of the study limitations, the results should be approached with some caution.

A scoping review of Rasch analysis and item response theory in otolaryngology: Implications and future possibilities

Objective

Item response theory (IRT) is a methodological approach to studying the psychometric performance of outcome measures. This study aims to determine and summarize the use of IRT in otolaryngological scientific literature.

Methods

A systematic search of the Medline, Embase, and the Cochrane Library databases was performed for original English-language published studies indexed up to January 28, 2023, per the following search strategy: ("item response theory" OR "irt" OR "rasch" OR "latent trait theory" OR "modern mental test theory") AND ("ent" OR "otorhinolaryngology" OR "ear" OR "nose" OR "throat" OR "otology" OR "audiology" OR "rhinology" OR "laryngology" OR "neurotology" OR "facial plastic surgery").

Results

Fifty-five studies were included in this review. IRT was used across all subspecialties in otolaryngology, and most studies utilizing IRT methodology were published within the last decade. Most studies analyzed polytomous response data, and the most commonly used IRT models were the partial credit and the rating scale model. There was considerable heterogeneity in reporting the main assumptions and results of IRT.

Conclusion

IRT is increasingly being used in the otolaryngological scientific literature. In the otolaryngology literature, IRT is most frequently used in the study of patient-reported outcome measures and many different IRT-based methods have been used. Future IRT-based outcome studies, using standardized reporting guidelines, might improve otolaryngology-outcome research sustainably by improving response rates and reducing patient response burden.

Level of evidence

2.

Evaluating Factors Associated With Patient-Perceived Applicability of the QuickDASH Survey in Reflecting Their Goals of Care Prior to Hand Surgery

PURPOSE

Patient-reported outcome measures intend to capture patients' perspectives on their health status. However, the patient-perceived applicability of many of these patient-reported outcome measures is unknown. We hypothesized that patients experiencing greater upper extremity disability and greater pain interference would be more likely to report that the Quick Disabilities of the Arm, Shoulder, and Hand questionnaire (QuickDASH) survey content is responsive to their daily lives and goals in seeking surgical care.

METHODS

Adult preoperative hand surgery patients at a single tertiary academic center were recruited prospectively. QuickDASH, Patient-Reported Outcomes Measurement Information System Pain Interference computerized-adaptive-testing, and the Godin Leisure-Time Exercise Questionnaire (GLTEQ)-a validated adult physical activity level metric-data were collected. The following two Likert response questions were also asked: question (1) "How applicable is the above questionnaire to your treatment goals for your upper extremity condition?" and question (2) "How applicable is the overall questionnaire to your daily life?" Multivariable binary logistic regression was performed to define the factors associated with patients reporting that the survey was "very applicable."

RESULTS

Of the 133 included patients, the mean age was 49 ± 18 years, 40% were women, and the mean GLTEQ score was 54.1 ± 5.5 (consistent with a high level of activity). For questions 1 and 2, 32% and 29% of the patients reported that QuickDASH was "very applicable," respectively. The multivariable model demonstrated that for every 10-point increase in QuickDASH, there was a 45% to 49% greater odds of respondents reporting that the survey was "very applicable," and for every 5-point increase in pain interference computerized-adaptive-testing, the odds increased by 55% to 70%. No association with GLTEQ was observed.

CONCLUSIONS

Patients with greater upper extremity disability and pain interference were more likely to find the QuickDASH content to be applicable to their daily lives and goals in seeking surgical care.

CLINICAL RELEVANCE

These findings suggest that QuickDASH may not be an optimal instrument when evaluating upper extremity function in cohorts with mild disability and low pain interference.

Continuous data capture of gait and mobility metrics using wearable devices for postoperative monitoring in common elective orthopaedic procedures of the hip, knee, and spine: a scoping review

BACKGROUND

Surgical procedures involving the hip, knee, or spine represent a majority of orthopaedic procedures performed electively in the health care system. Postoperative care is a key aspect of surgery and mobilisation without injury is the primary objective. Recent advances in wearable technologies allow objective evaluation of walking metrics to inform and guide postoperative care following orthopaedic surgery.

PURPOSE

The aim of this scoping review is to explore current applications of wearable devices, objective data capture and gait analysis in monitoring postoperative recovery following commonly performed elective orthopaedic procedures of the hip, knee and spine.

METHODS

A search against pre-defined criteria was performed on the following scientific databases from date of inception to February 28th, 2021: Medline (via OvidSP), Embase (via OvidSP) and Cochrane Library (via CENTRAL). Data were collected according to a predetermined checklist including study participants, surgery, wearable device (model), sensor location, and monitoring parameters such as mobility metrics, monitoring timepoints and monitoring duration for each study included in our review. Quality was assessed independently using the Newcastle Ottawa Scale (NOS).

CONCLUSIONS

To our knowledge, this is the first review of wearable monitoring (of postoperative recovery) following hip, knee and spine surgery. Patients undergoing elective orthopaedic procedures may benefit from wearable monitoring of their walking health and mobility metrics.

Presentation approaches for enhancing interpretability of patient-reported outcomes in meta-analyses: a systematic survey of Cochrane reviews

OBJECTIVES

To systematically survey Cochrane reviews' approaches to calculating, presenting, and interpreting pooled estimates of patient-reported outcome measures (PROMs).

STUDY DESIGN AND SETTING

We retrospectively selected 200 Cochrane reviews that met the eligibility criteria. Two researchers independently extracted the pooled effect measures and approaches for pooling and interpreting the effect measures, reaching consensus through discussions.

RESULTS

When primary studies used the same PROM, Cochrane review authors most often used mean differences (MDs) (81.9%) for calculating the pooled effect measures; when primary studies used different PROMs, the review authors often applied standardized mean differences (SMDs) (54.3%). Although in most cases (80.1%) the review authors interpreted the importance of effect, they failed, in 48.5% of the pooled effect measures, to report criteria for categorizing the magnitude of effect. When authors interpreted the importance of the effect, for those with primary studies using the same PROM, they most often referred to the minimally important differences (MIDs) (75.0%); for those with primary studies using different PROMs, the approaches used varied.

CONCLUSION

Cochrane review authors most often used MDs or SMDs for calculating and presenting the pooled effect measures of PROs but often failed to make explicit their criteria for categorizing the magnitude of effect.

Patient-Reported Outcomes Measurement Information System Is Not as Responsive as Legacy Scores in Detecting Patient Outcomes in Hip Preservation: A Systematic Review

PURPOSE

To evaluate publication trends of Patient-Reported Outcomes Measurement Information System (PROMIS) in hip preservation literature, assess the usage of PROMIS as an outcome measure, and evaluate correlations of all available published PROMIS domains with legacy patient-reported outcomes (PROs).

METHODS

The PubMed, Scopus, Cochrane Central Register of Controlled Trials, and Google Scholar databases were queried for articles evaluating PROMIS scores among hip preservation populations. Inclusion criteria consisted of studies with Level IV evidence or above (per the Sackett et al. levels of evidence), such as case series and cohort studies, reporting on perioperative use of hip PROMIS scores. Exclusion criteria consisted of arthroplasty and trauma studies. Patient demographics, PROMIS usage, and PROMIS Pearson or Spearman correlation coefficients to historic PROs were recorded for each study.

RESULTS

Fifteen articles published between 2017 and 2021 were included in the analysis, with the majority (75%) published between 2020 and 2021. Studies assessing postoperative outcomes had follow-up periods ranging from 6 months to 5 years. The most common PROMIS domain reported was Physical Function (PF), and there was varying usage of other domains including Pain Intensity, Anxiety, and Depression. PROMIS validity was most often assessed in comparison to the modified Harris Hip Score (mHHS) by calculating the Pearson coefficient, which assumes normal data distribution, or Spearman coefficient, which is rank-based and does not require normal data distribution. Studies comparing PROMIS-PF with mHHS reported Pearson coefficients ranging from 0.49 to 0.72 and Spearman coefficients ranging from 0.67 to 0.71.

CONCLUSIONS

There has been a chronologic increase in PROMIS usage in hip preservation literature. PROMIS demonstrates moderate-to-strong correlations with legacy PROs, but there is substantial heterogeneity in follow-up periods, PROMIS domains used, and statistical methodology. The current data show that PROMIS is not as responsive as historically used, validated PROs in quantitatively assessing function and pain in hip preservation patients.

CLINICAL RELEVANCE

Surgeons using PROMIS solely should be aware that the score may not be as responsive as legacy PROs in closely assessing improvements or deterioration in patient performance after hip preservation surgery. Rather than being used alone, PROMIS may be useful as a replacement for a group of legacy PROs. Thus, when used alongside select legacy PROs, overall questionnaire burden can be reduced while maintaining a high level of accuracy in assessing health status.

The concept of recovery kinetics: an observational study of continuous post-operative monitoring in spine surgery

BACKGROUND

The spine surgeon's understanding of an individual patient's burden of disease and functional disability in daily life is shaped by patient-reported outcome measures (PROMs). Although PROMs are useful in understanding the patient's perception of their disease, the use of PROMs constitutes a "snapshot" approach of single timepoint data capture, omitting day-to-day fluctuations in functional status. We introduce the concept of kinetics when considering continuous and objective postoperative patient monitoring with wearable sensors.

METHODS

A prospective single-centre series was performed using patients either undergoing lumbar decompression for lumbar spinal stenosis (LSS) (n=12), or posterior lumbar fusion for degenerative spondylolisthesis (n=12). The Oswestry Disability Index (PROM) was conducted preoperatively and 12-weeks postoperatively. During this timeframe, continuous measurements of step count and distance travelled were made using a wrist-based wearable accelerometer.

RESULTS

Over the 12-week study period, mean daily step count for all participants improved from 4,700 to 7,700 steps per day (P=0.013), following an initial dip in total steps taken. The mean daily distance travelled improved from 3,300 to 5,300 meters per day (P=0.003). Decompression group recovered at a faster rate than the fusion group.

CONCLUSIONS

Although overall improvement was similar between the decompression and fusion groups, the recovery kinetics varied. The recovery kinetics approach of continuous postoperative monitoring provides additional insight to postoperative patient progress.

Measurement Properties of the Patient-Specific Functional Scale and Its Current Uses: An Updated Systematic Review of 57 Studies Using COSMIN Guidelines

OBJECTIVE

To systematically review measurement properties, including acceptability, feasibility, and interpretability, and current uses of the Patient-Specific Functional Scale (PSFS).

DESIGN

Systematic review of a patient-reported outcome measure using the COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN) guidelines.

LITERATURE SEARCH

We searched 11 databases from January 2010 to July 2020 for articles on measurement properties or use of PSFS.

STUDY SELECTION CRITERIA

Published primary articles without language restrictions.

DATA SYNTHESIS

Two independent reviewers screened all records, extracted data, and performed risk of bias assessments using COSMIN guidelines. We qualitatively synthesized findings for each measurement property in musculoskeletal and nonmusculoskeletal conditions, and 2 reviewers independently performed Grading of Recommendations Assessment, Development and Evaluation assessments. This study was preregistered with the Open Science Framework (https://doi.org/10.17605/OSF.IO/42UZT).

RESULTS

Of the 985 articles screened, we included 57 articles on measurement properties and 255 articles on the use of PSFS. The PSFS had sufficient test-retest reliability in musculoskeletal (22 studies, 845 participants, low-to-moderate certainty) and nonmusculoskeletal conditions (6 studies, 197 participants, very low certainty), insufficient construct validity as a measure of physical function (21 studies, 2 945 participants, low-to-moderate certainty), and sufficient responsiveness (32 studies, 13 770 participants, moderate-to-high certainty). The standard error of measurement ranged from 0.35 to 1.5. The PSFS was used in 87 unique health conditions, some without prior evidence of validity.

CONCLUSION

The PSFS is an easy-to-use, reliable, and responsive scale in numerous musculoskeletal conditions, but the construct validity of PSFS remains uncertain. Further study of the measurement properties of the PSFS in nonmusculoskeletal conditions is necessary before clinical use. J Orthop Sports Phys Ther 2022;52(5):262-275. Epub: 05 Feb 2022. doi:10.2519/jospt.2022.10727.

A review of the content and psychometric properties of cancer-related fatigue (CRF) measures used to assess fatigue in intervention studies

PURPOSE

Cancer-related fatigue (CRF) is a common and debilitating consequence of cancer and its treatment. Numerous supportive care interventions have been developed to alleviate CRF; however, the diversity of outcome measures used to assess CRF limits comparability of findings. We aimed to evaluate the content and psychometric properties of measures used to assess CRF in interventions targeting fatigue, to inform the selection of suitable measures in future research.

METHODS

Included measures were identified from a systematic review of interventions targeting CRF. General characteristics of each measure were extracted, and item content was assessed against domains specified by the National Comprehensive Cancer Network (NCCN) definition of CRF. Psychometric properties were evaluated against COnsensus-based Standards for the selection of heath Measurement INstruments (COSMIN) criteria.

RESULTS

Of 54 measures identified, 25 met inclusion criteria. Seventeen were fatigue-specific and eight a fatigue subscale or single item within a broader measure. Only 14 (56%) were specifically developed for cancer populations. Content coverage according to the NCCN CRF definition ranged from 0 to 75%. Evidence for fulfilment of COSMIN criteria in cancer populations ranged from 0 to 93%, with only five measures meeting > 70% of the COSMIN criteria.

CONCLUSION

The Piper Fatigue Scale-Revised had good content coverage, but did not comprehensively address COSMIN criteria. The EORTC-FA12 and FACIT/FACT-F had excellent psychometric properties, with each capturing different aspects of fatigue. Ultimately, the choice of CRF measure should be guided by the research question and the CRF domains most relevant to the particular research context.

A review of trauma and orthopaedic randomised clinical trials published in high-impact general medical journals

Introduction

The recent past has seen a significant increase in the number of trauma and orthopaedic randomised clinical trials published in “the big five” general medical journals. The quality of this research has, however, not yet been established.

Methods

We therefore set out to critically appraise the quality of available literature over a 10-year period (April 2010–April 2020) through a systematic search of these 5 high-impact general medical journals (JAMA, NEJM, BMJ, Lancet and Annals). A standardised data extraction proforma was utilised to gather information regarding: trial design, sample size calculation, results, study quality and pragmatism. Quality assessment was performed using the Cochrane Risk of Bias 2 tool and the modified Delphi list. Study pragmatism was assessed using the PRECIS-2 tool.

Results

A total of 25 studies were eligible for inclusion. Over half of the included trials did not meet their sample size calculation for the primary outcome, with a similar proportion of these studies at risk of type II error for their non-significant results. There was a high degree of pragmatism according to PRECIS-2. Non-significant studies had greater pragmatism that those with statistically significant results (p < 0.001). Only 56% studies provided adequate justification for the minimum clinically important difference (MCID) in the population assessed. Overall, very few studies were deemed high quality/low risk of bias.

Conclusions

These findings highlight that there are some important methodological concerns present within the current evidence base of RCTs published in high-impact medical journals. Potential strategies that may improve future trial design are highlighted.

Level of evidence

Level 1.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00590-021-03137-3.

Asthma patients experience increased symptoms of anxiety, depression and fear during the COVID-19 pandemic

Population studies showed a decrease in psychological wellbeing during the COVID-19 pandemic. Asthma is associated with a negative effect on anxiety and depression, which might worsen during the COVID-19 lockdown. The aim of the study was to compare fear, anxiety and depression between asthma patients and patients wit hout asthma pre-COVID-19 and during COVID-19 pandemic.

This study compares fear, anxiety and depression in asthma patients and controls between pre-COVID-19 and during COVID-19 lockdown with a cross-sectional online survey. Participants were invited to fill out several questionnaires pertaining to fear, anxiety, depression, asthma control and quality of life.

Asthma patients (N = 37) displayed, during the course of the pandemic, a clinically relevant increase in anxiety (3.32 ± 2.95 vs. 6.68 ± 3.78; p < 0.001) and depression (1.30 ± 1.15 vs. 3.65 ± 3.31; p < 0.001), according to the hospital anxiety and depression levels (HADS) compared to pre-COVID-19 assessment. This was not seen in controls. Also, asthma patients displayed more anxiety about acquiring COVID-19 disease compared to controls ((5.11 ± 1.99 vs. 3.50 ± 2.79), p = 0.006).

Patients with asthma experienced an increase in anxiety and depression levels and were more afraid of acquiring COVID-19 disease compared to controls. Also, patients with asthma were more likely to avoid healthcare facilities due to fear of acquiring COVID-19 disease compared to controls. Therefore, we advise health care workers to address these possible negative effects on mental health by phone or e-consults.

Personalized quantification of facial normality: a machine learning approach

What is a normal face? A fundamental task for the facial reconstructive surgeon is to answer that question as it pertains to any given individual. Accordingly, it would be important to be able to place the facial appearance of a patient with congenital or acquired deformity numerically along their own continuum of normality, and to measure any surgical changes against such a personalized benchmark. This has not previously been possible. We have solved this problem by designing a computerized model that produces realistic, normalized versions of any given facial image, and objectively measures the perceptual distance between the raw and normalized facial image pair. The model is able to faithfully predict human scoring of facial normality. We believe this work represents a paradigm shift in the assessment of the human face, holding great promise for development as an objective tool for surgical planning, patient education, and as a means for clinical outcome measurement.