Comparing methods for assessing the reliability of health care quality measures

Quality measurement plays an increasing role in U.S. health care. Measures inform quality improvement efforts, public reporting of variations in quality of care across providers and hospitals, and high-stakes financial decisions. To be meaningful in these contexts, measures should be reliable and not heavily impacted by chance variations in sampling or measurement. Several different methods are used in practice by measure developers and endorsers to evaluate reliability; however, there is uncertainty and debate over differences between these methods and their interpretations. We review methods currently used in practice, pointing out differences that can lead to disparate reliability estimates. We compare estimates from 14 different methods in the case of two sets of mental health quality measures within a large health system. We find that estimates can differ substantially and that these discrepancies widen when sample size is reduced.

Split-sample reliability estimation in health care quality measurement: Once is not enough

OBJECTIVE

To examine the sensitivity of split-sample reliability estimates to the random split of the data and propose alternative methods for improving the stability of the split-sample method.

DATA SOURCES AND STUDY SETTING

Data were simulated to reflect a variety of real-world quality measure distributions and scenarios. There is no date range to report as the data are simulated.

STUDY DESIGN

Simulation studies of split-sample reliability estimation were conducted under varying practical scenarios.

DATA COLLECTION/EXTRACTION METHODS

All data were simulated using functions in R.

PRINCIPAL FINDINGS

Single split-sample reliability estimates can be very dependent on the random split of the data, especially in low sample size and low variability settings. Averaging split-sample estimates over many splits of the data can yield a more stable reliability estimate.

CONCLUSIONS

Measure developers and evaluators using the split-sample reliability method should average a series of reliability estimates calculated from many resamples of the data without replacement to obtain a more stable reliability estimate.

The reliability and validity of lung cancer and melanoma clinical quality survival measures

OBJECTIVE

To develop a risk adjustment approach and test reliability and validity for oncology survival measures.

DATA SOURCES AND STUDY SETTING

We used the National Cancer Institute's Surveillance, Epidemiology, and End Results (SEER)-Medicare data from 2010 to 2013, with mortality data through 2015.

STUDY DESIGN

We developed 2-year risk-standardized survival rates (RSSR) for melanoma, non-small cell lung cancer (NSCLC), and small cell lung cancer (SCLC). Patients were attributed to group practices based on the plurality of visits. We identified the risk-adjustment variables via bootstrap and calculated the RSSRs. Reliability was tested via three approaches: (1) signal-to-noise ratio (SNR) reliability, (2) split-half, and (3) test-retest using bootstrap. We tested known group validity by stage at diagnosis using Cohen's d.

DATA COLLECTION/EXTRACTION METHODS

We selected all patients enrolled in Medicare and linked to SEER during the measurement period with an incident first primary diagnosis of stage I-IV melanoma, NSCLC, or SCLC. We excluded patients with missing data on month and/or stage of diagnosis.

PRINCIPAL FINDINGS

Results are based on patients with melanoma (n = 4344); NSCLC (n = 16,080); and SCLC (n = 2807) diagnosed between 2012 and 2013. The median (interquartile range) for the RSSRs at the group practice-level were 0.89 (0.83-0.87) for melanoma, 0.37 (0.30-0.43) for NSCLC, and 0.19 (0.11-0.25) for SCLC. C-statistics for the models ranged from 0.725 to 0.825. The reliability varied by approach with median SNR 0.20, 0.25, and 0.13; median test-retest 0.59, 0.57, and 0.56; median split-half reliability 0.21, 0.29, and 0.29 for melanoma, NSCLC, and SCLC, respectively. Cohen's d for stage I-IIIa and IIIb+ was 1.27, 0.86, 0.60 for melanoma, NSCLC, and SCLC, respectively.

CONCLUSIONS

Our results suggest that these cancer survival measures demonstrated adequate test-retest reliability and expected findings for the known-group validity analysis. If data limitations and feasibility challenges can be addressed, implementation of these quality measures may provide a survival metric used for oncology quality improvement efforts.

The profile inter-unit reliability

To assess the quality of health care, patient outcomes associated with medical providers (eg, dialysis facilities) are routinely monitored in order to identify poor (or excellent) provider performance. Given the high stakes of such evaluations for payment as well as public reporting of quality, it is important to assess the reliability of quality measures. A commonly used metric is the inter-unit reliability (IUR), which is the proportion of variation in the measure that comes from inter-provider differences. Despite its wide use, however, the size of the IUR has little to do with the usefulness of the measure for profiling extreme outcomes. A large IUR can signal the need for further risk adjustment to account for differences between patients treated by different providers, while even measures with an IUR close to zero can be useful for identifying extreme providers. To address these limitations, we propose an alternative measure of reliability, which assesses more directly the value of a quality measure in identifying (or profiling) providers with extreme outcomes. The resulting metric reflects the extent to which the profiling status is consistent over repeated measurements. We use national dialysis data to examine this approach on various measures of dialysis facilities.

Predictors of instrumental activities of daily living in community-dwelling older adults

Psychiatric mental health clinicians often rely on proxy and self-report evaluations to determine the cognitive function of older adults however, performance measures have greater accuracy and predictive ability for everyday function. This study tested physical and cognitive predictors of functional abilities in fifty-one community residing older adults. We administered a computerized battery of executive function tasks, a performance-based measure of instrumental activities of daily living (IADL), and three physical function measures (grip strength, 30-second Chair Stand Test, and 8-foot Up and Go). Regression models assessed the associations of three components of executive function (updating, shifting, and inhibition) with IADLs and physical functions. Updating was a significant predictor of the Medications and Financial DAFS scores and of grip strength. Shifting also predicted grip strength. In conclusion, different executive functions predict different domains of IADL functioning. Working memory was a robust predictor of IADL functioning in older adults, especially medication management skills.

Performance on Electronic Clinical Quality Measures in the Comprehensive Primary Care Initiative

Electronic clinical quality measures (eCQMs) that capture data from electronic health records promise accurate and timely measurement, but their use has been limited in payment. The Comprehensive Primary Care initiative sponsored by the Centers for Medicare & Medicaid Services used eCQMs as part of a shared savings incentive. To assess performance, the authors developed benchmarks for 11 measures and compared performance of initiative and benchmark practices. Initiative practices outperformed benchmark practices on 8 measures in 2015 (between 6.3 and 17.7 percentage points) and 9 measures in 2016 (between 1.7 and 20 percentage points). Initiative practices improved significantly on 7 measures from 2015 to 2016 (between 3.3 and 8.6 percentage points). For 3 measures, the improvement was greater than benchmark practices that reported the same measures in a 2-year period (between 1 and 8.9 percentage points). The authors conclude that eCQMs can be used for payment.

Deviations in Monthly Staffing and Injurious Assaults Against Staff and Patients on Psychiatric Units

It is widely thought that low staffing levels are associated with higher risk of psychiatric inpatient violence. The purpose of this study was to determine whether odds of an injurious assault are higher in months in which unit staffing levels are higher or lower relative to unit average, using a design allowing each unit to serve as its own control. Using 2011-2013 National Database of Nursing Quality Indicators data from 480 adult and 90 geriatric units in 361 US hospitals, monthly assault odds were modeled as functions of unit staffing. Monthly RN and non-RN staffing (hours per patient day) were categorized as very low, low, average, high, or very high, based on deviation from the unit's average staffing across study months. Endpoints were binary indicators for one or more injurious assaults against staff during the month and for one or more injurious assaults against patients during the month. Despite large sample sizes, neither RN nor non-RN staffing was a statistically significant predictor of odds of assault, nor was there a consistent trend of odds of assault being higher at below- or above-average staffing levels. There was little evidence that monthly deviation in unit staffing is associated with the odds of an injurious assault on a unit. This suggests that staffing-assault rate associations in previous studies of monthly data are largely attributable to between-unit rather than within-unit staffing differences. Hospitals may need to look beyond below- or above-average nurse staffing as a cause of assaults. © 2016 Wiley Periodicals, Inc.