Alternative pay-for-performance scoring methods: implications for quality improvement and patient outcomes

Composite measures of quality of health care: Evidence mapping of methodology and reporting

Background

Quality indicators are used to quantify the quality of care. A large number of quality indicators makes assessment of overall quality difficult, time consuming and impractical. There is consequently an increasing interest for composite measures based on a combination of multiple indicators.

Objective

To examine the use of different approaches to construct composite measures of quality of care and to assess the use of methodological considerations and justifications.

Methods

We conducted a literature search on PubMed and EMBASE databases (latest update 1 December 2020). For each publication, we extracted information on the weighting and aggregation methodology that had been used to construct composite indicator(s).

Results

A total of 2711 publications were identified of which 145 were included after a screening process. Opportunity scoring with equal weights was the most used approach (86/145, 59%) followed by all-or-none scoring (48/145, 33%). Other approaches regarding aggregation or weighting of individual indicators were used in 32 publications (22%). The rationale for selecting a specific type of composite measure was reported in 36 publications (25%), whereas 22 papers (15%) addressed limitations regarding the composite measure.

Conclusion

Opportunity scoring and all-or-none scoring are the most frequently used approaches when constructing composite measures of quality of care. The attention towards the rationale and limitations of the composite measures appears low.

Discussion

Considering the widespread use and the potential implications for decision-making of composite measures, a high level of transparency regarding the construction process of the composite and the functionality of the measures is crucial.

Learning From England's Best Practice Tariff: Process Measure Pay-for-Performance Can Improve Hip Fracture Outcomes

OBJECTIVE

The objective of this study was to evaluate England's Best Practice Tariff (BPT) and consider potential implications for Medicare patients should the US adopt a similar plan.

SUMMARY BACKGROUND DATA

Since the beginning of the Affordable Care Act, Medicare has renewed efforts to improve the outcomes of older adults through introduction of an expanding set of alternative-payment models. Among trauma patients, recommended arrangements met with mixed success given concerns about the heterogeneous nature of trauma patients and resulting outcome variation. A novel approach taken for hip fractures in England could offer a viable alternative.

METHODS

Linear regression, interrupted time-series, difference-in-difference, and counterfactual models of 2000 to 2016 Medicare (US), HES-APC (England) death certificate-linked claims (≥65 years) were used to: track US hip fracture trends, look at changes in English hip fracture trends before-and-after BPT implementation, compare changes in US-versus-English mortality, and estimate total/theoretical lives saved.

RESULTS

A total of 806,036 English and 3,221,109 US hospitalizations were included. After BPT implementation, England's 30-day mortality decreased by 2.6 percentage-points (95%CI: 1.7-3.5) from a baseline of 9.9% (relative reduction 26.3%). 90- and 365-day mortality decreased by 5.6 and 5.4 percentage-points. 30/90/365-day readmissions also declined with a concurrent shortening of hospital length-of-stay. From 2000 to 2016, US outcomes were stagnant (P > 0.05), resulting in an inversion of the countries' mortality and >38,000 potential annual US lives saved.

CONCLUSIONS

Process measure pay-for-performance led to significant improvements in English hip fracture outcomes. As efforts to improve US older adult health continue to increase, there are important lessons to be learned from a successful initiative like the BPT.

Do Health Care Delivery System Reforms Improve Value? The Jury Is Still Out

BACKGROUND

Widespread restructuring of health delivery systems is underway in the United States to reduce costs and improve the quality of health care.

OBJECTIVE

To describe studies evaluating the impact of system-level interventions (incentives and delivery structures) on the value of US health care, defined as the balance between quality and cost.

RESEARCH DESIGN

We identified articles in PubMed (2003 to July 2014) using keywords identified through an iterative process, with reference and author tracking. We searched tables of contents of relevant journals from August 2014 through 11 August 2015 to update our sample.

SUBJECTS

We included prospective or retrospective studies of system-level changes, with a control, reporting both quality and either cost or utilization of resources.

MEASURES

Data about study design, study quality, and outcomes was extracted by one reviewer and checked by a second.

RESULTS

Thirty reports of 28 interventions were included. Interventions included patient-centered medical home implementations (n=12), pay-for-performance programs (n=10), and mixed interventions (n=6); no other intervention types were identified. Most reports (n=19) described both cost and utilization outcomes. Quality, cost, and utilization outcomes varied widely; many improvements were small and process outcomes predominated. Improved value (improved quality with stable or lower cost/utilization or stable quality with lower cost/utilization) was seen in 23 reports; 1 showed decreased value, and 6 showed unchanged, unclear, or mixed results.Study limitations included variability among specific endpoints reported, inconsistent methodologies, and lack of full adjustment in some observational trials. Lack of standardized MeSH terms was also a challenge in the search.

CONCLUSIONS

On balance, the literature suggests that health system reforms can improve value. However, this finding is tempered by the varying outcomes evaluated across studies with little documented improvement in outcome quality measures. Standardized measures of value would facilitate assessment of the impact of interventions across studies and better estimates of the broad impact of system change.

Composite Measures of Health Care Provider Performance: A Description of Approaches

CONTEXT

Since the Institute of Medicine's 2001 report Crossing the Quality Chasm, there has been a rapid proliferation of quality measures used in quality-monitoring, provider-profiling, and pay-for-performance (P4P) programs. Al-though individual performance measures are useful for identifying specific processes and outcomes for improvement and tracking progress, they do not easily provide an accessible overview of performance. Composite measures aggregate individual performance measures into a summary score. By reducing the amount of data that must be processed, they facilitate (1) benchmarking of an organization's performance, encouraging quality improvement initiatives to match performance against high-performing organizations, and (2) profiling and P4P programs based on an organization's overall performance.

METHODS

We describe different approaches to creating composite measures,discuss their advantages and disadvantages, and provide examples of their use.

FINDINGS

The major issues in creating composite measures are (1) whether to aggregate measures at the patient level through all-or-none approaches or the facility level, using one of the several possible weighting schemes; (2) when combining measures on different scales, how to rescale measures (using z scores,range percentages, ranks, or 5-star categorizations); and (3) whether to use shrinkage estimators, which increase precision by smoothing rates from smaller facilities but also decrease transparency.

CONCLUSIONS

Because provider rankings and rewards under P4P programs may be sensitive to both context and the data, careful analysis is warranted before deciding to implement a particular method. A better understanding of both when and where to use composite measures and the incentives created by composite measures are likely to be important areas of research as the use of composite measures grows.

Factors related to clinical quality improvement for small practices using an EHR

OBJECTIVE

To analyze the impact of three primary care practice transformation program models on performance: Meaningful Use (MU), Patient-Centered Medical Home (PCMH), and a pay-for-performance program (eHearts).

DATA SOURCES/STUDY SETTING

Data for seven quality measures (QM) were retrospectively collected from 192 small primary care practices between October 2009 and October 2012; practice demographics and program participation status were extracted from in-house data.

STUDY DESIGN

Bivariate analyses were conducted to measure the impact of individual programs, and a Generalized Estimating Equation model was built to test the impact of each program alongside the others.

DATA COLLECTION/EXTRACTION METHODS

Monthly data were extracted via a structured query data network and were compared to program participation status, adjusting for variables including practice size and patient volume. Seven QMs were analyzed related to smoking prevention, blood pressure control, BMI, diabetes, and antithrombotic therapy.

PRINCIPAL FINDINGS

In bivariate analysis, MU practices tended to perform better on process measures, PCMH practices on more complex process measures, and eHearts practices on measures for which they were incentivized; in multivariate analysis, PCMH recognition was associated with better performance on more QMs than any other program.

CONCLUSIONS

Results suggest each of the programs can positively impact performance. In our data, PCMH appears to have the most positive impact.

Transferability of health cost evaluation across locations in oncology: cluster and principal component analysis as an explorative tool

Background

The transferability of economic evaluation in health care is of increasing interest in today’s globalized environment. Here, we propose a methodology for assessing the variability of data elements in cost evaluations in oncology. This method was tested in the context of the European Network of Excellence “Connective Tissues Cancers Network”.

Methods

Using a database that was previously aimed at exploring sarcoma management practices in Rhône-Alpes (France) and Veneto (Italy), we developed a model to assess the transferability of health cost evaluation across different locations. A nested data structure with 60 final factors of variability (e.g., unit cost of chest radiograph) within 16 variability areas (e.g., unit cost of imaging) within 12 objects (e.g., diagnoses) was produced in Italy and France, separately. Distances between objects were measured by Euclidean distance, Mahalanobis distance, and city-block metric. A hierarchical structure using cluster analysis (CA) was constructed. The objects were also represented by their projections and area of variability through correlation studies using principal component analysis (PCA). Finally, a hierarchical clustering based on principal components was performed.

Results

CA suggested four clusters of objects: chemotherapy in France; follow-up with relapse in Italy; diagnosis, surgery, radiotherapy, chemotherapy, and follow-up without relapse in Italy; and diagnosis, surgery, and follow-up with or without relapse in France. The variability between clusters was high, suggesting a lower transferability of results. Also, PCA showed a high variability (i.e. lower transferability) for diagnosis between both countries with regard to the quantities and unit costs of biopsies.

Conclusion

CA and PCA were found to be useful for assessing the variability of cost evaluations across countries. In future studies, regression methods could be applied after these methods to elucidate the determinants of the differences found in these analyses.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-014-0537-x) contains supplementary material, which is available to authorized users.

Use of care management practices in small- and medium-sized physician groups: do public reporting of physician quality and financial incentives matter?

OBJECTIVE

To examine the effect of public reporting (PR) and financial incentives tied to quality performance on the use of care management practices (CMPs) among small- and medium-sized physician groups.

DATA

Survey data from The National Study of Small and Medium-sized Physician Practices were used. Primary data collection was also conducted to assess community-level PR activities. The final sample included 643 practices engaged in quality reporting; about half of these practices were subject to PR.

STUDY DESIGN

We used a treatment effects model. The instrumental variables were the community-level variables that capture the level of PR activity in each community in which the practices operate.

FINDINGS

(1) PR is associated with increased use of CMPs, but the estimate is not statistically significant; (2) financial incentives are associated with greater use of CMPs; (3) practices' awareness/sensitivity to quality reports is positively related to their use of CMPs; and (4) combined PR and financial incentives jointly affect CMP use to a greater degree than either of these factors alone.

CONCLUSION

Small- to medium-sized practices appear to respond to PR and financial incentives by greater use of CMPs. Future research needs to investigate the appropriate mix and type of incentive arrangements and quality reporting.

The importance of consistent, high-quality acute myocardial infarction and heart failure care results from the American Heart Association's Get with the Guidelines Program

OBJECTIVES

This study examined the degree to which hospital performance for acute myocardial infarction (AMI) and heart failure (HF) care processes are correlated.

BACKGROUND

Although AMI and HF care processes may be amenable to similar quality improvement interventions, whether these are indeed correlated and whether hospitals with consistently superior performance for both care metrics have the best outcomes remains unknown.

METHODS

We compared hospital performance of the Centers for Medicare & Medicaid Services AMI and HF core measures in 283 hospitals submitting 10 or more patients to the Get With The Guidelines AMI and HF programs between January 2005 and April 2009.

RESULTS

Median hospital adherence to AMI and HF composite measures were 93% (interquartile range: 87% to 97%) and 92% (interquartile range: 85% to 96%), respectively, with only a modest correlation between hospital performance on these 2 composite metrics (r = 0.50; 95% confidence interval: 0.41 to 0.58). Hospitals with superior performance to both AMI and HF processes had significantly longer duration of Get With The Guidelines participation and lower adjusted in-hospital mortality (odds ratio: 0.79; 95% confidence interval: 0.63 to 0.99) for AMI and HF patients, whereas hospitals with superior adherence to either alone had similar mortality rates as hospitals with superior adherence to neither measure.

CONCLUSIONS

Hospitals that had consistent, superior performance for both AMI and HF care had significantly lower risk-adjusted mortality than those with superior performance either alone or for neither measure. Whether a single scoring system to assess global, rather than condition-specific, quality of cardiovascular care would facilitate care quality improvement more consistently and would optimize patient outcomes merits further investigation.

Predictors of nonadherence to national hospital quality measures for heart failure and pneumonia

BACKGROUND

Understanding factors associated with process measure nonadherence may improve both patient care and future measure design.

METHODS

We analyzed 3401 patients with heart failure and 2186 patients with pneumonia who were eligible for at least 1 National Hospital Quality Measure at an urban tertiary medical center from July 1, 2004, to June 30, 2008. We investigated the association of socioeconomic, demographic, clinical, and care delivery factors with process measure nonadherence, using multivariable analysis.

RESULTS

Demographic and socioeconomic variables were unrelated to heart failure measure adherence. Nonadherence with angiotensin-converting enzyme inhibitor/angiotensin II receptor blocker use was more common in patients with renal failure (odds ratio [OR] 2.56; 95% confidence interval [CI], 1.46-4.49), and patients admitted to noncardiac units more often failed the heart failure all-or-none measure (OR 2.22; 95% CI, 1.79-2.75). Patients with pneumonia who were admitted via the emergency department were less likely to fail antibiotic timing (OR 0.41; 95% CI, 0.27-0.63), whereas those with a history of methicillin-resistant Staphylococcus aureus (OR 2.63; 95% CI, 1.31-5.28) or requiring intensive care unit admission (OR 11.4; 95% CI, 6.3-20.8) were more likely to fail the antibiotic selection measure.

CONCLUSION

Demographic and socioeconomic factors were generally unrelated to process measure nonadherence, demonstrating that excellent inpatient care can be delivered even to vulnerable populations. Clinical predictors suggest opportunities for improving both medical record documentation of appropriate exclusions and future measure specifications, especially for complex patients. Care delivery factors substantially affect process adherence.

Exploring and comparing the characteristics of nonlatent and latent composite scores: implications for pay-for-performance incentive design

A concise and reliable composite quality score would be helpful in judging the quality of a hospital's services, especially for pay-for-performance (P4P) initiatives. This study compared several nonlatent and latent composite quality scores to evaluate the quality of care using diabetes mellitus (DM) P4P data and discusses their characteristics and implications for P4P policy. The authors describe a cross-sectional study of the DM P4P data collected from the claims data of the Bureau of National Health Insurance (NHI) in Taiwan from January 2007 to December 2007. The DM patient outcome data, such as hemoglobin A1C values, were retrieved from the P4P database sponsored by the Bureau of NHI in Taiwan. The composite scores were derived from the following methods: 1) nonlatent scores methods (e.g., the raw sum score and the all-or-none score methods)and 2) latent scores methods (e.g., item-response theory-based Models I and II and the PRIDIT model). These scores are compared in terms of 2 aspects-agreement of hospital rankings (using Spearman's rank correlation) and reliability (using bootstrap methods). The latent methods were superior to the nonlatent methods because they were more reliable and had specific weighting themes. The correlations among the 3 latent methods were moderately high. The use of the PRIDIT approach, which is moderately difficult compared with item response theory-based model, is recommended if the insurer wants to balance convenience and precision.

Implementing composite quality metrics for bipolar disorder: towards a more comprehensive approach to quality measurement

OBJECTIVE

We implemented a set of processes of care measures for bipolar disorder that reflect psychosocial, patient preference and continuum of care approaches to mental health, and examined whether veterans with bipolar disorder receive care concordant with these practices.

METHOD

Data from medical record reviews were used to assess key processes of care for 433 VA mental health outpatients with bipolar disorder. Both composite and individual processes of care measures were operationalized.

RESULTS

Based on composite measures, 17% had documented assessment of psychiatric symptoms (e.g., psychotic, hallucinatory), 28% had documented patient treatment preferences (e.g., reasons for treatment discontinuation), 56% had documented substance abuse and psychiatric comorbidity assessment, and 62% had documentation of adequate cardiometabolic assessment. No-show visits were followed up 20% of the time, and monitoring of weight gain was noted in only 54% of the patient charts. In multivariate analyses, history of homelessness (OR=1.61; 95% CI=1.05-2.46) and nonwhite race (OR=1.74; 95%CI=1.02-2.98) were associated with documentation of psychiatric symptoms and comorbidities, respectively.

CONCLUSIONS

Only half of patients diagnosed with bipolar disorder received care in accordance with clinical practice guidelines. High-quality treatment of bipolar disorder includes not only adherence to treatment guidelines but also patient-centered care processes.