Are Improvements in Measured Performance Driven by Better Treatment or "Denominator Management"?

Healthcare quality measures in implementation research: advantages, risks and lessons learned

Implementation studies evaluate strategies to move evidence-based practices into routine clinical practice. Often, implementation scientists use healthcare quality measures to evaluate the integration of an evidence-based clinical practice into real-world healthcare settings. Healthcare quality measures have standardized definitions and are a method to operationalize and monitor guideline-congruent care. Implementation scientists can access existing data on healthcare quality measures through various sources (e.g. operations-calculated), or they can calculate the measures directly from healthcare claims and administrative data (i.e. researcher-calculated). Implementation scientists need a better understanding of the advantages and disadvantages of these methods of obtaining healthcare quality data for designing, planning and executing an implementation study. The purpose of this paper is to describe the advantages, risks and lessons learned when using operations- versus researcher-calculated healthcare quality measures in site selection, implementation monitoring and implementation outcome evaluation. A key lesson learned was that relying solely on operations-calculated healthcare quality measures during an implementation study poses risks to site selection, accurate feedback on implementation progress to stakeholders, and the integrity of study results. A possible solution is using operations-calculated quality measures for monitoring of evidence-based practice uptake and researcher-calculated measures for site section and outcomes evaluation. This approach provides researchers greater control over the data and consistency of the measurement from site selection to outcomes evaluation while still retaining measures that are familiar and understood by key stakeholders whom implementation scientists need to engage in practice change efforts.

Incentivizing performance in health care: a rapid review, typology and qualitative study of unintended consequences

BACKGROUND

Health systems are increasingly implementing policy-driven programs to incentivize performance using contracts, scorecards, rankings, rewards, and penalties. Studies of these "Performance Management" (PM) programs have identified unintended negative consequences. However, no single comprehensive typology of the negative and positive unintended consequences of PM in healthcare exists and most studies of unintended consequences were conducted in England or the United States. The aims of this study were: (1) To develop a comprehensive typology of unintended consequences of PM in healthcare, and (2) To describe multiple stakeholder perspectives of the unintended consequences of PM in cancer and renal care in Ontario, Canada.

METHODS

We conducted a rapid review of unintended consequences of PM in healthcare (n = 41 papers) to develop a typology of unintended consequences. We then conducted a secondary analysis of data from a qualitative study involving semi-structured interviews with 147 participants involved with or impacted by a PM system used to oversee 40 care delivery networks in Ontario, Canada. Participants included administrators and clinical leads from the networks and the government agency managing the PM system. We undertook a hybrid inductive and deductive coding approach using the typology we developed from the rapid review.

RESULTS

We present a comprehensive typology of 48 negative and positive unintended consequences of PM in healthcare, including five novel unintended consequences not previously identified or well-described in the literature. The typology is organized into two broad categories: unintended consequences on (1) organizations and providers and on (2) patients and patient care. The most common unintended consequences of PM identified in the literature were measure fixation, tunnel vision, and misrepresentation or gaming, while those most prominent in the qualitative data were administrative burden, insensitivity, reduced morale, and systemic dysfunction. We also found that unintended consequences of PM are often mutually reinforcing.

CONCLUSIONS

Our comprehensive typology provides a common language for discourse on unintended consequences and supports systematic, comparable analyses of unintended consequences across PM regimes and healthcare systems. Healthcare policymakers and managers can use the results of this study to inform the (re-)design and implementation of evidence-informed PM programs.

Assessing the postdeployment quality of treatment for substance use disorders among Army enlisted soldiers in the Military Health System

Little is known about the rates and predictors of substance use treatment received in the Military Health System among Army soldiers diagnosed with a postdeployment substance use disorder (SUD). We used data from the Substance Use and Psychological Injury Combat study to determine the proportion of active duty (n = 338,708) and National Guard/Reserve (n = 178,801) enlisted soldiers returning from an Afghanistan/Iraq deployment in fiscal years 2008 to 2011 who had an SUD diagnosis in the first 150 days postdeployment. Among soldiers diagnosed with an SUD, we examined the rates and predictors of substance use treatment initiation and engagement according to the Healthcare Effectiveness Data and Information Set criteria. In the first 150 days postdeployment 3.3% of active duty soldiers and 1.0% of National Guard/Reserve soldiers were diagnosed with an SUD. Active duty soldiers were more likely to initiate and engage in substance use treatment than National Guard/Reserve soldiers, yet overall, engagement rates were low (25.0% and 15.7%, respectively). Soldiers were more likely to engage in treatment if they received their index diagnosis in a specialty behavioral health setting. Efforts to improve substance use treatment in the Military Health System should include initiatives to more accurately identify soldiers with undiagnosed SUD. Suggestions to improve substance use treatment engagement in the Military Health System will be discussed.

Facility-level changes in receipt of pharmacotherapy for opioid use disorder: Implications for implementation science

BACKGROUND

The U.S. is facing an opioid epidemic, but despite mandates for pharmacotherapy for opioid use disorder to be available at Veterans Health Administration (VHA) facilities, the majority of veterans with opioid use disorder do not receive these medications. In implementation research, facilities are often targeted for qualitative inquiry or quality improvement efforts based on quality measure performance during a one-year period. However, sites that experience quality performance changes from one year to the next may be highly informative because mechanisms that impact facility change may be more discoverable. The current study examined changes in receipt of pharmacotherapy for opioid use disorder in a national healthcare system to determine the extent to which sites fluctuated in performance over a two-year period and illustrate how changes in quality measures over time may be useful for implementation research and healthcare surveillance of quality measures.

METHODS

Using national VHA data from Fiscal Years (FY) 2016 and 2017, we calculated quality measure performance as the number of patients who received pharmacotherapy for opioid use disorder (i.e., methadone, buprenorphine, and naltrexone) divided by the number of patients with a current non-remitted opioid use disorder diagnosis for each FY at each facility (n = 129) and examined change from FY16 to FY17.

RESULTS

The mean rate of receipt of pharmacotherapy for opioid use disorder was 38% (facility range = 3% to 74%) in FY16 and 41% (facility range = 2% to 76%) in FY17. The average facility-level change in performance was 3% and ranged from -19% to 26%. There were 32 facilities that decreased in provision of pharmacotherapy, 12 facilities with no change, and 85 facilities that increased.

CONCLUSIONS

For facilities with average or high performance, it was difficult to maintain their performance over time. Identifying and learning from facilities with recent fluctuations may be more informative to guide the design of future quality improvement efforts than studying facilities with stable high or low performance.

Pay-for-Performance and Veteran Care in the VHA and the Community: a Systematic Review

BACKGROUND

Although pay-for-performance (P4P) strategies have been used by the Veterans Health Administration (VHA) for over a decade, the long-term benefits of P4P are unclear. The use of P4P is further complicated by the increased use of non-VHA healthcare providers as part of the Veterans Choice Program. We conducted a systematic review and key informant interviews to better understand the effectiveness and potential unintended consequences of P4P, as well as the implementation factors and design features important in both VHA and non-VHA/community settings.

METHODS

We searched PubMed, PsycINFO, and CINAHL through March 2017 and reviewed reference lists. We included trials and observational studies of P4P targeting Veteran health. Two investigators abstracted data and assessed study quality. We interviewed VHA stakeholders to gain further insight.

RESULTS

The literature search yielded 1031 titles and abstracts, of which 30 studies met pre-specified inclusion criteria. Twenty-five examined P4P in VHA settings and 5 in community settings. There was no strong evidence supporting the effectiveness of P4P in VHA settings. Interviews with 17 key informants were consistent with studies that identified the potential for overtreatment associated with performance metrics in the VHA. Key informants' views on P4P in community settings included the need to develop relationships with providers and health systems with records of strong performance, to improve coordination by targeting documentation and data sharing processes, and to troubleshoot the limited impact of P4P among practices where Veterans make up a small fraction of the patient population.

DISCUSSION

The evidence to support the effectiveness of P4P on Veteran health is limited. Key informants recognize the potential for unintended consequences, such as overtreatment in VHA settings, and suggest that implementation of P4P in the community focus on relationship building and target areas such as documentation and coordination of care.

Promoting Adoption of Medication for Opioid and Alcohol Use Disorders Through System Change

OBJECTIVE

The Medication Research Partnership (MRP), a collaboration between a national commercial health plan and nine addiction treatment centers, implemented organizational and system changes to promote use of federally approved medications for treatment of alcohol and opioid use disorders.

METHOD

A difference-in-differences analysis examined change over time in the percentage of patients receiving a prescription medication for alcohol or opioid use disorders treated in MRP (n = 9) and comparison (n = 15) sites.

RESULTS

MRP clinics experienced a 2.4-fold increase in patients receiving an alcohol or opioid prescription (13.2% at baseline to 31.7% at 3 years after MRP initiation); comparison clinics experienced significantly less change (17.6% to 23.5%) with an adjusted difference-in-differences of 12.5% (95% CI [5.4, 19.6], p = .001). MRP sites increased the patients with prescriptions to treat opioid use disorder from 17.0% (baseline) to 36.8% (3 years after initiation), with smaller changes observed in comparison sites (23.2% to 24.0%) and a 3-year post-initiation adjusted difference-in-differences of 19% (95% CI [8.5, 29.5], p = .000). Medications for alcohol use disorders increased in both MRP (9.0% to 26.5%) and comparison sites (11.4% to 23.1%).

CONCLUSIONS

Promoting the use of medications to support recovery required complex interventions. The Advancing Recovery System Change Model, initially developed in publicly funded systems of care, was successfully adapted for commercial sector use. The model provides a framework for providers and commercial health plans to collaborate and increase patient access to medications.

Performance Measurement and Target-Setting in California's Safety Net Health Systems

Health policies encourage implementing quality measurement with performance targets. The 2010-2015 California Medicaid waiver mandated quality measurement and reporting. In 2013, California safety net hospitals participating in the waiver set a voluntary performance target (the 90th percentile for Medicare preferred provider organization plans) for mammography screening and cholesterol control in diabetes. They did not reach the target, and the difference-in-differences analysis suggested that there was no difference for mammography ( P = .39) and low-density lipoprotein control ( P = .11) performance compared to measures for which no statewide quality improvement initiative existed. California's Medicaid waiver was associated with improved performance on a number of metrics, but this performance was not attributable to target setting on specific health conditions. Performance may have improved because of secular trends or systems improvements related to waiver funding. Relying on condition-specific targets to measure performance may underestimate improvements and disadvantage certain health systems. Achieving ambitious targets likely requires sustained fiscal, management, and workforce investments.