Pay-for-performance as a cost-effective implementation strategy: results from a cluster randomized trial

A Randomized Controlled Trial of a Pay-for-Performance Initiative to Reduce Costs of Care for High-Need Psychiatric Patients

OBJECTIVE

Pay-for-performance (P4P) initiatives hold promise for improving health care delivery but are rarely applied to behavioral health or tested in randomized controlled trials (RCTs). This RCT examined the effectiveness of a P4P initiative to reduce total cost of 24-hour care among patients with high needs for psychiatric care in a large county in California.

METHODS

From August 2016 to March 2022, a total of 652 adult residents of Santa Clara County, California, were enrolled in a P4P initiative (mean±SD age=46.7±13.3 years, 61% male, 51% White, and 60% diagnosed as having a bipolar or psychotic disorder). Participants were randomly assigned to usual full-service partnerships from the county (N=327) or a comparable level of care from a contractor who agreed to a schedule of financial penalties and rewards based on whether enrollees (N=325) used more or less care than a historical cohort of similar county patients. The primary outcome was total cost of 24-hour psychiatric services. Secondary outcomes were costs of each of the 24-hour care services.

RESULTS

The proportion of the total sample that used 24-hour psychiatric services decreased over the 36-month study period. Intent-to-treat analyses revealed no differences between the two study conditions in total care costs during the follow-up period. No significant care utilization differences were observed between the two conditions in most of the individual 24-hour services.

CONCLUSIONS

A P4P initiative for high-need patients was no more effective than usual care for reducing costs of 24-hour psychiatric care.

Testing the incremental effectiveness of pay-for-performance to improve implementation of a motivational interviewing brief intervention for substance use disorders in HIV settings: Results of a cluster-randomized type 3 hybrid trial

Background

Substance use disorders (SUDs) have a serious adverse impact on people living with HIV. Previously, using a 39-site dual-randomized type 2 hybrid trial design, findings from the Substance Abuse Treatment to HIV Care Project supported the Implementation and Sustainment Facilitation (ISF) strategy to improve implementation and effectiveness of a motivational interviewing brief intervention (MIBI) for SUD within HIV service settings across the United States (US). Building on this trial, this cluster-randomized type 3 hybrid trial aimed to test the incremental effectiveness of a pay-for-performance (P4P), a form of the "alter incentive/allowance structures" strategy.

Methods

Twenty-six HIV service organizations, their staff participants (N=87), and their client participants (N=341) were cluster-randomized to one of two implementation conditions. The control condition included staff-focused training, feedback, and consultation (TFC) and team-focused implementation and sustainment (ISF). The experimental condition included TFC+ISF as well as P4P (TFC+ISF+P4P). P4P used financial incentives to reward MIBI implementation (US$10 per MIBI delivered) and MIBI implementation at or above a pre-defined level of quality (US$10 per demonstration). In addition to these outcomes, past 4-week changes/reductions in client participant's days of primary substance use and anxiety symptoms were examined.

Results

The addition of P4P had a large and significant effect on the number of MIBIs implemented (d=1.30, p<.05) and reduction in anxiety (d=-1.54), but there was no impact on days of substance use. P4P had large effects on MIBI quality (d=1.24) and MIBI implementation effectiveness (d=1.28), but these were not significant (p<.10).

Conclusions

P4P is a form of the "alter incentive/allowance structures" strategy Its function is to reward the implementation of a clinical innovation. Rewarding implementation is consistent with the theory of implementation effectiveness, which suggests implementation climate (i.e., the extent to which implementation is expected, supported, and rewarded) is a key antecedent of implementation effectiveness (i.e., the consistency and quality of implementation). We found that P4P had a significant, positive impact on MIBI implementation in HIV service settings, but client-level outcomes were mixed. Future research should examine the cost-effectiveness of this strategy, as well as to examine the effectiveness of P4P to improve the implementation of other evidence-based innovations.

Trial registration

ClinicalTrials.gov: NCT04687917. Registered 12/18/2020.

Partnering with social service staff to implement pragmatic clinical trials: an interim analysis of implementation strategies

BACKGROUND

With recent growth in the conduct of pragmatic clinical trials, the reliance on frontline staff to contribute to trial-related activities has grown as well. Active partnerships with staff members are often critical to pragmatic trial implementation, but rarely do research teams track and evaluate the specific "implementation strategies" used to support staff's involvement in trial procedures (e.g., participant recruitment). Accordingly, we adapted implementation science methodologies and conducted an interim analysis of the strategies deployed with social service staff involved in one multi-site pragmatic clinical trial.

METHODS

We used a naturalistic, observational study design to characterize strategies our research team deployed with staff during monthly, virtual meetings. Data were drawn from meeting notes and recordings from the trial's 4-month Preparation phase and 8-month Implementation phase. Strategies were mapped to the Expert Recommendations for Implementing Change taxonomy and categorized into nine implementation clusters. Survey data were also collected from staff to identify the most useful strategies the research team should deploy when onboarding new staff members in the trial's second year.

RESULTS

A total of 287 strategies were deployed. Strategies in the develop stakeholder interrelationships cluster predominated in both the Preparation (35%) and Implementation (31%) phases, followed by strategies in the use iterative and evaluative approaches cluster, though these were more prevalent during trial Preparation (24%) as compared to trial Implementation (18%). When surveyed on strategy usefulness, strategies in the provide interactive assistance, use financial approaches, and support staff clusters were most useful, per staff responses.

CONCLUSIONS

While strategies to develop stakeholder interrelationships were used most frequently during trial Preparation and Implementation, program staff perceived strategies that provided technical assistance, supported clinicians, and used financial approaches to be most useful and should be deployed when onboarding new staff members. Research teams are encouraged to adapt and apply implementation strategy tracking methods when partnering with social service staff and deploy practical strategies that support pragmatic trial success given staff needs and preferences.

TRIAL REGISTRATION

NCT05357261. May 2, 2022.

From innovative applications of the effectiveness-implementation hybrid trial design to the dissemination, implementation, effectiveness, sustainment, economics, and level-of-scaling hybrid trial design

To address the enduring gap between research and practice, there is a need to improve the speed and efficiency of research across the translational research spectrum. In 2012, the effectiveness-implementation hybrid trial design (HTD) was codified as a design with the potential to improve the speed and efficiency of translation, especially as part of T2 (clinical research) translational research. Building on this and other recent efforts to stimulate greater use of this novel HTD, the current article highlights an innovative application of each effectiveness-implementation HTD type. The completed application of the Type 1 effectiveness-implementation HTD tested the effectiveness of a clinical intervention for reducing HIV viral load and retaining people with HIV in care, as well as conducted a longitudinal mixed-methods examination to test for significant changes over time in three key measures of context, and economic evaluation of the clinical intervention. The completed application of the Type 2 effectiveness-implementation HTD used a dual-randomized design to simultaneously test the effectiveness of a clinical intervention for addressing substance use disorder among people with HIV and effectiveness of a blended strategy called the Implementation and Sustainment Facilitation (ISF) Strategy. This Type 2 hybrid trial was also innovative due to its focus on both sustainment and economic outcomes. The innovative Type 3 application (funded in 2008 and completed in 2012) tested the effectiveness and cost-effectiveness of using pay-for-performance to improve both implementation outcomes and client outcomes. This article also codifies a HTD called the Dissemination, Implementation, effectiveness, Sustainment, Economics, and Level-of-scaling (DIeSEL) HTD.

Virtual Training Is More Cost-Effective Than In-Person Training for Preparing Staff to Implement Contingency Management

Behavior therapy implementation relies in part on training to foster counselor skills in preparation for delivery with fidelity. Amidst Covid-19, the professional education arena witnessed a rapid shift from in-person to virtual training, yet these modalities' relative utility and expense is unknown. In the context of a cluster-randomized hybrid type 3 trial of contingency management (CM) implementation in opioid treatment programs (OTPs), a multi-cohort design presented rare opportunity to compare cost-effectiveness of virtual vs. in-person training. An initial counselor cohort (n = 26) from eight OTPs attended in-person training, and a subsequent cohort (n = 31) from ten OTPs attended virtual training. Common training elements were the facilitator, learning objectives, and educational strategies/activities. All clinicians submitted a post-training role-play, independently scored with a validated fidelity instrument for which performances were compared against benchmarks representing initial readiness and advanced proficiency. To examine the utility and expense of in-person and virtual trainings, cohort-specific rates for benchmark attainment were computed, and per-clinician expenses were estimated. Adjusted between-cohort differences were estimated via ordinary least squares, and an incremental cost effectiveness ratio (ICER) was calculated. Readiness and proficiency benchmarks were attained at rates 12-14% higher among clinicians attending virtual training, for which aggregated costs indicated a $399 per-clinician savings relative to in-person training. Accordingly, the ICER identified virtual training as the dominant strategy, reflecting greater cost-effectiveness across willingness-to-pay values. Study findings document greater utility, lesser expense, and cost-effectiveness of virtual training, which may inform post-pandemic dissemination of CM and other therapies.

Scoping review of costs of implementation strategies in community, public health and healthcare settings

OBJECTIVES

To identify existing evidence concerning the cost of dissemination and implementation (D&I) strategies in community, public health and health service research, mapped with the 'Expert Recommendations for Implementing Change' (ERIC) taxonomy.

DESIGN

Scoping review.

DATA SOURCES

MEDLINE, EMBASE, CINAHL, PsycINFO, Scopus and the Cochrane Library were searched to identify any English language reports that had been published between January 2008 and December 2019 concerning the cost of D&I strategies.

DATA EXTRACTION

We matched the strategies identified in each article using ERIC taxonomies; further classified them into five areas (eg, dissemination, implementation, integration, capacity building and scale-up); and extracted the corresponding costs (total costs and cots per action target and per evidence-based programme (EBP) participant). We also recorded the reported level of costing methodology used for cost assessment of D&I strategies.

RESULTS

Of the 6445 articles identified, 52 studies were eligible for data extraction. Lack of D&I strategy cost data was the predominant reason (55% of the excluded studies) for study exclusion. Predominant topic, setting, country and research design in the included studies were mental health (19%), primary care settings (44%), the US (35%) and observational (42%). Thirty-five (67%) studies used multicomponent D&I strategies (ranging from two to five discrete strategies). The most frequently applied strategies were Conduct ongoing training (50%) and Conduct educational meetings (23%). Adoption (42%) and reach (27%) were the two most frequently assessed outcomes. The overall costs of Conduct ongoing training ranged from $199 to $105 772 ($1-$13 973 per action target and $0.02-$412 per EBP participant); whereas the cost of Conduct educational meetings ranged from $987 to $1.1-$2.9 million/year ($33-$54 869 per action target and $0.2-$146 per EBP participant). The wide range of costs was due to the varying scales of the studies, intended audiences/diseases and the complexities of the strategy components. Most studies presented limited information on costing methodology, making interpretation difficult.

CONCLUSIONS

The quantity of published D&I strategy cost analyses is increasing, yet guidance on conducting and reporting of D&I strategy cost analysis is necessary to facilitate and promote the application of comparative economic evaluation in the field of D&I research.

The implementation & sustainment facilitation (ISF) strategy: Cost and cost-effectiveness results from a 39-site cluster randomized trial integrating substance use services in community-based HIV service organizations

Background: As part of the Substance Abuse Treatment to HIV Care Project, the Implementation & Sustainment Facilitation (ISF) strategy was found to be an effective adjunct to the Addiction Technology Transfer Center (ATTC) strategy for integrating a motivational interviewing-based brief intervention (MIBI) for substance use disorders. This study presents the cost and cost-effectiveness results. Methods: Thirty-nine HIV service organizations were randomized to receive the ATTC-only condition or the ATTC + ISF condition. Two staff from each organization received the ATTC-training. In ATTC + ISF organizations, the same two staff and additional support staff participated in facilitation sessions to support MIBI implementation. We estimated costs using primary data on the time spent in each strategy and the time spent delivering 409 MIBIs to clients. We estimated staff-level cost-effectiveness for the number of MIBIs delivered, average MIBI quality scores, and total client days abstinent per staff. We used sensitivity analyses to test how changes to key variables affect the results. Results: Adjusted per-staff costs were $2,915 for the ATTC strategy and $5,371 for ATTC + ISF, resulting in an incremental cost of $2,457. ATTC + ISF significantly increased the number of MIBIs delivered (3.73) and the average MIBI quality score (61.45), yielding incremental cost effectiveness ratios (ICERs) of $659 and $40. Client days abstinent increased by 59 days per staff with a quality-adjusted life-year ICER of $40,578 (95% confidence interval $29,795–$61,031). Conclusions: From the perspective of federal policymakers, ISF as an adjunct to the ATTC strategy may be cost-effective for improving the integration of MIBIs within HIV service organizations, especially if scaled up to reach more clients. Travel accounted for nearly half of costs, and virtual implementation may further increase value. We also highlight two considerations for cost-effectiveness analysis with hybrid trials: study protocols kept recruitment low and modeling choices affect how we interpret the effects on client-level outcomes.

Project MIMIC (Maximizing Implementation of Motivational Incentives in Clinics): A cluster-randomized type 3 hybrid effectiveness-implementation trial

BACKGROUND

Opioid-related overdoses and harms have been declared a public health emergency in the United States, highlighting an urgent need to implement evidence-based treatments. Contingency management (CM) is one of the most effective behavioral interventions when delivered in combination with medication for opioid use disorder, but its implementation in opioid treatment programs is woefully limited. Project MIMIC (Maximizing Implementation of Motivational Incentives in Clinics) was funded by the National Institute on Drug Abuse to identify effective strategies for helping opioid treatment programs improve CM implementation as an adjunct to medication. Specific aims will test the impact of two different strategies on implementation outcomes (primary aim) and patient outcomes (secondary aims), as well as test putative mediators of implementation effectiveness (exploratory aim).

METHODS

A 3-cohort, cluster-randomized, type 3 hybrid design is used with the opioid treatment programs as the unit of randomization. Thirty programs are randomized to one of two conditions. The control condition is the Addiction Technology Transfer Center (ATTC) Network implementation strategy, which consists of three core approaches: didactic training, performance feedback, and on-going consultation. The experimental condition is an enhanced ATTC strategy, with the same core ATTC elements plus two additional theory-driven elements. The two additional elements are Pay-for-Performance, which aims to increase implementing staff's extrinsic motivations, and Implementation & Sustainment Facilitation, which targets staff's intrinsic motivations. Data will be collected using a novel, CM Tracker tool to document CM session delivery, session audio recordings, provider surveys, and patient surveys. Implementation outcomes include CM Exposure (number of CM sessions delivered per patient), CM Skill (ratings of CM fidelity), and CM Sustainment (number of patients receiving CM after removal of support). Patient outcomes include self-reported opioid abstinence and opioid-related problems (both assessed at 3- and 6-months post-baseline).

DISCUSSION

There is urgent public health need to improve the implementation of CM as an adjunct to medication for opioid use disorder. Consistent with its hybrid type 3 design, Project MIMIC is advancing implementation science by comparing impacts of these two multifaceted strategies on both implementation and patient outcomes, and by examining the extent to which the impacts of those strategies can be explained by putative mediators.

TRIAL REGISTRATION

This clinical trial has been registered with clinicaltrials.gov (NCT03931174). Registered April 30, 2019. https://clinicaltrials.gov/ct2/show/NCT03931174?term=project+mimic&draw=2&rank=1.

Priority domains, aims, and testable hypotheses for implementation research: Protocol for a scoping review and evidence map

BACKGROUND

The challenge of implementing evidence-based innovations within practice settings is a significant public health issue that the field of implementation research (IR) is focused on addressing. Significant amounts of funding, time, and effort have been invested in IR to date, yet there remains significant room for advancement, especially regarding IR's development of scientific theories as defined by the National Academy of Sciences (i.e., a comprehensive explanation of the relationship between variables that is supported by a vast body of evidence). Research priority setting (i.e., promoting consensus about areas where research effort will have wide benefits to society) is a key approach to helping accelerate research advancements. Thus, building upon existing IR, general principles of data reduction, and a general framework for moderated mediation, this article identifies four priority domains, three priority aims, and four testable hypotheses for IR, which we organize in the priority aims and testable hypotheses (PATH) diagram.

METHODS

The objective of this scoping review is to map the extent to which IR has examined the identified PATH priorities to date. Our sample will include IR published in leading implementation-focused journals (i.e., Implementation Science, Implementation Science Communications, and Implementation Research and Practice) between their inception and December 2020. The protocol for the current scoping review and evidence map has been developed in accordance with the approach developed by Arksey and O'Malley and advanced by Levac, Colquhoun, and O'Brien. Because scoping reviews seek to provide an overview of the identified evidence base rather than synthesize findings from across studies, we plan to use our data-charting form to provide a descriptive overview of implementation research to date and summarize the research via one or more summary tables. We will use the PATH diagram to organize a map of the evidence to date.

DISCUSSION

This scoping review and evidence map is intended to help accelerate IR focused on suggested priority aims and testable hypotheses, which in turn will accelerate IR's development of National Academy of Sciences-defined scientific theories and, subsequently, improvements in public health.

SYSTEMATIC REVIEW REGISTRATION

Open Science Framework https://osf.io/3vhuj/.

Association of Alternative Payment and Delivery Models With Outcomes for Mental Health and Substance Use Disorders: A Systematic Review

IMPORTANCE

Health care spending in the United States continues to grow. Mental health and substance use disorders (MH/SUDs) are prevalent and associated with worse health outcomes and higher health care spending; alternative payment and delivery models (APMs) have the potential to facilitate higher quality, integrated, and more cost-effective MH/SUD care.

OBJECTIVE

To systematically review and summarize the published literature on populations and MH/SUD conditions examined by APM evaluations and the associations of APMs with MH/SUD outcomes.

EVIDENCE REVIEW

A literature search of MEDLINE, PsychInfo, Scopus, and Business Source was conducted from January 1, 1997, to May 17, 2019, for publications examining APMs for MH/SUD services, assessing at least 1 MH/SUD outcome, and having a comparison group. A total of 27 articles met these criteria, and each was classified according to the Health Care Payment Learning and Action Network's APM framework. Strength of evidence was graded using a modified Oxford Centre for Evidence-Based Medicine framework.

FINDINGS

The 27 included articles evaluated 17 APM implementations that spanned 3 Health Care Payment Learning and Action Network categories and 6 subcategories, with no single category predominating the literature. APMs varied with regard to their assessed outcomes, funding sources, target populations, and diagnostic focuses. The APMs were primarily evaluated on their associations with process-of-care measures (15 [88.2%]), followed by utilization (11 [64.7%]), spending (9 [52.9%]), and clinical outcomes (5 [29.4%]). Medicaid and publicly funded SUD programs were most common, with each representing 7 APMs (41.2%). Most APMs focused on adults (11 [64.7%]), while fewer (2 [11.8%]) targeted children or adolescents. More than half of the APMs (9 [52.9%]) targeted populations with SUD, while 4 (23.5%) targeted MH populations, and the rest targeted MH/SUD broadly defined. APMs were most commonly associated with improvements in MH/SUD process-of-care outcomes (12 of 15 [80.0%]), although they were also associated with lower spending (4 of 8 [50.0%]) and utilization (5 of 11 [45.5%]) outcomes, suggesting gains in value from APMs. However, clinical outcomes were rarely measured (5 APMs [29.4%]). A total of 8 APMs (47.1%) assessed for gaming (ie, falsification of outcomes because of APM incentives) and adverse selection, with 1 (12.5%) showing evidence of gaming and 3 (37.5%) showing evidence of adverse selection. Other than those assessing accountable care organizations, few studies included qualitative evaluations.

CONCLUSIONS AND RELEVANCE

In this study, APMs were associated with improvements in process-of-care outcomes, reductions in MH/SUD utilization, and decreases in spending. However, these findings cannot fully substitute for assessments of clinical outcomes, which have rarely been evaluated in this context. Additionally, this systematic review identified some noteworthy evidence for gaming and adverse selection, although these outcomes have not always been duly measured or analyzed. Future research is needed to better understand the varied qualitative experiences across APMs, their successful components, and their associations with clinical outcomes among diverse populations and settings.

Financial Incentives to Increase Cardiac Rehabilitation Participation Among Low-Socioeconomic Status Patients: A Randomized Clinical Trial

OBJECTIVES

This study sought to examine the efficacy of financial incentives to increase Medicaid patient participation in and completion of cardiac rehabilitation (CR).

BACKGROUND

Participation in CR reduces morbidity, mortality, and hospitalizations while improving quality of life. Lower-socioeconomic status (SES) patients are much less likely to attend and complete CR, despite being at increased risk for recurrent cardiovascular events.

METHODS

A total of 130 individuals enrolled in Medicaid with a CR-qualifying cardiac event were randomized 1:1 to receive financial incentives on an escalating schedule ($4 to $50) for completing CR sessions or to receive usual care. Primary outcomes were CR participation (number of sessions completed) and completion (≥30 sessions completed). Secondary outcomes included changes in sociocognitive measurements (depressive/anxious symptoms, executive function), body composition (waist circumference, body mass index), fitness (peak VO₂) over 4 months, and combined number of hospitalizations and emergency department (ED) contacts over 1 year.

RESULTS

Patients randomized to the incentive condition completed more sessions (22.4 vs. 14.7, respectively; p = 0.013) and were almost twice as likely to complete CR (55.4% vs. 29.2%, respectively; p = 0.002) as controls. Incentivized patients were also more likely to experience improvements in executive function (p < 0.001), although there were no significant effects on other secondary outcomes. Patients who completed ≥30 sessions had 47% fewer combined hospitalizations and ED visits (p = 0.014), as reflected by a nonsignificant trend by study condition with 39% fewer hospital contacts in the incentive condition group (p = 0.079).

CONCLUSIONS

Financial incentives improve CR participation among lower-SES patients following a cardiac event. Increasing participation among lower-SES patients in CR is critical for positive longer-term health outcomes. (Increasing Cardiac Rehabilitation Participation Among Medicaid Enrollees; NCT02172820).