Merging Implementation Practice and Science to Scale Up Promising Practices: The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) Program

Understanding pathways from implementation to sustainment: a longitudinal, mixed methods analysis of promising practices implemented in the Veterans Health Administration

BACKGROUND

The Veterans Health Administration (VHA) is the United States largest learning health system. The Diffusion of Excellence (DoE) program is a large-scale model of diffusion that identifies and diffuses evidence-informed practices across VHA. During the period of 2016-2021, 57 evidence-informed practices were implemented across 82 VHA facilities. This setting provides a unique opportunity to understand sustainment determinants and pathways. Our objective was to characterize the longitudinal pathways of practices as they transition from initial implementation to long-term sustainment at each facility.

METHODS

A longitudinal, mixed-methods evaluation of 82 VHA facilities. Eighty-two facility representatives, chosen by leadership as points-of-contact for 57 DoE practices, were eligible for post-implementation interviews and annual sustainment surveys. Primary outcomes (implementation, sustainment), and secondary outcomes (institutionalization, effectiveness, anticipated sustainment) at four time-points were collected. We performed descriptive statistics and directed content analysis using Hailemariam et al.'s factors influencing sustainment.

RESULTS

After approximately five years post-implementation (e.g., 2021 sustainment outcomes), of the 82 facilities, about one-third fully sustained their practice compared to one-third that did not fully sustain their practice because it was in a "liminal" stage (neither sustained nor discontinued) or permanently discontinued. The remaining one-third of facilities had missing 2021 sustainment outcomes. A higher percentage of facilities (70%) had inconsistent primary outcomes (changing over time) compared to facilities (30%) with consistent primary outcomes (same over time). Thirty-four percent of facilities with sustained practices reported resilience since they overcame implementation and sustainment barriers. Facilities with sustained practices reported more positive secondary outcomes compared to those that did not sustain their practice. Key factors facilitating practice sustainment included: demonstrating practice effectiveness/benefit, sufficient organizational leadership, sufficient workforce, and adaptation/alignment with local context. Key factors hindering practice sustainment included: insufficient workforce, not able to maintain practice fidelity/integrity, critical incidents related to the COVID-19 pandemic, organizational leadership did not support sustainment of practice, and no ongoing support.

CONCLUSIONS

We identified diverse pathways from implementation to sustainment, and our data underscore that initial implementation outcomes may not determine long-term sustainment outcomes. This longitudinal evaluation contributes to understanding impacts of the DoE program, including return on investment, achieving learning health system goals, and insights into achieving high-quality healthcare in VHA.

Diffusion of excellence: evaluating a system to identify, replicate, and spread promising innovative practices across the Veterans health administration

Introduction

The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program provides a system to identify, replicate, and spread promising practices across the largest integrated healthcare system in the United States. DoE identifies innovations that have been successfully implemented in the VHA through a Shark Tank style competition. VHA facility and regional directors bid resources needed to replicate promising practices. Winning facilities/regions receive external facilitation to aid in replication/implementation over the course of a year. DoE staff then support diffusion of successful practices across the nationwide VHA.

Methods

Organized around the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework, we summarize results of an ongoing long-term mixed-methods implementation evaluation of DoE. Data sources include: Shark Tank application and bid details, tracking practice adoptions through a Diffusion Marketplace, characteristics of VHA facilities, focus groups with Shark Tank bidders, structured observations of DoE events, surveys of DoE program participants, and semi-structured interviews of national VHA program office leaders, VHA healthcare system/facility executives, practice developers, implementation teams and facilitators.

Results

In the first eight Shark Tanks (2016-2022), 3,280 Shark Tank applications were submitted; 88 were designated DoE Promising Practices (i.e., practices receive facilitated replication). DoE has effectively spread practices across the VHA, with 1,440 documented instances of adoption/replication of practices across the VHA. This includes 180 adoptions/replications in facilities located in rural areas. Leadership decisions to adopt innovations are often based on big picture considerations such as constituency support and linkage to organizational goals. DoE Promising Practices that have the greatest national spread have been successfully replicated at new sites during the facilitated replication process, have close partnerships with VHA national program offices, and tend to be less expensive to implement. Two indicators of sustainment indicate that 56 of the 88 Promising Practices are still being diffused across the VHA; 56% of facilities originally replicating the practices have sustained them, even up to 6 years after the first Shark Tank.

Conclusion

DoE has developed a sustainable process for the identification, replication, and spread of promising practices as part of a learning health system committed to providing equitable access to high quality care.

Implementation of Direct-to-Patient Mobile Teledermatology in VA

BACKGROUND

Innovative technology can enhance patient access to healthcare but must be successfully implemented to be effective.

OBJECTIVE

We evaluated Department of Veterans Affairs' (VA's) implementation of My VA Images, a direct-to-patient asynchronous teledermatology mobile application enabling established dermatology patients to receive follow-up care remotely instead of in-person.

DESIGN /PARTICIPANTS/APPROACH

Following pilot testing at 3 facilities, the app was introduced to 28 facilities (4 groups of 7) every 3 months using a stepped-wedge cluster-randomized design. Using the Organizational Theory of Implementation Effectiveness, we examined the app's implementation using qualitative and quantitative data consisting of encounter data from VA's corporate data warehouse; app usage from VA's Mobile Health database; bi-monthly reports from facility representatives; phone interviews with clinicians; and documented communications between the operational partner and facility staff.

KEY RESULTS

Implementation policies and practices included VA's vision to expand home telehealth and marketing/communication strategies. The COVID-19 pandemic dominated the implementation climate by stressing staffing, introducing competing demands, and influencing stakeholder attitudes to the app, including its fit to their values. These factors were associated with mixed implementation effectiveness, defined as high quality consistent use. Nineteen of 31 exposed facilities prepared to use the app; 10 facilities used it for actual patient care, 7 as originally intended. Residents, nurse practitioners, and physician assistants were more likely than attendings to use the app. Facilities exposed to the app pre-pandemic were more likely to use and sustain the new process.

CONCLUSIONS

Considerable heterogeneity existed in implementing mobile teledermatology, despite VA's common mission, integrated healthcare system, and stakeholders' broad interest. Identifying opportunities to target favorable facilities and user groups (such as teaching facilities and physician extenders, respectively) while addressing internal implementation barriers including incomplete integration with the electronic health record as well as inadequate staffing may help optimize the initial impact of direct-to-patient telehealth. The COVID pandemic was a notable extrinsic barrier.

CLINICAL TRIALS REGISTRATION

NCT03241589.

Hurdles of innovation-insights from a new healthcare delivery innovation program

Introduction

Healthcare systems are actively working to innovate their care delivery models, seeking to improve service quality, improve patient and provider satisfaction, and reduce cost.

Methods

By critically evaluating our experiences to date, this article highlights challenges systems may face in the process of trying to redesign healthcare and offers insights on how to navigate hurdles. We identify barriers to-and ultimately approaches to promote-rapid, scalable, sustainable, and transformative care redesign.

Results

Dedicated electronic health record IT and analytic support, and ongoing leadership engagement and communication, play a valuable role in enabling redesign efforts. Flexible, but guided, innovation support helps teams stay accountable and motivated, while accommodating new project needs and directions. Understanding the change ecosystem and evaluating and sharing outcomes on an ongoing basis, enables teams to adapt as needed. Facilitation and support help realize the value of diverse, engaged teams; novel approaches and techniques draw out innovative perspectives and promote creative thinking.

Conclusions

Although not an exhaustive list of challenges or strategies to overcome them, we hope these insights will contribute to a culture of innovation and support other institutions in their healthcare redesign initiatives.

Sustainment of diverse evidence-informed practices disseminated in the Veterans Health Administration (VHA): initial development and piloting of a pragmatic survey tool

BACKGROUND

There are challenges associated with measuring sustainment of evidence-informed practices (EIPs). First, the terms sustainability and sustainment are often falsely conflated: sustainability assesses the likelihood of an EIP being in use in the future while sustainment assesses the extent to which an EIP is (or is not) in use. Second, grant funding often ends before sustainment can be assessed. The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program is one of few large-scale models of diffusion; it seeks to identify and disseminate practices across the VHA system. The DoE sponsors "Shark Tank" competitions, in which leaders bid on the opportunity to implement a practice with approximately 6 months of implementation support. As part of an ongoing evaluation of the DoE, we sought to develop and pilot a pragmatic survey tool to assess sustainment of DoE practices.

METHODS

In June 2020, surveys were sent to 64 facilities that were part of the DoE evaluation. We began analysis by comparing alignment of quantitative and qualitative responses; some facility representatives reported in the open-text box of the survey that their practice was on a temporary hold due to COVID-19 but answered the primary outcome question differently. As a result, the team reclassified the primary outcome of these facilities to Sustained: Temporary COVID-Hold. Following this reclassification, the number and percent of facilities in each category was calculated. We used directed content analysis, guided by the Consolidated Framework for Implementation Research (CFIR), to analyze open-text box responses.

RESULTS

A representative from forty-one facilities (64%) completed the survey. Among responding facilities, 29/41 sustained their practice, 1/41 partially sustained their practice, 8/41 had not sustained their practice, and 3/41 had never implemented their practice. Sustainment rates increased between Cohorts 1-4.

CONCLUSIONS

The initial development and piloting of our pragmatic survey allowed us to assess sustainment of DoE practices. Planned updates to the survey will enable flexibility in assessing sustainment and its determinants at any phase after adoption. This assessment approach can flex with the longitudinal and dynamic nature of sustainment, including capturing nuances in outcomes when practices are on a temporary hold. If additional piloting illustrates the survey is useful, we plan to assess the reliability and validity of this measure for broader use in the field.

Implementation of an Intensive Telehealth Intervention for Rural Patients with Clinic-Refractory Diabetes

BACKGROUND

Rural patients with type 2 diabetes (T2D) may experience poor glycemic control due to limited access to T2D specialty care and self-management support. Telehealth can facilitate delivery of comprehensive T2D care to rural patients, but implementation in clinical practice is challenging.

OBJECTIVE

To examine the implementation of Advanced Comprehensive Diabetes Care (ACDC), an evidence-based, comprehensive telehealth intervention for clinic-refractory, uncontrolled T2D. ACDC leverages existing Veterans Health Administration (VHA) Home Telehealth (HT) infrastructure, making delivery practical in rural areas.

DESIGN

Mixed-methods implementation study.

PARTICIPANTS

230 patients with clinic-refractory, uncontrolled T2D.

INTERVENTION

ACDC bundles telemonitoring, self-management support, and specialist-guided medication management, and is delivered over 6 months using existing VHA HT clinical staffing/equipment. Patients may continue in a maintenance protocol after the initial 6-month intervention period.

MAIN MEASURES

Implementation was evaluated using the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. The primary effectiveness outcome was hemoglobin A1c (HbA1c).

KEY RESULTS

From 2017 to 2020, ACDC was delivered to 230 patients across seven geographically diverse VHA sites; on average, patients were 59 years of age, 95% male, 80% white, and 14% Hispanic/Latinx. Patients completed an average of 10.1 of 12 scheduled encounters during the 6-month intervention period. Model-estimated mean baseline HbA1c was 9.56% and improved to 8.14% at 6 months (- 1.43%, 95% CI: - 1.64, - 1.21; P < .001). Benefits persisted at 12 (- 1.26%, 95% CI: - 1.48, - 1.05; P < .001) and 18 months (- 1.08%, 95% CI - 1.35, - 0.81; P < .001). Patients reported increased engagement in self-management and awareness of glycemic control, while clinicians and HT nurses reported a moderate workload increase. As of this submission, some sites have maintained delivery of ACDC for up to 4 years.

CONCLUSIONS

When strategically designed to leverage existing infrastructure, comprehensive telehealth interventions can be implemented successfully, even in rural areas. ACDC produced sustained improvements in glycemic control in a previously refractory population.

Balancing reality in embedded research and evaluation: Low vs high embeddedness

Embedding research and evaluation into organizations is one way to generate “practice‐based” evidence needed to accelerate implementation of evidence‐based innovations within learning health systems. Organizations and researchers/evaluators vary greatly in how they structure and operationalize these collaborations. One key aspect is the degree of embeddedness: from low embeddedness where researchers/evaluators are located outside organizations (eg, outside evaluation consultants) to high embeddedness where researchers/evaluators are employed by organizations and thus more deeply involved in program evolution and operations. Pros and cons related to the degree of embeddedness (low vs high) must be balanced when developing these relationships. We reflect on this process within the context of an embedded, mixed‐methods evaluation of the Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program. Considerations that must be balanced include: (a) low vs high alignment of goals; (b) low vs high involvement in strategic planning; (c) observing what is happening vs being integrally involved with programmatic activities; (d) reporting findings at the project's end vs providing iterative findings and recommendations that contribute to program evolution; and (e) adhering to predetermined aims vs adapting aims in response to evolving partner needs.

Rapid versus traditional qualitative analysis using the Consolidated Framework for Implementation Research (CFIR)

Background

Qualitative approaches, alone or in mixed methods, are prominent within implementation science. However, traditional qualitative approaches are resource intensive, which has led to the development of rapid qualitative approaches. Published rapid approaches are often inductive in nature and rely on transcripts of interviews. We describe a deductive rapid analysis approach using the Consolidated Framework for Implementation Research (CFIR) that uses notes and audio recordings. This paper compares our rapid versus traditional deductive CFIR approach.

Methods

Semi-structured interviews were conducted for two cohorts of the Veterans Health Administration (VHA) Diffusion of Excellence (DoE). The CFIR guided data collection and analysis. In cohort A, we used our traditional CFIR-based deductive analysis approach (directed content analysis), where two analysts completed independent in-depth manual coding of interview transcripts using qualitative software. In cohort B, we used our new rapid CFIR-based deductive analysis approach (directed content analysis), where the primary analyst wrote detailed notes during interviews and immediately “coded” notes into a MS Excel CFIR construct by facility matrix; a secondary analyst then listened to audio recordings and edited the matrix. We tracked time for our traditional and rapid deductive CFIR approaches using a spreadsheet and captured transcription costs from invoices. We retrospectively compared our approaches in terms of effectiveness and rigor.

Results

Cohorts A and B were similar in terms of the amount of data collected. However, our rapid deductive CFIR approach required 409.5 analyst hours compared to 683 h during the traditional deductive CFIR approach. The rapid deductive approach eliminated $7250 in transcription costs. The facility-level analysis phase provided the greatest savings: 14 h/facility for the traditional analysis versus 3.92 h/facility for the rapid analysis. Data interpretation required the same number of hours for both approaches.

Conclusion

Our rapid deductive CFIR approach was less time intensive and eliminated transcription costs, yet effective in meeting evaluation objectives and establishing rigor. Researchers should consider the following when employing our approach: (1) team expertise in the CFIR and qualitative methods, (2) level of detail needed to meet project aims, (3) mode of data to analyze, and (4) advantages and disadvantages of using the CFIR.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-021-01111-5.