How do researchers conceptualize and plan for the sustainability of their NIH R01 implementation projects?

Changing patterns of routine laboratory testing over time at children's hospitals

BACKGROUND

Research into low-value routine testing at children's hospitals has not consistently evaluated changing patterns of testing over time.

OBJECTIVES

To identify changes in routine laboratory testing rates at children's hospitals over ten years and the association with patient outcomes.

DESIGN, SETTINGS, AND PARTICIPANTS

We performed a multi-center, retrospective cohort study of children aged 0-18 hospitalized with common, lower-severity diagnoses at 28 children's hospitals in the Pediatric Health Information Systems database.

MAIN OUTCOMES AND MEASURES

We calculated average annual testing rates for complete blood counts, electrolytes, and inflammatory markers between 2010 and 2019 for each hospital. A > 2% average testing rate change per year was defined as clinically meaningful and used to separate hospitals into groups: increasing, decreasing, and unchanged testing rates. Groups were compared for differences in length of stay, cost, and 30-day readmission or ED revisit, adjusted for demographics and case mix index.

RESULTS

Our study included 576,572 encounters for common, low-severity diagnoses. Individual hospital testing rates in each year of the study varied from 0.3 to 1.4 tests per patient day. The average yearly change in hospital-specific testing rates ranged from -6% to +7%. Four hospitals remained in the lowest quartile of testing and two in the highest quartile throughout all ten years of the study. We grouped hospitals with increasing (8), decreasing (n = 5), and unchanged (n = 15) testing rates. No difference was found across subgroups in costs, length of stay, 30-day ED revisit, or readmission rates. Comparing resource utilization trends over time provides important insights into achievable rates of testing reduction.

Reinforcing Feedbacks for Sustainable Implementation of Rural Drinking-Water Treatment Technology

Progress toward universal access to safe drinking water depends on rural water service delivery models that incorporate water safety management. Water supplies of all types have high rates of fecal contamination unless water safety risks are actively managed through water source protection, treatment, distribution, and storage. Recognizing the role of treatment within this broader risk-based framework, this study focuses on the implementation of passive chlorination and ultraviolet (UV) disinfection technologies in rural settings. These technologies can reduce the health risk from microbiological contaminants in drinking water; however, technology-focused treatment interventions have had limited sustainability in rural settings. This study examines the requirements for sustainable implementation of rural water treatment through qualitative content analysis of 26 key informant interviews, representing passive chlorination and UV disinfection projects in rural areas in South America, Africa, and Asia. The analysis is aligned with the RE-AIM framework and delivers insight into 18 principal enablers and barriers to rural water treatment sustainability. Analysis of the interrelationships among these factors identifies leverage points and encourages fit-for-purpose intervention design reinforced by collaboration between facilitating actors through hybrid service delivery models. Further work should prioritize health impact evidence, water quality reporting guidance, and technological capabilities that optimize trade-offs in fit-for-purpose treatment design.

Navigating the field of implementation science towards maturity: challenges and opportunities

BACKGROUND

The field of implementation science has significantly expanded in size and scope over the past two decades, although work related to understanding implementation processes have of course long preceded the more systematic efforts to improve integration of evidence-based interventions into practice settings. While this growth has had significant benefits to research, practice, and policy, there are some clear challenges that this period of adolescence has uncovered.

MAIN BODY

This invited commentary reflects on the development of implementation science, its rapid growth, and milestones in its establishment as a viable component of the biomedical research enterprise. The authors reflect on progress in research and training, and then unpack some of the consequences of rapid growth, as the field has grappled with the competing challenges of legitimacy among the research community set against the necessary integration and engagement with practice and policy partners. The article then enumerates a set of principles for the field's next developmental stage and espouses the aspirational goal of a "big tent" to support the next generation of impactful science.

CONCLUSION

For implementation science to expand its relevance and impact to practice and policy, researchers must not lose sight of the original purpose of the field-to support improvements in health and health care at scale, the importance of building a community of research and practice among key partners, and the balance of rigor, relevance, and societal benefit.

Knowledge translation strategies used for sustainability of an evidence-based intervention in child health: a multimethod qualitative study

BACKGROUND

Sustainability of evidence-based interventions (EBIs) is suboptimal in healthcare. Evidence on how knowledge translation (KT) strategies are used for the sustainability of EBIs in practice is lacking. This study examined what and how KT strategies were used to facilitate the sustainability of Alberta Family Integrated Care (FICare)™, a psychoeducational model of care scaled and spread across 14 neonatal intensive care units, in Alberta, Canada.

METHODS

First, we conducted an environmental scan of relevant documents to determine the use of KT strategies to support the sustainability of Alberta FICare™. Second, we conducted semi-structured interviews with decision makers and operational leaders to explore what and how KT strategies were used for the sustainability of Alberta FICare™, as well as barriers and facilitators to using the KT strategies for sustainability. We used the Expert Recommendations for Implementation Change (ERIC) taxonomy to code the strategies. Lastly, we facilitated consultation meetings with the Alberta FICare™ leads to share and gain insights and clarification on our findings.

RESULTS

We identified nine KT strategies to facilitate the sustainability of Alberta FICare™: Conduct ongoing training; Identify and prepare local champions; Research co-production; Remind clinicians; Audit and provide feedback; Change record systems; Promote adaptability; Access new funding; and Involve patients/consumers and family members. A significant barrier to the sustainability of Alberta FICare™ was a lack of clarity on who was responsible for the ongoing maintenance of the intervention. A key facilitator to sustainability of Alberta FICare was its alignment with the Maternal, Newborn, Child & Youth Strategic Clinical Network (MNCY SCN) priorities. Co-production between researchers and health system partners in the design, implementation, and scale and spread of Alberta FICare™ was critical to sustainability.

CONCLUSION

This research highlights the importance of clearly articulating who is responsible for continued championing for the sustainability of EBIs. Additionally, our research demonstrates that the adaptation of interventions must be considered from the onset of implementation so interventions can be tailored to align with contextual barriers for sustainability. Clear guidance is needed to continually support researchers and health system leaders in co-producing strategies that facilitate the long-term sustainability of effective EBIs in practice.

Essential content for teaching implementation practice in healthcare: a mixed-methods study of teams offering capacity-building initiatives

BACKGROUND

Applying the knowledge gained through implementation science can support the uptake of research evidence into practice; however, those doing and supporting implementation (implementation practitioners) may face barriers to applying implementation science in their work. One strategy to enhance individuals' and teams' ability to apply implementation science in practice is through training and professional development opportunities (capacity-building initiatives). Although there is an increasing demand for and offerings of implementation practice capacity-building initiatives, there is no universal agreement on what content should be included. In this study we aimed to explore what capacity-building developers and deliverers identify as essential training content for teaching implementation practice.

METHODS

We conducted a convergent mixed-methods study with participants who had developed and/or delivered a capacity-building initiative focused on teaching implementation practice. Participants completed an online questionnaire to provide details on their capacity-building initiatives; took part in an interview or focus group to explore their questionnaire responses in depth; and offered course materials for review. We analyzed a subset of data that focused on the capacity-building initiatives' content and curriculum. We used descriptive statistics for quantitative data and conventional content analysis for qualitative data, with the data sets merged during the analytic phase. We presented frequency counts for each category to highlight commonalities and differences across capacity-building initiatives.

RESULTS

Thirty-three individuals representing 20 capacity-building initiatives participated. Study participants identified several core content areas included in their capacity-building initiatives: (1) taking a process approach to implementation; (2) identifying and applying implementation theories, models, frameworks, and approaches; (3) learning implementation steps and skills; (4) developing relational skills. In addition, study participants described offering applied and pragmatic content (e.g., tools and resources), and tailoring and evolving the capacity-building initiative content to address emerging trends in implementation science. Study participants highlighted some challenges learners face when acquiring and applying implementation practice knowledge and skills.

CONCLUSIONS

This study synthesized what experienced capacity-building initiative developers and deliverers identify as essential content for teaching implementation practice. These findings can inform the development, refinement, and delivery of capacity-building initiatives, as well as future research directions, to enhance the translation of implementation science into practice.

Aging with Pride: Innovations in Dementia Empowerment and Action (IDEA)

Background

Sexual and gender minority (SGM) older adults and their care partners, compared to the general population, face unique vulnerabilities that exacerbate living with dementia, including elevated disparities in comorbidities, social isolation, and structural inequities, such as discrimination and lack of access to supports.

Methods

This paper describes the virtual adaptation process of the first-ever randomized controlled clinical trial intervention, Aging with Pride: Innovations in Dementia Empowerment and Action (IDEA), that was designed for SGM older adults living with dementia and their care partners and built upon the foundation of RDAD and NHAS.

Results

The virtual adaptation of IDEA was guided by the goals of accessibility, quality, ease of delivery, sustainability, and cultural relevance. The implementation required the development of a HIPPA-compliant online virtual platform, coach and participant virtual training, and modification of necessary intervention elements and materials, as needed. Based on the preliminary findings, the participants and intervention coaches responded well to the virtual adaptation of IDEA. When comparing to in-person delivery, the virtual delivery decreased attrition among both intervention participants and coaches.

Discussion

The virtual adaptation of the IDEA intervention resulted in preliminary, unexpected, yet potentially important benefits, including the ability to expand the reach of the intervention and decreased attrition. Virtual interventions are an emerging field for people living with dementia and their care partners and additional systematic research is needed to fully assess the benefits and limitations as well as to evaluate if specific subgroups are better served by differing delivery modalities.

A conceptual model for building program sustainability in public health settings: Learning from the implementation of the program sustainability action planning model and training curricula

Background

The emergence of implementation science has driven an increase in research examining the implementation of evidence-based programs and policies. However, there has been less attention through program sustainability. To achieve the full benefit of investment in program development and implementation, there must be an understanding of the factors that relate to sustainability; additionally, there is a need for a robust set of tools and trainings to support strategic long-term program sustainability. This paper presents results of our sustainability training intervention and a new conceptual model of sustainability. The proposed conceptual model builds upon the intervention design, further specifying the implementation strategy, strategy-mechanism linkages, and effect modifiers.

Methods

This research is part of the larger randomized control trial evaluating the effectiveness of the Program Sustainability Action Planning Model and Training Curricula. Specifically, this multimethod study establishes a conceptual model for program sustainability and related capacity-building interventions. The training intervention was delivered through workshops and technical assistance to 11 state tobacco control programs, principally entailing the development and implementation of a sustainability action plan. We utilize descriptive statistics and participant perspectives to evaluate the training intervention and propose an empirically-grounded conceptual model for sustainability capacity-building interventions in public health settings.

Results

Participants found intervention components (workshop, workbook, instructor and resources) to be effective. Overall, participants found the intervention improved their ability to develop sustainability action plans and assess their program and partners. Throughout the study, program managers emphasized the importance of the workshop in providing direction for their sustainability work and the value of robust, ongoing technical assistance. Program managers identified several factors that interfered with intervention reception including staff turnover, competing priorities, partnership challenges, and the COVID-19 pandemic.

Conclusion

The present study documents the development and implementation of a novel Program Sustainability Action Planning Model and Training Curricula, one of the first interventions designed to improve program sustainability. In addition, we present an empirically-grounded conceptual model for program sustainability. Considering the paucity of research in this understudied and undefined topic area, this is an important contribution that can serve as a framework for similar intervention designs and implementation efforts. Clinical Trail Registration: ClinicalTrails.gov identification number is NCT03598114.

Sustaining HIV Research in Resource-Limited Settings Using PLAN (People, Learning, Adapting, Nurturing): Evidence from the 4 Youth by Youth Project in Nigeria

PURPOSE OF REVIEW

Sustaining evidence-based interventions in resource-limited settings is a perennial challenge. Despite growing research on the significance of sustainability, few frameworks describe why and how to plan for sustainability in settings limited with resources. Drawing on a synthesis of the literature on sustainability, including the Dynamic Sustainability Framework, we review lessons learned from research to date, to point out a path forward for sustaining evidence-based interventions in resource-limited settings.

RECENT FINDINGS

We describe PLAN or why people learning, adapting, and nurturing the core values of an intervention can enhance its sustainability over time. PLAN is a dynamic framework that simplifies the process of planning for sustainability of evidence-based interventions throughout the lifecyle of an intervention, taking into consideration the people that matter as well as the learning, adaptation, and nurturing involved with understanding and studying the interactions between interventions/innovations, practice settings, intervention fit, and the broader ecological contexts in which implementation occurs. We use case-study data from our ongoing pragmatic HIV implementation trial, the 4 Youth by Youth project, to detail the value and implications of why people learning, adapting, and nurturing HIV interventions implemented in resource-limited settings matter. PLAN is designed to further the dialogue on ways research and practice teams can critically work to ensure the sustainability of their evidence-based interventions from the onset, particularly in settings and with populations limited with resources. It also illustrates how attention to sustainability from the beginning may foster actions necessary for sustained program → sustained benefits → sustained capacity → sustained value, but in the absence of early and active planning, none of this will occur. Ultimately, we hope to accelerate the sustainability of evidence-based HIV interventions, and making a PLAN at the bare minimum may ensure that the goals of continuing and maintaining desirable features of any evidence-based interventions can be realized.

Obesity prevention practices in early care and education settings: an adaptive implementation trial

Background

Despite the potential for Early Care and Education (ECE) settings to promote healthy habits, a gap exists between current practices and evidence-based practices (EBPs) for obesity prevention in childhood.

Methods

We will use an enhanced non-responder trial design to determine the effectiveness and incremental cost-effectiveness of an adaptive implementation strategy for Together, We Inspire Smart Eating (WISE), while examining moderators and mediators of the strategy effect. WISE is a curriculum that aims to increase children’s intake of carotenoid-rich fruits and vegetables through four evidence-based practices in the early care and education setting. In this trial, we will randomize sites that do not respond to low-intensity strategies to either (a) continue receiving low-intensity strategies or (b) receive high-intensity strategies. This design will determine the effect of an adaptive implementation strategy that adds high-intensity versus one that continues with low-intensity among non-responder sites. We will also apply explanatory, sequential mixed methods to provide a nuanced understanding of implementation mechanisms, contextual factors, and characteristics of sites that respond to differing intensities of implementation strategies. Finally, we will conduct a cost effectiveness analysis to estimate the incremental effect of augmenting implementation with high-intensity strategies compared to continuing low-intensity strategies on costs, fidelity, and child health outcomes.

Discussion

We expect our study to contribute to an evidence base for structuring implementation support in real-world ECE contexts, ultimately providing a guide for applying the adaptive implementation strategy in ECE for WISE scale-up. Our work will also provide data to guide implementation decisions of other interventions in ECE. Finally, we will provide the first estimate of relative value for different implementation strategies in this setting.

Trial registration

NCT05050539; 9/20/21.

Sustainability of breastfeeding interventions to reduce child mortality rates in low, middle-income countries: A systematic review of randomized controlled trials

Child mortality is the lowest it has ever been, but the burden of death in low- and middle-income countries (LMICs) is still prevalent, and the numbers average above the global mean. Breastfeeding contributes to the reduction of child mortality by improving chance of survival beyond childhood. Therefore, it is essential to examine how evidence-based breastfeeding interventions are being maintained in resource-constrained settings. Guided by Scheirer and Dearing's sustainability framework, the aim of this systematic review was to explore how evidence-based breastfeeding interventions implemented to address child mortality in LMICs are sustained. The literature search included randomized controlled trials (RCTs) of breastfeeding interventions from the following electronic databases: Cochrane Library, Global Health, PubMed, Scopus, and Web of Science. Literature selection and data extraction were completed according to the PRISMA guidelines. A narrative synthesis was used to investigate factors that contributed to sustainability failure or success. A total of 497 articles were identified through the database search. Only three papers were included in the review after the removal of duplicates and assessment for eligibility. The three RCTs included breastfeeding interventions predominately focusing on breastfeeding initiation and exclusivity in rural, semi-rural, and peri-urban areas in South Africa, Kenya, and India. The number of women included in the studies ranged from 901 to 3,890, and the duration of studies stretched from 6 weeks to 2.5 years. In two studies, sustainability was reported as the continuation of the intervention, and the other study outlined program dissemination and scale-up. Facilitators and barriers that influenced the sustainability of breastfeeding interventions were largely related to specific characteristics of the interventions (i.e., strong intervention implementers-facilitator; small number of CHWs involved-barrier). Optimizing the sustainability of breastfeeding interventions in LMICs is imperative to reduce child mortality. The focal point of implementation must be planning for sustainability to lead to continued benefits and changes in population outcomes. A defined action plan for sustainability needs to be included in both funding and research.

Contextual factors and mechanisms that influence sustainability: a realist evaluation of two scaled, multi-component interventions

Background

In 2012, Alberta Health Services created Strategic Clinical Networks^TM (SCNs) to develop and implement evidence-informed, clinician-led and team-delivered health system improvement in Alberta, Canada. SCNs have had several provincial successes in improving health outcomes. Little research has been done on the sustainability of these evidence-based implementation efforts.

Methods

We conducted a qualitative realist evaluation using a case study approach to identify and explain the contextual factors and mechanisms perceived to influence the sustainability of two provincial SCN evidence-based interventions, a delirium intervention for Critical Care and an Appropriate Use of Antipsychotics (AUA) intervention for Senior’s Health. The context (C) + mechanism (M) = outcome (O) configurations (CMOcs) heuristic guided our research.

Results

We conducted thirty realist interviews in two cases and found four important strategies that facilitated sustainability: Learning collaboratives, audit & feedback, the informal leadership role, and patient stories. These strategies triggered certain mechanisms such as sense-making, understanding value and impact of the intervention, empowerment, and motivation that increased the likelihood of sustainability. For example, informal leaders were often hands-on and influential to front-line staff. Learning collaboratives broke down professional and organizational silos and encouraged collective sharing and learning, motivating participants to continue with the intervention. Continual audit-feedback interventions motivated participants to want to perform and improve on a long-term basis, increasing the likelihood of sustainability of the two multi-component interventions. Patient stories demonstrated the interventions’ impact on patient outcomes, motivating staff to want to continue doing the intervention, and increasing the likelihood of its sustainability.

Conclusions

This research contributes to the field of implementation science, providing evidence on key strategies for sustainability and the underlying causal mechanisms of these strategies that increases the likelihood of sustainability. Identifying causal mechanisms provides evidence on the processes by which implementation strategies operate and lead to sustainability. Future work is needed to evaluate the impact of informal leadership, learning collaboratives, audit-feedback, and patient stories as strategies for sustainability, to generate better guidance on planning sustainable improvements with long term impact.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-07214-5.

A cross-case analysis of developing program sustainability and institutionalization in early stages of a multisite biomedical student diversity initiative

BACKGROUND

Grant funding often drives innovative programming in efforts to enhance diversity in biomedical fields, yet strategies for sustainability of grant-funded biomedical intervention are not well understood. Additionally, as funding agencies shift toward supporting institutional change to biomedical training, less is known about the extent to which sustainability strategies can support long-term institutionalization of the original goals of the grant-funded initiative. The purpose of this study is twofold: to identify strategies used by grant-funded programs for promoting sustainability, and to examine the interrelations between the concepts of sustainability and institutionalization during early stages of grant-funded biomedical career training efforts.

METHODS

We employed a multiple case study design and cross-case analysis using interviews of program administrators and participants from 10 undergraduate institutions that received Building Infrastructure Leading to Diversity (BUILD) awards funded by the National Institutes of Health (NIH).

RESULTS

BUILD sites engaged in the following strategies to develop program sustainability: 1) scaling and adapting to expand programmatic impact, 2) identifying additional funding and cost-cutting measures, 3) developing and maintaining infrastructure and structural operations, 4) leveraging relationships and with intra-and inter-institutional partners, and 5) and addressing hiring, policies, and reward systems at the institution. Senior administrative support supported program sustainability and early institutionalization, although we also identified situations where participants felt that they were on track for sustainable changes without administrative support or institutional change. Of the strategies identified, those that involve organizational and infrastructural changes contribute to early stages of institutionalization.

CONCLUSIONS

This study contributes to literature on organizational change by providing evidence of distinctions and interrelations between program sustainability efforts and institutionalization of change efforts in that some sustainability strategies can overlap with strategies to move toward institutionalization. The findings indicate the importance of program administrators developing early sustainability plans that also lead to institutionalization, as well as an opportunity for funding agencies to develop technical assistance on sustainability, organizational change, and institutionalization as a resource to support program administrators' efforts toward making lasting, structural change on their campuses.

PhD programs and the advancement of nursing science

Nurses are well-positioned to be groundbreaking researchers, scientists, leaders, and innovators to improve the health and well-being of individuals, families, and communities. Nurse scientists are needed to contribute to scientific discoveries that inform effective strategies to improve patient care and outcomes and to inform future policies. Thoughtful consideration is required about the preparation of nurse scientists to ensure they are equipped with the knowledge and skill sets to meet the needs of society. Evolving health needs and priority areas of inquiry along with an ever-increasing array of sophisticated methodologies and centrality of interdisciplinary teams to solve complex problems should drive how we prepare PhD students. This paper reflects a panel and subsequent dialogue with nurse leaders at the PhD summit held at the University of Pennsylvania in October 2019. Three aspects of PhD education and the advancement of nursing science are discussed 1) examining important elements to support nurse scientist development; 2) identifying key gaps in science that the discipline needs to address in educating the next generation of nurse scientists; and 3) preparing nurse scientists for the competitive funding environment.

Achieving Optimal Population Cardiovascular Health Requires an Interdisciplinary Team and a Learning Healthcare System: A Scientific Statement From the American Heart Association

Population cardiovascular health, or improving cardiovascular health among patients and the population at large, requires a redoubling of primordial and primary prevention efforts as declines in cardiovascular disease mortality have decelerated over the past decade. Great potential exists for healthcare systems-based approaches to aid in reversing these trends. A learning healthcare system, in which population cardiovascular health metrics are measured, evaluated, intervened on, and re-evaluated, can serve as a model for developing the evidence base for developing, deploying, and disseminating interventions. This scientific statement on optimizing population cardiovascular health summarizes the current evidence for such an approach; reviews contemporary sources for relevant performance and clinical metrics; highlights the role of implementation science strategies; and advocates for an interdisciplinary team approach to enhance the impact of this work.

The influence of healthcare financing on cardiovascular disease prevention in people living with HIV

BACKGROUND

People living with HIV are diagnosed with age-related chronic health conditions, including cardiovascular disease, at higher than expected rates. Medical management of these chronic health conditions frequently occur in HIV specialty clinics by providers trained in general internal medicine, family medicine, or infectious disease. In recent years, changes in the healthcare financing for people living with HIV in the U.S. has been dynamic due to changes in the Affordable Care Act. There is little evidence examining how healthcare financing characteristics shape primary and secondary cardiovascular disease prevention among people living with HIV. Our objective was to examine the perspectives of people living with HIV and their healthcare providers on how healthcare financing influences cardiovascular disease prevention.

METHODS

As part of the EXTRA-CVD study, we conducted in-depth, semi-structured interviews with 51 people living with HIV and 34 multidisciplinary healthcare providers and at three U.S. HIV clinics in Ohio and North Carolina from October 2018 to March 2019. Thematic analysis using Template Analysis techniques was used to examine healthcare financing barriers and enablers of cardiovascular disease prevention in people living with HIV.

RESULTS

Three themes emerged across sites and disciplines (1): healthcare payers substantially shape preventative cardiovascular care in HIV clinics (2); physician compensation tied to relative value units disincentivizes cardiovascular disease prevention efforts by HIV providers; and (3) grant-based services enable tailored cardiovascular disease prevention, but sustainability is limited by sponsor priorities.

CONCLUSIONS

With HIV now a chronic disease, there is a growing need for HIV-specific cardiovascular disease prevention; however, healthcare financing complicates effective delivery of this preventative care. It is important to understand the effects of evolving payer models on patient and healthcare provider behavior. Additional systematic investigation of these models will help HIV specialty clinics implement cardiovascular disease prevention within a dynamic reimbursement landscape.

TRIAL REGISTRATION

Clinical Trial Registration Number: NCT03643705 .

Using Collective Intelligence to identify barriers to implementing and sustaining effective Fundamental Movement Skill interventions: A rationale and application example

To have population-level impact, interventions must be effectively implemented and sustained under real-world conditions. Few Fundamental Movement Skill (FMS) interventions are implemented at scale, and even fewer are sustained in a way that allows for ongoing evaluation. There has been increasing recognition of applying systems thinking to investigate the multitude of influences on interventions. To improve research-practice translations, investigations need to incorporate synthesised perspective and collective input from intervention stakeholders. This study trials Collective Intelligence (CI) - an applied systems science approach - to understand barriers to the adoption, implementation and institutionalisation of effective FMS interventions for children and adolescents. A total of 58 barriers were generated and organised into 13 barrier categories. Participants voted to select 10 critical barriers and generated a structural map among the barriers to guide future action mapping. Barriers related to Government and Institutional factors and Curricular Conflicts were structured as fundamental drivers of the system of barriers. By presenting this application example, we aim to underline the considerations and alleviate barriers to conducting much needed implementation and sustainability studies in FMS interventions. CI also adds to the "tool box" to understand the complexity and functioning of public health interventions, such as those targeting physical activity behaviours.

Making change last? Exploring the value of sustainability approaches in healthcare: a scoping review

Background

Numerous models, tools and frameworks have been produced to improve the sustainability of evidence-based interventions. Due to the vast number available, choosing the most appropriate one is increasingly difficult for researchers and practitioners. To understand the value of such approaches, evidence warranting their use is needed. However, there is limited understanding of how sustainability approaches have been used and how they have impacted research or practice. This review aims to consolidate evidence on the application and impact of sustainability approaches in healthcare settings.

Methods

A systematic scoping review was designed to search for peer-reviewed publications detailing the use of sustainability approaches in practice. A 5-stage framework for scoping reviews directed the search strategy, and quality assessment was performed using the Mixed Method Appraisal Tool. Searches were performed through electronic citation tracking and snowballing of references. Articles were obtained through Web of Science, PubMed and Google Scholar. Six outcome variables for sustainability were explored to ascertain impact of approaches.

Results

This review includes 68 articles demonstrating the application of sustainability approaches in practice. Results show an increase in the use of sustainability approaches in peer-reviewed studies. Approaches have been applied across a range of healthcare settings, including primary, secondary, tertiary and community healthcare. Approaches are used for five main purposes, namely analysis, evaluation, guidance, assessment and planning. Results outline benefits (e.g. improved conceptualisation of sustainability constructs and improved ability to interpret sustainability data) and challenges (e.g. issues with approach constructs and difficulty in application) associated with using a sustainability approach in practice. Few articles (14/68) reported the sustainability outcome variables explored; therefore, the impact of approaches on sustainability remains unclear. Additional sustainability outcome variables reported in retrieved articles are discussed.

Conclusions

This review provides practitioners and researchers with a consolidated evidence base on sustainability approaches. Findings highlight the remaining gaps in the literature and emphasise the need for improved rigour and reporting of sustainability approaches in research studies. To guide future assessment and study of sustainability in healthcare settings an updated list of sustainability outcome variables is proposed.

Trial Registration

This review was registered on the PROSPERO database CRD 42016040081 in June 2016.

Advancing the pragmatic measurement of sustainment: a narrative review of measures

BACKGROUND

Sustainment, an outcome indicating an intervention continues to be implemented over time, has been comparatively less studied than other phases of the implementation process. This may be because of methodological difficulties, funding cycles, and minimal attention to theories and measurement of sustainment. This review synthesizes the literature on sustainment measures, evaluates the qualities of each measure, and highlights the strengths and gaps in existing sustainment measures. Results of the review will inform recommendations for the development of a pragmatic, valid, and reliable measure of sustainment.

METHODS

A narrative review of published sustainment outcome and sustainability measures (i.e., factors that influence sustainment) was conducted, including appraising measures in the Society of Implementation Research Collaboration (SIRC) instrument review project (IRP) and the Dissemination and Implementation Grid-Enabled Measures database initiative (GEM-D&I). The narrative review used a snowballing strategy by searching the reference sections of literature reviews and definitions of sustainability and sustainment. Measures used frequently and judged to be comprehensive and/or validated by a team of implementation scientists were extracted for analysis.

RESULTS

Eleven measures were evaluated. Three of the included measures were found in the SIRC-IRP, three in the GEM-D&I database, (one measure was in both databases) and six were identified in our additional searches. Thirteen constructs relating to sustainment were coded from selected measures. Measures covered a range of determinants for sustainment (i.e., construct of sustainability) as well as constructs of sustainment as an outcome. Strengths of the measures included, development by expert panels knowledgeable about particular interventions, fields or contexts, and utility in specific scenarios. A number of limitations were found in the measures analyzed including inadequate assessment of psychometric characteristics, being overly intervention or context specific, being lengthy and/or complex, and focusing on outer context factors.

CONCLUSION

There is a lack of pragmatic and psychometrically sound measures of sustainment that can be completed by implementation stakeholders within inner context settings (e.g., frontline providers, supervisors).

Ten recommendations for using implementation frameworks in research and practice

Background

Recent reviews of the use and application of implementation frameworks in implementation efforts highlight the limited use of frameworks, despite the value in doing so. As such, this article aims to provide recommendations to enhance the application of implementation frameworks, for implementation researchers, intermediaries, and practitioners.

Discussion

Ideally, an implementation framework, or multiple frameworks should be used prior to and throughout an implementation effort. This includes both in implementation science research studies and in real-world implementation projects. To guide this application, outlined are ten recommendations for using implementation frameworks across the implementation process. The recommendations have been written in the rough chronological order of an implementation effort; however, we understand these may vary depending on the project or context: (1) select a suitable framework(s), (2) establish and maintain community stakeholder engagement and partnerships, (3) define issue and develop research or evaluation questions and hypotheses, (4) develop an implementation mechanistic process model or logic model, (5) select research and evaluation methods (6) determine implementation factors/determinants, (7) select and tailor, or develop, implementation strategy(s), (8) specify implementation outcomes and evaluate implementation, (9) use a framework(s) at micro level to conduct and tailor implementation, and (10) write the proposal and report. Ideally, a framework(s) would be applied to each of the recommendations. For this article, we begin by discussing each recommendation within the context of frameworks broadly, followed by specific examples using the Exploration, Preparation, Implementation, Sustainment (EPIS) framework.

Summary

The use of conceptual and theoretical frameworks provides a foundation from which generalizable implementation knowledge can be advanced. On the contrary, superficial use of frameworks hinders being able to use, learn from, and work sequentially to progress the field. Following the provided ten recommendations, we hope to assist researchers, intermediaries, and practitioners to improve the use of implementation science frameworks.

Implementation and sustainment of diverse practices in a large integrated health system: a mixed methods study

BACKGROUND

One goal of health systems seeking to evolve into learning health systems is to accelerate the implementation and sustainment of evidence-based practices (EBPs). As part of this evolution, the Veterans Health Administration (VHA) developed the Innovation Ecosystem, which includes the Diffusion of Excellence (DoE), a program that identifies and diffuses Gold Status Practices (GSPs) across facilities. The DoE hosts an annual "Shark Tank" competition in which leaders bid on the opportunity to implement a GSP with 6 months of implementation support. Over 750 diverse practices were submitted in cohorts 2 and 3 of Shark Tank; 23 were designated GSPs and were implemented in 31 VA networks or facilities. As part of a national evaluation of the DoE, we identified factors contributing to GSP implementation and sustainment.

METHODS

Our sequential mixed methods evaluation of cohorts 2 and 3 of Shark Tank included semi-structured interviews with at least one representative from 30/31 implementing teams (N = 78/105 people invited) and survey responses from 29/31 teams (N = 39/47 invited). Interviews focused on factors influencing implementation and future sustainment. Surveys focused on sustainment 1.5-2 years after implementation. The Consolidated Framework for Implementation Research (CFIR) informed data collection and directed content analysis. Ordinal scales were developed inductively to rank implementation and sustainment outcomes.

RESULTS

Over 50% of teams (17/30) successfully implemented their GSP within the 6-month implementation period. Despite extensive implementation support, significant barriers related to centralized decision-making, staffing, and resources led to partial (n = 6) or no (n = 7) implementation for the remaining teams. While 12/17 initially successful implementation teams reported sustained use of their GSP, over half of the initially unsuccessful teams (n = 7/13) also reported sustained GSP use 1.5 years after the initial implementation period. When asked at 6 months, 18/27 teams with complete data accurately anticipated their future sustainability based on reported sustainment an average of 1.5 years later.

CONCLUSIONS

Most teams implemented within 6 months and/or sustained their GSP 1.5 years later. High levels of implementation and sustainment across diverse practices and teams suggest that VHA's DoE is a successful large-scale model of diffusion. Team predictions about sustainability after the first 6 months of implementation provide a promising early assessment and point of intervention to increase sustainability.

An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time

RE-AIM is a widely adopted, robust implementation science (IS) framework used to inform intervention and implementation design, planning, and evaluation, as well as to address short-term maintenance. In recent years, there has been growing focus on the longer-term sustainability of evidence-based programs, policies and practices (EBIs). In particular, investigators have conceptualized sustainability as the continued health impact and delivery of EBIs over a longer period of time (e.g., years after initial implementation) and incorporated the complex and evolving nature of context. We propose a reconsideration of RE-AIM to integrate recent conceptualizations of sustainability with a focus on addressing dynamic context and promoting health equity. In this Perspective, we present an extension of the RE-AIM framework to guide planning, measurement/evaluation, and adaptations focused on enhancing sustainability. We recommend consideration of: (1) extension of "maintenance" within RE-AIM to include recent conceptualizations of dynamic, longer-term intervention sustainability and "evolvability" across the life cycle of EBIs, including adaptation and potential de-implementation in light of changing and evolving evidence, contexts, and population needs; (2) iterative application of RE-AIM assessments to guide adaptations and enhance long-term sustainability; (3) explicit consideration of equity and cost as fundamental, driving forces that need to be addressed across RE-AIM dimensions to enhance sustainability; and (4) use or integration of RE-AIM with other existing frameworks that address key contextual factors and examine multi-level determinants of sustainability. Finally, we provide testable hypotheses and detailed research questions to inform future research in these areas.

Ten recommendations for using implementation frameworks in research and practice

Background

Recent reviews of the use and application of implementation frameworks in implementation efforts highlight the limited use of frameworks, despite the value in doing so. As such, this article aims to provide recommendations to enhance the application of implementation frameworks, for implementation researchers, intermediaries, and practitioners.

Discussion

Ideally, an implementation framework, or multiple frameworks should be used prior to and throughout an implementation effort. This includes both in implementation science research studies and in real-world implementation projects. To guide this application, outlined are ten recommendations for using implementation frameworks across the implementation process. The recommendations have been written in the rough chronological order of an implementation effort; however, we understand these may vary depending on the project or context: (1) select a suitable framework(s), (2) establish and maintain community stakeholder engagement and partnerships, (3) define issue and develop research or evaluation questions and hypotheses, (4) develop an implementation mechanistic process model or logic model, (5) select research and evaluation methods (6) determine implementation factors/determinants, (7) select and tailor, or develop, implementation strategy(s), (8) specify implementation outcomes and evaluate implementation, (9) use a framework(s) at micro level to conduct and tailor implementation, and (10) write the proposal and report. Ideally, a framework(s) would be applied to each of the recommendations. For this article, we begin by discussing each recommendation within the context of frameworks broadly, followed by specific examples using the Exploration, Preparation, Implementation, Sustainment (EPIS) framework.

Summary

The use of conceptual and theoretical frameworks provides a foundation from which generalizable implementation knowledge can be advanced. On the contrary, superficial use of frameworks hinders being able to use, learn from, and work sequentially to progress the field. Following the provided ten recommendations, we hope to assist researchers, intermediaries, and practitioners to improve the use of implementation science frameworks.

Conceptualizing and measuring sustainability of prevention programs, policies, and practices

A large knowledge gap exists regarding the measurement of sustainability of evidence-based prevention programs for mental and behavioral health. We interviewed 45 representatives of 10 grantees and 9 program officers within 4 Substance Abuse and Mental Health Services Administration prevention grant initiatives to identify experiences with implementation and sustainability barriers and facilitators; what "sustainability" means and what it will take to sustain their programs; and which Consolidated Framework for Implementation Research (CFIR) elements are important for sustainability. Lists of sustainability determinants and outcomes were then compiled from each data set and compared with one another. Analysis of themes from interviews and free lists revealed considerable overlap between sustainability determinants and outcomes. Four sustainability elements were identified by all three data sets (ongoing coalitions, collaborations, and networks and partnerships; infrastructure and capacity to support sustainability; community need for program; and ongoing evaluation of performance and outcomes), and 11 elements were identified by two of three data sets (availability of funding; consistency with organizational culture; evidence of positive outcomes; development of a plan for implementation and sustainment; presence of a champion; institutionalization and integration of program; institutional support and commitment; community buy-in and support; program continuity; supportive leadership; and opportunities for staff training). All but one of the CFIR domain elements (pressure from other states, tribes, or communities) were endorsed as important to sustainability by 50% or more of participants. It may be more important to implement a standardized process of eliciting determinants and outcomes of sustainability than to implement a single standardized instrument.

Conceptualizing and measuring sustainability of prevention programs, policies, and practices

A large knowledge gap exists regarding the measurement of sustainability of evidence-based prevention programs for mental and behavioral health. We interviewed 45 representatives of 10 grantees and 9 program officers within 4 Substance Abuse and Mental Health Services Administration prevention grant initiatives to identify experiences with implementation and sustainability barriers and facilitators; what "sustainability" means and what it will take to sustain their programs; and which Consolidated Framework for Implementation Research (CFIR) elements are important for sustainability. Lists of sustainability determinants and outcomes were then compiled from each data set and compared with one another. Analysis of themes from interviews and free lists revealed considerable overlap between sustainability determinants and outcomes. Four sustainability elements were identified by all three data sets (ongoing coalitions, collaborations, and networks and partnerships; infrastructure and capacity to support sustainability; community need for program; and ongoing evaluation of performance and outcomes), and 11 elements were identified by two of three data sets (availability of funding; consistency with organizational culture; evidence of positive outcomes; development of a plan for implementation and sustainment; presence of a champion; institutionalization and integration of program; institutional support and commitment; community buy-in and support; program continuity; supportive leadership; and opportunities for staff training). All but one of the CFIR domain elements (pressure from other states, tribes, or communities) were endorsed as important to sustainability by 50% or more of participants. It may be more important to implement a standardized process of eliciting determinants and outcomes of sustainability than to implement a single standardized instrument.

Sustainability of an Interprofessional Collaborative Practice Model for Population Health

Sustainability is an important concept in implementation science, yet little about sustainability is published in leadership journals. Leaders are charged on a daily basis with initiating programs that make a difference; however, they are often not well prepared to design effective strategies to sustain their efforts. In a value-based health care industry where facilitating access to care, enhancing the patient experience, improving health outcomes, and reducing the cost of care are imperative, creating sustainability strategies that achieve these results is key. In this article, we describe the successful efforts within an academic-practice partnership to implement a sustainable interprofessional collaborative practice model emphasizing transitional care coordination in chronic disease management for advancing population health with underserved populations. A sustainability framework is presented along with lessons learned.

Sustainment, Sustainability, and Spread Study (SSaSSy): protocol for a study of factors that contribute to the sustainment, sustainability, and spread of practice changes introduced through an evidence-based quality-improvement intervention in Canadian nursing homes

Background

Implementation scientists and practitioners, alike, recognize the importance of sustaining practice change, however post-implementation studies of interventions are rare. This is a protocol for the Sustainment, Sustainability and Spread Study (SSaSSy). The purpose of this study is to contribute to knowledge on the sustainment (sustained use), sustainability (sustained benefits), and spread of evidence-based practice innovations in health care. Specifically, this is a post-implementation study of an evidence-informed, Care Aide-led, facilitation-based quality-improvement intervention called SCOPE (Safer Care for Older Persons (in long-term care) Environments). SCOPE has been implemented in nursing homes in the Canadian Provinces of Manitoba (MB), Alberta (AB) and British Columbia (BC). Our study has three aims: (i) to determine the role that adaptation/contextualization plays in sustainment, sustainability and spread of the SCOPE intervention; (ii) to study the relative effects on sustainment, sustainability and intra-organizational spread of high-intensity and low-intensity post-implementation “boosters”, and a “no booster” condition, and (iii) to compare the relative costs and impacts of each booster condition.

Methods/design

SSaSSy is a two-phase mixed methods study. The overarching design is convergent, with qualitative and quantitative data collected over a similar timeframe in each of the two phases, analyzed independently, then merged for analysis and interpretation. Phase 1 is a pilot involving up to 7 units in 7 MB nursing homes in which SCOPE was piloted in 2016 to 2017, in preparation for phase 2. Phase 2 will comprise a quasi-experiment with two treatment groups of low- and high-intensity post-implementation “boosters”, and an untreated control group (no booster), using pretests and post-tests of the dependent variables relating to sustained care and management practices, and resident outcomes. Phase 2 will involve 31 trial sites in BC (17 units) and AB (14 units) nursing homes, where the SCOPE trial concluded in May 2019.

Discussion

This project stands to advance understanding of the factors that influence the sustainment of practice changes introduced through evidence-informed practice change interventions, and their associated sustainability. Findings will inform our understanding of the nature of the relationship of fidelity and adaptation to sustainment and sustainability, and afford insights into factors that influence the intra-organizational spread of practice changes introduced through complex interventions.

Health professional-delivered obesity prevention interventions during the first 1,000 days: A systematic review of external validity reporting

Background: Childhood obesity prevention interventions delivered by health professionals during the first 1,000 days show some evidence of effectiveness, particularly in relation to behavioural outcomes. External validity refers to how generalisable interventions are to populations or settings beyond those in the original study. The degree to which external validity elements are reported in such studies is unclear however. This systematic review aimed to determine the extent to which childhood obesity interventions delivered by health professionals during the first 1,000 days report on elements that can be used to inform generalizability across settings and populations. Methods: Eligible studies meeting study inclusion and exclusion criteria were identified through a systematic review of 11 databases and three trial registers. An assessment tool based on the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was used to assess the external validity of included studies. It comprised five dimensions: reach and representativeness of individuals, reach and representativeness of settings, implementation and adaptation, outcomes for decision making maintenance and/or institutionalisation. Two authors independently assessed the external validity of 20% of included studies; discrepancies were resolved, and then one author completed assessments of the remaining studies. Results: In total, 39 trials involving 46 interventions published between 1999 and 2019 were identified. The majority of studies were randomized controlled trials (n=24). Reporting varied within and between dimensions. External validity elements that were poorly described included: representativeness of individuals and settings, treatment receipt, intervention mechanisms and moderators, cost effectiveness, and intervention sustainability and acceptability. Conclusions: Our review suggests that more emphasis is needed on research designs that consider generalisability, and the reporting of external validity elements in early life childhood obesity prevention interventions. Important gaps in external validity reporting were identified that could facilitate decisions around the translation and scale-up of interventions from research to practice. Registration: PROSPERO CRD42016050793 03/11/16.

Health professional-delivered obesity prevention interventions during the first 1,000 days: A systematic review of external validity reporting

Background: Childhood obesity prevention interventions delivered by health professionals during the first 1,000 days show some evidence of effectiveness, particularly in relation to behavioural outcomes. External validity refers to how generalisable interventions are to populations or settings beyond those in the original study. The degree to which external validity elements are reported in such studies is unclear however. This systematic review aimed to determine the extent to which childhood obesity interventions delivered by health professionals during the first 1,000 days report on elements that can be used to inform generalizability across settings and populations. Methods: Eligible studies meeting study inclusion and exclusion criteria were identified through a systematic review of 11 databases and three trial registers. An assessment tool based on the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was used to assess the external validity of included studies. It comprised five dimensions: reach and representativeness of individuals, reach and representativeness of settings, implementation and adaptation, outcomes for decision making maintenance and/or institutionalisation. Two authors independently assessed the external validity of 20% of included studies; discrepancies were resolved, and then one author completed assessments of the remaining studies. Results: In total, 39 trials involving 46 interventions published between 1999 and 2019 were identified. The majority of studies were randomized controlled trials (n=24). Reporting varied within and between dimensions. External validity elements that were poorly described included: representativeness of individuals and settings, treatment receipt, intervention mechanisms and moderators, cost effectiveness, and intervention sustainability and acceptability. Conclusions: Our review suggests that more emphasis is needed on research designs that consider generalisability, and the reporting of external validity elements in early life childhood obesity prevention interventions. Important gaps in external validity reporting were identified that could facilitate decisions around the translation and scale-up of interventions from research to practice. Registration: PROSPERO CRD42016050793 03/11/16.