Advancing the pragmatic measurement of sustainment: a narrative review of measures

Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C): development and psychometric evaluation of a measure of sustainability determinants in the early childhood education and care setting

BACKGROUND

There is a need for valid and reliable measures of determinants of sustainability of public health interventions in early childhood education and care (ECEC) settings. This study aimed to develop and evaluate the psychometric and pragmatic properties of such a measure - the Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C).

METHODS

We undertook a two-phase process guided by the COnsensus-based Standards for the selection of health status Measurement INstruments checklist (COSMIN) and Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Phase 1 involved measure development; i.e., determining items and scales through an iterative process and assessment of face and content validity. Phase 2 involved the evaluation of psychometric and pragmatic properties. The 29-item measure completed by service executives (directors and nominated supervisors) was embedded in a larger survey from a national sample of Australian ECEC services assessing their implementation of nutrition and physical activity programs. Structural validity, concurrent validity, known groups validity, internal consistency, floor and ceiling effects, norms, and pragmatic qualities of the measure were assessed according to the PAPERS criteria.

RESULTS

The final measure contained 26 items, with respondents reporting how strongly they agreed or disagreed on a five-point Likert scale. Phase 1 assessments confirmed the relevance, and face and content validity of the scale. In Phase 2, we obtained 482 completed surveys, of which 84% (n = 405) completed the entire measure across 405 ECEC settings (one executive per service). Three of the four fit indices for the confirmatory factor analysis met the pre-specified criteria (SRMR = 0.056, CFI = 0.993, RMSEA = 0.067) indicating 'good' structural validity. The IMPRESS-C illustrated: 'good' internal consistency, with Cronbach's alpha values from 0.53 to 0.92; 'emerging' concurrent validity; 'poor' known groups validity; 'good' norms; and 'good' overall pragmatic qualities (cost, readability, length, and assessor burden).

CONCLUSIONS

The IMPRESS-C possesses strong psychometric and pragmatic qualities for assessing service executive-level perceptions of determinants influencing sustainment of public health interventions within ECEC settings. To achieve a full range of perspectives in this setting, future work should be directed to also develop and test measures of sustainability determinants at the implementer level (e.g., among individual educators and staff).

Randomized trial of the leadership and organizational change for implementation (LOCI) strategy in substance use treatment clinics

INTRODUCTION

Evidence-based practice (EBP) implementation represents a strategic change that requires alignment of leadership and support throughout organizations. Leadership and Organizational Change for Implementation (LOCI) is a multifaceted implementation strategy that aims to improve implementation leadership and climate within organizations through iterative cycles of leadership and climate assessment and feedback, leadership training and coaching, and strategic planning with upper-level leaders. This study tested the effects of LOCI on transformational and implementation leadership, implementation climate, implementation citizenship behavior, and EBP reach.

METHODS

A multiple cohort, cluster randomized trial tests the effect of LOCI in 60 clinics across nine behavioral health organizations in California and Arizona, USA. The study randomized clinics within organizations to either LOCI or a leadership training webinar control condition in three consecutive cohorts. Repeated web-based surveys of direct service providers (nLOCI = 201, nControl = 179) assessed leadership, implementation climate, and implementation citizenship over time. Multilevel autoregressive modeling was the primary statistical analysis such that providers (level-1) were nested within clinics (level-2). The study predicted between-condition differences at 4-, 8-, and 12-month follow-up assessments. Provider engagement in a fidelity monitoring process assessed reach of motivational interviewing (i.e., number of sessions recorded/submitted for fidelity coding). An independent sample t-test explored between condition differences in motivational interviewing reach.

RESULTS

Results indicated between condition differences at 4 months for implementation leadership, implementation climate, and implementation citizenship behavior such that greater improvements were evidenced in the LOCI condition compared to the control condition. Reach of MI was significantly greater in the LOCI vs control condition such that LOCI providers were significantly more likely to engage in the fidelity monitoring process (chi-square (1, n = 370) = 5.59, p = .018).

CONCLUSIONS

LOCI was developed based on organizational theories of strategic leadership and climate to affect organizational change processes that communicate that innovation implementation is expected, supported, and recognized as a value of the organization. The LOCI implementation strategy resulted in more positive hypothesized outcomes compared to the control condition. Organizational change strategies have utility for implementing health innovations in complex, multilevel contexts and for greater sustainment of facilitative leader behaviors, strategic implementation climate, and improved implementation outcomes.

TRIAL REGISTRATION

This study is registered with Clinicaltrials.gov gov (NCT03042832, 2 February 2017; retrospectively registered).

Sustainability of an outcome measure in outpatient stroke rehabilitation: A realist evaluation

RATIONALE

Only half of newly implemented evidence-based practices are sustained. Though poor sustainment can lead to negative consequences for clinical teams, organizations and patients, the causal explanations of sustainment are largely unknown.

AIMS AND OBJECTIVES

We aimed to ascertain how (mechanisms) and in what circumstances (context) a newly implemented outcome measure in rehabilitation was sustained or not (outcome).

METHODOLOGY

Informed by an integrated knowledge translation approach, we conducted a realist evaluation using a mixed method, embedded single case study design with data collection up to 18 months following the implementation of the Mayo-Portland Adaptability Inventory - version 4 (MPAI-4), a rehabilitation outcome measure. Quantitative data (survey and patient charts) was analysed using descriptive statistics, then integrated with qualitative data (interviews with 10 key informants) and analysed using inductive and deductive retroduction. We integrated the data to develop a case description and ultimately, to refine the programme theory to better understand the sustainability of the MPAI-4.

RESULTS

We linked context, mechanisms and outcomes, and also emphasized sustainability strategies in 18 explanations of how sustainability works. These explanations provide evidence for four overarching patterns: (1) implementation and sustainability phases are interconnected, (2) outcomes build on each other recursively, with patient benefits as the keystone outcome, (3) sustainment is achieved to varying levels across different sustainability outcomes [e.g., high level (e.g., rate of MPAI-4 scoring: 77.7%) to low level (e.g., rate of MPAI-4 application to clinical decision-making: 3.7%)] and (4) the work of sustaining the MPAI-4 is shared amongst different stakeholders.

CONCLUSION

Implementation teams can draw from this programme theory to improve the sustainment of outcomes measures while researchers could continue to refine the theory. Continued investigation of sustainability, including diverse and continuous sustainability outcomes, is needed to understand how to maintain improvements in quality of care and patient outcomes.

Understanding pathways from implementation to sustainment: a longitudinal, mixed methods analysis of promising practices implemented in the Veterans Health Administration

BACKGROUND

The Veterans Health Administration (VHA) is the United States largest learning health system. The Diffusion of Excellence (DoE) program is a large-scale model of diffusion that identifies and diffuses evidence-informed practices across VHA. During the period of 2016-2021, 57 evidence-informed practices were implemented across 82 VHA facilities. This setting provides a unique opportunity to understand sustainment determinants and pathways. Our objective was to characterize the longitudinal pathways of practices as they transition from initial implementation to long-term sustainment at each facility.

METHODS

A longitudinal, mixed-methods evaluation of 82 VHA facilities. Eighty-two facility representatives, chosen by leadership as points-of-contact for 57 DoE practices, were eligible for post-implementation interviews and annual sustainment surveys. Primary outcomes (implementation, sustainment), and secondary outcomes (institutionalization, effectiveness, anticipated sustainment) at four time-points were collected. We performed descriptive statistics and directed content analysis using Hailemariam et al.'s factors influencing sustainment.

RESULTS

After approximately five years post-implementation (e.g., 2021 sustainment outcomes), of the 82 facilities, about one-third fully sustained their practice compared to one-third that did not fully sustain their practice because it was in a "liminal" stage (neither sustained nor discontinued) or permanently discontinued. The remaining one-third of facilities had missing 2021 sustainment outcomes. A higher percentage of facilities (70%) had inconsistent primary outcomes (changing over time) compared to facilities (30%) with consistent primary outcomes (same over time). Thirty-four percent of facilities with sustained practices reported resilience since they overcame implementation and sustainment barriers. Facilities with sustained practices reported more positive secondary outcomes compared to those that did not sustain their practice. Key factors facilitating practice sustainment included: demonstrating practice effectiveness/benefit, sufficient organizational leadership, sufficient workforce, and adaptation/alignment with local context. Key factors hindering practice sustainment included: insufficient workforce, not able to maintain practice fidelity/integrity, critical incidents related to the COVID-19 pandemic, organizational leadership did not support sustainment of practice, and no ongoing support.

CONCLUSIONS

We identified diverse pathways from implementation to sustainment, and our data underscore that initial implementation outcomes may not determine long-term sustainment outcomes. This longitudinal evaluation contributes to understanding impacts of the DoE program, including return on investment, achieving learning health system goals, and insights into achieving high-quality healthcare in VHA.

Maintaining Program Fidelity in a Changing World: National Implementation of a School-Based HIV Prevention Program

Large-scale, evidence-based interventions face challenges to program fidelity of implementation. We developed implementation strategies to support teachers implementing an evidence-based HIV prevention program in schools, Focus on Youth in The Caribbean (FOYC) and Caribbean Informed Parents and Children Together (CImPACT) in The Bahamas. We examined the effects of these implementation strategies on teachers' implementation in the subsequent year after the initial implementation during the COVID-19 pandemic. Data were collected from 79 Grade 6 teachers in 24 government elementary schools. Teachers completed training workshops and a pre-implementation questionnaire to record their characteristics and perceptions that might affect their program fidelity. School coordinators and peer mentors provided teachers with monitoring, feedback, and mentoring. In Year 1, teachers on average taught 79.3% of the sessions and 80.8% of core activities; teachers in Year 2 covered 84.2% of sessions and 72.9% of the core activities. Teachers with "good" or "excellent" school coordinators in the second year taught significantly more sessions on average (7.8 vs. 7.0, t = 2.04, P < 0.05) and more core activities (26.3 vs. 23.0, t = 2.41, P < 0.05) than teachers with "satisfactory" coordinators. Teachers who had a "good" or "satisfactory" mentor taught more sessions than teachers who did not have a mentor (7.9 vs. 7.3; t = 2.22; P = 0.03). Two-level mixed-effects model analysis indicated that teachers' program fidelity in Year 1, confidence in the execution of core activities, and school coordinators' performance were significantly associated with Year 2 implementation dose. Implementation of FOYC + CImPACT was significantly associated with improved student outcomes. Teachers maintained high fidelity to a comprehensive HIV prevention program over 2 years during the COVID-19 pandemic. Future program implementers should consider additional implementation support to improve the implementation of school-based programs.

Sustaining health obesity prevention programs: Lessons from real-world population settings

Current understanding of what sustains public health programs is derived primarily from evaluations of programs delivered for only a few years. To improve program planning and knowledge on program sustainment, we examined real-world population programs that have been delivered for >2 years. Our focus was physical activity and nutrition programs for obesity prevention. We identified programs through published literature and searched through publicly available information on the theoretical factors determining sustainment. We reviewed 90 programs and explored intervention characteristics, provider, delivery and process level factors, and the outer environment influences. Programs were sustained on average 15 years and more frequently characterised as behavioural change skills programs targeting children, funded by non-government organisations and delivered by community organisations. Most programs had undergone some modification (n = 55). Differences between programs were observed across key factors: Programs sustained 15+ years were mainly behaviour skills programs (n = 21); and characterised by interactive designs (i.e., face-to-face or digital) over static innovations (i.e., print material) compared with those sustained <15 years (p = 0.024). While government funding supported many of the programs (n = 45;), those sustained 15+ years were more likely to have been commercially funded (p = 0.044); and were less likely to have current community involvement (p = 0.013). Differences in ownership and funding were also observed across countries. While multiple factors may influence program sustainment, longer delivered programs where distinguished by their program characteristics and outer contextual factors, suggesting such factors may have an important role in sustaining programs in the longer-term.

A bibliographic review of sustainability research output and investment in 10 leading public health journals across three time periods

Objectives

Long-term delivery, or sustainability, of evidence-based interventions is necessary for public health benefits to be realised. However, sustainment of effective interventions is poor. Understanding the evidence-base and identifying potential gaps is necessary to inform where future research efforts are most warranted.

Study design

We undertook a repeat cross-sectional bibliographic review of research published in 10 public health journals across three time periods (2010, 2015 and 2020/2021).

Methods

Studies were eligible if they were a data-based study or review article. Studies were assessed as to whether they focused on sustainability. The percentage of public health research studies assessing sustainability overall and by the three time periods was calculated. The association between time period and the proportion of sustainability articles was assessed using logistic regression. Descriptive statistics were used to summarise study characteristics overall and by time period.

Results

10,588 data-based articles were identified, of which 1.3 % (n = 136) focused on sustainability. There was a statistically significant association between time period and the proportion of sustainability research, with a slight increase across the three time periods: 0.3 % (95 % CI: 0.1 %, 0.7 %) in 2010, 1.4 % (95 % CI: 1.0 %, 1.9 %) in 2015 and 1.6 % (95 % CI: 1.3 %, 1.9 %) in 2020/2021. Most research was descriptive/epidemiological (n = 69, 51 %), few focused on measurement (n = 2, 1.5 %) and none on cost effectiveness. Only one intervention study assessed the effect of specific sustainability strategies.

Conclusions

This bibliographic review highlights the need for more public health research on sustainability, particularly in the areas of measurement, sustainability interventions, and cost effectiveness.

Implementation and dissemination of home- and community-based interventions for informal caregivers of people living with dementia: a systematic scoping review

BACKGROUND

Informal caregivers of people with dementia (PwD) living at home are often the primary source of care, and, in their role, they often experience loss of quality of life. Implementation science knowledge is needed to optimize the real-world outcomes of evidence-based interventions (EBIs) for informal caregivers. This scoping review aims to systematically synthesize the literature that reports implementation strategies employed to deliver home- and community-based EBIs for informal caregivers of PwD, implementation outcomes, and the barriers and facilitators to implementation of these EBIs.

METHODS

Embase, MEDLINE, Web of Science, and Cochrane Library were searched from inception to March 2021; included studies focused on "implementation science," "home- and community-based interventions," and "informal caregivers of people with dementia." Titles and abstracts were screened using ASReview (an innovative AI-based tool for evidence reviews), and data extraction was guided by the ERIC taxonomy, the Implementation Outcome Framework, and the Consolidated Framework for Implementation Science Research; each framework was used to examine a unique element of implementation.

RESULTS

Sixty-seven studies were included in the review. Multicomponent (26.9%) and eHealth (22.3%) interventions were most commonly reported, and 31.3% of included studies were guided by an implementation science framework. Training and education-related strategies and provision of interactive assistance were the implementation strategy clusters of the ERIC taxonomy where most implementation strategies were reported across the reviewed studies. Acceptability (82.1%), penetration (77.6%), and appropriateness (73.1%) were the most frequently reported implementation outcomes. Design quality and packaging (intervention component suitability) and cosmopolitanism (partnerships) constructs, and patient's needs and resources and available resources (infrastructure) constructs as per the CFIR framework, reflected the most frequently reported barriers and facilitators to implementation.

CONCLUSION

Included studies focused largely on intervention outcomes rather than implementation outcomes and lacked detailed insights on inner and outer setting determinants of implementation success or failure. Recent publications suggest implementation science in dementia research is developing but remains in nascent stages, requiring future studies to apply implementation science knowledge to obtain more contextually relevant findings and to structurally examine the mechanisms through which implementation partners can strategically leverage existing resources and regional networks to streamline local implementation. Mapping local evidence ecosystems will facilitate structured implementation planning and support implementation-focused theory building.

TRIAL REGISTRATION

Not applicable.

Implementing strategies to prevent infections in acute-care settings

This document introduces and explains common implementation concepts and frameworks relevant to healthcare epidemiology and infection prevention and control and can serve as a stand-alone guide or be paired with the "SHEA/IDSA/APIC Compendium of Strategies to Prevent Healthcare-Associated Infections in Acute Care Hospitals: 2022 Updates," which contain technical implementation guidance for specific healthcare-associated infections. This Compendium article focuses on broad behavioral and socio-adaptive concepts and suggests ways that infection prevention and control teams, healthcare epidemiologists, infection preventionists, and specialty groups may utilize them to deliver high-quality care. Implementation concepts, frameworks, and models can help bridge the "knowing-doing" gap, a term used to describe why practices in healthcare may diverge from those recommended according to evidence. It aims to guide the reader to think about implementation and to find resources suited for a specific setting and circumstances by describing strategies for implementation, including determinants and measurement, as well as the conceptual models and frameworks: 4Es, Behavior Change Wheel, CUSP, European and Mixed Methods, Getting to Outcomes, Model for Improvement, RE-AIM, REP, and Theoretical Domains.

A Citation Review of 83 Dissemination and Implementation Theories, Models, or Frameworks Utilized in U.S.-Based Aging Research

BACKGROUND AND OBJECTIVES

Dissemination-implementation.org outlines 110 theories, models, and frameworks (TMFs): we conducted a citation analysis on 83 TMFs, searching Web of Science and PubMed databases.

RESEARCH DESIGN AND METHODS

Search terms were broad and included "aging," "older," "elderly," and "geriatric." We extracted each TMF in identified articles from inception through January 28, 2022. Included articles must have used a TMF in research or quality improvement work directly linked to older adults within the United States.

RESULTS

We reviewed 2,681 articles of which 295 articles cited at least one of 56 TMFs. Five TMFs represented 50% of the citations: Reach, Effectiveness, Adoption, Implementation, and Maintenance 1.0, Consolidated Framework for Implementation Research, Greenhalgh Diffusion of Innovation in Service Organizations, Quality Enhancement Research Initiative, Community-Based Participatory Research, and Promoting Action on Research Implementation in Health Services. TMF application varied and there was a steady increase in TMF citations over time, with a 2- to 3-fold increase in citations in 2020-2021. We identified that only 41% of TMF use was meaningful.

DISCUSSION AND IMPLICATIONS

Our results suggest TMF utilization is increasing in aging research, but there is a need to more meaningful utilize TMFs. As the population of older adults continues to grow, there will be increasing demand for effective evidence-based practices and models of care to be quickly and effectively translated into routine care. Use of TMFs is critical to building such evidence and to identifying and evaluating methods to support this translation.

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review

BACKGROUND

Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings.

METHODS

Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed.

RESULTS

A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants.

CONCLUSIONS

This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field.

TRIAL REGISTRATION

This review was prospectively registered with Research Registry (reviewregistry1097), March 2021.

Understanding How Newly Implemented Rehabilitation Best Practices Are Sustained: A Realist Review

OBJECTIVE

We conducted a realist review to understand how (mechanism) and in what circumstances (context) evidence-based practices are sustained in rehabilitation (outcome).

DATA SOURCES

MEDLINE, Embase, reference lists, and targeted websites.

STUDY SELECTION

Two independent reviewers calibrated study selection; then 1 reviewer screened all titles and abstracts, while the second reviewer screened a random 20%. We repeated this process for full texts. We included 115 documents representing 61 implementation projects (8.9% of identified documents). Included documents described implementation projects in which physical therapists, occupational therapists, and/or speech-language pathologists were the target users of an evidence-based practice.

DATA EXTRACTION

Two reviewers repeated the independent process described in study selection to extract basic study and sustainability characteristics as well as context, mechanism, outcome, and strategy text.

DATA SYNTHESIS

Using basic numerical analyses, we found that only 54% of evidence-based practices in rehabilitation are sustained. Furthermore, while authors who reported sustainability planning sustained the practice 94% of the time, sustainability planning in rehabilitation is rare (only reported 26% of the time). Extracted text was synthesized using the realist technique of inductive and deductive retroduction in which context, mechanism, outcome, and strategy text are combined into narrative explanations of how sustainability works. To inform these explanations, we applied normalization process theory and the theory of planned behavior. Collectively, the 52 identified narratives provide evidence for 3 patterns: (1) implementation and sustainability phases are interconnected, (2) continued use of the evidence-based practice can be interpreted as the ultimate sustainability outcome, and (3) intermediate sustainability outcomes (ie, fit/alignment, financial support, benefits, expertise) can become contextual features influencing other sustainability outcomes.

CONCLUSIONS

Implementation teams can use the narrative explanations generated in this review to optimize sustainability planning. This can sustain practice changes and improve quality of care and patient outcomes. Future research should seek to iteratively refine the proposed narrative explanations.

School-level factors associated with the sustainment of weekly physical activity scheduled in Australian elementary schools: an observational study

Background

We aimed to: (1) identify school-level factors associated with the sustainment of weekly physical activity (PA) scheduled in elementary schools following withdrawal of effective implementation support; and (2) determine teacher’s perceived usefulness of suggested strategies for sustaining the scheduling of weekly PA.

Methods

A secondary exploratory analysis was employed of data from the intervention arm (n = 31 schools) of a randomised controlled trial. Self-report survey data from 134 classroom teachers in New South Wales, Australia, collected following withdrawal of initial implementation support (follow-up T1) and six-months following completion of support (follow-up T2) were used. The outcomes of sustainment of weekly overall PA and energisers (short classroom PA breaks) scheduled were measured via teachers’ completion of a daily activity logbook, with results presented as the difference in mean minutes of PA and energisers scheduled at T1 and T2. An adapted version of the Program Sustainability Assessment Tool (PSAT) was used to measure capacity for program sustainability across seven key domains at follow-up T2. Linear mixed regressions were conducted to evaluate associations between school-level sociodemographic characteristics (e.g., school size, remoteness, and type), teacher-reported school factors (i.e., seven adapted PSAT domains) and the sustainment of PA and energisers scheduled across the school week. Perceived usefulness of 14 proposed sustainability strategies was measured via the teacher survey at follow-up T2 and reported descriptively.

Results

No school-level factor was statistically associated with the sustainment of overall weekly PA or energisers scheduled. Teacher-reported factors in two PSAT domains – ‘strategic planning’ and ‘program evaluation’ were statistically negatively associated with the sustainment of weekly energisers scheduled (− 6.74, 95% CI: − 13.02; − 0.47, p = 0.036 and − 6.65, 95% CI: − 12.17; − 1.12, p = 0.019 respectively). The proposed support sustainability strategy – ‘provision of PA equipment packs that enable energisers or integrated lessons’ was perceived useful by the most teachers (85%).

Conclusions

Further research is required to explore additional contextual-specific, and end-user appropriate factors associated with schools’ sustainment of weekly PA scheduled. This will help accurately inform the development of strategies to address these determinants and support the sustainment and long-term benefits of school-based health interventions more broadly.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-022-13732-6.

Sustainability in pediatric hospitals: An exploration at the intersection of quality improvement and implementation science

Background

Although new evidence-based practices are frequently implemented in clinical settings, many are not sustained, limiting the intended impact. Within implementation science, there is a gap in understanding sustainability. Pediatric healthcare settings have a robust history of quality improvement (QI), which includes a focus on continuation of change efforts. QI capability and sustainability capacity, therefore, serve as a useful concept for connecting the broader fields of QI and implementation science to provide insights on improving care. This study addresses these gaps in understanding of sustainability in pediatric settings and its relationship to QI.

Methods

This is a cross-sectional observational study conducted within pediatric academic medical centers in the United States. Clinicians surveyed worked with one of three evidence-based clinical programs: perioperative antimicrobial stewardship prescribing, early mobility in the intensive care unit, and massive blood transfusion administration. Participants completed two assessments: (1) the Clinical Sustainability Assessment Tool (CSAT) and (2) a 19-question assessment that included demographics and validation questions, specifically a subset of questions from the Change Process Capability Questionnaire, a QI scale. Initial descriptive and bivariate analyses were conducted prior to building mixed-effects models relating perceived QI to clinical sustainability capacity.

Results

A total of 181 individuals from three different programs and 30 sites were included in the final analyses. QI capability scores were assessed as a single construct (5-point Likert scale), with an average response of 4.16 (higher scores indicate greater QI capability). The overall CSAT score (7-point Likert scale) was the highest for massive transfusion programs (5.51, SD = 0.91), followed by early mobility (5.25, SD = 0.92) and perioperative antibiotic prescribing (4.91, SD = 1.07). Mixed-effects modeling illustrated that after controlling for person and setting level variables, higher perceptions of QI capabilities were significantly related to overall clinical sustainability.

Conclusion

Organizations and programs with higher QI capabilities had a higher sustainability capacity, even when controlling for differences at the individual and intervention levels. Organizational factors that enable evidence-based interventions should be further studied, especially as they relate to sustainability. Issues to be considered by practitioners when planning for sustainability include bedside provider perceptions, intervention achievability, frequency of delivery, and organizational influences.

Do the Expert Recommendations for Implementing Change (ERIC) strategies adequately address sustainment?

Background

Sustainability science is an emerging area within implementation science. There is limited evidence regarding strategies to best support the continued delivery and sustained impact of evidence-based interventions (EBIs). To build such evidence, clear definitions, and ways to operationalize strategies specific and/or relevant to sustainment are required. Taxonomies and compilations such as the Expert Recommendations for Implementing Change (ERIC) were developed to describe and organize implementation strategies. This study aimed to adapt, refine, and extend the ERIC compilation to incorporate an explicit focus on sustainment. We also sought to classify the specific phase(s) of implementation when the ERIC strategies could be considered and applied.

Methods

We used a two-phase iterative approach to adapt the ERIC. This involved: (1) adapting through consensus (ERIC strategies were mapped against barriers to sustainment as identified via the literature to identify if existing implementation strategies were sufficient to address sustainment, needed wording changes, or if new strategies were required) and; (2) preliminary application of this sustainment-explicit ERIC glossary (strategies described in published sustainment interventions were coded against the glossary to identify if any further amendments were needed). All team members independently reviewed changes and provided feedback for subsequent iterations until consensus was reached. Following this, and utilizing the same consensus process, the Exploration, Preparation, Implementation and Sustainment (EPIS) Framework was applied to identify when each strategy may be best employed across phases.

Results

Surface level changes were made to the definitions of 41 of the 73 ERIC strategies to explicitly address sustainment. Four additional strategies received deeper changes in their definitions. One new strategy was identified: Communicate with stakeholders the continued impact of the evidence-based practice. Application of the EPIS identified that at least three-quarters of strategies should be considered during preparation and implementation phases as they are likely to impact sustainment.

Conclusion

A sustainment-explicit ERIC glossary is provided to help researchers and practitioners develop, test, or apply strategies to improve the sustainment of EBIs in real-world settings. Whilst most ERIC strategies only needed minor changes, their impact on sustainment needs to be tested empirically which may require significant refinement or additions in the future.

Review article: A primer for clinical researchers in the emergency department: Part XII. Sustainability of improvements in care: An introduction

Despite an increased focus on ways to improve implementation of evidence and de-implementation of practices with no known benefit, there is limited guidance on how to sustain these improvements. This review provides an introduction to sustainability of improvements in care and sustainability research, discussing how to support sustainability in practice and detailing a sustainability research agenda for the emergency medicine setting.

Applying dissemination and implementation research methods to translate a school-based asthma program

Our school-based asthma program has reduced asthma exacerbations for youth with health disparities in the Denver metropolitan area, due partly to addressing social determinants of health, such as access to health care and medications. Dissemination and implementation (D&I) science approaches accelerate the translation of evidence-based programs into routine practice. D&I methods are being applied more commonly to improve health equity. The purpose of this publication was to give an overview of D&I research methods, using our school-based asthma program as an example. To successfully scale out our program across the state of Colorado, we are applying a D&I framework that guides the adaptation of our existing implementation approach to better meet our stakeholders' local context-the Exploration, Preparation, Implementation, Sustainment framework. In a pragmatic trial design, we will evaluate the outcomes of implementing the program across 5 Colorado regions, with attention to health equity, using a second commonly used D&I framework-Reach, Effectiveness, Adoption, Implementation, and Maintenance. Our central hypothesis is that our program will have broad and equitable reach to eligible students (primary outcome) and will reduce asthma attacks and symptoms. This D&I approach accelerates dissemination of our program and is an applicable process for translating other effective allergy/asthma programs to address asthma and allergy-related disparities.

Sustainability of evidence-based practices in the management of infants with bronchiolitis in hospital settings - a PREDICT study protocol

Background

Understanding how and why de-implementation of low-value practices is sustained remains unclear. The Paediatric Research in Emergency Departments International CollaboraTive (PREDICT) Bronchiolitis Knowledge Translation (KT) Study was a cluster randomised controlled trial conducted in 26 Australian and New Zealand hospitals (May-November 2017). Results showed targeted, theory-informed interventions (clinical leads, stakeholder meetings, train-the-trainer workshop, targeted educational package, audit/feedback) were effective at reducing use of five low-value practices for bronchiolitis (salbutamol, glucocorticoids, antibiotics, adrenaline and chest x-ray) by 14.1% in acute care settings. The primary aim of this study is to determine the sustainability (continued receipt of benefits) of these outcomes at intervention hospitals two-years after the removal of study supports. Secondary aims are to determine sustainability at one-year after removal of study support at intervention hospitals; improvements one-and-two years at control hospitals; and explore factors that influence sustainability at intervention hospitals and contribute to improvements at control hospitals.

Methods

A mixed-methods study design. The quantitative component is a retrospective medical record audit of bronchiolitis management within 24 hours of emergency department (ED) presentations at 26 Australian (n = 20) and New Zealand (n = 6) hospitals, which participated in the PREDICT Bronchiolitis KT Study. Data for a total of 1800 infants from intervention and control sites (up to 150 per site) will be collected to determine if improvements (i.e., no use of all five low-value practices) were sustained two- years (2019) post-trial (primary outcome; composite score); and a further 1800 infants from intervention and control sites will be collected to determine sustained improvements one- year (2018) post-trial (secondary outcome). An a priori definition of sustainability will be used. The qualitative component will consist of semi-structured interviews with three to five key emergency department and paediatric inpatient medical and nursing staff per site (total n = 78-130). Factors that may have contributed to sustaining outcomes and/or interventions will be explored and mapped to an established sustainability framework.

Discussion

This study will improve our understanding of the sustainability of evidence-based bronchiolitis management in infants. Results will also advance implementation science research by informing future de-implementation strategies to reduce low-value practices and sustain practice change in paediatric acute care.

Trial registration

Australian and New Zealand Clinical Trials Registry No: ACTRN12621001287820.

Developing an implementation research logic model: using a multiple case study design to establish a worked exemplar

Background

Implementation science frameworks explore, interpret, and evaluate different components of the implementation process. By using a program logic approach, implementation frameworks with different purposes can be combined to detail complex interactions. The Implementation Research Logic Model (IRLM) facilitates the development of causal pathways and mechanisms that enable implementation. Critical elements of the IRLM vary across different study designs, and its applicability to synthesizing findings across settings is also under-explored. The dual purpose of this study is to develop an IRLM from an implementation research study that used case study methodology and to demonstrate the utility of the IRLM to synthesize findings across case sites.

Method

The method used in the exemplar project and the alignment of the IRLM to case study methodology are described. Cases were purposely selected using replication logic and represent organizations that have embedded exercise in routine care for people with cancer or mental illness. Four data sources were selected: semi-structured interviews with purposely selected staff, organizational document review, observations, and a survey using the Program Sustainability Assessment Tool (PSAT). Framework analysis was used, and an IRLM was produced at each case site. Similar elements within the individual IRLM were identified, extracted, and re-produced to synthesize findings across sites and represent the generalized, cross-case findings.

Results

The IRLM was embedded within multiple stages of the study, including data collection, analysis, and reporting transparency. Between 33-44 determinants and 36-44 implementation strategies were identified at sites that informed individual IRLMs. An example of generalized findings describing “intervention adaptability” demonstrated similarities in determinant detail and mechanisms of implementation strategies across sites. However, different strategies were applied to address similar determinants. Dependent and bi-directional relationships operated along the causal pathway that influenced implementation outcomes.

Conclusions

Case study methods help address implementation research priorities, including developing causal pathways and mechanisms. Embedding the IRLM within the case study approach provided structure and added to the transparency and replicability of the study. Identifying the similar elements across sites helped synthesize findings and give a general explanation of the implementation process. Detailing the methods provides an example for replication that can build generalizable knowledge in implementation research.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-022-00337-8.

Using qualitative comparative analysis to uncover multiple pathways to program sustainment: implications for community-based youth substance misuse prevention

Background

In order to achieve wide-scale impact in community settings, programs must be sustained. Theory and empirical evidence suggest that intervention characteristics, organizational context, capacity for program implementation, and processes related to implementation are associated with continued program delivery. However, few studies examine how combinations of these factors work together in different settings to influence program sustainment.

Methods

Using scales specified in the Program Sustainability Assessment Tool (PSAT), the current cross-sectional study aims to identify the necessary and sufficient conditions for the sustainment of the Strengthening Families Program for Parents and Youth 10-14 (SFP 10-14). Staff (n = 59) at SFP 10-14 implementation sites across Washington State completed an online survey reporting on their current level of SFP 10-14 sustainment. They also completed PSAT, with eight scales designed to assess conditions that consistently produce sustainment. Data were analyzed using qualitative comparative analysis.

Results

Environmental support was the only necessary condition for sustainment success. Four solutions sufficient to achieve sustainment were also identified. These included the combined presence of (1) environmental support, organizational capacity, and funding stability; (2) environmental support, organizational capacity, communication, and program evaluation, in the absence of strategic planning; (3) environmental support, organizational capacity, program evaluation, and partnerships, in the absence of strategic planning; and (4) environmental support, communication, partnerships, and funding stability, in the absence of program evaluation.

Conclusions

Environmental support in combination with organizational capacity appeared to most consistently produce sustainment of SFP 10-14 programs in Washington State. Program providers will benefit from a focusing on enhancing those conditions to increase program sustainment.

Adaptation and Validation of the Program Sustainability Assessment Tool (PSAT) for Use in the Elementary School Setting

There is a lack of valid and reliable measures of determinants of sustainability specific to public health interventions in the elementary school setting. This study aimed to adapt and evaluate the Program Sustainability Assessment Tool (PSAT) for use in this setting. An expert reference group adapted the PSAT to ensure face validity. Elementary school teachers participating in a multi-component implementation intervention to increase their scheduling of physical activity completed the adapted PSAT. Structural validity was assessed via confirmatory factor analysis. Convergent validity was assessed using linear mixed regression evaluating the associations between scheduling of physical activity and adapted PSAT scores. Cronbach's alpha was used to evaluate internal consistency and intracluster correlation coefficients for interrater reliability. Floor and ceiling effects were also evaluated. Following adaptation and psychometric evaluation, the final measure contained 26 items. Domain Cronbach's alpha ranged from 0.77 to 0.92. Only one domain illustrated acceptable interrater reliability. Evidence for structural validity was mixed and was lacking for convergent validity. There were no floor and ceiling effects. Efforts to adapt and validate the PSAT for the elementary school setting were mixed. Future work to develop and improve measures specific to public health program sustainment that are relevant and psychometrically robust for elementary school settings are needed.

Reliability and validity of a Spanish-language measure assessing clinical capacity to sustain Paediatric Early Warning Systems (PEWS) in resource-limited hospitals

BACKGROUND

Paediatric Early Warning Systems (PEWSs) improve identification of deterioration, however, their sustainability has not been studied. Sustainability is critical to maximise impact of interventions like PEWS, particularly in low-resource settings. This study establishes the reliability and validity of a Spanish-language Clinical Sustainability Assessment Tool (CSAT) to assess clinical capacity to sustain interventions in resource-limited hospitals.

METHODS

Participants included PEWS implementation leadership teams of 29 paediatric cancer centres in Latin America involved in a collaborative to implement PEWS. The CSAT, a sustainability assessment tool validated in high-resource settings, was translated into Spanish and distributed to participants as an anonymous electronic survey. Psychometric, confirmatory factor analysis (CFA), and multivariate analyses were preformed to assess reliability, structure and initial validity. Focus groups were conducted after participants reviewed CSAT reports to assess their interpretation and utility.

RESULTS

The CSAT survey achieved an 80% response rate (n=169) with a mean score of 4.4 (of 5; 3.8-4.8 among centres). The CSAT had good reliability with an average internal consistency of 0.77 (95% CI 0.71 to 0.81); and CFAs supported the seven-domain structure. CSAT results were associated with respondents' perceptions of the evidence for PEWS, its implementation and use in their centre, and their assessment of the hospital culture and implementation climate. The mean CSAT score was higher among respondents at centres with longer time using PEWS (p<0.001). Focus group participants noted the CSAT report helped assess their centre's clinical capacity to sustain PEWS and provided constructive feedback for improvement.

CONCLUSIONS

We present information supporting the reliability and validity of the CSAT tool, the first Spanish-language instrument to assess clinical capacity to sustain evidence-based interventions in hospitals of variable resource levels. This assessment demonstrates a high capacity to sustain PEWS in these resource-limited centres with improvement over time from PEWS implementation.

Provider REport of Sustainment Scale (PRESS): development and validation of a brief measure of inner context sustainment

BACKGROUND

Implementation scientists and practitioners often rely on frontline providers for reporting on implementation outcomes. Furthermore, measures of sustainment are few, and available sustainment measures are mainly setting or evidenced-based practice (EBP) specific, require organizational and system-level knowledge to complete, and often lack psychometric rigor. The aim of this study was to develop a brief, pragmatic, and generalizable measure for completion by frontline service providers of the implementation outcome, sustainment.

METHODS

We utilized a Rasch measurement theory approach to scale the development and testing of psychometric parameters. Sustainment items were developed to be relevant for direct service providers to complete. In order to promote generalizability, data were collected and items were tested across four diverse psychosocial evidence-based practices (motivational interviewing [MI], SafeCare®, classroom pivotal response training [CPRT], and an individualized mental health intervention for children with autism spectrum disorder [AIM-HI]) and in four service settings (substance use disorder treatment, child welfare, education, and specialty mental health). Associations between the sustainment measure and sustainment leadership, sustainment climate, and attitudes towards the adoption and use of each of the EBPs were assessed to confirm construct validity.

RESULTS

Three items for the Provider REport of Sustainment Scale (PRESS) were assessed for measuring the core component of sustainment: continued use of the EBP. Internal consistency reliability was high. The scale indicated fit to the Rasch measurement model with no response dependency, ordered thresholds, no differential item functioning, and supported unidimensionality. Additionally, construct validity evidence was provided based on the correlations with related variables.

CONCLUSION

The PRESS measure is a brief, three-item measure of sustainment that is both pragmatic and useable across different EBPs, provider types, and settings. The PRESS captures frontline staffs' report of their clinic, team, or agency's continued use of an EBP. Future testing of the PRESS for concurrent and predictive validity is recommended.

Recidivism Treatment for Justice-Involved Veterans: Evaluating Adoption and Sustainment of Moral Reconation Therapy in the US Veterans Health Administration

Moral Reconation Therapy (MRT), an evidence-based intervention to reduce risk for criminal recidivism among justice-involved adults, was developed and primarily tested in correctional settings. Therefore, a better understanding of the implementation potential of MRT within non-correctional settings is needed. To address this gap in the literature, we evaluated the adoption and sustainment of MRT in the US Veterans Health Administration (VHA) following a national training initiative in fiscal years 2016 and 2017. In February 2019, surveys with 66 of the 78 VHA facilities that participated in the training were used to estimate the prevalence of MRT adoption and sustainment, and qualitative interviews with key informants from 20 facilities were used to identify factors associated with sustainment of MRT groups. Of the 66 facilities surveyed, the majority reported adopting (n = 52; 79%) and sustaining their MRT group until the time of the survey (n = 38; 58%). MRT sustainment was facilitated by strong intra-facility (e.g., between veterans justice and behavioral health services) and inter-agency collaborations (e.g., between VHA and criminal justice system stakeholders), which provided a reliable referral source to MRT groups, external incentives for patient engagement, and sufficient staffing to maintain groups. Additional facilitators of MRT sustainment were adaptations to the content and delivery of MRT for patients and screening of referrals to the groups. The findings provide guidance to clinics and healthcare systems that are seeking to implement MRT with justice-involved patient populations, and inform development of implementation strategies to be formally tested in future trials.