Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale

A mixed-methods, theory-driven assessment of the sustainability of a multi-sectoral preventive intervention for South Asian Americans at risk for cardiovascular disease

BACKGROUND

South Asian Americans bear a high burden of atherosclerotic cardiovascular disease (ASCVD), but little is known about the sustainability of evidence-based interventions (EBI) to prevent ASCVD in this population. Using community-based participatory research, we previously developed and implemented the South Asian Healthy Lifestyle Intervention (SAHELI), a culturally-adapted EBI targeting diet, physical activity, and stress management. In this study, we use the Integrated Sustainability Framework to investigate multisectoral partners' perceptions of organizational factors influencing SAHELI sustainability and strategies for ensuring sustainability.

METHODS

From 2022 to 2023, we conducted a mixed-methods study (quant- > QUAL) with 17 SAHELI partners in the Chicago area. Partners' settings included: community organization, school district, public health department, and healthcare system. Descriptive statistics summarized quantitative results. Two coders used a hybrid thematic analysis approach to identify qualitative themes. Qualitative and quantitative data were integrated and analyzed using mixed methods.

RESULTS

Surveys (score range 1-5: higher scores indicate facilitators; lower scores indicate barriers) indicated SAHELI sustainability facilitators to be its "responsiveness to community values and needs" (mean = 4.9). Barriers were "financial support" (mean = 3.5), "infrastructure/capacity to support sustainment" (mean = 4.2), and "implementation leadership" (mean = 4.3). Qualitative findings confirmed quantitative findings that SAHELI provided culturally-tailored cardiovascular health education responsive to the needs of the South Asian American community, increased attention to health issues, and transformed perceptions of research among community members. Qualitative findings expanded upon quantitative findings, showing that the organizational fit of SAHELI was a facilitator to sustainability while competing priorities were barriers for partners from the public health department and health system. Partners from the public health department and health system discussed challenges in offering culturally-tailored programming exclusively for one targeted population. Sustainability strategies envisioned by partners included: transitioning SAHELI to a program delivered by community members; integrating components of SAHELI into other programs; and expanding SAHELI to other populations. Modifications made to SAHELI (i.e., virtual instead of in-person delivery) had both positive and negative implications for sustainability.

DISCUSSION

This study identifies common sustainability barriers and facilitators across different sectors, as well as those specific to certain settings. Aligning health equity interventions with community needs and values, organizational activities, and local context and resources is critical for sustainability. Challenges also arise from balancing the needs of specific populations against providing programming for broader audiences.

Mothers and Babies Virtual Group (MBVG) for perinatal Latina women: study protocol for a hybrid type-1 effectiveness-implementation randomized controlled trial

BACKGROUND

Immigrant Latinas (who are foreign-born but now reside in the USA) are at greater risk for developing postpartum depression than the general perinatal population, but many face barriers to treatment. To address these barriers, we adapted the Mothers and Babies Course-an evidence-based intervention for postpartum depression prevention-to a virtual group format. Additional adaptations are inclusion of tailored supplemental child health content and nutrition benefit assistance. We are partnering with Early Learning Centers (ELC) across the state of Maryland to deliver and test the adapted intervention.

METHODS

The design is a Hybrid Type I Effectiveness-Implementation Trial. A total of 300 participants will be individually randomized to immediate (N = 150) versus delayed (N = 150) receipt of the intervention, Mothers and Babies Virtual Group (MB-VG). The intervention will be delivered by trained Early Learning Center staff. The primary outcomes are depressive symptoms (measured via the Center for Epidemiologic Studies-Depression Scale), parenting self-efficacy (measured via the Parental Cognition and Conduct Towards the Infant Scale (PACOTIS) Parenting Self-Efficacy subscale), and parenting responsiveness (measured via the Maternal Infant Responsiveness Instrument) at 1-week, 3-month, and 6-month post-intervention. Depressive episodes (Structured Clinical Interview for DSM-V- Disorders Research Version) at 3-month and 6-month post-intervention will also be assessed. Secondary outcomes include social support, mood management, anxiety symptoms, perceived stress, food insecurity, and mental health stigma at 1-week, 3-month, and 6-month post-intervention. Exploratory child outcomes are dysregulation and school readiness at 6-month post-intervention. Intervention fidelity, feasibility, acceptability, and appropriateness will also be assessed guided by the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework.

DISCUSSION

This study will be one of the first to test the efficacy of a group-based virtual perinatal depression intervention with Latina immigrants, for whom stark disparities exist in access to health services. The hybrid effectiveness-implementation design will allow rigorous examination of barriers and facilitators to delivery of the intervention package (including supplemental components) which will provide important information on factors influencing intervention effectiveness and the scalability of intervention components in Early Learning Centers and other child-serving settings.

REGISTRATION

ClinicalTrials.gov NCT05873569.

A review of implementation research funded by the National Institute on Drug Abuse, 2007-2023: Progress and opportunities

BACKGROUND

The ongoing and evolving overdose epidemic highlights the need to translate research results into routine clinical practice to address urgent service delivery needs. Implementation science is a relatively new discipline intended to develop systematic, replicable, scalable strategies to accelerate this translation. This article presents a comprehensive review of implementation research funded by the National Institute on Drug Abuse (NIDA).

METHODS

The study identified all NIDA-funded research grants awarded in fiscal years 2007 through 2023 in treatment services or prevention research (n = 1111) and screened them to find those with a pre-specified implementation science component (n = 248). Using the text of the grant application, two reviewers independently coded the key characteristics of each study.

RESULTS

The characteristics of these grants, and trends over time, are described, and priority gap areas are identified. NIDA's implementation research grants have demonstrated increasing rigor in design and measurement.

CONCLUSIONS

Growth in the portfolio has been driven in part by NIDA's investments in research-practice partnerships in the criminal-legal system, and by recent efforts to address the overdose epidemic.

Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C): development and psychometric evaluation of a measure of sustainability determinants in the early childhood education and care setting

BACKGROUND

There is a need for valid and reliable measures of determinants of sustainability of public health interventions in early childhood education and care (ECEC) settings. This study aimed to develop and evaluate the psychometric and pragmatic properties of such a measure - the Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C).

METHODS

We undertook a two-phase process guided by the COnsensus-based Standards for the selection of health status Measurement INstruments checklist (COSMIN) and Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Phase 1 involved measure development; i.e., determining items and scales through an iterative process and assessment of face and content validity. Phase 2 involved the evaluation of psychometric and pragmatic properties. The 29-item measure completed by service executives (directors and nominated supervisors) was embedded in a larger survey from a national sample of Australian ECEC services assessing their implementation of nutrition and physical activity programs. Structural validity, concurrent validity, known groups validity, internal consistency, floor and ceiling effects, norms, and pragmatic qualities of the measure were assessed according to the PAPERS criteria.

RESULTS

The final measure contained 26 items, with respondents reporting how strongly they agreed or disagreed on a five-point Likert scale. Phase 1 assessments confirmed the relevance, and face and content validity of the scale. In Phase 2, we obtained 482 completed surveys, of which 84% (n = 405) completed the entire measure across 405 ECEC settings (one executive per service). Three of the four fit indices for the confirmatory factor analysis met the pre-specified criteria (SRMR = 0.056, CFI = 0.993, RMSEA = 0.067) indicating 'good' structural validity. The IMPRESS-C illustrated: 'good' internal consistency, with Cronbach's alpha values from 0.53 to 0.92; 'emerging' concurrent validity; 'poor' known groups validity; 'good' norms; and 'good' overall pragmatic qualities (cost, readability, length, and assessor burden).

CONCLUSIONS

The IMPRESS-C possesses strong psychometric and pragmatic qualities for assessing service executive-level perceptions of determinants influencing sustainment of public health interventions within ECEC settings. To achieve a full range of perspectives in this setting, future work should be directed to also develop and test measures of sustainability determinants at the implementer level (e.g., among individual educators and staff).

Defining the transition from new to normal: a qualitative investigation of the clinical change process

Background

Understanding how and when a new evidence-based clinical intervention becomes standard practice is crucial to ensure that healthcare is delivered in alignment with the most up-to-date knowledge. However, rigorous methods are needed to determine when a new clinical practice becomes normalized to the standard of care. To address this gap, this study qualitatively explores how, when, and why a clinical practice change becomes normalized within healthcare organizations.

Methods

We used purposive sampling to recruit clinical leaders who worked in implementation science across diverse health contexts. Enrolled participants completed semi-structured interviews. Qualitative data analysis was guided by a modified version of the Normalization Process Theory (NPT) framework to identify salient themes. Identified normalization strategies were mapped to the Expert Recommendations for Implementation Change (ERIC) project.

Results

A total of 17 individuals were interviewed. Participants described four key signals for identifying when a novel clinical practice becomes the new normal: 1) integration into existing workflows; 2) scaling across the entire organizational unit; 3) staff buy-in and ownership; and 4) sustainment without ongoing monitoring. Participants identified salient strategies to normalize new clinical interventions: 1) taking a patient approach; 2) gaining staff buy-in and ownership; and 3) conducting ongoing measurement of progress towards normalization.

Conclusions

The results offer valuable insight into the indicators that signify when a novel clinical practice becomes normalized, and the strategies employed to facilitate this transition. These findings can inform future research to develop instruments that implementation leaders can use to systematically measure the clinical change process.

Sustaining health obesity prevention programs: Lessons from real-world population settings

Current understanding of what sustains public health programs is derived primarily from evaluations of programs delivered for only a few years. To improve program planning and knowledge on program sustainment, we examined real-world population programs that have been delivered for >2 years. Our focus was physical activity and nutrition programs for obesity prevention. We identified programs through published literature and searched through publicly available information on the theoretical factors determining sustainment. We reviewed 90 programs and explored intervention characteristics, provider, delivery and process level factors, and the outer environment influences. Programs were sustained on average 15 years and more frequently characterised as behavioural change skills programs targeting children, funded by non-government organisations and delivered by community organisations. Most programs had undergone some modification (n = 55). Differences between programs were observed across key factors: Programs sustained 15+ years were mainly behaviour skills programs (n = 21); and characterised by interactive designs (i.e., face-to-face or digital) over static innovations (i.e., print material) compared with those sustained <15 years (p = 0.024). While government funding supported many of the programs (n = 45;), those sustained 15+ years were more likely to have been commercially funded (p = 0.044); and were less likely to have current community involvement (p = 0.013). Differences in ownership and funding were also observed across countries. While multiple factors may influence program sustainment, longer delivered programs where distinguished by their program characteristics and outer contextual factors, suggesting such factors may have an important role in sustaining programs in the longer-term.

Sustaining the collaborative chronic care model in outpatient mental health: a matrixed multiple case study

BACKGROUND

Sustaining evidence-based practices (EBPs) is crucial to ensuring care quality and addressing health disparities. Approaches to identifying factors related to sustainability are critically needed. One such approach is Matrixed Multiple Case Study (MMCS), which identifies factors and their combinations that influence implementation. We applied MMCS to identify factors related to the sustainability of the evidence-based Collaborative Chronic Care Model (CCM) at nine Department of Veterans Affairs (VA) outpatient mental health clinics, 3-4 years after implementation support had concluded.

METHODS

We conducted a directed content analysis of 30 provider interviews, using 6 CCM elements and 4 Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) domains as codes. Based on CCM code summaries, we designated each site as high/medium/low sustainability. We used i-PARIHS code summaries to identify relevant factors for each site, the extent of their presence, and the type of influence they had on sustainability (enabling/neutral/hindering/unclear). We organized these data into a sortable matrix and assessed sustainability-related cross-site trends.

RESULTS

CCM sustainability status was distributed among the sites, with three sites each being high, medium, and low. Twenty-five factors were identified from the i-PARIHS code summaries, of which 3 exhibited strong trends by sustainability status (relevant i-PARIHS domain in square brackets): "Collaborativeness/Teamwork [Recipients]," "Staff/Leadership turnover [Recipients]," and "Having a consistent/strong internal facilitator [Facilitation]" during and after active implementation. At most high-sustainability sites only, (i) "Having a knowledgeable/helpful external facilitator [Facilitation]" was variably present and enabled sustainability when present, while (ii) "Clarity about what CCM comprises [Innovation]," "Interdisciplinary coordination [Recipients]," and "Adequate clinic space for CCM team members [Context]" were somewhat or less present with mixed influences on sustainability.

CONCLUSIONS

MMCS revealed that CCM sustainability in VA outpatient mental health clinics may be related most strongly to provider collaboration, knowledge retention during staff/leadership transitions, and availability of skilled internal facilitators. These findings have informed a subsequent CCM implementation trial that prospectively examines whether enhancing the above-mentioned factors within implementation facilitation improves sustainability. MMCS is a systematic approach to multi-site examination that can be used to investigate sustainability-related factors applicable to other EBPs and across multiple contexts.

Development and Application of the Scale-Up Reflection Guide (SRG)

Scaling up effective interventions in public health is complex and comprehensive, and published accounts of the scale-up process are scarce. Key aspects of the scale-up experience need to be more comprehensively captured. This study describes the development of a guide for reflecting on and documenting the scale-up of public health interventions, to increase the depth of practice-based information of scaling up. Reviews of relevant scale-up frameworks along with expert input informed the development of the guide. We evaluated its acceptability with potential end-users and applied it to two real-world case studies. The Scale-up Reflection Guide (SRG) provides a structure and process for reflecting on and documenting key aspects of the scale-up process of public health interventions. The SRG is comprised of eight sections: context of completion; intervention delivery, history/background; intervention components; costs/funding strategies and partnership arrangements; the scale-up setting and delivery; scale-up process; and evidence of effectiveness and long-term outcomes. Utilization of the SRG may improve the consistency and reporting for the scale-up of public health interventions and facilitate knowledge sharing. The SRG can be used by a variety of stakeholders including researchers, policymakers or practitioners to more comprehensively reflect on and document scale-up experiences and inform future practice.

Sustainment of diverse evidence-informed practices disseminated in the Veterans Health Administration (VHA): initial development and piloting of a pragmatic survey tool

BACKGROUND

There are challenges associated with measuring sustainment of evidence-informed practices (EIPs). First, the terms sustainability and sustainment are often falsely conflated: sustainability assesses the likelihood of an EIP being in use in the future while sustainment assesses the extent to which an EIP is (or is not) in use. Second, grant funding often ends before sustainment can be assessed. The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program is one of few large-scale models of diffusion; it seeks to identify and disseminate practices across the VHA system. The DoE sponsors "Shark Tank" competitions, in which leaders bid on the opportunity to implement a practice with approximately 6 months of implementation support. As part of an ongoing evaluation of the DoE, we sought to develop and pilot a pragmatic survey tool to assess sustainment of DoE practices.

METHODS

In June 2020, surveys were sent to 64 facilities that were part of the DoE evaluation. We began analysis by comparing alignment of quantitative and qualitative responses; some facility representatives reported in the open-text box of the survey that their practice was on a temporary hold due to COVID-19 but answered the primary outcome question differently. As a result, the team reclassified the primary outcome of these facilities to Sustained: Temporary COVID-Hold. Following this reclassification, the number and percent of facilities in each category was calculated. We used directed content analysis, guided by the Consolidated Framework for Implementation Research (CFIR), to analyze open-text box responses.

RESULTS

A representative from forty-one facilities (64%) completed the survey. Among responding facilities, 29/41 sustained their practice, 1/41 partially sustained their practice, 8/41 had not sustained their practice, and 3/41 had never implemented their practice. Sustainment rates increased between Cohorts 1-4.

CONCLUSIONS

The initial development and piloting of our pragmatic survey allowed us to assess sustainment of DoE practices. Planned updates to the survey will enable flexibility in assessing sustainment and its determinants at any phase after adoption. This assessment approach can flex with the longitudinal and dynamic nature of sustainment, including capturing nuances in outcomes when practices are on a temporary hold. If additional piloting illustrates the survey is useful, we plan to assess the reliability and validity of this measure for broader use in the field.

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review

BACKGROUND

Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings.

METHODS

Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed.

RESULTS

A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants.

CONCLUSIONS

This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field.

TRIAL REGISTRATION

This review was prospectively registered with Research Registry (reviewregistry1097), March 2021.

Assessing the sustainability capacity of evidence-based programs in community and health settings

Background

Within many public health settings, there remain large challenges to sustaining evidence-based practices. The Program Sustainability Assessment Tool has been developed and validated to measure sustainability capacity of public health, social service, and educational programs. This paper describes how this tool was utilized between January 2014 and January 2019. We describe characteristics of programs that are associated with increased capacity for sustainability and ultimately describe the utility of the PSAT in sustainability research and practice.

Methods

The PSAT is comprised of 8 subscales, measuring sustainability capacity in eight distinct conceptual domains. Each subscale is made up of five items, all assessed on a 7-point Likert scale. Data were obtained from persons who used the PSAT on the online website (https://sustaintool.org/), from 2014 to 2019. In addition to the PSAT scale, participants were asked about four program-level characteristics. The resulting dataset includes 5,706 individual assessments reporting on 2,892 programs.

Results

The mean overall PSAT score was 4.73, with the lowest and highest scoring subscales being funding stability and program adaptation, respectively. Internal consistency for each subscale was excellent (average Cronbach's alpha = 0.90, ranging from 0.85 to 0.94). Confirmatory factor analysis highlighted good to excellent fit of the PSAT measurement model (eight distinct conceptual domains) to the observed data, with a comparative fit index of 0.902, root mean square error of approximation equal to 0.054, and standardized root mean square residual of 0.054. Overall sustainability capacity was significantly related to program size (F = 25.6; p < 0.001). Specifically, smaller programs (with staff sizes of ten or below) consistently reported lower program sustainability capacity. Capacity was not associated with program age and did not vary significantly by program level.

Discussion

The PSAT maintained its excellent reliability when tested with a large and diverse sample over time. Initial criterion validity was explored through the assessment of program characteristics, including program type and program size. The data collected reinforces the ability of the PSAT to assess sustainability capacity for a wide variety of public health and social programs.

Factors contributing to the sustained implementation of an early childhood obesity prevention intervention: The INFANT Program

Background

The INFANT Program is an efficacious, group-based program for first-time parents, delivered at three-monthly intervals when INFANT are aged 3-18 months through an existing universal care service in Victoria, Australia. Many lessons have been learnt from its origins as a cluster randomized control trial to its small-scale, community-level implementation. This study aimed to describe factors contributing to its sustained implementation to inform large-scale implementation across Australia.

Methods

This study used a multi-site qualitative exploratory approach. INFANT facilitators trained between 2013 and 2017 were sent an online survey, with optional telephone interviews. The Consolidated Framework for Implementation Research (CFIR) was selected as the underpinning theoretical framework as it offered the opportunity to explore a breadth of possible barriers and enablers across patterns of implementation (never, discontinued, ongoing).

Results

All participants were female (n = 31), the majority were Maternal and Child Health Nurses (48%), representing five regional and nine metro local government areas (LGAs), across all patterns of implementation (never implemented n = 4; discontinued implementation n = 5; ongoing implementation n = 5). All consenting participants were interviewed (n = 11) representing four regional and seven metro LGAs, across all patterns of implementation (never implemented n = 3; discontinued implementation n = 4; ongoing implementation n = 4). The main reason for attending INFANT Program training was to become skilled to implement the program. Mapping identified barriers and enablers to the CFIR revealed the inner and outer settings and implementation process to be of greatest influence. Main differences between LGAs with ongoing and discontinued implementation related to funding availability, organizational management support and endorsement, organizational resourcing and capacity, integration into routine practice and establishing role clarity with partner organizations, and planning for sustained implementation from the start.

Conclusion

This study provides important insights into the barriers and enablers to the sustained implementation of an evidence-based intervention (the INFANT Program) during small scale community-level implementation. The authors therefore contend that the pre-requisite for scale-up of a population health intervention is not just proof of effectiveness but also proof of sustained implementation at the local/organizational level. Study findings have broad transferability given their similarity to those identified for health promotion interventions implemented globally, in healthcare, education and community settings.

Profile of the Portfolio of NIH-Funded HIV Implementation Research Projects to Inform Ending the HIV Epidemic Strategies

BACKGROUND

The US government created an initiative to end the HIV epidemic in the United States by the year 2030 (EHE). This multiagency initiative was structured around four pillars: Prevent, Diagnose, Treat, and Respond to improve HIV programs, resources, and service delivery infrastructure. In support of its research mission, the National Institutes of Health (NIH) has funded implementation research (IR) projects by addressing the four pillars and encouraging investigators to collaborate with local partners and Health and Human Services (HHS) grantees in 57 priority jurisdictions.

METHODS

This paper analyzed data from the NIH funded CFAR/ARC supplement projects from 2019 to 2021. The Exploration, Preparation, Implementation, Sustainment (EPIS) framework was used to characterize projects by stage of implementation.

RESULTS

The Prevent pillar was most frequently studied, with Pre-Exposure Prophylaxis (PrEP) being the most studied intervention. The most common partners were health departments, community-based organizations (CBOs), and Federally Qualified Health Centers (FQHCs). The Consolidated Framework for Implementation Research (CFIR) framework was the most utilized to investigate implementation determinants, followed by the RE-AIM framework and Proctor model to assess implementation outcomes.

CONCLUSION

Monitoring the projects resulting from NIH investments is fundamental to understanding the response to EHE, and achieving these results requires systematic and continuous effort that can support the generalizable implementation knowledge emerging from individual studies. There are some remaining gaps in the project portfolio, including geographical coverage, range of implementation outcomes being measured, and interventions still requiring further research to ensure equitable scale-up of evidence based interventions and achieve EHE goals.

Understanding the sustainment of population health programmes from a whole-of-system approach

BACKGROUND

Population health prevention programmes are needed to reduce the prevalence of chronic diseases. Nevertheless, sustaining programmes at a population level is challenging. Population health is highly influenced by social, economic and political environments and is vulnerable to these system-level changes. The aim of this research was to examine the factors and mechanisms contributing to the sustainment of population prevention programmes taking a systems thinking approach.

METHODS

We conducted a qualitative study through interviews with population health experts working within Australian government and non-government agencies experienced in sustaining public health programs at the local, state or national level (n = 13). We used a deductive thematic approach, grounded in systems thinking to analyse data.

RESULTS

We identified four key barriers affecting program sustainment: 1) short term political and funding cycles; 2) competing interests; 3) silo thinking within health service delivery; and 4) the fit of a program to population needs. To overcome these barriers various approaches have centred on the importance of long-range planning and resourcing, flexible program design and management, leadership and partnerships, evidence generation, and system support structures.

CONCLUSION

This study provides key insights for overcoming challenges to the sustainment of population health programmes amidst complex system-wide changes.

Designing for Dissemination and Sustainability to Promote Equitable Impacts on Health

Designing for dissemination and sustainability (D4DS) refers to principles and methods for enhancing the fit between a health program, policy, or practice and the context in which it is intended to be adopted. In this article we first summarize the historical context of D4DS and justify the need to shift traditional health research and dissemination practices. We present a diverse literature according to a D4DS organizing schema and describe a variety of dissemination products, design processes and outcomes, and approaches to messaging, packaging, and distribution. D4DS design processes include stakeholder engagement, participatory codesign, and context and situation analysis, and leverage methods and frameworks from dissemination and implementation science, marketing and business, communications and visualarts, and systems science. Finally, we present eight recommendations to adopt a D4DS paradigm, reflecting shifts in ways of thinking, skills and approaches, and infrastructure and systems for training and evaluation.

Measuring capacity to use evidence-based interventions in community-based organizations: A comprehensive, scoping review

Introduction:

Community-based organizations (CBOs) are well-positioned to incorporate research evidence, local expertise, and contextual factors to address health inequities. However, insufficient capacity limits use of evidence-based interventions (EBIs) in these settings. Capacity-building implementation strategies are popular, but a lack of standard models and validated measures hinders progress in the field. To advance the literature, we conducted a comprehensive scoping review.

Methods:

With a reference librarian, we executed a comprehensive search strategy of PubMed/Medline, Web of Science Core Collection, and EBSCO Global Health. We included articles that addressed implementation science, capacity-building, and CBOs. Of 5527 articles, 99 met our inclusion criteria, and we extracted data using a double-coding process

Results:

Of the 99 articles, 47% defined capacity explicitly, 31% defined it indirectly, and 21% did not define it. Common concepts in definitions were skills, knowledge/expertise, and resources. Of the 57 articles with quantitative analysis, 48 (82%) measured capacity, and 11 (23%) offered psychometric data for the capacity measures. Of the 99 studies, 40% focused exclusively on populations experiencing inequities and 22% included those populations to some extent. The bulk of the studies came from high-income countries.

Conclusions:

Implementation scientists should 1) be explicit about models and definitions of capacity and strategies for building capacity, 2) specify expected multi-level implementation outcomes, 3) develop and use validated measures for quantitative work, and 4) integrate equity considerations into the conceptualization and measurement of capacity-building efforts. With these refinements, we can ensure that the necessary supports reach CBO practitioners and critical partners for addressing health inequities.

Sustainment of proactive physical therapy for individuals with early-stage Parkinson's disease: a quality improvement study over 4 years

Background

Implementation science frameworks aided the development of a new, evidence-based clinical physical therapy program. The purpose of this report is to describe the process of sustaining a clinical program in practice for over 4 years. We present a framework for integrating tools for sustainability with the Knowledge-to-Action model in the context of a proactive physical therapy (PAPT) program for individuals with early-stage Parkinson’s disease.

Methods

Sustainability of implementation strategies was addressed using the Dynamic Sustainability Framework and sustainability assessment tools. Repeated retrospective medical record reviews and phone interviews were used to evaluate the reach and adoption of the PAPT over 4 years. Characteristics of those who engaged with PAPT, implementation fidelity, and clinical effectiveness were assessed for year 1 and year 3. Sustainability was measured using RE-AIM, NHS Sustainability Model, and Clinical Sustainability Assessment Tool.

Results

Reach increased from 28 to 110 total patients per year and spread occurred from one to three sites. PAPT user age, sex, Hoehn and Yahr rating, time since diagnosis, and type of insurance were similar in year 1 and year 3 (p > 0.05). The program sustained its effect to help participants increase or maintain self-reported exercise (Y1, 95%; Y3, 100%). However, upon evaluation PAPT users in year 3 had longer time since symptom onset and worse UPDRS motor scores compared to the PAPT users in year 1 (p < 0.05). All sites sustained the core intervention components, with sustainability scores of 71/100 (± 9.9) on the NHS Sustainability Model and 6.1/7 (± 0.9) on the Clinical Sustainability Assessment Tool.

Conclusions

Integrating multiple sustainability models and assessments supported continued effectiveness, spread, and sustainment of PAPT for 4 years. Effective planning, anticipating common healthcare changes, and addressing sustainability determinants early in program implementation were essential aspects of long-term success.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-021-00205-x.

Increasing Access to Contraception in the United States: Assessing Achievement and Sustainability

Background: During October 2016 through May 2018, a learning community was convened to focus on policies and programs to increase access to the full range of contraceptive options for women of reproductive age. The Increasing Access to Contraception (IAC) Learning Community included 27 jurisdictions, with teams from each jurisdiction consisting of state health department leaders, program staff, and provider champions. At the kick-off meeting, teams from each jurisdiction created action plans that outlined their goals. Methods: We contacted jurisdictions during May-June 2019, 1 year after the learning community ended, and invited them to complete a post-assessment of goal achievement and sustainment through semi-structured interviews over the telephone or via email. Results: Follow-up information was collected from 26 jurisdictions (96%) that participated in the learning community. The teams from these jurisdictions had created 79 total goals. At the time of the learning community closing meeting in May 2018, 35 goals (44%) had been achieved. Three jurisdictions achieved all their goals by the close of the learning community. At the time of the post-assessment 1 year later, jurisdictions were sustaining efforts for 69 (87%) of the total goals. In every jurisdiction, work on at least one goal that originated in the learning community was sustained. Conclusions: The jurisdictions that participated in the IAC Learning Community continued the work of their action plan goals 1 year after the formal closure of the learning community, indicating sustainability of the learning community activities, beyond what jurisdictions accomplished during formal participation.

Provider REport of Sustainment Scale (PRESS): development and validation of a brief measure of inner context sustainment

BACKGROUND

Implementation scientists and practitioners often rely on frontline providers for reporting on implementation outcomes. Furthermore, measures of sustainment are few, and available sustainment measures are mainly setting or evidenced-based practice (EBP) specific, require organizational and system-level knowledge to complete, and often lack psychometric rigor. The aim of this study was to develop a brief, pragmatic, and generalizable measure for completion by frontline service providers of the implementation outcome, sustainment.

METHODS

We utilized a Rasch measurement theory approach to scale the development and testing of psychometric parameters. Sustainment items were developed to be relevant for direct service providers to complete. In order to promote generalizability, data were collected and items were tested across four diverse psychosocial evidence-based practices (motivational interviewing [MI], SafeCare®, classroom pivotal response training [CPRT], and an individualized mental health intervention for children with autism spectrum disorder [AIM-HI]) and in four service settings (substance use disorder treatment, child welfare, education, and specialty mental health). Associations between the sustainment measure and sustainment leadership, sustainment climate, and attitudes towards the adoption and use of each of the EBPs were assessed to confirm construct validity.

RESULTS

Three items for the Provider REport of Sustainment Scale (PRESS) were assessed for measuring the core component of sustainment: continued use of the EBP. Internal consistency reliability was high. The scale indicated fit to the Rasch measurement model with no response dependency, ordered thresholds, no differential item functioning, and supported unidimensionality. Additionally, construct validity evidence was provided based on the correlations with related variables.

CONCLUSION

The PRESS measure is a brief, three-item measure of sustainment that is both pragmatic and useable across different EBPs, provider types, and settings. The PRESS captures frontline staffs' report of their clinic, team, or agency's continued use of an EBP. Future testing of the PRESS for concurrent and predictive validity is recommended.

The Clinical Sustainability Assessment Tool: measuring organizational capacity to promote sustainability in healthcare

Background

Few validated assessment tools are available to increase understanding and measure factors associated with sustainment of clinical practices, an increasingly recognized need among clinicians. We describe the development of the Clinical Sustainability Assessment Tool (CSAT), designed to assess factors that contribute to sustainable practices in clinical settings.

Methods

Sixty-four participants from clinical and research fields participated in concept mapping and were recruited to brainstorm factors that lead to sustained clinical practices. Once repeated factors were removed, participants sorted items based on similarity and rated them by importance and feasibility. Using concept mapping analyses, items were grouped into meaningful domains to develop an initial tool. We then recruited pilot sites and early adopters, for a total of 286 practicing clinicians, to pilot and evaluate the tool. Individuals were recruited from clinical settings across pediatric and adult medical and surgical subspecialties. The data were analyzed using confirmatory factor analysis (CFA) to test hypothesized subscale structure in the instrument. We used root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) to assess fit and thus the ability of CSAT to measure the identified domains.

Results

The concept mapping produced sorted statements that were edited into items that could be responded to, resulting in the creation of a tool with seven determinant domains and 47 items. The pilot and CFA testing resulted in a final CSAT instrument made up 35 items, five per domain. CFA results demonstrated very good fit of the seven domain structure of the CSAT (RMSEA = 0.049; SRMR = 0.049). Usability testing indicated the CSAT is brief, easy to use, easy to learn, and does not require extensive training. Additionally, the measure scored highly (18/20) on the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The seven final CSAT domains were engaged staff and leadership, engaged stakeholders, organizational readiness, workflow integration, implementation and training, monitoring and evaluation, and outcomes and effectiveness.

Conclusions

The CSAT is a new reliable assessment tool which allows for greater practical and scientific understanding of contextual factors that enable sustainable clinical practices over time.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-021-00181-2.

Perceived sustainability of the school-based social and behavior change communication (SBCC) approach on malaria prevention in rural Ethiopia: stakeholders' perspectives

BACKGROUND

Evidence on what makes the school-engaged social and behavior change communication (SBCC) interventions on malaria prevention more sustainable are limited in literature partly due to its recent emergence. Enrolling the key stakeholders, this study explored the perceived sustainability of the SBCC interventions on malaria prevention through primary school communities in rural Ethiopia.

METHODS

The SBCC interventions were implemented from 2017 to 2019 in 75 primary schools and villages in rural Jimma to promote malaria preventive practices. As a part of program evaluation, this study employed a mixed-method to collect qualitative and quantitative data from 205 stakeholders following the end of the program. Data were collected using interview guides and structured questionnaires. The SPSS version 26 and Atlas ti7.1 software were used to analyze the data. Multivariable linear regression modeling was used to identify predictors of the perceived sustainability of the program (SOP).

RESULTS

The mean score of SOP was 25.93 (SD = 4.32; range 6-30). Multivariable linear regression modeling showed that the perceived risk to malaria (β = 0.150; P = 0.029), self-efficacy (β = 0.192; P = 0.003), and perceived fidelity of implementation (β = 0.292; P = 0.000) and degree of adoption (β = 0.286; P = 0.000) were positively predicted the perceived SOP. The qualitative result identified various barriers and opportunities to sustaining the program that summarized under three themes which include perceptions about the quality of program delivery (e.g inadequate involvement of stakeholders and staffs, concerns over short project life, immature sustainability efforts), school settings (e.g schools' malaria priority, schools' climate and quality of coaching) and the outer settings (e.g existing structures in the health and education systems).

CONCLUSION

The study identified key predictive variables such as stakeholders' perceived risk to malaria, self-efficacy, perceived fidelity of implementation and degree of adoption that could help to improve the sustainment of the school-based SBCC approach on malaria prevention and control. Further longitudinal study should be conducted to examine the rate of decline in program components over time and how improved sustainability would contribute to the effectiveness on malaria preventive behaviors among students.

Sustainability planning in the US response to the opioid crisis: An examination using expert and text mining approaches

Between January 2016 and June 2020, the Substance Abuse and Mental Health Services Administration rapidly distributed $7.5 billion in response to the U.S. opioid crisis. These funds are designed to increase access to medications for addiction treatment, reduce unmet treatment need, reduce overdose death rates, and provide and sustain effective prevention, treatment and recovery activities. It is unclear whether or not the services developed using these funds will be sustained beyond the start-up period. Based on 34 (64%) State Opioid Response (SOR) applications, we assessed the states' sustainability plans focusing on potential funding sources, policies, and quality monitoring. We found variable commitment to sustainability across response plans with less than half the states adequately describing sustainability plans. States with higher proportions of opioid prescribing, opioid misuse, and poverty had somewhat higher scores on sustainment. A text mining/machine learning approach automatically rated sustainability in SOR applications with an 82% accuracy compared to human ratings. Because life saving evidence-based programs and services may be lost, intentional commitment to sustainment beyond the bolus of start-up funding is essential.