Translating economic evaluations into financing strategies for implementing evidence-based practices

Multi-level alignment processes in the sustainment of a youth substance use treatment model following a federal implementation initiative: A mixed method study

INTRODUCTION

Government agencies have identified evidence-based practice (EBP) dissemination as a pathway to high-quality behavioral health care for youth. However, gaps remain about how to best sustain EBPs in treatment organizations in the U.S., especially in resource-constrained settings like publicly-funded youth substance use services. One important, but understudied, determinant of EBP sustainment is alignment: the extent to which multi-level factors that influence sustainment processes and outcomes are congruent, consistent, and/or coordinated. This study examined the role of alignment in U.S. states' efforts to sustain the Adolescent Community Reinforcement Approach (A-CRA), an EBP for youth substance use disorders, during the COVID-19 pandemic.

METHODS

In this mixed methods study, the qualitative investigation preceded and informed the quantitative investigation. We interviewed state administrators and providers (i.e., supervisors and clinicians) from 15 states that had completed a federal A-CRA implementation grant; providers also completed surveys. The sample included 50 providers from 35 treatment organizations that reported sustaining A-CRA when the COVID-19 pandemic began, and 20 state administrators. In qualitative thematic analyses, we applied the EPIS (Exploration, Preparation, Implementation, Sustainment) framework to characterize alignment processes that interviewees described as influential on sustainment. We then used survey items to quantitatively explore the associations described in qualitative themes, using bivariate linear regressions.

RESULTS

At the time of interview, staff from 80 % of the treatment organizations (n = 28), reported sustaining A-CRA. Providers from both sustainer and non-sustainer organizations, as well as state administrators, described major sources of misalignment when state agencies ceased technical assistance post-grant, and because limited staff capacity conflicted with A-CRA's training model, which was perceived as time-intensive. Participants described the pandemic as exacerbating preexisting challenges, including capacity issues. Sustainer organizations reported seeking new funding to help sustain A-CRA. Quantitative associations between self-rated extent of sustainment and other survey items largely followed the pattern predicted from the qualitative findings.

CONCLUSIONS

The COVID-19 pandemic amplified longstanding A-CRA sustainment challenges, but treatment organizations already successfully sustaining A-CRA pre-pandemic largely continued. There are missed opportunities for state-level actors to coordinate with providers on the shared goal of EBP sustainment. A greater focus on alignment processes in research and practice could help states and providers strengthen sustainability planning.

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022

BACKGROUND

Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies and the Reach Effectiveness Adoption Implementation and Maintenance (RE-AIM) framework to classify outcomes.

METHODS

We conducted a systematic review of studies examining implementation strategies from 2010-2022 and registered with PROSPERO (CRD42021235592). We searched databases using terms "implementation strategy", "intervention", "bundle", "support", and their variants. We also solicited study recommendations from implementation science experts and mined existing systematic reviews. We included studies that quantitatively assessed the impact of at least one implementation strategy to improve health or health care using an outcome that could be mapped to the five evaluation dimensions of RE-AIM. Only studies meeting prespecified methodologic standards were included. We described the characteristics of studies and frequency of implementation strategy use across study arms. We also examined common strategy pairings and cooccurrence with significant outcomes.

FINDINGS

Our search resulted in 16,605 studies; 129 met inclusion criteria. Studies tested an average of 6.73 strategies (0-20 range). The most assessed outcomes were Effectiveness (n=82; 64%) and Implementation (n=73; 56%). The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59). These strategies were often used in combination. Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes. However, many strategies were not tested sufficiently to draw conclusions.

CONCLUSION

This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings. We present recommendations for improving future similar efforts.

Informing evidence-based medicine for opioid use disorder using pharmacoeconomic studies

INTRODUCTION

The health and economic consequences of inadequately treated opioid use disorder (OUD) are substantial. Healthcare systems in the United States (US) and other countries are facing a growing healthcare crisis due to opioids. Although effective medications for OUD exist, relying solely on clinical information is insufficient for addressing the opioid crisis.

AREAS COVERED

In this review, the role of pharmacoeconomic studies in informing evidence-based medication treatment for OUD is discussed, with a particular emphasis on the US healthcare system, where the economic burden is significantly higher than the global average. The scope/objective of pharmacoeconomics as a distinct scientific research program is briefly defined, followed by a discussion of existing evidence informed by data from systematic reviews, in addition to a convenience sample of recently published pharmacoeconomic studies and protocols. The review also explores the need for methodological advancements in the field.

EXPERT OPINION

Despite the potential of pharmacoeconomic research in shaping evidence-based medicine for OUD, significant challenges limiting its real-world application remain. How to address these challenges are explored, including how to combine cost-effectiveness and budget impact analyses to address the needs of the healthcare system as a whole and specific stakeholders interested in adopting new OUD treatment strategies.

It takes a village: A pilot study of a group telehealth intervention for support persons affected by opioid use disorder

INTRODUCTION

Opioid use disorder (OUD) has devastating effects on individuals, families, and communities. The Community Reinforcement and Family Training (CRAFT) is a Support Person (SP)-focused intervention that aims to increase SPs' communication strategies, positive reinforcement/rewards, and social support. This pilot study, called eINSPIRE (INtegrating Support Persons Into REcovery), adapted CRAFT for delivery via group telehealth. The aims were to evaluate the feasibility, acceptability, and preliminary effectiveness of this intervention on patient buprenorphine retention and SP mental health.

METHODS

The study recruited patients receiving buprenorphine treatment in a primary care setting across five community health centers with their SP (N = 100 dyads). SP participants were randomly assigned to receive usual care (UC) or the eINSPIRE intervention. We interviewed Patients and SPs at baseline and three months later. The study collected patient buprenorphine retention data from the electronic medical record three months post-baseline.

RESULTS

About 88 % (656/742) of potentially eligible patients were able to nominate a SP and 69 % (100/145) of nominated SPs were eligible and consented to the study. eINSPIRE groups had low reach (25 % of SPs attended), but high exposure (M = 7 of 10 sessions attended) and acceptability (classes helped them with their patient's OUD). The proportion of eINSPIRE patients (68 %) and UC patients (53 %) retained on buprenorphine at follow-up were similar (p = 0.203). SPs in both conditions reported similar reductions in their depression, anxiety, and impairment symptoms.

CONCLUSIONS

Preliminary data suggest that eINSPIRE groups may not be feasible in primary care without further adaptations for this population. A future study with a larger sample size is needed to elucidate the observed distribution differences in buprenorphine retention. Future research should also explore methods to reduce barriers to SP session attendance to improve the reach of this evidence-based intervention.

Implementation costs of sugary drink policies in the United States

To support implementation of important public health policies, policymakers need information about implementation costs over time and across stakeholder groups. We assessed implementation costs of two federal sugar-sweetened beverage (SSB) policies of current policy interest and with evidence to support their effects: excise taxes and health warning labels. Our analysis encompassed the entire policy life cycle using the Exploration, Preparation, Implementation, and Sustainment framework. We identified implementation actions using key informant interviews and developed quantitative estimates of implementation costs using published literature and government documents. Results show that implementation costs vary over time and among stakeholders. Explicitly integrating implementation science theory and using mixed methods improved the comprehensiveness of our results. Although this work is specific to federal SSB policies, the process can inform how we understand the costs of many public health policies, providing crucial information for public health policy making.

Costs to Implement a Pediatric Weight Management Program Across 3 Distinct Contexts

BACKGROUND

The Connect for Health program is an evidence-based program that aligns with national recommendations for pediatric weight management and includes clinical decision support, educational handouts, and community resources. As implementation costs are a major driver of program adoption and maintenance decisions, we assessed the costs to implement the Connect for Health program across 3 health systems that primarily serve low-income communities with a high prevalence of childhood obesity.

METHODS

We used time-driven activity-based costing methods. Each health system (site) developed a process map and a detailed report of all implementation actions taken, aligned with major implementation requirements (eg, electronic health record integration) or strategies (eg, providing clinician training). For each action, sites identified the personnel involved and estimated the time they spent, allowing us to estimate the total costs of implementation and breakdown costs by major implementation activities.

RESULTS

Process maps indicated that the program integrated easily into well-child visits. Overall implementation costs ranged from $77,103 (Prisma Health) to $84,954 (Denver Health) to $142,721 (Massachusetts General Hospital). Across implementation activities, setting up the technological aspects of the program was a major driver of costs. Other cost drivers included training, engaging stakeholders, and audit and feedback activities, though there was variability across systems based on organizational context and implementation choices.

CONCLUSIONS

Our work highlights the major cost drivers of implementing the Connect for Health program. Accounting for context-specific considerations when assessing the costs of implementation is crucial, especially to facilitate accurate projections of implementation costs in future settings.

Perspectives on Financing Strategies for Evidence-Based Treatment Implementation in Youth Mental Health Systems

BACKGROUND

Evidence-based treatments (EBTs) are critical to effectively address mental health problems among children and adolescents, but costly for mental health service agencies to implement and sustain. Financing strategies help agencies overcome cost-related barriers by obtaining financial resources to support EBT implementation and/or sustainment.

AIMS

We sought to (i) understand how youth mental health system decision-makers involved with EBT implementation and sustainment view key features (e.g., relevance, feasibility) that inform financing strategy selection and (ii) compare service agency, funding agency, and intermediary representative perspectives.

METHOD

Two surveys were disseminated to 48 representatives across U.S. youth mental health service agencies, funding agencies, and intermediaries who were participating in a larger study of financing strategies. Quantitative and qualitative data were gathered on 23 financing strategies through quantitative ratings and open-ended responses. Data were analyzed using descriptive statistics and rapid content analysis.

RESULTS

The financing strategies rated as most relevant include braided funding streams, contracts for EBTs, credentialing/rostering providers, fee-for-service reimbursement (regular and increased), and grant funding. All other strategies were unfamiliar to 1/3 to 1/2 of participants. The six strategies were rated between somewhat and quite available, feasible, and effective for EBT sustainment. For sustaining different EBT components (e.g., delivery, materials), the mix of financing strategies was rated as somewhat adequate. Qualitative analysis revealed challenges with strategies being non-recurring or unavailable in representatives' regions. Ratings were largely similar across participant roles, though funding agency representatives were the most familiar with financing strategies.

DISCUSSION

Despite the breadth of innovative financing strategies, expert representatives within the youth mental health services ecosystem had limited knowledge of most options. Experts relied on strategies that were familiar but often did not adequately support EBT implementation or sustainment. These findings underscore more fundamental issues with under-resourced mental health systems in the U.S.; financing strategies can help agencies navigate EBT use but must be accompanied by larger-scale system reforms. Limitations include difficulties generalizing results due to using a small sample familiar with EBTs, high agreement as a potential function of snowball recruiting, and limited responses to the open-ended survey questions.

IMPLICATIONS FOR HEALTH CARE PROVISION AND USE

Although EBTs have been found to effectively address mental health problems in children and adolescents, available strategies for financing their implementation and sustainment in mental health systems are insufficient. This constraint prevents many children and adolescents from receiving high-quality services.

IMPLICATIONS FOR HEALTH POLICIES

Financing strategies alone cannot solve systematic issues that prevent youth mental health service agencies from providing EBTs. Policy changes may be required, such as increased financial investment from the U.S. government into mental health services to support basic infrastructure (e.g., facility operations, measuring outcomes).

IMPLICATIONS FOR FURTHER RESEARCH

Future work should examine expert perspectives on EBT financing strategies in different contexts (e.g., substance use services), gathering targeted feedback on financing strategies that are less well known, and exploring topics such as strategic planning, funding stability, and collaborative decision-making as they relate to EBT implementation and sustainment.

Costs of implementing a multi-site facilitation intervention to increase access to medication treatment for opioid use disorder

BACKGROUND

The United States has been grappling with the opioid epidemic, which has resulted in over 75,000 opioid-related deaths between April 2020 and 2021. Evidence-based pharmaceutical interventions (buprenorphine, methadone, and naltrexone) are available to reduce opioid-related overdoses and deaths. However, adoption of these medications for opioid use disorder has been stifled due to individual- and system-level barriers. External facilitation is an evidence-based implementation intervention that has been used to increase access to medication for opioid use disorder (MOUD), but the implementation costs of external facilitation have not been assessed. We sought to measure the facility-level direct costs of implementing an external facilitation intervention for MOUD to provide decision makers with estimates of the resources needed to implement this evidence-based program.

METHODS

We performed a cost analysis of the pre-implementation and implementation phases, including an itemization of external facilitation team and local site labor costs. We used labor estimates from the Bureau of Labor and Statistics, and sensitivity analyses were performed using labor estimates from the Veterans Health Administration (VHA) Financial Management System general ledger data.

RESULTS

The average total costs for implementing an external facilitation intervention for MOUD per site was $18,847 (SD 6717) and ranged between $11,320 and $31,592. This translates to approximately $48 per patient with OUD. Sites with more encounters and participants with higher salaries in attendance had higher costs. This was driven mostly by the labor involved in planning and implementation activities. The average total cost of the pre-implementation and implementation activities were $1031 and $17,816 per site, respectively. In the sensitivity analysis, costs for VHA were higher than BLS estimates likely due to higher wages.

CONCLUSIONS

Implementing external facilitation to increase MOUD prescribing may be affordable depending on the payer's budget constraints. Our study reported that there were variations in the time invested at each phase of implementation and the number and type of participants involved with implementing an external facilitation intervention. Participant composition played an important role in total implementation costs, and decision makers will need to identify the most efficient and optimal number of stakeholders to involve in their implementation plans.

A budget impact analysis of cost to implement a whole child health focused, family-based intervention in primary care for children with elevated BMI

BACKGROUND

Although the cost of implementing evidence-based interventions (EBIs) is a key determinant of adoption, lack of cost information is widespread. We previously evaluated the cost of preparing to implement Family Check-Up 4 Health (FCU4Health), an individually tailored, evidence-based parenting program that takes a whole child approach, with effects on both behavioral health and health behavior outcomes, in primary care settings. This study estimates the cost of implementation, including preparation.

METHODS

We assessed the cost of FCU4Health across the preparation and implementation phases spanning 32 months and 1 week (October 1, 2016-June 13, 2019) in a type 2 hybrid effectiveness-implementation study. This family-level randomized controlled trial took place in Arizona with n = 113 predominantly low-income, Latino families with children ages > 5.5 to < 13 years. Using electronic cost capture and time-based activity-driven methods, budget impact analysis from the perspective of a future FCU4Health adopting entity-namely, ambulatory pediatric care clinicians-was used to estimate the cost of implementation. Labor costs were based on 2021 Bureau of Labor Statistics Occupational Employment Statistics, NIH-directed salary cap levels or known salaries, plus fringe benefits at a standard rate of 30%. Non-labor costs were based on actual amounts spent from receipts and invoices.

RESULTS

The cost of FCU4Health implementation to 113 families was $268,886 ($2380 per family). Actual per family cost varied widely, as individual tailoring resulted in families receiving a range of 1-15 sessions. The estimated cost of replicating implementation for future sites ranged from $37,636-$72,372 ($333-$641 per family). Using our previously reported preparation costs (i.e., $174,489; $1544 per family), with estimated replication costs of $18,524-$21,836 ($164-$193 per family), the total cost of delivering FCU4Health was $443,375 ($3924 per family), with total estimated replication costs of $56,160-$94,208 ($497-$834 per family).

CONCLUSIONS

This study provides a baseline for costs associated with implementation of an individually tailored parenting program. Results provide critical information for decision makers and a model for future economic analysis and can be used to inform optimization thresholds for implementation and, when necessary, benchmarks for program adaptation to promote scale-up.

TRIAL REGISTRATION

This trial was prospectively registered on January 6, 2017, at ClinicalTrials.gov (NCT03013309).

Programmatic Costs of Project ImPACT for Children with Autism: A Time-Driven Activity Based Costing Study

Programmatic cost assessment of clinical interventions can inform future dissemination and implementation efforts. We conducted a randomized trial of Project ImPACT (Improving Parents As Communication Teachers) in which community early intervention (EI) providers coached caregivers in techniques to improve young children's social communication skills. We estimated implementation and intervention costs while demonstrating an application of Time-Driven Activity-Based Costing (TDABC). We defined Project ImPACT implementation and intervention as processes that can be broken down successively into a set of procedures. We created process maps for both implementation and intervention delivery. We determined resource use and costs, per unit procedure in the first year of the program, from a payer perspective. We estimated total implementation cost per clinician and per site, intervention cost per child, and provided estimates of total hours spent and associated costs for implementation strategies, intervention activities and their detailed procedures. Total implementation cost was $43,509 per clinic and $14,503 per clinician. Clinician time (60%) and coach time (12%) were the most expensive personnel resources. Implementation coordination and monitoring (47%), ongoing consultation (26%) and clinician training (19%) comprised most of the implementation cost, followed by fidelity assessment (7%), and stakeholder engagement (1%). Per-child intervention costs were $2619 and $9650, respectively, at a dose of one hour per week and four hours per week Project ImPACT. Clinician and clinic leader time accounted for 98% of per child intervention costs. Highest cost intervention activity was ImPACT delivery to parents (89%) followed by assessment for child's ImPACT eligibility (10%). The findings can be used to inform funding and policy decision-making to enhance early intervention options for young children with autism. Uncompensated time costs of clinicians are large which raises practical and ethical concerns and should be considered in planning of implementation initiatives. In program budgeting, decisionmakers should anticipate resource needs for coordination and monitoring activities. TDABC may encourage researchers to assess costs more systematically, relying on process mapping and gathering prospective data on resource use and costs concurrently with their collection of other trial data.

A Natural Helper Intervention to Address Disparities in Parent Child-Interaction Therapy: A Randomized Pilot Study

OBJECTIVE

Parent-child interaction therapy (PCIT) is an effective intervention to address child externalizing behaviors. However, disparities in access and retention are pervasive, which relate to the availability of PCIT in low-income communities, inadequate workforces to provide culturally appropriate care, and distrust in services due to systemic discrimination. This study incorporated natural helpers who had been trained as community health workers into PCIT delivery to improve disparities in engagement and outcomes.

METHOD

Families from three low-income, predominately Latino/a/x and Black neighborhoods in Miami qualified for services if they had a child aged 2-8 with clinically elevated externalizing behaviors. Families were randomly assigned into either Standard-PCIT group (N = 30 families; 80% boys, 57% Latino/a/x, 27% Black) or a PCIT plus Natural helper (PCIT+NH) group (N = 51 families; 66% boys, 76% Latino/a/x, 18% Black). Families in the PCIT+NH group received home visits and support addressing barriers to care from a natural helper. Path analyses within an intention-to-treat framework examined group-differences in treatment engagement, child behavior, and parenting skills and stress.

RESULTS

Families in both groups demonstrated large improvements in child externalizing behavior, caregiver stress, and parenting skills from pre-to-post-treatment. Externalizing behavior improved significantly more in the PCIT+NH group compared to the Standard-PCIT group. There were no significant group differences in parenting skills or caregiver stress. Though differences in engagement were not significant, the PCIT+NH group had a small effect on treatment retention.

CONCLUSIONS

Natural helpers may help to address structural barriers that systematically impact communities of color, apply treatment in naturalistic environments, and promote improved treatment outcomes.

Assessing the sustainability capacity of evidence-based programs in community and health settings

Background

Within many public health settings, there remain large challenges to sustaining evidence-based practices. The Program Sustainability Assessment Tool has been developed and validated to measure sustainability capacity of public health, social service, and educational programs. This paper describes how this tool was utilized between January 2014 and January 2019. We describe characteristics of programs that are associated with increased capacity for sustainability and ultimately describe the utility of the PSAT in sustainability research and practice.

Methods

The PSAT is comprised of 8 subscales, measuring sustainability capacity in eight distinct conceptual domains. Each subscale is made up of five items, all assessed on a 7-point Likert scale. Data were obtained from persons who used the PSAT on the online website (https://sustaintool.org/), from 2014 to 2019. In addition to the PSAT scale, participants were asked about four program-level characteristics. The resulting dataset includes 5,706 individual assessments reporting on 2,892 programs.

Results

The mean overall PSAT score was 4.73, with the lowest and highest scoring subscales being funding stability and program adaptation, respectively. Internal consistency for each subscale was excellent (average Cronbach's alpha = 0.90, ranging from 0.85 to 0.94). Confirmatory factor analysis highlighted good to excellent fit of the PSAT measurement model (eight distinct conceptual domains) to the observed data, with a comparative fit index of 0.902, root mean square error of approximation equal to 0.054, and standardized root mean square residual of 0.054. Overall sustainability capacity was significantly related to program size (F = 25.6; p < 0.001). Specifically, smaller programs (with staff sizes of ten or below) consistently reported lower program sustainability capacity. Capacity was not associated with program age and did not vary significantly by program level.

Discussion

The PSAT maintained its excellent reliability when tested with a large and diverse sample over time. Initial criterion validity was explored through the assessment of program characteristics, including program type and program size. The data collected reinforces the ability of the PSAT to assess sustainability capacity for a wide variety of public health and social programs.

Economic evaluation of implementation science outcomes in low- and middle-income countries: a scoping review

BACKGROUND

Historically, the focus of cost-effectiveness analyses has been on the costs to operate and deliver interventions after their initial design and launch. The costs related to design and implementation of interventions have often been omitted. Ignoring these costs leads to an underestimation of the true price of interventions and biases economic analyses toward favoring new interventions. This is especially true in low- and middle-income countries (LMICs), where implementation may require substantial up-front investment. This scoping review was conducted to explore the topics, depth, and availability of scientific literature on integrating implementation science into economic evaluations of health interventions in LMICs.

METHODS

We searched Web of Science and PubMed for papers published between January 1, 2010, and December 31, 2021, that included components of both implementation science and economic evaluation. Studies from LMICs were prioritized for review, but papers from high-income countries were included if their methodology/findings were relevant to LMIC settings.

RESULTS

Six thousand nine hundred eighty-six studies were screened, of which 55 were included in full-text review and 23 selected for inclusion and data extraction. Most papers were theoretical, though some focused on a single disease or disease subset, including: mental health (n = 5), HIV (n = 3), tuberculosis (n = 3), and diabetes (n = 2). Manuscripts included a mix of methodology papers, empirical studies, and other (e.g., narrative) reviews. Authorship of the included literature was skewed toward high-income settings, with 22 of the 23 papers featuring first and senior authors from high-income countries. Of nine empirical studies included, no consistent implementation cost outcomes were measured, and only four could be mapped to an existing costing or implementation framework. There was also substantial heterogeneity across studies in how implementation costs were defined, and the methods used to collect them.

CONCLUSION

A sparse but growing literature explores the intersection of implementation science and economic evaluation. Key needs include more research in LMICs, greater consensus on the definition of implementation costs, standardized methods to collect such costs, and identifying outcomes of greatest relevance. Addressing these gaps will result in stronger links between implementation science and economic evaluation and will create more robust and accurate estimates of intervention costs.

TRIAL REGISTRATION

The protocol for this manuscript was published on the Open Science Framework. It is available at: https://osf.io/ms5fa/ (DOI: 10.17605/OSF.IO/32EPJ).

Required sample size to detect mediation in 3-level implementation studies

Background

Statistical tests of mediation are important for advancing implementation science; however, little research has examined the sample sizes needed to detect mediation in 3-level designs (e.g., organization, provider, patient) that are common in implementation research. Using a generalizable Monte Carlo simulation method, this paper examines the sample sizes required to detect mediation in 3-level designs under a range of conditions plausible for implementation studies.

Method

Statistical power was estimated for 17,496 3-level mediation designs in which the independent variable (X) resided at the highest cluster level (e.g., organization), the mediator (M) resided at the intermediate nested level (e.g., provider), and the outcome (Y) resided at the lowest nested level (e.g., patient). Designs varied by sample size per level, intraclass correlation coefficients of M and Y, effect sizes of the two paths constituting the indirect (mediation) effect (i.e., X→M and M→Y), and size of the direct effect. Power estimates were generated for all designs using two statistical models—conventional linear multilevel modeling of manifest variables (MVM) and multilevel structural equation modeling (MSEM)—for both 1- and 2-sided hypothesis tests.

Results

For 2-sided tests, statistical power to detect mediation was sufficient (≥0.8) in only 463 designs (2.6%) estimated using MVM and 228 designs (1.3%) estimated using MSEM; the minimum number of highest-level units needed to achieve adequate power was 40; the minimum total sample size was 900 observations. For 1-sided tests, 808 designs (4.6%) estimated using MVM and 369 designs (2.1%) estimated using MSEM had adequate power; the minimum number of highest-level units was 20; the minimum total sample was 600. At least one large effect size for either the X→M or M→Y path was necessary to achieve adequate power across all conditions.

Conclusions

While our analysis has important limitations, results suggest many of the 3-level mediation designs that can realistically be conducted in implementation research lack statistical power to detect mediation of highest-level independent variables unless effect sizes are large and 40 or more highest-level units are enrolled. We suggest strategies to increase statistical power for multilevel mediation designs and innovations to improve the feasibility of mediation tests in implementation research.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-022-01235-2.

Using decision analysis to support implementation planning in research and practice

BACKGROUND

The process of implementing evidence-based interventions, programs, and policies is difficult and complex. Planning for implementation is critical and likely plays a key role in the long-term impact and sustainability of interventions in practice. However, implementation planning is also difficult. Implementors must choose what to implement and how best to implement it, and each choice has costs and consequences to consider. As a step towards supporting structured and organized implementation planning, we advocate for increased use of decision analysis.

MAIN TEXT

When applied to implementation planning, decision analysis guides users to explicitly define the problem of interest, outline different plans (e.g., interventions/actions, implementation strategies, timelines), and assess the potential outcomes under each alternative in their context. We ground our discussion of decision analysis in the PROACTIVE framework, which guides teams through key steps in decision analyses. This framework includes three phases: (1) definition of the decision problems and overall objectives with purposeful stakeholder engagement, (2) identification and comparison of different alternatives, and (3) synthesis of information on each alternative, incorporating uncertainty. We present three examples to illustrate the breadth of relevant decision analysis approaches to implementation planning.

CONCLUSION

To further the use of decision analysis for implementation planning, we suggest areas for future research and practice: embrace model thinking; build the business case for decision analysis; identify when, how, and for whom decision analysis is more or less useful; improve reporting and transparency of cost data; and increase collaborative opportunities and training.

Understanding implementation costs of a pediatric weight management intervention: an economic evaluation protocol

Background

Understanding the cost and/or cost-effectiveness of implementation strategies is crucial for organizations to make informed decisions about the resources needed to implement and sustain evidence-based interventions (EBIs). This economic evaluation protocol describes the methods and processes that will be used to assess costs and cost-effectiveness across implementation strategies used to improve the reach, adoption, implementation, and organizational maintenance of an evidence-based pediatric weight management intervention- Building Health Families (BHF).

Methods

A within-trial cost and cost-effectiveness analysis (CEA) will be completed as part of a hybrid type III effectiveness-implementation trial (HEI) designed to examine the impact of an action Learning Collaborative (LC) strategy consisting of network weaving, consultee-centered training, goal-setting and feedback, and sustainability action planning to improve the adoption, implementation, organizational maintenance, and program reach of BHF in micropolitan and surrounding rural communities in the USA, over a 12-month period. We discuss key features of implementation strategy components and the associated cost collection and outcome measures and present brief examples on what will be included in the CEA for each discrete implementation strategy and how the results will be interpreted. The cost data will be collected by identifying implementation activities associated with each strategy and using a digital-based time tracking tool to capture the time associated with each activity. Costs will be assessed relative to the BHF program implementation and the multicomponent implementation strategy, included within and external to a LC designed to improve reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) of BHF. The CEA results will be reported by RE-AIM outcomes, using the average cost-effectiveness ratio or incremental cost-effectiveness ratio. All the CEAs will be performed from the community perspective.

Discussion

The proposed costing approach and economic evaluation framework for dissemination and implementation strategies and EBI implementation will contribute to the evolving but still scant literature on economic evaluation of implementation and strategies used and facilitate the comparative economic analysis.

Trial registration

ClinicalTrials.gov NCT04719442. Registered on January 22, 2021.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-022-00287-1.

Moving beyond Aim Three: a need for a transdisciplinary approach to build capacity for economic evaluations in implementation science

BACKGROUND

Understanding the costs and economic benefits of implementation has been identified by policymakers and researchers as critical to increase the uptake and sustainment of evidence-based practices, but this topic remains relatively understudied. Conducting team science with health economists has been proposed as a solution to increase economic evaluation in implementation science; however, these recommendations ignore the differences in goals and perspectives in these two fields. Our recent qualitative research identified that implementation researchers predominantly approach health economists to examine costs, whereas the majority of health economists expressed limited interest in conducting economic evaluations and a desire to be more integrated within implementation science initiatives. These interviews pointed to challenges in establishing fruitful partnerships when health economists are relegated to the "Third Aim" (i.e., lowest-priority research objective) in implementation science projects by their research partners.

DISCUSSION

In this debate paper, we argue that implementation researchers and health economists need to focus on team science research principles to expand capacity to address pressing research questions that cut across the two fields. Specifically, we use the four-phase model of transdisciplinary research to outline the goals and processes needed to build capacity in this area (Hall et al., Transl Behav Med 2:415-30, 2012). The first phase focuses on the development of transdisciplinary research teams, including identifying appropriate partners (e.g., considering policy or public health researchers in addition to health economists) and building trust. The conceptual phase focuses on strategies to consider when developing joint research questions and methodology across fields. In the implementation phase, we outline the effective processes for conducting research projects, such as team learning. Finally, in the translation phase, we highlight how a transdisciplinary approach between health economists and implementation researchers can impact real-world practice and policy. The importance of investigating the economic impact of evidence-based practice implementation is widely recognized, but efforts have been limited due to the challenges in conducting team science across disciplines. Training in team science can help advance transdisciplinary efforts, which has the potential to increase the rigor and impact of economic evaluations in implementation science while expanding the roles taken by health economists.