Process Evaluation and Investigation of Cultural Adaptations for an Online Parent-Based Intervention Using a Mixed-Method Approach

Most alcohol intervention research focuses on program efficacy, yet few studies have investigated the acceptability of a program's design and implementation to the target population or adapting existing alcohol interventions to different populations. To address these gaps in the literature, we (1) examined participant responsiveness to and implementation quality of FITSTART+, a web-app delivered parent-based alcohol intervention designed for incoming first-year college students in the United States, and (2) gathered feedback on how this intervention could be adapted to other populations of parents. A sample of U.S. parents of 17-20-year-old first-year college students (N = 109) participated in FITSTART+ during their child's first year of college and completed a survey about parents' responsiveness to the app and its quality. Next, a sample of non-U.S. parents of adolescents aged 13 to 19 (N = 44) participated in one of 11 focus groups in which they briefly explored the app and then discussed how it could be adapted to be applicable and culturally relevant for them and their context. Results revealed that U.S. parents rated the intervention's quality as high and parents were responsive to the web-app's content, but some did not visit one of the most critical aspects of the intervention (i.e., alcohol-related parenting resources). Non-U.S. participants provided a range of suggestions for adapting the intervention to their context, which varied by culture. Results identify areas for improvement, particularly regarding the use of alcohol-related parenting resources, in this intervention and for web-delivered PBIs more broadly.

Feasibility, usability, and acceptability of MobileCoach-Teen: A smartphone app-based preventative intervention for risky adolescent drinking behavior

BACKGROUND

Older adolescence (ages 15-18) is a critical period for experimentation with substance use, especially alcohol. Adolescent drinking poses hazards to physical and mental health, amplifies risk associated with other activities typically initiated during this life stage (e.g., driving, sexual activity), and is associated with adverse outcomes in adolescence and adulthood. Existing preventative interventions are expensive and have questionable long-term efficacy. Digital interventions may represent an accessible and personalized approach to providing preventative intervention content to youth.

METHODS

This study recruited 29 adolescents aged 16-18 (M = 17.24, SD = 0.74) for a pilot feasibility trial of the MobileCoach-Teen (MC-Teen) smartphone app-based intervention. The study team randomized participants to receive either the alcohol intervention (MC-Teen) or attention control pseudo-intervention (MC-Fit). MC-Teen participants received 12 weeks of content adapted from a prior Swiss-based trial of a preventative alcohol intervention. Participants provided qualitative and quantitative feedback at baseline, via six biweekly surveys during and post-intervention.

RESULTS

Both groups rated the application as easy to download (M = 4.31, SD = 0.93; 5-point Likert). All participants completed the baseline survey in less than the estimated time of 10 min (M = 7:42, SD = 2:15) and rated the survey as easy to complete (M = 4.69, SD = 0.60; 5-point Likert). MC-Teen participants favorably assessed application user experience, message user experience, and digital working alliance with application. Qualitative themes included a desire for increased rate/amount and diversity of content, greater representation via coach options, user interface/user experience improvements, and additional features.

CONCLUSION

The MC-Teen intervention is feasible and acceptable based on a pilot feasibility trial with a sample of U.S. adolescents.

Effects of the FITSTART + PBI on drinking and negative alcohol-related consequences during the first year of college

Objective: This study evaluated FITSTART+, a parent-based intervention (PBI), for preventing risky drinking among first-year college students. Participants: Participants were traditional first-year students aged between 17 and 20. Method: In total, 391 eligible students completed a baseline survey and their parents were then invited to use the FITSTART+ PBI or Control web-applications. 266 students had a parent who created a profile in the app (FITSTART+ PBI, n = 134; FITSTART+ Control, n = 132). Additionally, 58 parents randomized to the FITSTART+ PBI did not complete the personalized normative feedback (PNF) component of the intervention. Results: No significant effect on drinking or related consequences was detected between Intervention and Control groups. However, exploratory analyses revealed that completing PNF in the FITSTART+ PBI condition was associated with a greater likelihood of remaining zero on consequences at follow-ups. Conclusion: The PNF component of the FITSTART+ PBI showed potential in preventing risky drinking, warranting further research.

The Association Between Facilitator Competent Adherence and Outcomes in Parenting Programs: a Systematic Review and SWiM Analysis

There is increasing interest about the fidelity with which interventions are implemented because it is theorized that better implementation fidelity by facilitators is associated with better participant outcomes. However, in the parenting program literature, there is mixed evidence on the relationship between implementation fidelity and outcomes. This paper provides a synthesis of the evidence on the relationship between facilitator delivery and outcomes in the parenting program literature. Following PRISMA guidelines, this paper synthesizes the results of a systematic review of studies on parenting programs aiming to reduce violence against children and child behavior problems. Specifically, it examines associations between observational measures of facilitator competent adherence and parent and child outcomes. A meta-analysis was not feasible due to study heterogeneity. As a result, Synthesis Without Meta-Analysis guidelines were followed. Searches in electronic databases, reference searching, forward citation tracking, and expert input identified 9653 articles. After screening using pre-specified criteria, 18 articles were included. The review found that most studies (n = 13) reported a statistically significant positive relationship with at least one parent or child outcome. However, eight studies reported inconsistent findings across outcomes, and four studies found no association with outcomes. The results suggest that better facilitator competent adherence is generally associated with positive parent and child outcomes. However, this finding is weakened by the methodological heterogeneity of included studies and due to the wide variety of ways in which studies conceptualized competent adherence-outcome relationships.

Measuring the dosage of brief and skill-targeted social-emotional learning (SEL) activities in humanitarian settings

Introduction

In humanitarian settings, social-emotional learning (SEL) programs for children are often delivered using a field-feasible approach where the programs are more easily deployable and adaptable in the field, require minimal training, and depend less on the strict sequence and structure of the program components to elicit the intended treatment effect. However, evidence is lacking on what aspects of this implementation approach enable the SEL programming to be more beneficial to children's SEL development.

Method

In this study, we propose and evaluate measures for three dimensions of dosage (quantity, duration, and temporal pattern) of two sets of brief and skill-targeted SEL activities (Mindfulness and Brain Games) implemented in 20 primary schools in two low-income chiefdoms of Sierra Leone.

Results

We find preliminary evidence of predictive validity that these dosage measures could predict children's attendance and classroom adaptive behavior.

Discussion

This study is the first to develop procedures to measure the dimensions of dosage of brief SEL activities in humanitarian settings. Our findings illuminate the need for future research on optimizing the dosage and implementation design of SEL programming using brief SEL activities.

[Evaluating population health interventions : The contributions of theory-driven evaluations]

Population health intervention research has been characterized by the deployment of scientific methods designed to produce knowledge on policy and program interventions directly or indirectly involving the health sector, and potentially affecting population health. The proposed solutions encompass a multitude of interventions of variable types, scale, focus and implementation, rendering them particularly complex and difficult to understand. This complexity raises major conceptual and methodological issues because in reality, we assess not an intervention, but rather a set of interventional elements interacting with those specific to the context in which they appear. It is the interactions that produce effects, to the extent that it makes little sense to speak of an intervention, and more sense to consider an interventional system defined by the interactions. To grasp the numerous elements brought into play, it behooves us to amalgamate evaluation paradigms and approaches. In a precise context, theory-driven evaluations are of pronounced interest. This article presents the main principles of this type of evaluation by focusing on its capacity to shed light on the stakes involved in intervention/context interplay, and by putting forward conclusions transferable to population health research.

Understanding the Needs of Primary School Teachers in Supporting Their Students' Emotion Regulation

BACKGROUND

Accumulating evidence has underscored the importance of fostering children's emotion regulation (ER) within primary school settings and the role of teachers in such efforts. This study sought to assess the needs of teachers in supporting students' ER, through a better understanding of teachers' perceptions and use of healthy versus unhealthy ER strategies in the classroom.

METHODS

Primary school teachers (n = 212; 91% female) completed an online, researcher-developed needs assessment survey assessing their perceptions regarding the importance of ER instruction and challenges surrounding children's ER, as well as the perceived effectiveness and reported use of healthy and unhealthy ER strategies in the classroom.

RESULTS

Cochran's Q and chi-square analyses revealed misperceptions regarding the effectiveness of healthy and unhealthy strategies, as well as discrepancies between teachers' perceptions regarding the effectiveness of specific healthy strategies (eg, meditation) and their reported use of them.

CONCLUSIONS

While teachers recognize the growing importance of fostering ER in the classroom, the present findings suggest that there is a need for more professional development regarding the effectiveness and implementation of ER strategies in the primary school context. Efforts should be made to provide teachers with concrete recommendations for the implementation of ER strategies in the classroom.

From raw data to a score: comparing quantitative methods that construct multi-level composite implementation strength scores of family planning programs in Malawi

BACKGROUND

Data that capture implementation strength can be combined in multiple ways across content and health system levels to create a summary measure that can help us to explore and compare program implementation across facility catchment areas. Summary indices can make it easier for national policymakers to understand and address variation in strength of program implementation across jurisdictions. In this paper, we describe the development of an index that we used to describe the district-level strength of implementation of Malawi's national family planning program.

METHODS

To develop the index, we used data collected during a 2017 national, health facility and community health worker Implementation Strength Assessment survey in Malawi to test different methods to combine indicators within and then across domains (4 methods-simple additive, weighted additive, principal components analysis, exploratory factor analysis) and combine scores across health facility and community health worker levels (2 methods-simple average and mixed effects model) to create a catchment area-level summary score for each health facility in Malawi. We explored how well each model captures variation and predicts couple-years protection and how feasible it is to conduct each type of analysis and the resulting interpretability.

RESULTS

We found little difference in how the four methods combined indicator data at the individual and combined levels of the health system. However, there were major differences when combining scores across health system levels to obtain a score at the health facility catchment area level. The scores resulting from the mixed effects model were able to better discriminate differences between catchment area scores compared to the simple average method. The scores using the mixed effects combination method also demonstrated more of a dose-response relationship with couple-years protection.

CONCLUSIONS

The summary measure that was calculated from the mixed effects combination method captured the variation of strength of implementation of Malawi's national family planning program at the health facility catchment area level. However, the best method for creating an index should be based on the pros and cons listed, not least, analyst capacity and ease of interpretability of findings. Ultimately, the resulting summary measure can aid decision-makers in understanding the combined effect of multiple aspects of programs being implemented in their health system and comparing the strengths of programs across geographies.

Adaptations and modifications to a co-designed intervention and its clinical implementation: a qualitative study in Denmark

BACKGROUND

There is a long-standing debate in implementation research on whether adaptations to evidence-based interventions (EBIs) are desirable in health care. If an intervention is adapted and not delivered as conceived and planned, it is said to have low fidelity. The WALK-Cph project was developed based on the assumption that involving stakeholders in co-design processes would facilitate the fidelity of an intervention to increase the mobility of acutely admitted older medical patients and its implementation in two hospitals in Denmark. The purpose of this study is to describe and analyse adaptations and modifications that were made to the co-designed WALK-Cph intervention and its implementation.

METHODS

This study used a qualitative design. An ethnographic field study was performed using participant observations, workshops and semi-structured interviews. Data were analysed twice using the Framework Method. The first analysis was based on the frameworks from Stirman, Moore and Proctor. The second analysis, a retrospective modifications analysis, was based on the Adaptation-Impact Framework.

RESULTS

Many different types of adaptations and modifications were made to the WALK-Cph intervention and its implementation plan. Most of the modifications were made on the contents of the intervention. In total, 44 adaptations and modifications were made, of which 21 were planned (adaptations) and 23 were made haphazardly (modifications). Most of the content and context adaptations and modifications made on the intervention had a mixed result regarding enhanced fidelity. The retrospective modifications analysis showed that modifications were ongoing and both situationally and contextually shaped.

CONCLUSIONS

Although an extensive co-design process was carried out to facilitate the fidelity of the WALK-Cph intervention, this study showed that many adaptations and modifications were still made to both the intervention and its implementation plan. It could indicate that the co-design process had a small effect or that adaptations and modifications are ongoing and both situationally and contextually shaped, which challenge the assumption and the desire to be able to plan and control changes.

A Turkish-Speaking Community Program in A Primary Care Psychotherapy Setting: How to Support and Engage A Complex Patient Population

This paper presents a description of a multigroup-based psychotherapy program for Turkish-speaking individuals provided by an innovative primary care mental health service in London. As of 2021, the project offers two types of therapy: a year-long horticultural group, which is a blend between talking therapy and gardening, and a 20-week psychotherapy group. Both were designed to engage the particularly complex Turkish-speaking patient population in the London Borough of Hackney. The project is grounded in group psychotherapy and borrows elements from different theoretical orientations and therapeutic modalities including community psychology, group analytic therapy, horticultural therapy, attachment theory, and dynamic therapies. Here, we present how the program was created and developed and the specifics of the groups, together with the main themes and dynamics that emerged in the therapeutic process. The program evaluation and outcomes are illustrated by several clinical case vignettes throughout the paper.

Continuing education meetings and workshops: effects on professional practice and healthcare outcomes

BACKGROUND

Educational meetings are used widely by health personnel to provide continuing medical education and to promote implementation of innovations or translate new knowledge to change practice within healthcare systems. Previous reviews have concluded that educational meetings can result in small changes in behaviour, but that effects vary considerably. Investigations into which characteristics of educational meetings might lead to greater impact have yielded varying results, and factors that might explain heterogeneity in effects remain unclear. This is the second update of this Cochrane Review.

OBJECTIVES

• To assess the effects of educational meetings on professional practice and healthcare outcomes • To investigate factors that might explain the heterogeneity of these effects SEARCH METHODS: We searched CENTRAL, MEDLINE, Embase, ERIC, Science Citation Index Expanded (ISI Web of Knowledge), and Social Sciences Citation Index (last search in November 2016).

SELECTION CRITERIA

We sought randomised trials examining the effects of educational meetings on professional practice and patient outcomes.

DATA COLLECTION AND ANALYSIS

Two review authors independently extracted data and assessed risk of bias. One review author assessed the certainty of evidence (GRADE) and discussed with a second review author. We included studies in the primary analysis that reported baseline data and that we judged to be at low or unclear risk of bias. For each comparison of dichotomous outcomes, we measured treatment effect as risk difference adjusted for baseline compliance. We expressed adjusted risk difference values as percentages, and we noted that values greater than zero favour educational meetings. For continuous outcomes, we measured treatment effect as per cent change relative to the control group mean post test, adjusted for baseline performance; we expressed values as percentages and noted that values greater than zero favour educational meetings. We report means and 95% confidence intervals (CIs) and, when appropriate, medians and interquartile ranges to facilitate comparisons to previous versions of this review. We analysed professional and patient outcomes separately and analysed 22 variables that were hypothesised a priori to explain heterogeneity. We explored heterogeneity by using univariate meta-regression and by inspecting violin plots.

MAIN RESULTS

We included 215 studies involving more than 28,167 health professionals, including 142 new studies for this update. Educational meetings as the single intervention or the main component of a multi-faceted intervention compared with no intervention • Probably slightly improve compliance with desired practice when compared with no intervention (65 comparisons, 7868 health professionals for dichotomous outcomes (adjusted risk difference 6.79%, 95% CI 6.62% to 6.97%; median 4.00%; interquartile range 0.29% to 13.00%); 28 comparisons, 2577 health professionals for continuous outcomes (adjusted relative percentage change 44.36%, 95% CI 41.98% to 46.75%; median 20.00%; interquartile range 6.00% to 65.00%)) • Probably slightly improve patient outcomes compared with no intervention (15 comparisons, 2530 health professionals for dichotomous outcomes (adjusted risk difference 3.30%, 95% CI 3.10% to 3.51%; median 0.10%; interquartile range 0.00% to 4.00%); 28 comparisons, 2294 health professionals for continuous outcomes (adjusted relative percentage change 8.35%, 95% CI 7.46% to 9.24%; median 2.00%; interquartile range -1.00% to 21.00%)) The certainty of evidence for this comparison is moderate. Educational meetings alone compared with other interventions • May improve compliance with desired practice when compared with other interventions (6 studies, 1402 health professionals for dichotomous outcomes (adjusted risk difference 9.99%, 95% CI 9.47% to 10.52%; median 16.5%; interquartile range 0.80% to 16.50%); 2 studies, 72 health professionals for continuous outcomes (adjusted relative percentage change 12.00%, 95% CI 9.16% to 14.84%; median 12.00%; interquartile range 0.00% to 24.00%)) No studies met the inclusion criteria for patient outcome measurements. The certainty of evidence for this comparison is low. Interactive educational meetings compared with didactic (lecture-based) educational meetings • We are uncertain of effects on compliance with desired practice (3 studies, 370 health professionals for dichotomous outcomes; 1 study, 192 health professionals for continuous outcomes) or on patient outcomes (1 study, 54 health professionals for continuous outcomes), as the certainty of evidence is very low Any other comparison of different formats and durations of educational meetings • We are uncertain of effects on compliance with desired practice (1 study, 19 health professionals for dichotomous outcomes; 1 study, 20 health professionals for continuous outcomes) or on patient outcomes (1 study, 113 health professionals for continuous outcomes), as the certainty of evidence is very low. Factors that might explain heterogeneity of effects Meta-regression suggests that larger estimates of effect are associated with studies judged to be at high risk of bias, with studies that had unit of analysis errors, and with studies in which the unit of analysis was the provider rather than the patient. Improved compliance with desired practice may be associated with: shorter meetings; poor baseline compliance; better attendance; shorter follow-up; professionals provided with additional take-home material; explicit building of educational meetings on theory; targeting of low- versus high-complexity behaviours; targeting of outcomes with high versus low importance; goal of increasing rather than decreasing behaviour; teaching by opinion leaders; and use of didactic versus interactive teaching methods. Pre-specified exploratory analyses of behaviour change techniques suggest that improved compliance with desired practice may be associated with use of a greater number of behaviour change techniques; goal-setting; provision of feedback; provision for social comparison; and provision for social support. Compliance may be decreased by the use of follow-up prompts, skills training, and barrier identification techniques.

AUTHORS' CONCLUSIONS

Compared with no intervention, educational meetings as the main component of an intervention probably slightly improve professional practice and, to a lesser extent, patient outcomes. Educational meetings may improve compliance with desired practice to a greater extent than other kinds of behaviour change interventions, such as text messages, fees, or office systems. Our findings suggest that multi-strategy approaches might positively influence the effects of educational meetings. Additional trials of educational meetings compared with no intervention are unlikely to change the review findings; therefore we will not further update this review comparison in the future. However, we note that randomised trials comparing different types of education are needed.

Impact of Implementing an Academic Detailing Program on Opioid-Benzodiazepine Co-Prescribing Trends at the U.S. Department of Veterans Affairs

OBJECTIVES

To assess the process and outcomes of academic detailing to enhance the Opioid Safety Initiative and the Psychotropic Drug Safety Initiative to reduce co-prescribing of opioid-benzodiazepine combinations in veterans.

METHODS

A retrospective cohort design was conducted to evaluate the impact of implementing an academic detailing program on opioid-benzodiazepine co-prescribing between October 2014 through March 2019 at the U.S. Department of Veterans Affairs (VA). The primary outcome was the monthly prevalence of veterans (number per 1,000 population) who were co-prescribed opioid-benzodiazepine combination. Process measure was evaluated using implementation reach (proportion of providers who received academic detailing). Station-level analysis was performed using a linear fixed effects regression model to evaluate the rate of change in the prevalence of veterans co-prescribed opioid-benzodiazepine.

RESULTS

Altogether 130 VA stations was included for analysis; 119 stations implemented opioid-related or benzodiazepine-related academic detailing, and 11 stations did not. Stations that had implemented academic detailing had a 33% greater monthly reduction on the opioid-benzodiazepine co-prescribing prevalence compared to stations that did not implement academic detailing (P = .036). In the linear fixed effects regression model, stations that were expected to have 100% of providers exposed to academic detailing were statistically associated with a greater decrease in the monthly prevalence of Veterans co-prescribed opioid-benzodiazepine by 4.9 veterans per 1,000 population (P < .001) compared to stations with 0% of providers exposed to academic detailing.

CONCLUSIONS

Stations that implemented academic detailing and had a higher proportion of providers who were exposed to opioid- or benzodiazepine-related academic detailing had a significant decrease in the monthly prevalence of Veterans co-prescribed opioid-benzodiazepine combinations.

Designing process evaluations using case study to explore the context of complex interventions evaluated in trials

Background

Process evaluations are an important component of an effectiveness evaluation as they focus on understanding the relationship between interventions and context to explain how and why interventions work or fail, and whether they can be transferred to other settings and populations. However, historically, context has not been sufficiently explored and reported resulting in the poor uptake of trial results. Therefore, suitable methodologies are needed to guide the investigation of context. Case study is one appropriate methodology, but there is little guidance about what case study design can offer the study of context in trials. We address this gap in the literature by presenting a number of important considerations for process evaluation using a case study design.

Main text

In this paper, we define context, the relationship between complex interventions and context, and describe case study design methodology. A well-designed process evaluation using case study should consider the following core components: the purpose; definition of the intervention; the trial design, the case, the theories or logic models underpinning the intervention, the sampling approach and the conceptual or theoretical framework. We describe each of these in detail and highlight with examples from recently published process evaluations.

Conclusions

There are a number of approaches to process evaluation design in the literature; however, there is a paucity of research on what case study design can offer process evaluations. We argue that case study is one of the best research designs to underpin process evaluations, to capture the dynamic and complex relationship between intervention and context during implementation. We provide a comprehensive overview of the issues for process evaluation design to consider when using a case study design.

Trial registration

DQIP - ClinicalTrials.gov number, NCT01425502 - OPAL - ISRCTN57746448

Implementation of initiatives to prevent student stress: process evaluation findings from the Healthy High School study

Process evaluation of public health interventions is important for understanding intervention results and can help explain why interventions succeed or fail. This study evaluated implementation of a school-based intervention combining educational and environmental strategies to prevent stress among Danish high school students. We investigated dose delivered, dose received, fidelity, appreciation, barriers and facilitators at the 15 intervention schools using mixed methods and multiple data sources: questionnaires among students, teachers and school coordinators; semi-structured interviews with school coordinators; telephone interviews with student counsellors; and focus group interviews with students and teachers. Implementation varied by schools and classes. Half of the intervention schools delivered the environmental strategies. For the educational strategies, dose delivered differed according to intervention provider. Students reported a lower dose received compared with dose delivered reported by school staff. Overall, student counsellors, school coordinators and students-especially those with low perceived stress-were satisfied with the stress preventive initiatives while teacher satisfaction varied. Five main barriers and three facilitators for implementation were identified. The use of multiple data sources and data methods created new knowledge of the implementation process which is important for the interpretation of effect evaluation and development of future interventions.

Dimensions of the Complexity of Health Interventions: What Are We Talking About? A Review

Many recommendations and innovative approaches are available for the development and evaluation of complex health interventions. We investigated the dimensions of complexity described in health research and how these descriptions may affect the adopted research methodology (e.g., the choice of designs and methods). We used a mixed method approach to review the scientific literature evaluating complex interventions in the health field. Of 438 articles identified, 179 were subjected to descriptive analysis and 48 to content analysis. The three principal dimensions of complexity were: stakeholder characteristics, intervention multimodality and context. Recognition of such dimensions influenced the methodological choices made during evaluation of the interventions with their use of designs and methods, which aimed to address the complexity. We analysed not only how researchers view complexity but also the effects of such views on researcher practices. Our results highlight the need for clarification of what complexity means and to consider complexity when deciding how to evaluate research interventions.

Measuring fidelity of delivery of the Community Occupational Therapy in Dementia-UK intervention

Background

Interpreting data about intervention effectiveness requires an understanding of which intervention components were delivered and whether they were delivered as planned (fidelity of delivery). These studies aimed to develop a reliable measure for assessing fidelity of delivery of the Community Occupational Therapy in Dementia-UK intervention (COTiD-UK) (Study 1) and measure fidelity of delivery of COTiD-UK across sessions, sites and occupational therapists (Study 2).

Methods

The studies used a longitudinal observational design nested within a multi-site randomised controlled trial. Where practicable, all intervention sessions were audio-recorded. Fidelity checklists and coding guidelines were developed, piloted and refined until good agreement was achieved between two coders. Ten percent of sessions were purposively sampled from 12 sites and 31 occupational therapists. Transcripts were coded using checklists developed in Study 1; 10% of sets of intervention session transcripts were double coded to ensure that agreement was maintained. Percentages of components that were delivered were calculated for each session, site and occupational therapist.

Results

A reliable measure of fidelity of delivery for COTiD-UK was developed after several rounds of piloting and amendments. COTiD-UK was delivered with moderate fidelity across all six sessions (range: 52.4–75.5%). The mean range of fidelity varied across sites (26.7–91.2%) and occupational therapists (26.7–94.1%).

Conclusions

A reliable, systematic method for measuring fidelity of delivery of COTiD-UK was developed and applied, and can be adapted for use in similar interventions. As COTiD-UK was delivered with moderate fidelity, there is a reasonable degree of confidence that intervention effects were attributable to COTiD-UK.

Mixed methods evaluation of implementation and outcomes in a community-based cancer prevention intervention

Background

Community-based educational programs can complement clinical strategies to increase cancer screenings and encourage healthier lifestyles to reduce cancer burden. However, implementation quality can influence program outcomes and is rarely formally evaluated in community settings. This mixed-methods study aimed to characterize implementation of a community-based cancer prevention program using the Consolidated Framework for Implementation Research (CFIR), determine if implementation was related to participant outcomes, and identify barriers and facilitators to implementation that could be addressed.

Methods

This study utilized quantitative participant evaluation data (n = 115) and quantitative and qualitative data from semi-structured interviews with program instructors (N = 13). At the participant level, demographic data (age, sex, insurance status) and behavior change intention were captured. Instructor data included implementation of program components and program attendance to create a 7-point implementation score of fidelity and reach variables. Degree of program implementation (high and low) was operationalized based on these variables (low: 0–4, high: 5–7). Relationships among degree of implementation, participant demographics, and participant outcomes (e.g., intent to be physically active or limit alcohol) were assessed using linear or ordinal logistic mixed effects models as appropriate. Interview data were transcribed and coded deductively for CFIR constructs, and constructs were then rated for magnitude and valence. Patterns between ratings of high and low implementation programs were used to determine constructs that manifested as barriers or facilitators.

Results

Program implementation varied with scores ranging from 4 to 7. High implementation was related to greater improvements in intention to be physically active (p <  0.05), achieve a healthy weight (p <  0.05), and limit alcohol (p <  0.01). Eight constructs distinguished between high and low implementation programs. Design quality and packaging, compatibility, external change agents, access to knowledge and information, and experience were facilitators of implementation and formally appointed internal implementation leaders was a barrier to implementation.

Conclusions

As higher implementation was related to improved participant outcomes, program administrators should emphasize the importance of fidelity in training for program instructors. The CFIR can be used to identify barriers and/or facilitators to implementation in community interventions, but results may be unique from clinical contexts.

Electronic supplementary material

The online version of this article (10.1186/s12889-019-7315-y) contains supplementary material, which is available to authorized users.

From intervention to interventional system: towards greater theorization in population health intervention research

Background

Population health intervention research raises major conceptual and methodological issues. These require us to clarify what an intervention is and how best to address it.

This paper aims to clarify the concepts of intervention and context and to propose a way to consider their interactions in evaluation studies, especially by addressing the mechanisms and using the theory-driven evaluation methodology.

Main text

This article synthesizes the notions of intervention and context. It suggests that we consider an “interventional system”, defined as a set of interrelated human and non-human contextual agents within spatial and temporal boundaries generating mechanistic configurations – mechanisms – which are prerequisites for change in health. The evaluation focal point is no longer the interventional ingredients taken separately from the context, but rather mechanisms that punctuate the process of change. It encourages a move towards theorization in evaluation designs, in order to analyze the interventional system more effectively. More particularly, it promotes theory-driven evaluation, either alone or combined with experimental designs.

Conclusion

Considering the intervention system, hybridizing paradigms in a process of theorization within evaluation designs, including different scientific disciplines, practitioners and intervention beneficiaries, may allow researchers a better understanding of what is being investigated and enable them to design the most appropriate methods and modalities for characterizing the interventional system. Evaluation methodologies should therefore be repositioned in relation to one another with regard to a new definition of “evidence”, repositioning practitioners’ expertise, qualitative paradigms and experimental questions in order to address the intervention system more profoundly.

Program implementation and effectiveness of a national workplace physical activity intervention: UPnGO with ParticipACTION

INTERVENTION

UPnGO with ParticipACTION (UPnGO) is a 6-week workplace physical activity (PA) initiative aiming to increase habitual PA (steps) during the workday. Core intervention components included (1) self-monitoring of steps and action planning behaviours using a Web/mobile app with incentives and (2) organizational support, which included senior management's role modeling and endorsement of the program.

RESEARCH QUESTION

What is the effectiveness and levels of implementation of the UPnGO intervention? What is the relationship between effectiveness and levels of implementation?

METHODS

A single-arm, pre-/post-test study design was used. Participants were 660 employees from nine organizations who had valid step data and complete socio-demographic information at baseline. The primary outcome (mean daily steps) was assessed by Garmin VivoFit. Using the usage data from the UPnGO web-based system, a composite score for levels of implementation was calculated based on participant's compliance with the self-monitoring component and senior management's role modeling. Associations of interest were analyzed using linear mixed-effects models.

RESULTS

Levels of implementation were highly variable across organizations (mean = 68.22% ± 18.75, range = 19.8 to 100%). A significant Time × Implementation (IM) status interaction effect was observed. When stratified by IM status, a significant increase in mean daily steps at week 6 was found among participants in the high (β = 540.01 ± 202.69, p = 0.011) but not low (β = - 81.54 ± 291.96, p = 0.78) implementation group.

CONCLUSION

Findings suggest significant intervention effects in increasing average daily steps among participants who were exposed to optimal levels of implementation (~ 70%). UPnGO may be a scalable workplace PA intervention at a national level, although this needs further verification with more rigorous study designs.

Effect Evaluation of a Web-Based Coaching Intervention to Support Implementation of Sex Education Among Secondary School Teachers: Randomized Controlled Trial

BACKGROUND

The quality of implementation is important to ensure the effectiveness of behavioral change interventions in practice. Implementing such programs with completeness and adherence is not an automatic process and may require additional support. In school settings, the support teachers receive during implementation is often limited and appears to fall short when attempting to preserve completeness and adherence in program delivery. With the aim to improve completeness and adherence of teachers' delivery of a sexual health promoting intervention ("Long Live Love" [LLL]) in secondary education, a Web-based e-coach was developed ("lesgevenindeliefde.nl"or"teachinglove.nl"). The effectiveness of the e-coach, as part of a broader implementation strategy, in influencing teachers' implementation was evaluated.

OBJECTIVE

This study aimed to report on the effect evaluation to determine the effect of the Web-based e-coach on teacher implementation of a school-based sex education program called LLL and on its determinants.

METHODS

A cluster randomized controlled trial (e-coaching vs waiting list control) was conducted with a baseline assessment (T0) and follow-up (T1) 2 weeks after completing the LLL program. A total of 43 schools with 83 teachers participated in the study. In the follow-up, 38 schools participated, 23 in the e-coaching condition with 41 teachers and 15 in the control condition with 26 teachers. Multilevel regression analysis was used to evaluate the effect of the e-coaching website on implementation behavior, namely, completeness and adherence to LLL implementation, and on its determinants.

RESULTS

The e-coaching intervention was not found to have an effect on teachers' implementation behavior; teachers assigned to the experimental e-coaching website did not score higher on completeness (P=.60) or adherence (P=.67) as compared with teachers in the control condition. When comparing the 30 teachers who made actual use of the e-coaching website with the 37 teachers who did not, no significant differences were found either (P≥.54). In addition, there was no effect of e-coaching on the determinants of teacher implementation behavior (t_67-75≤0.69; P≥.22).

CONCLUSIONS

E-coaching was not found to be effective in enhancing completeness of and adherence to LLL by teachers. The lack of effect may be attributed to the intervention content, the limited use, or the study design itself. The e-coaching intervention may not have adequately addressed adherence and completeness of LLL to bring about behavioral change. Furthermore, the e-coaching intervention was not or insufficiently used by teachers. A possible biased sample of motivated, able teachers may have agreed to participate in the study, and a possible "ceiling effect" may have been present because of the high implementation grade. This, however, does not imply that Web-based coaching in itself is an ineffective strategy to promote adherence and completeness of program implementation. A process evaluation is required as follow-up.

TRIAL REGISTRATION

International Standard Randomised Controlled Trial Number ISRCTN11754581; http://www.isrctn.com/ISRCTN11754581 (Archived by WebCite at http://www.webcitation.org/70C5TUOOh).

Evaluating the implementation of Check & Connect in various school settings: Is intervention fidelity necessarily associated with positive outcomes?

There are numerous school dropout prevention programs. However, few of them have undergone a rigorous implementation evaluation to understand their effects. This research presents two studies that evaluated the intervention fidelity and differential effects of Check & Connect (C&C), a targeted school dropout prevention program aimed at promoting student engagement and achievement. A total of 145 elementary school students (Study 1) and 200 secondary school students (Study 2) from two French-Canadian school boards (regional districts grouping elementary and secondary schools) received the C&C intervention for two years. In both studies, a clinical monitoring form was used to compare the intervention fidelity of each program component and active ingredient with what was initially planned. The relation between intervention fidelity and the effects of C&C on student engagement and achievement was analyzed using multiple linear regressions. Overall, the results show that intervention fidelity varies across elementary and secondary schools from one component to another and from one site to another. Furthermore, the association between the fidelity of each component and positive outcomes varies, depending on the implementation site. This evaluation supports the relevance of every component of C&C to favor engagement and academic achievement among at-risk elementary and secondary school students, while suggesting that the importance of certain program components may vary, depending on contextual influences on implementation and outcomes.

Peer support to improve diabetes care: an implementation evaluation of the Australasian Peers for Progress Diabetes Program

BACKGROUND

Several studies have now demonstrated the benefits of peer support in promoting diabetes control. The aim of this study is to evaluate the implementation of a cluster randomised controlled trial of a group-based, peer support program to improve diabetes self-management and thereby, diabetes control in people with Type 2 Diabetes in Victoria, Australia.

METHODS

The intervention program was designed to address four key peer support functions i.e. 1) assistance in daily management, 2) social and emotional support, 3) regular linkage to clinical care, and 4) ongoing and sustained support to assist with the lifelong needs of diabetes self-care management. The intervention participants attended monthly group meetings facilitated by a trained peer leader for 12 months. Data was collected on the intervention's reach, participation, implementation fidelity, groups' effectiveness and participants' perceived support and satisfaction with the intervention. The RE-AIM and PIPE frameworks were used to guide this evaluation.

RESULTS

The trial reached a high proportion (79%) of its target population through mailed invitations. Out of a total of 441 eligible individuals, 273 (61.9%) were willing to participate. The intervention fidelity was high (92.7%). The proportion of successful participants who demonstrated a reduction in 5 years cardiovascular disease risk score was 65.1 and 44.8% in the intervention and control arm respectively. Ninety-four percent (94%) of the intervention participants stated that the program helped them manage their diabetes on a day to day basis. Overall, attending monthly group meetings provided 'a lot of support' to 57% and 'moderate' support to 34% of the participants.

CONCLUSION

Peer support programs are feasible, acceptable and can be used to supplement treatment for patients motivated to improve behaviours related to diabetes. However, program planners need to focus on the participation component in designing future programs. The use of two evaluation frameworks allowed a comprehensive evaluation of the trial from the provider-, participant- and public health perspective. The learnings gained from this evaluation will guide and improve future implementation by improving program feasibility for adoption and acceptability among participants, and will ultimately increase the likelihood of program effectiveness for the participants.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry (ANZCTR) ACTRN12609000469213 . Registered 16 June 2009.

An application of Extended Normalisation Process Theory in a randomised controlled trial of a complex social intervention: Process evaluation of the Strengthening Families Programme (10-14) in Wales, UK

Purpose

Process evaluations generate important data on the extent to which interventions are delivered as intended. However, the tendency to focus only on assessment of pre-specified structural aspects of fidelity has been criticised for paying insufficient attention to implementation processes and how intervention-context interactions influence programme delivery. This paper reports findings from a process evaluation nested within a randomised controlled trial of the Strengthening Families Programme 10–14 (SFP 10–14) in Wales, UK. It uses Extended Normalisation Process Theory to theorise how interaction between SFP 10–14 and local delivery systems - particularly practitioner commitment/capability and organisational capacity - influenced delivery of intended programme activities: fidelity (adherence to SFP 10–14 content and implementation requirements); dose delivered; dose received (participant engagement); participant recruitment and reach (intervention attendance).

Methods

A mixed methods design was utilised. Fidelity assessment sheets (completed by practitioners), structured observation by researchers, and routine data were used to assess: adherence to programme content; staffing numbers and consistency; recruitment/retention; and group size and composition. Interviews with practitioners explored implementation processes and context.

Results

Adherence to programme content was high - with some variation, linked to practitioner commitment to, and understanding of, the intervention’s content and mechanisms. Variation in adherence rates was associated with the extent to which multi-agency delivery team planning meetings were held. Recruitment challenges meant that targets for group size/composition were not always met, but did not affect adherence levels or family engagement. Targets for staffing numbers and consistency were achieved, though capacity within multi-agency networks reduced over time.

Conclusions

Extended Normalisation Process Theory provided a useful framework for assessing implementation and explaining variation by examining intervention-context interactions. Findings highlight the need for process evaluations to consider both the structural and process components of implementation to explain whether programme activities are delivered as intended and why.

•

Uses Extended Normalisation Process Theory to theorise intervention implementation.

•

The Strengthening Families Programme was delivered with good overall fidelity.

•

Adherence to planned content and staffing guidelines was high, with some variation.

•

Recruitment of families was challenging and affected group size/composition

•

Intervention-context interactions shaped variation in fidelity and implementation.

Intervention fidelity in the definitive cluster randomised controlled trial of the Healthy Lifestyles Programme (HeLP) trial: findings from the process evaluation

Background

The Healthy Lifestyles Programme (HeLP) was a novel school-located intervention for 9–10 year olds, designed to prevent obesity by changing patterns of child behaviour through the creation of supportive school and home environments using dynamic and creative delivery methods. This paper reports on both the quantitative and qualitative data regarding the implementation of the HeLP intervention in the definitive cluster randomised controlled trial, which was part of the wider process evaluation.

Methods

Mixed methods were used to collect data on intervention uptake, fidelity of delivery in terms of content and quality of delivery of the intervention, as well as school and child engagement with the programme. Data were collected using registers of attendance, observations and checklists, field notes, focus groups with children and semi-structured interviews with teachers. Qualitative data were analysed thematically and quantitative data were summarized using descriptive statistics.

Results

All 16 intervention schools received a complete or near complete programme (94–100%), which was delivered in the spirit in which it had been designed. Of the 676 children in the intervention schools, over 90% of children participated in each phase of HeLP; 92% of children across the socio-economic spectrum were deemed to be engaged with HeLP and qualitative data revealed a high level of enjoyment by all children, particularly to the interactive drama workshops. Further evidence of child engagment with the programme was demonstrated by children’s clear understanding of programme messages around marketing, moderation and food labelling. Thirteen of the intervention schools were deemed to be fully engaged with HeLP and qualitative data revealed a high level of teacher ‘buy in’, due to the programme’s compatability with the National Curriculum, level of teacher support and use of innovative and creative delivery methods by external drama practitioners.

Conclusion

Our trial shows that it is possible to successfully scale up complex school-based interventions, engage schools and children across the socio-economic spectrum and deliver an intervention as designed. As programme integrity was maintained throughout the HeLP trial, across all intervention schools, we can be confident that the trial findings are a true reflection of the effectiveness of the intervention, enabling policy recommendations to be made.

Trial registration

ISRCTN15811706.

Electronic supplementary material

The online version of this article (10.1186/s12966-017-0616-6) contains supplementary material, which is available to authorized users.

Measures of fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions: A systematic review of measure quality

PURPOSE

Understanding the effectiveness of complex, face-to-face health behaviour change interventions requires high-quality measures to assess fidelity of delivery and engagement. This systematic review aimed to (1) identify the types of measures used to monitor fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions and (2) describe the reporting of psychometric and implementation qualities.

METHODS

Electronic databases were searched, systematic reviews and reference lists were hand-searched, and 21 experts were contacted to identify articles. Studies that quantitatively measured fidelity of delivery of, and/or engagement with, a complex, face-to-face health behaviour change intervention for adults were included. Data on interventions, measures, and psychometric and implementation qualities were extracted and synthesized using narrative analysis.

RESULTS

Sixty-six studies were included: 24 measured both fidelity of delivery and engagement, 20 measured fidelity of delivery, and 22 measured engagement. Measures of fidelity of delivery included observation (n = 17; 38.6%), self-report (n = 15; 34%), quantitatively rated qualitative interviews (n = 1; 2.3%), or multiple measures (n = 11; 25%). Measures of engagement included self-report (n = 18; 39.1%), intervention records (n = 11; 24%), or multiple measures (n = 17; 37%). Fifty-one studies (77%) reported at least one psychometric or implementation quality; 49 studies (74.2%) reported at least one psychometric quality, and 17 studies (25.8%) reported at least one implementation quality.

CONCLUSION

Fewer than half of the reviewed studies measured both fidelity of delivery of, and engagement with complex, face-to-face health behaviour change interventions. More studies reported psychometric qualities than implementation qualities. Interpretation of intervention outcomes from fidelity of delivery and engagement measurements may be limited due to a lack of reporting of psychometric and implementation qualities. Statement of contribution What is already known on this subject? Evidence of fidelity and engagement is needed to understand effectiveness of complex interventions Evidence of fidelity and engagement are rarely reported High-quality measures are needed to measure fidelity and engagement What does this study add? Evidence that indicators of quality of measures are reported in some studies Evidence that psychometric qualities are reported more frequently than implementation qualities A recommendation for intervention evaluations to report indicators of quality of fidelity and engagement measures.

Measuring the Implementation of Youth Empowerment Solutions

Scholars have increasingly emphasized the importance of using evidence-based programs to promote health and prevent disease. While theoretically and empirically based programs may be effective in carefully controlled conditions, many fail to achieve desired outcomes when implemented in real-world settings. Ensuring high-quality implementation of health promotion programs is critically important as variation in implementation is closely associated with program effectiveness. The purpose of this article is to present methods used to document and assess the implementation of the Youth Empowerment Solutions (YES) program. We collected process evaluation data on 25 YES groups from 12 schools over a period of 4 years. The evaluation assessed four key aspects of delivery: fidelity, dose delivered, dose received, and program quality. We found wide variations in delivery for some measures, while others were more consistent across groups. These indicators of program delivery provided a strong basis for evaluating program implementation, taking actions to improve it, and ultimately, deepening understanding of program effectiveness. The study suggests a model for using multiple methods to collect and analyze data about aspects of program delivery to guide future implementations.

Advancing the physical activity curriculum at the collegiate level

Purpose

The purpose of this paper is to assess college students’ pre- and post- health-related, fitness levels, as determined by the American College of Sports Medicine’s (ACSM) five components of fitness, in a one-credit, graded college course and to objectively measure any differences between those pre- and post- health-related fitness levels.

Design/methodology/approach

In a field setting, the investigators conducted health-related, fitness assessments using the ACSM validated protocols. In addition, descriptive statistics were collected including demographic information, such as, age and sex.

Findings

Paired-sample t tests were used to calculate the pre- and post-test scores for six fitness- and health-related categories across four semesters. There were statistically significant (p<0.001) improvements in six different areas in each of the four semesters with the exception of the resting heart rate and VO2 Max measurements in the fall semester of 2014.

Originality/value

This study builds upon the current body of work tracking trends in physical activity, college courses. The results answer health promotion scientists’ call for more research on the implementation and evaluation of programmatic interventions (Domitrovich and Greendberg, 2000; Durlack, 1998; Durlak and DuPre, 2008) “in real-world settings in order to understand if and how an intervention works” (Søvik et al., 2016, p. 238). This results in addressing a research gap in assessing the effectiveness of physical activity courses in higher education (Keating et al., 2005).

Scaling Up a Water, Sanitation, and Hygiene Program in Rural Bangladesh: The Role of Program Implementation

OBJECTIVES

To evaluate whether the quality of implementation of a water, sanitation, and hygiene program called SHEWA-B and delivered by UNICEF to 20 million people in rural Bangladesh was associated with health behaviors and sanitation infrastructure access.

METHODS

We surveyed 33 027 households targeted by SHEWA-B and 1110 SHEWA-B hygiene promoters in 2011 and 2012. We developed an implementation quality index and compared the probability of health behaviors and sanitation infrastructure access in counterfactual scenarios over the range of implementation quality.

RESULTS

Forty-seven percent of households (n = 14 622) had met a SHEWA-B hygiene promoter, and 47% of hygiene promoters (n = 527) could recall all key program messages. The frequency of hygiene promoter visits was not associated with improved outcomes. Higher implementation quality was not associated with better health behaviors or infrastructure access. Outcomes differed by only 1% to 3% in scenarios in which all clusters received low versus high implementation quality.

CONCLUSIONS

SHEWA-B did not meet UNICEF's ideal implementation quality in any area. Improved implementation quality would have resulted in marginal changes in health behaviors or infrastructure access. This suggests that SHEWA-B's design was suboptimal for improving these outcomes.

Alcohol Prevention and Evaluation in the Era of Evidence Based Practice – the Need for a Systematic Approach to Evaluation

AIMS

This article focuses on the shortcomings of experimental outcome evaluations.

MATERIAL

The study uses two studies of the alcohol prevention program Prime for Life (PFL) to illustrate problems associated with a ‘one dimensional’ goal focus and how implementation may affect the efficiency of an intervention.

Results

As evidence based practice often fails to acknowledge the importance of context and implementation quality in its quest to find ‘what works’, it is argued in the article that a wider perspective on efficiency is needed in evaluations of alcohol prevention.

CONSLUSIONS

To be able to find out what works and during what circumstances, evaluators need to look for a wider set of traits that constitute effective interventions.

Methods used to address fidelity of receipt in health intervention research: a citation analysis and systematic review

BACKGROUND

The American Behaviour Change Consortium (BCC) framework acknowledges patients as active participants and supports the need to investigate the fidelity with which they receive interventions, i.e. receipt. According to this framework, addressing receipt consists in using strategies to assess or enhance participants' understanding and/or performance of intervention skills. This systematic review aims to establish the frequency with which receipt is addressed as defined in the BCC framework in health research, and to describe the methods used in papers informed by the BCC framework and in the wider literature.

METHODS

A forward citation search on papers presenting the BCC framework was performed to determine the frequency with which receipt as defined in this framework was addressed. A second electronic database search, including search terms pertaining to fidelity, receipt, health and process evaluations was performed to identify papers reporting on receipt in the wider literature and irrespective of the framework used. These results were combined with forward citation search results to review methods to assess receipt. Eligibility criteria and data extraction forms were developed and applied to papers. Results are described in a narrative synthesis.

RESULTS

19.6% of 33 studies identified from the forward citation search to report on fidelity were found to address receipt. In 60.6% of these, receipt was assessed in relation to understanding and in 42.4% in relation to performance of skill. Strategies to enhance these were present in 12.1% and 21.1% of studies, respectively. Fifty-five studies were included in the review of the wider literature. Several frameworks and operationalisations of receipt were reported, but the latter were not always consistent with the guiding framework. Receipt was most frequently operationalised in relation to intervention content (16.4%), satisfaction (14.5%), engagement (14.5%), and attendance (14.5%). The majority of studies (90.0%) included subjective assessments of receipt. These relied on quantitative (76.0%) rather than qualitative (42.0%) methods and studies collected data on intervention recipients (50.0%), intervention deliverers (28.0%), or both (22.0%). Few studies (26.0%) reported on the reliability or validity of methods used.

CONCLUSIONS

Receipt is infrequently addressed in health research and improvements to methods of assessment and reporting are required.

Countdown to 2015 country case studies: systematic tools to address the "black box" of health systems and policy assessment

BACKGROUND

Evaluating health systems and policy (HSP) change and implementation is critical in understanding reproductive, maternal, newborn and child health (RMNCH) progress within and across countries. Whilst data for health outcomes, coverage and equity have advanced in the last decade, comparable analyses of HSP changes are lacking. We present a set of novel tools developed by Countdown to 2015 (Countdown) to systematically analyse and describe HSP change for RMNCH indicators, enabling multi-country comparisons.

METHODS

International experts worked with eight country teams to develop HSP tools via mixed methods. These tools assess RMNCH change over time (e.g. 1990-2015) and include: (i) Policy and Programme Timeline Tool (depicting change according to level of policy); (ii) Health Policy Tracer Indicators Dashboard (showing 11 selected RMNCH policies over time); (iii) Health Systems Tracer Indicators Dashboard (showing four selected systems indicators over time); and (iv) Programme implementation assessment. To illustrate these tools, we present results from Tanzania and Peru, two of eight Countdown case studies.

RESULTS

The Policy and Programme Timeline tool shows that Tanzania's RMNCH environment is complex, with increased funding and programmes for child survival, particularly primary-care implementation. Maternal health was prioritised since mid-1990s, yet with variable programme implementation, mainly targeting facilities. Newborn health only received attention since 2005, yet is rapidly scaling-up interventions at facility- and community-levels. Reproductive health lost momentum, with re-investment since 2010. Contrastingly, Peru moved from standalone to integrated RMNCH programme implementation, combined with multi-sectoral, anti-poverty strategies. The HSP Tracer Indicators Dashboards show that Peru has adopted nine of 11 policy tracer indicators and Tanzania has adopted seven. Peru costed national RMNCH plans pre-2000, whereas Tanzania developed a national RMNCH plan in 2006 but only costed the reproductive health component. Both countries included all lifesaving RMNCH commodities on their essential medicines lists. Peru has twice the health worker density of Tanzania (15.4 vs. 7.1/10,000 population, respectively), although both are below the 22.8 WHO minimum threshold.

CONCLUSIONS

These are the first HSP tools using mixed methods to systematically analyse and describe RMNCH changes within and across countries, important in informing accelerated progress for ending preventable maternal, newborn and child mortality in the post-2015 era.

Innovatively Supporting Teachers' Implementation of School-Based Sex Education: Developing A Web-Based Coaching Intervention From Problem to Solution

BACKGROUND

Full program implementation is crucial for effectiveness but is often overlooked or insufficiently considered during development of behavioral change interventions. For school-based health promotion programs, teachers are key players in program implementation, but teacher support in this phase is mostly limited to technical support and information. To ensure optimal implementation of the Dutch school-based sexual health program Long Live Love, a Web-based coaching website was developed to support teachers in completeness and fidelity of program implementation.

OBJECTIVE

The aim of this paper is to provide insight into the process of systematic development of a Web-based coaching intervention to support teachers in their implementation of a school-based sexual health program.

METHODS

The intervention mapping (IM) protocol was applied for the development of a theory- and evidence-based intervention. The IM process begins with (1) a needs assessment, followed by (2) the formulation of change objectives, (3) the selection of theory-based intervention methods and practical applications that take the parameters for effectiveness into consideration, (4) integration of practical applications into an organized program, (5) planning for adoption, implementation, and sustainability of the program, and finally, (6) generating an evaluation plan to measure program effectiveness.

RESULTS

Teacher's implementation behavior was characterized by inconsistently selecting parts of the program and not delivering (all) lessons as intended by program developers. Teachers, however, did not perceive this behavior as problematic, revealing the discrepancy between teacher's actual and perceived need for support in delivering Long Live Love lessons with completeness and fidelity. Teachers did, however, acknowledge different difficulties they encountered which could potentially negatively influence the quality of implementation. With the IM protocol, this Web-based coaching intervention was developed based on a concept of unobtrusive coaching, by and for teachers, to bring about change in teachers' implementation behavior.

CONCLUSIONS

This paper provides an example of a Web-based intervention to bring about behavioral change in a target group of intermediaries who lack intrinsic motivation for coaching and who's perceptions differ from their actual problematic behavior. The IM protocol is a useful tool for guiding the scientific development of interventions and making them compatible with the needs and preferences of the target group.

Program Provider and Observer Ratings of School-Based Preventive Intervention Implementation: Agreement and Relation to Youth Outcomes

Few prevention studies have examined the degree to which different measures of program implementation adherence predict youth outcomes. The current study was conducted with rural middle school youth participating in a longitudinal school-based preventive intervention program. Study participants’ average age at the pretest assessment was 12.3 years. The association between program implementation ratings supplied by provider self-reports and trained independent observer reports were evaluated. In addition, the relationship between measures of implementation and youth outcomes were examined. Results indicated that although program providers tended to report higher implementation than independent observers, most ratings were correlated significantly across raters. Observer-reported implementation ratings significantly predicted several youth substance-related outcomes, while provider-reported self-ratings did not.Program provider characteristics predicted several youth outcomes. Findings suggest that there might be a social desirability bias in provider self-reported ratings of implementation and that caution must be used when interpreting self-reported ratings of implementation.

Evaluating the implementation of the Empowering Coaching™ programme

Purpose

– The purpose of this paper is to explore the implementation of a theoretically grounded coach education training programme for youth football coaches in Norway, through observational methods. In particular, it focuses on implementation fidelity and programme adaptation, and possible differences between the coach educators (CEs) according to their level of experience.

Design/methodology/approach

– Implementation fidelity and programme adaptations for seven CEs were explored through the use of a fidelity scale and in-depth qualitative analyses. Participant responsiveness was applied to inform the observational analyses.

Findings

– Results showed that most of the programme was implemented with moderate to high fidelity, and that it was adapted when delivered. Most of the adaptations seemed to be positive, aligning with the programme’s theoretical foundation and goals. A few negative and neutral adaptations also occurred. The most experienced CEs seemed to deliver the programme with highest fidelity and they also made most positive adaptations. The findings, supported by the participants’ evaluation of the programme delivery, indicate that these CEs also delivered the programme with high quality.

Research limitations/implications

– In the field of implementation research there is a need for further studies applying observational methods to explore programme effectiveness in relation to implementation fidelity, adaptations made and the quality of programme delivery.

Originality/value

– This study applies observational methods to evaluate the fidelity of implementation and adaptations made when implementing a coach education programme, and based on these findings quality of delivery is discussed.

Assessing the Quality of Goal Setting in Behavioural Support for Smoking Cessation and its Association with Outcomes

BACKGROUND

Smoking cessation behavioural support can be effective but practitioners differ markedly in effectiveness, possibly due to variation in the quality of delivery of key behaviour change techniques, such as goal setting (i.e. setting a quit date).

OBJECTIVES

This study aimed to (i) develop a reliable method for assessing the quality of practitioners' support in setting quit dates and (ii) assess whether quality predicts initiation of abstinence as a first step to quitting.

METHODS

A scale for scoring the quality of goal setting was developed from national guidance documents and applied to 85 transcribed behavioural support sessions. Inter-rater reliability was assessed. Associations between quality scores and quit attempts were assessed.

RESULTS

The 10-item scale produced had good inter-rater reliability (Kappa = 0.68). Higher quality goal setting was associated with increased self-reported quit attempts (p < .001; OR = 2.60, 95 % CI 1.54-4.40). The scale components 'set a clear quit date' (χ (2) (2, N = 85) = 22.3, p < .001) and 'within an appropriate timeframe' (χ (2) (2, N = 85) = 15.5, p < .001) were independently associated with quit attempts.

CONCLUSIONS

It is possible to reliably assess the quality of goal setting in smoking cessation behavioural support. Higher quality of goal setting is associated with greater likelihood of initiating quit attempts.

Assessing Fidelity of Implementation (FOI) for School-Based Mindfulness and Yoga Interventions: A Systematic Review

As school-based mindfulness and yoga programs gain popularity, the systematic study of fidelity of program implementation (FOI) is critical to provide a more robust understanding of the core components of mindfulness and yoga interventions, their potential to improve specified teacher and student outcomes, and our ability to implement these programs consistently and effectively. This paper reviews the current state of the science with respect to inclusion and reporting of FOI in peer-reviewed studies examining the effects of school-based mindfulness and/or yoga programs targeting students and/or teachers implemented in grades kindergarten through twelve (K-12) in North America. Electronic searches in PsychInfo and Web of Science from their inception through May 2014, in addition to hand searches of relevant review articles, identified 312 publications, 48 of which met inclusion criteria. Findings indicated a relative paucity of rigorous FOI. Fewer than 10% of studies outlined potential core program components or referenced a formal theory of action, and fewer than 20% assessed any aspect of FOI beyond participant dosage. The emerging nature of the evidence base provides a critical window of opportunity to grapple with key issues relevant to FOI of mindfulness-based and yoga programs, including identifying essential elements of these programs that should be faithfully implemented and how we might develop rigorous measures to accurately capture them. Consideration of these questions and suggested next steps are intended to help advance the emerging field of school-based mindfulness and yoga interventions.

Adherence and delivery: implementation quality and program outcomes for the seventh-grade keepin' it REAL program

Poor implementation quality (IQ) is known to reduce program effects making it important to consider IQ for evaluation and dissemination of prevention programs. However, less is known about the ways specific implementation variables relate to outcomes. In this study, two versions of keepin' it REAL, a seventh-grade drug prevention intervention, were implemented in 78 classrooms in 25 schools in rural districts in Pennsylvania and Ohio. IQ was measured through observational coding of 276 videos. IQ variables included adherence to the curriculum, teacher engagement (attentiveness, enthusiasm, seriousness, clarity, positivity), student engagement (attention, participation), and a global rating of teacher delivery quality. Factor analysis showed that teacher engagement, student engagement, and delivery quality formed one factor, which was labeled delivery. A second factor was adherence to the curriculum. Self-report student surveys measured substance use, norms (beliefs about prevalence and acceptability of use), and efficacy (beliefs about one's ability to refuse substance offers) at two waves (pretest, immediate posttest). Mixed model regression analysis which accounted for missing data and controlled for pretest levels examined implementation quality's effects on individual level outcomes, statistically controlling for cluster level effects. Results show that when implemented well, students show positive outcomes compared to students receiving a poorly implemented program. Delivery significantly influenced substance use and norms, but not efficacy. Adherence marginally significantly predicted use and significantly predicted norms, but not efficacy. Findings underscore the importance of comprehensively measuring and accounting for IQ, particularly delivery, when evaluating prevention interventions.

An analysis of the adaptability of a professional development program in public health: results from the ALPS Study

Background

Professional development is a key component of effective public health infrastructures. To be successful, professional development programs in public health and health promotion must adapt to practitioners’ complex real-world practice settings while preserving the core components of those programs’ models and theoretical bases. An appropriate balance must be struck between implementation fidelity, defined as respecting the core nature of the program that underlies its effects, and adaptability to context to maximize benefit in specific situations. This article presents a professional development pilot program, the Health Promotion Laboratory (HPL), and analyzes how it was adapted to three different settings while preserving its core components. An exploratory analysis was also conducted to identify team and contextual factors that might have been at play in the emergence of implementation profiles in each site.

Methods

This paper describes the program, its core components and adaptive features, along with three implementation experiences in local public health teams in Quebec, Canada. For each setting, documentary sources were analyzed to trace the implementation of activities, including temporal patterns throughout the project for each program component. Information about teams and their contexts/settings was obtained through documentary analysis and semi-structured interviews with HPL participants, colleagues and managers from each organization.

Results

While each team developed a unique pattern of implementing the activities, all the program’s core components were implemented. Differences of implementation were observed in terms of numbers and percentages of activities related to different components of the program as well as in the patterns of activities across time. It is plausible that organizational characteristics influencing, for example, work schedule flexibility or learning culture might have played a role in the HPL implementation process.

Conclusions

This paper shows how a professional development program model can be adapted to different contexts while preserving its core components. Capturing the heterogeneity of the intervention’s exposure, as was done here, will make possible in-depth impact analyses involving, for example, the testing of program–context interactions to identify program outcomes predictors. Such work is essential to advance knowledge on the action mechanisms of professional development programs.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-0903-3) contains supplementary material, which is available to authorized users.

Process evaluation of "Girls on the Run": exploring implementation in a physical activity-based positive youth development program

Many positive youth development programs rely on physical activity as a primary program component. Referred to as physical activity-based youth development programs, these program designs have great potential for promoting healthy youth development. This study examined how one such physical activity-based positive youth development program was implemented in order to identify design features critical to maximizing positive youth outcomes. This mixed method, multi-site process evaluation of Girls on the Run (GOTR) utilized focus groups, site visits, and self-report implementation checklists. Implementation scores were calculated to assess implementation fidelity across twenty-nine sites, and qualitative data were inductively analyzed to identify factors influential for implementation. Results reveal variability in how GOTR was implemented. Five themes emerged from the data that represented factors serving as facilitators or barriers to programmatic implementation. These included contextual/environmental factors (e.g., parental involvement, relationships with school personnel), organizational factors (e.g., implementation support and responsiveness of staff), program-specific factors (e.g., curriculum design), coach factors (e.g., existing relationships with participants, responsiveness to participant's needs), and youth factors (e.g., behavioral and discipline issues). Study findings have implications for improving the design of physical activity-based and other positive youth development programs, with relevance to evaluators, program planners, youth development leaders, and others working with children and youth.

Area-based interventions to ameliorate deprived Dutch neighborhoods in practice: does the Dutch District Approach address the social determinants of health to such an extent that future health impacts may be expected?

OBJECTIVE

We studied the local manifestation of a national procedural program that addressed problems regarding employment, education, housing and the physical neighborhood environment, social cohesion, and safety in the most deprived neighborhoods in the Netherlands. We aimed to assess if such a program, without the explicit aim to improve health, results in area-based interventions that address the social determinants of health to such an extent that future health impacts may be expected.

METHODS

We used standardized questionnaires and face-to-face interviews with 39 local district managers. We analyzed the content of the area-based interventions to assess if the activities addressed the social determinants of health. We assessed the duration and scale of the activities in order to estimate their potential to change social determinants of health.

RESULTS

Most districts addressed all six categories of social determinants of health central to the procedural program. Investments in broad-based primary schools, housing stock, green space, and social safety seemed to have the potential to result in district-level changes in social determinants. The scale of activities aimed at employment, income, educational attainment, and the social environment seemed too small to expect an impact at the district level.

CONCLUSION

We conclude that the area-based interventions addressed the neighborhood environment to such an extent that future health impacts of the Dutch District Approach may be expected. The health effects in the long term might be more substantial when area-based interventions were devoted more to the improvement of the socioeconomic circumstances of residents.

Understanding the critical ingredients for facilitating consumer change in housing first programming: a case study approach

Housing First is a form of permanent supportive housing for chronically homeless consumers with mental health and substance abuse issues. In light of the model's growing popularity and wide diffusion, researchers and policy makers have identified a need to better understand its critical ingredients and the processes through which they affect consumer outcomes. Researchers used a bottom-up approach to understand the critical ingredients of Housing First within community-based programs. Interviews and focus groups were conducted with 60 informants (staff and consumers) across 4 "successful" Housing First programs. Qualitative analysis demonstrated six program ingredients to be essential: (1) a low-threshold admissions policy, (2) harm reduction, (3) eviction prevention, (4) reduced service requirements, (5) separation of housing and services, and (6) consumer education.

Addressing core challenges for the next generation of type 2 translation research and systems: the translation science to population impact (TSci Impact) framework

Evidence-based preventive interventions developed over the past two decades represent great potential for enhancing public health and well-being. Research confirming the limited extent to which these interventions have been broadly and effectively implemented, however, indicates much progress is needed to achieve population-level impact. In part, progress requires Type 2 translation research that investigates the complex processes and systems through which evidence-based interventions are adopted, implemented, and sustained on a large scale, with a strong orientation toward devising empirically-driven strategies for increasing their population impact. In this article, we address two core challenges to the advancement of T2 translation research: (1) building infrastructure and capacity to support systems-oriented scaling up of evidence-based interventions, with well-integrated practice-oriented T2 research, and (2) developing an agenda and improving research methods for advancing T2 translation science. We also summarize a heuristic "Translation Science to Population Impact (TSci Impact) Framework." It articulates key considerations in addressing the core challenges, with three components that represent: (1) four phases of translation functions to be investigated (pre-adoption, adoption, implementation, and sustainability); (2) the multiple contexts in which translation occurs, ranging from community to national levels; and (3) necessary practice and research infrastructure supports. Discussion of the framework addresses the critical roles of practitioner-scientist partnerships and networks, governmental agencies and policies at all levels, plus financing partnerships and structures, all required for both infrastructure development and advances in the science. The article concludes with two sets of recommended action steps that could provide impetus for advancing the next generation of T2 translation science and, in turn, potentially enhance the health and well-being of subsequent generations of youth and families.