Using process data to explain outcomes. An illustration from the Child and Adolescent Trial for Cardiovascular Health (CATCH)

Evaluating the Mentors in Violence Prevention Program: A Process Examination of How Implementation Can Affect Gender-Based Violence Outcomes

Gender-based violence is a global public health issue and major human rights concern. It is also a type of violence that is disproportionately experienced by women and girls. This study is the first to examine multiple implementation process (dosage, fidelity, and adaptation) effects on changes in anticipated outcomes of a school-based bystander program targeting gender-based violence, Mentors in Violence Prevention (MVP). Data were collected from two participant groups: mentees (students receiving MVP) and mentors (students delivering MVP), across nine participating high schools. The mentee sample comprised 698 students (about 48.9% males and 49.7% females), aged 11 to 14 years old (M = 11.86, SD = 0.64). The mentor sample comprised 118 students (17.80% males, 82.20% females), aged 15 to 18 years old (M = 16.42, SD = 0.60). Anticipated outcomes were changes in bystanders' attitudes, social influences, control perceptions, intentions, willingness, and intervention behavior, measured using mentees' self-reports at two time points approximately 1 year apart. Implementation processes were measured using mentors' self-reports. Analyses revealed no effects for any of the implementation variables across changes in any of the outcomes measured. These results highlight important implications for the implementation of the MVP program going forward, given its widespread implementation in the United Kingdom. Possible ways that MVP may be enhanced in future are discussed. For example, furthering understanding into how gender-based violence and bystander intervention are addressed and framed during MVP lessons would give more insight into how the current implementation of the program can be improved to maximize its potential benefits.

Addressing the Adherence-Adaptation Debate: Lessons from the Replication of an Evidence-Based Sexual Health Program in School Settings

Whether high adherence to programs is necessary to achieve program outcomes is an area of great debate. The objectives of this study were to determine the frequency, type, and rationale of adaptations made in the implementation of an evidence-based program and to determine program outcomes for intervention program participants, as compared to comparison participants, by the level of adaptations. A total of 1608 participants in 45 classrooms participated. Percent adaptations was calculated by classroom. Thematic qualitative analysis was used to categorize types and rationales for adaptations. Program outcomes by level of adaptations were determined using logistic regression analyses and mean differences. Propensity score matching methods were used to create comparability between adaptation subgroup participants and comparison participants. Adaptations ranged from 2 to 97% across classrooms, with mean adaptations of 63%. Thematic analysis revealed that the adaptations made were related to delivery of content, rather than to the content itself and in response to participant needs and setting constraints. Program outcomes did not appear to be reduced for the high-adaptation subgroup. Understanding both rationale (intent) and type of adaptation made is crucial to understanding the complexity of adaptations. These finding support the argument for allowing facilitators some flexibility and autonomy to adapt the delivery of prescribed content to participant needs and setting constraints.

Statistical process control assessed implementation fidelity of patient-reported outcome measures (PROMs) in routine care

OBJECTIVES

Ensuring implementation fidelity of patient-reported outcome measures (PROMs) in the complex clinical setting remains exigent. We aimed to integrate the routine use of PROM reports in day-to-day patient care and assess the implementation outcomes using statistical process control (SPC) methods.

STUDY DESIGN AND SETTING

This prospective pilot study used an implementation science framework to integrate PROMs. SPC analysis was applied to track patient completion rates and staff acknowledgment rates over time. Daily observational data across clinical areas were collected to assess potential variations.

RESULTS

Data were available from 324 patient encounters over 14 weeks. On average, 78% (52% to 100%) of eligible patients entering the clinic completed the PROMs; staff acknowledged 78% (50-100%) of these patient reports. Most patterns of fluctuation were inside the control frame. Dips in compliance relating to changes in clinic daily routine, including the introduction of the second randomized clinic, were quickly resolved with a structured response.

CONCLUSION

Implementation of PROMs was feasible with reasonable patient completion and staff acknowledgment rates. Fidelity was vulnerable to barriers impacting the daily routine or unusual events in the clinics, suggesting that greater standardization and integration into clinic processes may yet further improve compliance and consistency of reporting.

Electronic Health Program to Empower Patients in Returning to Normal Activities After Colorectal Surgical Procedures: Mixed-Methods Process Evaluation Alongside a Randomized Controlled Trial

BACKGROUND

Long-term recovery takes longer than expected despite improved surgical techniques and Enhanced Recovery After Surgery programs. An electronic health (eHealth) care program ("ikherstel") was developed to partially substitute perioperative care for patients undergoing colorectal surgical procedures. Successfully tested eHealth programs are not always implemented in usual care, and it is, therefore, important to evaluate the process to optimize future implementation.

OBJECTIVE

The aim of this study was to evaluate whether the eHealth intervention was executed as planned.

METHODS

A mixed-methods process evaluation was carried out alongside a multicenter randomized controlled trial (RCT). This evaluation was performed using the Linnan and Steckler framework for the quantitative part of this study, measuring the components reach, dose delivered, dose received, fidelity, and participants' attitudes. Total implementation scores were calculated using the averaging approach, in which the sum of all data points is divided by the number of data points and the total adherence to the protocol is measured. For the qualitative part, the Unified Theory of Acceptance and Use of Technology framework was used. The quantitative data were based on participants' questionnaires, a logistic database, a weblog, and participants' medical files and were obtained by performing semistructured interviews with participants of the RCT.

RESULTS

A total of 151 participants of 340 eligible patients were included in the RCT, of which 73 participants were allocated to the intervention group. On the basis of the quantitative process data, total implementation scores for the website, mobile app, electronic consult, and activity tracker were 64%, 63%, 44%, and 67%, respectively. Participants in the qualitative part experienced the program as supportive and provided guidance on their recovery process after colorectal surgery. Most frequently mentioned barriers were the limited interaction with and feedback from health care professionals and the lack of tailoring of the convalescence plan in case of a different course of recovery.

CONCLUSIONS

The intervention needs more interaction with and feedback from health care professionals and needs more tailored guidance in case of different recovery or treatment courses. To ensure a successful implementation of the program in daily practice, some adjustments are required to optimize the program in a blended care form.

TRIAL REGISTRATION

Netherlands Trial Registry NTR5686; http://www.trialregister.nl/trialreg/admin/rctview.asp?TC= 5686 (Archieved by WebCite at http://www.webcitation.org/75LrJaHrr).

Measuring implementation fidelity of school-based obesity prevention programmes: a systematic review

Background

Until now, there is no clear overview of how fidelity is assessed in school-based obesity prevention programmes. In order to move the field of obesity prevention programmes forward, the current review aimed to 1) identify which fidelity components have been measured in school-based obesity prevention programmes; 2) identify how fidelity components have been measured; and 3) score the quality of these methods.

Methods

Studies published between January 2001–October 2017 were selected from searches in PubMed, EMBASE, PsycINFO, CINAHL, Cochrane Library and ERIC. We included studies examining the fidelity of obesity prevention programmes (nutrition and/or physical activity and/or sitting) at school (children aged 4–18 year) measuring at least one component of implementation fidelity. A data extraction was performed to identify which and how fidelity components were measured. Thereafter, a quality assessment was performed to score the quality of these methods. We scored each fidelity component on 7 quality criteria. Each fidelity component was rated high (> 75% positive), moderate (50–75%) or low (< 50%).

Results

Of the 26,294 retrieved articles, 73 articles reporting on 63 different studies were included in this review. In 17 studies a process evaluation was based on a theoretical framework. In total, 120 fidelity components were measured across studies: dose was measured most often (N = 50), followed by responsiveness (N = 36), adherence (N = 26) and quality of delivery (N = 8). There was substantial variability in how fidelity components were defined as well as how they were measured. Most common methods were observations, logbooks and questionnaires targeting teachers. The quality assessment scores ranged from 0 to 86%; most fidelity components scored low quality (n = 77).

Conclusions

There is no consensus on the operationalisation of concepts and methods used for assessing fidelity in school-based obesity prevention programmes and the quality of methods used is weak. As a result, we call for more consensus on the concepts and clear reporting on the methods employed for measurements of fidelity to increase the quality of fidelity measurements. Moreover, researchers should focus on the relation between fidelity and programme outcomes and determine to what extent adaptations to programmes have been made, whilst still being effective.

Electronic supplementary material

The online version of this article (10.1186/s12966-018-0709-x) contains supplementary material, which is available to authorized users.

Exposure to a multi-level multi-component childhood obesity prevention community-randomized controlled trial: patterns, determinants, and implications

BACKGROUND

For community interventions to be effective in real-world conditions, participants need to have sufficient exposure to the intervention. It is unclear how the dose and intensity of the intervention differ among study participants in low-income areas. We aimed to understand patterns of exposure to different components of a multi-level multi-component obesity prevention program to inform our future impact analyses.

METHODS

B'more Healthy Communities for Kids (BHCK) was a community-randomized controlled trial implemented in 28 low-income zones in Baltimore in two rounds (waves). Exposure to three different intervention components (corner store/carryout restaurants, social media/text messaging, and youth-led nutrition education) was assessed via post-intervention interviews with 385 low-income urban youths and their caregivers. Exposure scores were generated based on self-reported viewing of BHCK materials (posters, handouts, educational displays, and social media posts) and participating in activities, including taste tests during the intervention. For each intervention component, points were assigned for exposure to study materials and activities, then scaled (0-1 range), yielding an overall BHCK exposure score [youths: mean 1.1 (range 0-7.6 points); caregivers: 1.1 (0-6.7), possible highest score: 13]. Ordered logit regression analyses were used to investigate correlates of youths' and caregivers' exposure level (quartile of exposure).

RESULTS

Mean intervention exposure scores were significantly higher for intervention than comparison youths (mean 1.6 vs 0.5, p < 0.001) and caregivers (mean 1.6 vs 0.6, p < 0.001). However, exposure scores were low in both groups and 10% of the comparison group was moderately exposed to the intervention. For each 1-year increase in age, there was a 33% lower odds of being highly exposed to the intervention (odds ratio 0.77, 95% confidence interval 0.69; 0.88) in the unadjusted and adjusted model controlling for youths' sex and household income.

CONCLUSION

Treatment effects may be attenuated in community-based trials, as participants may be differentially exposed to intervention components and the comparison group may also be exposed. Exposure should be measured to provide context to impact evaluations in multi-level trials. Future analyses linking exposure scores to the outcome should control for potential confounders in the treatment-on-the-treated approach, while recognizing that confounding and selection bias may exist affecting causal inference.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT02181010 . Retrospectively registered on 2 July 2014.

Factors Influencing Implementation of a Physical Activity Intervention in Residential Children's Homes

The Environmental Intervention in Children's Homes (ENRICH) study was the first published physical activity intervention undertaken in residential children's homes (RCHs). The study revealed differences in implementation across the homes, which may be a key factor that affects program effectiveness. The purpose of this study was to examine the direct and indirect effects of organizational capacity, provider characteristics, and quality of prevention support system on level of implementation of the ENRICH intervention. This study analyzed the ENRICH process evaluation data collected from 24 RCHs. Bayesian Path analysis was used to examine the direct and indirect effects of organizational capacity, provider characteristics, and quality of prevention support system on level of implementation. Level of implementation across RCHs was variable, ranging from 38 to 97 % (M = 68.3, SD = 14.45). Results revealed that organizational capacity and provider characteristics had significant direct associations with level of implementation. Neither direct nor indirect associations between quality of prevention support system and level of implementation reached statistical significance. Conducting formative assessments on organizational capacity and provider characteristics and incorporating such information in implementation planning may increase the likelihood of achieving higher levels of implementation in future studies.

Directions for the Advancement of Culturally Adapted Preventive Interventions: Local Adaptations, Engagement, and Sustainability

To advance the implementation and dissemination of culturally adapted interventions to diverse populations, greater attention should be devoted to three underdeveloped topics: (a) local adaptations of interventions when they are implemented in community settings, (b) participant engagement, and (c) the sustainability of adapted interventions. Several typologies have been developed for studying local adaptations, and some research indicates that such adaptations might add to intervention effectiveness. There is suggestive evidence of ethnocultural group disparities in intervention engagement and in the success of efforts to boost engagement. Theory and limited data indicate that interventions' flexibility and fit with organizational culture and resources can be achieved through cultural adaptations. Furthermore, those adaptations should be associated with sustainability, but research has yet to test that hypothesis adequately. Several recommendations are made for advancing culturally adapted interventions through additional research on local adaptations during implementation, the many facets of participant engagement, and sustainability.

Implementation of a consumer-focused eHealth intervention for people with moderate-to-high cardiovascular disease risk: protocol for a mixed-methods process evaluation

INTRODUCTION

Technology-mediated strategies have potential to engage patients in modifying unhealthy behaviour and improving medication adherence to reduce morbidity and mortality from cardiovascular disease (CVD). Furthermore, electronic tools offer a medium by which consumers can more actively navigate personal healthcare information. Understanding how, why and among whom such strategies have an effect can help determine the requirements for implementing them at a scale. This paper aims to detail a process evaluation that will (1) assess implementation fidelity of a multicomponent eHealth intervention; (2) determine its effective features; (3) explore contextual factors influencing and maintaining user engagement; and (4) describe barriers, facilitators, preferences and acceptability of such interventions.

METHODS AND ANALYSIS

Mixed-methods sequential design to derive, examine, triangulate and report data from multiple sources. Quantitative data from 3 sources will help to inform both sampling and content framework for the qualitative data collection: (1) surveys of patients and general practitioners (GPs); (2) software analytics; (3) programme delivery records. Qualitative data from interviews with patients and GPs, focus groups with patients and field notes taken by intervention delivery staff will be thematically analysed. Concurrent interview data collection and analysis will enable a thematic framework to evolve inductively and inform theory building, consistent with a realistic evaluation perspective. Eligible patients are those at moderate-to-high CVD risk who were randomised to the intervention arm of a randomised controlled trial of an eHealth intervention and are contactable at completion of the follow-up period; eligible GPs are the primary healthcare providers of these patients.

ETHICS AND DISSEMINATION

Ethics approval has been received from the University of Sydney Human Research Ethics Committee and the Aboriginal Health and Medical Research Council (AH&MRC) of New South Wales. Results will be disseminated via scientific forums including peer-reviewed publications and national and international conferences.

TRIAL REGISTRATION NUMBER

ANZCTR 12613000715774.

Process Evaluation of Making HEPA Policy Practice: A Group Randomized Trial

This study examines the link between implementation of Strategies to Enhance Practice (STEPs) and outcomes. Twenty after-school programs (ASPs) participated in an intervention to increase children's accumulation of 30 minutes/day of moderate to vigorous physical activity (MVPA) and quality of snacks served during program time. Outcomes were measured via accelerometer (MVPA) and direct observation (snacks). STEPs implementation data were collected via document review and direct observation. Based on implementation data, ASPs were divided into high/low implementers. Differences between high/low implementers' change in percentage of boys accumulating 30 minutes/day of MVPA were observed. There was no difference between high/low implementers for girls. Days fruits and/or vegetables and water were served increased in the high/low implementation groups, while desserts and sugar-sweetened beverages decreased. Effect sizes (ES) for the difference in changes between the high and low group ranged from low (ES = 0.16) to high (ES = 0.97). Higher levels of implementation led to increased MVPA for boys, whereas girls MVPA benefited from the intervention regardless of high/low implementation. ESs of the difference between high/low implementers indicate that increased implementation of STEPs increases days healthier snacks are served. Programs in the high-implementation group implemented a variety of STEPs strategies, suggesting local adoption/adaptation is key to implementation.

Evaluating the Relationships Among Teacher Characteristics, Implementation Factors, and Student Outcomes of Children Participating in an Experiential School-Based Nutrition Program

The purpose of this study was to examine whether teacher characteristics and implementation factors influenced the effectiveness of a multicomponent nutrition education program to improve nutrition-related knowledge and behavior among fourth-grade children. Schools implemented the Shaping Healthy Choices Program (SHCP), a multicomponent nutrition intervention, during 2013-2014. Fourth-grade teachers ( n = 8) at two schools facilitated nutrition education using an experiential, garden-enhanced nutrition curriculum among students ( n = 173) in their classrooms. Teacher characteristics of teaching experience, self-efficacy in teaching nutrition, and knowledge about nutrition were collected at baseline. Implementation factors of fidelity and completion were measured using self-reports and qualitative observations. Student outcomes included body mass index percentile-for-age, nutrition knowledge, critical thinking skills, and identification of vegetables and were measured pre/post. Relationships were analyzed using multivariate mixed-model regression with classroom as a random effect. Pre/post changes were analyzed using t tests. Students improved body mass index percentile-for-age (−1.99 ± 7.95) and increased nutrition knowledge (2.28 ± 3.40), critical thinking skills (1.98 ± 3.28), and the ability to identify vegetables (0.87 ± 1.09). Teachers’ characteristics of teaching experience, experience teaching nutrition, attendance in a college nutrition course, self-efficacy, and nutrition knowledge were significantly related to fidelity and completeness of program implementation. This study demonstrated implementation of SHCP by teachers can positively influence student outcomes, though program implementation depends on teacher characteristics.

Deaf Adolescents' Learning of Cardiovascular Health Information: Sources and Access Challenges

Deaf individuals have more cardiovascular risks than the general population that are believed to be related to their cardiovascular health knowledge disparities. This phenomenological study describes where 20 deaf sign language-using adolescents from Rochester, New York, many who possess many positive characteristics to support their health literacy, learn cardiovascular health information and their lived experiences accessing health information. The goal is to ultimately use this information to improve the delivery of cardiovascular health education to this population and other deaf adolescents at a higher risk for weak health literacy. Deaf bilingual researchers interviewed deaf adolescents, transcribed and coded the data, and described the findings. Five major sources of cardiovascular health information were identified including family, health education teachers, healthcare providers, printed materials, and informal sources. Despite possessing advantageous characteristics contributing to stronger health literacy, study participants described significant challenges with accessing health information from each source. They also demonstrated inconsistencies in their cardiovascular health knowledge, especially regarding heart attack, stroke, and cholesterol. These findings suggest a great need for additional public funding to research deaf adolescents' informal health-related learning, develop accessible and culturally appropriate health surveys and health education programming, improve interpreter education, and disseminate information through social media.

Linking implementation process to intervention outcomes in a middle school obesity prevention curriculum, 'Choice, Control and Change'

This study investigates the link between process evaluation components and the outcomes of a school-based nutrition curriculum intervention, 'Choice, Control and Change'. Ten New York City public middle schools were recruited and randomly assigned into intervention or control condition. The curriculum was to improve sixth to seventh grade students' energy balance related behaviors, based on social cognitive and self-determination theories, and implemented during the 2006-2007 school year (n = 1136). Behaviors and psychosocial variables were measured by self-reported questionnaires. Process components were evaluated with classroom observations, teacher interviews, and a student questionnaire. Using 'Teacher Implementation' (dose delivered) and 'Student Reception' (dose received) process data; intervention group was further categorized into medium- and high-implementation groups. Analysis of covariance revealed that, compared with control group, only high-implementation group showed significant improvement in students' behavior and psychosocial outcomes. Hierarchical linear models showed that 'Teacher Implementation' and 'Student Reception' significantly predicted students' sweetened beverage outcomes (P < 0.05). 'Student Satisfaction' was also greater when these implementation components were higher, and significantly associated with behavior and psychosocial outcomes (P < 0.05). Implementation process influenced the effectiveness of the 'Choice, Control and Change' intervention study. It is important to take into account the process components when interpreting the results of such research.

Putting program evaluation into practice: enhancing the Girls Just Wanna Have Fun program

In recent years there has been a call for increased community physical activity and sport programs for female youth that are deliberately structured to foster positive developmental outcomes. In addition, researchers have recognized the need to empirically evaluate such programs to ensure that youth are provided with optimal opportunities to thrive. This study represents a utilization-focused evaluation of Girls Just Wanna Have Fun, a female-only physical activity-based life skills community program. A utilization-focused evaluation is particularly important when the evaluation is to help stakeholders utilize the findings in practice. The purpose of this study was twofold: (a) to gain an understanding of the ongoing successes and challenges after year two of program implementation and (b) to examine how the adaptations made based on feedback from the first year evaluation were perceived as impacting the program. From interviews with youth participants and program leaders, three main themes with eight sub-themes emerged. The main themes were: (a) applying lessons learned can make a significant difference, (b) continually implementing successful strategies, and (c) ongoing challenges. Overall, this evaluation represents an important step in understanding how to improve program delivery to better meet the needs of the participants in community-based programming.

The effects of teacher fidelity of implementation of pathways to health on student outcomes

Previous research has demonstrated the importance of ensuring that programs are implemented as intended by program developers in order to achieve desired program effects. The current study examined implementation fidelity of Pathways to Health (Pathways), a newly developed obesity prevention program for fourth- through sixth-grade children. We explored the associations between self-reported and observed implementation fidelity scores and whether implementation fidelity differed across the first 2 years of program implementation. Additionally, we examined whether implementation fidelity affected program outcomes and whether teacher beliefs were associated with implementation fidelity. The program was better received, and implementation fidelity had more effects on program outcomes in fifth grade than in fourth grade. Findings suggest that implementation in school-based obesity programs may affect junk food intake and intentions to eat healthfully and exercise. School support was associated with implementation fidelity, suggesting that prevention programs may benefit from including a component that boosts school-wide support.

Three Cs of Translating Evidence-Based Programs for Youth and Families to Practice Settings

Despite the growing number of evidence-based programs (EBPs) for youth and families, few are well-integrated in service systems or widely adopted by communities. One set of challenges to widespread adoption of EBPs relates to the transfer of programs from research and development to practice settings. This is often because program developers have limited guidance on how to prepare their programs for broad dissemination in practice settings. We describe Three Cs of Translation, which are key areas that are essential for developers to translate their EBPs from research to practice settings: (1) Communicate the underlying theory in terms easily understandable to end users, (2) Clarify fidelity and flexibility, and (3) Codify implementation lessons and examples. Program developers are in the best position to describe their interventions, to define intervention core components, to clarify fidelity and flexibility, and to codify implementation lessons from intervention studies. We note several advantages for developers to apply the Three Cs prior to intervention dissemination and provide specific recommendations for translation.

Effects of a structural intervention and implementation on physical activity among youth in residential children's homes

This study reports the effects of a structural intervention, ENRICH (Environmental Interventions in Children's Homes) which targeted the physical and social environment within residential children's homes (RCHs) to increase physical activity (PA) among residents (n=799). Participating RCHs (n=29) were randomized to Early (n=17) or Delayed (n=12) groups from 2004 to 2006 and 2006 to 2008, respectively. Children's PA was measured at three time periods (2004, 2006, 2008). Intent-to-treat analysis revealed no intervention impact on PA. Subsequent analyses used process evaluation data to group organizations into high and low PA-promoting RCHs to compare PA level, controlling for assignment to condition. Organizations with high PA-promoting environments were found to have more active youth. Utility of a comprehensive implementation monitoring plan and the need for formative assessment of organizational capacity is discussed.

Community, intervention and provider support influences on implementation: reflections from a South African illustration of safety, peace and health promotion

Background

The development, implementation and evaluation of community interventions are important for reducing child violence and injuries in low- to middle-income contexts, with successful implementation critical to effective intervention outcomes. The assessment of implementation processes is required to identify the factors that influence effective implementation. This article draws on a child safety, peace and health initiative to examine key factors that enabled or hindered its implementation, in a context characterised by limited resources.

Methods

A case study approach was employed. The research team was made up of six researchers and intervention coordinators, who led the development and implementation of the Ukuphepha Child Study in South Africa, and who are also the authors of this article. The study used author observations, reflections and discussions of the factors perceived to influence the implementation of the intervention. The authors engaged in an in-depth and iterative dialogic process aimed at abstracting the experiences of the intervention, with a recursive cycle of reflection and dialogue. Data were analysed utilising inductive content analysis, and categorised using classification frameworks for understanding implementation.

Results

The study highlights key factors that enabled or hindered implementation. These included the community context and concomitant community engagement processes; intervention compatibility and adaptability issues; community service provider perceptions of intervention relevance and expectations; and the intervention support system, characterised by training and mentorship support.

Conclusions

This evaluation illustrated the complexity of intervention implementation. The study approach sought to support intervention fidelity by fostering and maintaining community endorsement and support, a prerequisite for the unfolding implementation of the intervention.

The tug-of-war: fidelity versus adaptation throughout the health promotion program life cycle

Researchers across multiple fields have described the iterative and nonlinear phases of the translational research process from program development to dissemination. This process can be conceptualized within a "program life cycle" framework that includes overlapping and nonlinear phases: development, adoption, implementation, maintenance, sustainability or termination, and dissemination or diffusion, characterized by tensions between fidelity to the original plan and adaptation for the setting and population. In this article, we describe the life cycle (phases) for research-based health promotion programs, the key influences at each phase, and the issues related to the tug-of-war between fidelity and adaptation throughout the process using a fictionalized case study based on our previous research. This article suggests the importance of reconceptualizing intervention design, involving stakeholders, and monitoring fidelity and adaptation throughout all phases to maintain implementation fidelity and completeness. Intervention fidelity should be based on causal mechanisms to ensure effectiveness, while allowing for appropriate adaption to ensure maximum implementation and sustainability. Recommendations for future interventions include considering the determinants of implementation including contextual factors at each phase, the roles of stakeholders, and the importance of developing a rigorous, adaptive, and flexible definition of implementation fidelity and completeness.

Worksite wellness program implementation: a model of translational effectiveness

Occupational health promotion programs with documented efficacy have not penetrated worksites. Establishing an implementation model would allow focusing on mediating aspects to enhance installation and use of evidence-based occupational wellness interventions. The purpose of the study was to implement an established wellness program in fire departments and define predictors of program exposure/dose to outcomes to define a cross-sectional model of translational effectiveness. The study is a prospective observational study among 12 NW fire departments. Data were collected before and following installation, and findings were used to conduct mediation analysis and develop a translational effectiveness model. Worker age was examined for its impact. Leadership, scheduling/competing demands, and tailoring were confirmed as model components, while organizational climate was not a factor. The established model fit data well (χ (2)(9) = 25.57, CFI = 0.99, RMSEA = 0.05, SRMR = 0.03). Older firefighters, nearing retirement, appeared to have influences that both enhanced and hindered participation. Findings can inform implementation of worksite wellness in fire departments, and the prioritized influences and translational model can be validated and manipulated in these and other settings to more efficiently move health promotion science to service.

Measuring Fidelity and Adaptation: Reliability of a Instrument for School-Based Prevention Programs

There is a need to standardize methods for assessing fidelity and adaptation. Such standardization would allow program implementation to be examined in a manner that will be useful for understanding the moderating role of fidelity in dissemination research. This article describes a method for collecting data about fidelity of implementation for school-based prevention programs, including measures of adherence, quality of delivery, dosage, participant engagement, and adaptation. We report about the reliability of these methods when applied by four observers who coded video recordings of teachers delivering All Stars, a middle school drug prevention program. Interrater agreement for scaled items was assessed for an instrument designed to evaluate program fidelity. Results indicated sound interrater reliability for items assessing adherence, dosage, quality of teaching, teacher understanding of concepts, and program adaptations. The interrater reliability for items assessing potential program effectiveness, classroom management, achievement of activity objectives, and adaptation valences was improved by dichotomizing the response options for these items. The item that assessed student engagement demonstrated only modest interrater reliability and was not improved through dichotomization. Several coder pairs were discordant on items that overall demonstrated good interrater reliability. Proposed modifications to the coding manual and protocol are discussed.

Understanding the art and science of implementation in the SAAF efficacy trial

PURPOSE

The purpose of this paper is to demonstrate the importance of considering both fidelity and adaptation in assessing the implementation of evidence-based programs.

DESIGN/METHODOLOGY/APPROACH

The current study employs a multi-method strategy to understand two dimensions of implementation (fidelity and adaptation) in the Strong African American Families (SAAF) program. Data were video recordings of program delivery and pre-test and post-test interviews from the efficacy trial. Multilevel regression in Mplus was used to assess the impact of fidelity to the manual, coded by independent observers, on racial socialization outcomes. One activity on racial socialization, a core component of the program, was selected for an in-depth examination using conversation analysis (a qualitative method of analyzing talk in interactions).

FINDINGS

Results of the quantitative analyses demonstrated that fidelity of the selected activity was associated with increases in parent's use of racial socialization from pre-test to post-test, but only when participant attendance was included in the model. Results of the qualitative analyses demonstrated that facilitators were making adaptations to the session and that these adaptations appeared to be in line with cultural competence.

RESEARCH LIMITATIONS/IMPLICATIONS

The development of quantitative fidelity measures can be problematic, with many decision points to consider. The current study contributes to the evidence base to develop a quantitative measure of adaptation for family-based parenting programs.

ORIGINALITY/VALUE

Many researchers examining implementation of evidence-based programs consider fidelity and adaptation to be polar ends of a single spectrum. This paper provides evidence for the importance of examining each independently.

The quality implementation framework: a synthesis of critical steps in the implementation process

Implementation science is growing in importance among funders, researchers, and practitioners as an approach to bridging the gap between science and practice. We addressed three goals to contribute to the understanding of the complex and dynamic nature of implementation. Our first goal was to provide a conceptual overview of the process of implementation by synthesizing information from 25 implementation frameworks. The synthesis extends prior work by focusing on specific actions (i.e., the "how to") that can be employed to foster high quality implementation. The synthesis identified 14 critical steps that were used to construct the Quality Implementation Framework (QIF). These steps comprise four QIF phases: Initial Considerations Regarding the Host Setting, Creating a Structure for Implementation, Ongoing Structure Once Implementation Begins, and Improving Future Applications. Our second goal was to summarize research support for each of the 14 QIF steps and to offer suggestions to direct future research efforts. Our third goal was to outline practical implications of our findings for improving future implementation efforts in the world of practice. The QIF's critical steps can serve as a useful blueprint for future research and practice. Applying the collective guidance synthesized by the QIF to the Interactive Systems Framework for Dissemination and Implementation (ISF) emphasizes that accountability for quality implementation does not rest with the practitioner Delivery System alone. Instead, all three ISF systems are mutually accountable for quality implementation.

Subjective outcome evaluation of the project P.A.T.H.S. In different cohorts of students

The project P.A.T.H.S. is an indigenously developed positive youth development program in Hong Kong. In the extension phase (2009/2010 school year), subjective outcome evaluation data were collected from 231 schools involving 89,068 participants after completion of the curricula-based Tier 1 Program. With schools as the units of analysis, results showed that participants generally had positive perceptions of the program content and implementers, with over four-fifth of the participants regarded the program as helpful to them. There were some significant grade differences in the subjective outcome evaluation findings, although the related effect size was not strong. Multiple regression analyses revealed that program content and program implementers predicted perceived effectiveness of the program. The present study suggests that irrespective of cohorts, students in the junior secondary years perceived the program to be beneficial to them.

Putting the pieces together: an integrated model of program implementation

Considerable evidence indicates that variability in implementation of prevention programs is related to the outcomes achieved by these programs. However, while implementation has been conceptualized as a multidimensional construct, few studies examine more than a single dimension, and no theoretical framework exists to guide research on the effects of implementation. We seek to address this need by proposing a theoretical model of the relations between the dimensions of implementation and outcomes of prevention programs that can serve to guide future implementation research. In this article, we focus on four dimensions of implementation, which we conceptualize as behaviors of program facilitators (fidelity, quality of delivery, and adaptation) and behaviors of participants (responsiveness) and present the evidence supporting these as predictors of program outcomes. We then propose a theoretical model by which facilitator and participant dimensions of implementation influence participant outcomes. Finally, we provide recommendations and directions for future implementation research.

Putting the pieces together: an integrated model of program implementation

Considerable evidence indicates that variability in implementation of prevention programs is related to the outcomes achieved by these programs. However, while implementation has been conceptualized as a multidimensional construct, few studies examine more than a single dimension, and no theoretical framework exists to guide research on the effects of implementation. We seek to address this need by proposing a theoretical model of the relations between the dimensions of implementation and outcomes of prevention programs that can serve to guide future implementation research. In this article, we focus on four dimensions of implementation, which we conceptualize as behaviors of program facilitators (fidelity, quality of delivery, and adaptation) and behaviors of participants (responsiveness) and present the evidence supporting these as predictors of program outcomes. We then propose a theoretical model by which facilitator and participant dimensions of implementation influence participant outcomes. Finally, we provide recommendations and directions for future implementation research.

Rationale, design and methods for process evaluation in the HEALTHY study

The HEALTHY study was a multi-site randomized trial designed to determine whether a 3-year school-based intervention targeting nutrition and physical activity behaviors could effectively reduce risk factors associated with type 2 diabetes in middle school children. Pilot and formative studies were conducted to inform the development of the intervention components and the process evaluation methods for the main trial. During the main trial, both qualitative and quantitative assessments monitored the fidelity of the intervention and motivated modifications to improve intervention delivery. Structured observations of physical education classes, total school food environments, classroom-based educational modules, and communications and promotional campaigns provided verification that the intervention was delivered as intended. Interviews and focus groups yielded a multidimensional assessment of how the intervention was delivered and received, as well as identifying the barriers to and facilitators of the intervention across and within participating schools. Interim summaries of process evaluation data were presented to the study group as a means of ensuring standardization and quality of the intervention across the seven participating centers. Process evaluation methods and procedures documented the fidelity with which the HEALTHY study was implemented across 21 intervention schools and identified ways in which the intervention delivery might be enhanced throughout the study.

An impact evaluation of a federal mine safety training regulation on injury rates among US stone, sand, and gravel mine workers: an interrupted time-series analysis

OBJECTIVES

We evaluated the impact of a safety training regulation, implemented by the US Department of Labor's Mine Safety and Health Administration (MSHA) in 1999, on injury rates at stone, sand, and gravel mining operations.

METHODS

We applied a time-series design and analyses with quarterly counts of nonfatal injuries and employment hours from 7998 surface aggregate mines from 1995 through 2006. Covariates included standard industrial classification codes, ownership, and injury severity.

RESULTS

Overall crude rates of injuries declined over the 12-year period. Reductions in incident rates for medical treatment only, restricted duty, and lost-time injuries were consistent with temporal trends and provided no evidence of an intervention effect attributable to the MSHA regulation. Rates of permanently disabling injuries (PDIs) declined markedly. Regression analyses documented a statistically significant reduction in the risk rate in the postintervention time period (risk rate = 0.591; 95% confidence interval = 0.529, 0.661).

CONCLUSIONS

Although a causal relationship between the regulatory intervention and the decline in the rate of PDIs is plausible, inconsistency in the results with the other injury-severity categories preclude attributing the observed outcome to the MSHA regulation. Further analyses of these data are needed.

Using process evaluation for program improvement in dose, fidelity and reach: the ACT trial experience

Background

The purpose of this study was to demonstrate how formative program process evaluation was used to improve dose and fidelity of implementation, as well as reach of the intervention into the target population, in the "Active by Choice Today" (ACT) randomized school-based trial from years 1 to 3 of implementation.

Methods

The intervention integrated constructs from Self-Determination Theory and Social Cognitive Theory to enhance intrinsic motivation and behavioral skills for increasing long-term physical activity (PA) behavior in underserved adolescents (low income, minorities). ACT formative process data were examined at the end of each year to provide timely, corrective feedback to keep the intervention "on track".

Results

Between years 1 and 2 and years 2 and 3, three significant changes were made to attempt to increase dose and fidelity rates in the program delivery and participant attendance (reach). These changes included expanding the staff training, reformatting the intervention manual, and developing a tracking system for contacting parents of students who were not attending the after-school programs regularly. Process outcomes suggest that these efforts resulted in notable improvements in attendance, dose, and fidelity of intervention implementation from years 1 to 2 and 2 to 3 of the ACT trial.

Conclusion

Process evaluation methods, particularly implementation monitoring, are useful tools to ensure fidelity in intervention trials and for identifying key best practices for intervention delivery.

Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation

The first purpose of this review was to assess the impact of implementation on program outcomes, and the second purpose was to identify factors affecting the implementation process. Results from over quantitative 500 studies offered strong empirical support to the conclusion that the level of implementation affects the outcomes obtained in promotion and prevention programs. Findings from 81 additional reports indicate there are at least 23 contextual factors that influence implementation. The implementation process is affected by variables related to communities, providers and innovations, and aspects of the prevention delivery system (i.e., organizational functioning) and the prevention support system (i.e., training and technical assistance). The collection of implementation data is an essential feature of program evaluations, and more information is needed on which and how various factors influence implementation in different community settings.

An overview of "The Active by Choice Today" (ACT) trial for increasing physical activity

BACKGROUND

Although school-based behavioral interventions for increasing physical activity (PA) in children and adolescents have been conducted, little evidence suggests that these curriculum-based approaches lead to increases in overall activity outside of program days. The overall goal of the "Active by Choice Today" (ACT) trial is to expand the body of knowledge concerning the factors that influence long-term increases in PA in underserved adolescents (low socioeconomic status, minorities) during their middle school years.

DESIGN AND SETTING

An overview of the ACT study design, theoretical framework, process evaluation, and primary hypotheses is presented. The trial involves twenty-four middle schools (1560 6th graders) in South Carolina that are randomly assigned to one of two after-school programs (motivational and life skills intervention, or general health education).

INTERVENTION

The intervention integrates constructs from Self-Determination and Social Cognitive Theories to enhance intrinsic motivation and behavioral skills for PA. The intervention targets skill development for PA outside of program days and the after-school program social environment (autonomy, choice, participation, belongingness, fun, enjoyment, support) is designed to positively impact cognitive mediators (self-efficacy, perceived competence), and motivational orientation (intrinsic motivation, commitment, positive self-concept). MAIN HYPOTHESES/OUTCOMES: It is hypothesized that the 17-week motivational and life skills intervention will lead to greater increases in moderate-to-vigorous PA (based on 7-day accelerometry estimates) at post-intervention as compared to the general health education program.

CONCLUSIONS

Implications of this innovative school-based trial are discussed.

Examining the link between program implementation and behavior outcomes in the lifestyle education for activity program (LEAP)

Lifestyle Education for Activity Program (LEAP) was a comprehensive, school-based intervention designed to promote physical activity in high school girls. The intervention focused on changes in instructional practices and the school environment to affect personal, social, and environmental factors related to physical activity. Multiple process evaluation tools and an organizational assessment tool were developed to monitor program implementation from a framework called the LEAP essential elements, which characterized complete and acceptable intervention delivery; secular trends were also monitored. Using process data, LEAP intervention schools were categorized into low- and high-implementing groups and compared with control schools on nine essential elements assessed at the organizational level. The Wilcoxon scores test revealed that low- and high-implementing intervention, and control schools differed significantly on two of nine administrator-reported organizational-level components: having a physical activity team and having a faculty-staff health promotion program. A mixed-model analysis of covariance indicated that, compared to control schools, a greater percentage of girls in high-implementing schools reported engaging in vigorous physical activity. Process evaluation can be used to understand the relationship between level of implementation and successful program outcome.

Coaching Process Outcomes of a Family Visit Nutrition and Physical Activity Intervention

The purpose of this article is to report the process outcomes of a coaching methodology used in a study designed to increase fruit and vegetable consumption and physical activity in families. Eighty-eight families with second graders were recruited from a rural, biethnic community in Colorado and randomized to intervention and delayed intervention conditions. This article reports on the 27 families in the delayed intervention group. Families received up to 10 home visits over 10 months from a family advisor and completed activities to improve their dietary and physical activity behaviors. Coaching conversations took place during each home visit. Coaching process outcomes were evaluated by analysis of visit documentation, participant survey, and qualitative interviews. Results indicated that coaching, in conjunction with family activities, engaged families in the process of change and facilitated movement toward the achievement of their weekly nutrition or physical activity goals. Coaching methodology may be particularly useful for participatory research.

Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide

Process evaluation is used to monitor and document program implementation and can aid in understanding the relationship between specific program elements and program outcomes. The scope and implementation of process evaluation has grown in complexity as its importance and utility have become more widely recognized. Several practical frameworks and models are available to practitioners to guide the development of a comprehensive evaluation plan, including process evaluation for collaborative community initiatives. However, frameworks for developing a comprehensive process-evaluation plan for targeted programs are less common. Building from previous frameworks, the authors present a comprehensive and systematic approach for developing a process-evaluation plan to assess the implementation of a targeted health promotion intervention. Suggested elements for process-evaluation plans include fidelity, dose (delivered and received), reach, recruitment, and context. The purpose of this article is to describe and illustrate the steps involved in developing a process evaluation plan for any health promotion program.

Design, implementation, and quality control in the Pathways American-Indian multicenter trial

BACKGROUND

Pathways was the first multicenter American-Indian school-based study to test the effectiveness of an obesity prevention program promoting healthy eating and physical activity.

METHODS

Pathways employed a nested cohort design in which 41 schools were randomized to intervention or control conditions and students within these schools were followed as a cohort (1,704 third graders at baseline). The study's primary endpoint was percent body fat. Secondary endpoints were levels of fat in school lunches; time spent in physical activity; and knowledge, attitudes, and behaviors regarding diet and exercise. Quality control (QC) included design of data management systems which provided standardization and quality assurance of data collection and processing. Data QC procedures at study centers included manuals of operation, training and certification, and monitoring of performance. Process evaluation was conducted to monitor dose and fidelity of the interventions. Registration and tracking systems were used for students and schools.

RESULTS

No difference in mean percent body fat at fifth grade was found between the intervention and control schools. Percent of calories from fat and saturated fat in school lunches was significantly reduced in the intervention schools as was total energy intake from 24-hour recalls. Significant increases in self-reported physical activity levels and knowledge of healthy behaviors were found for the intervention school students.

CONCLUSIONS

The Pathways study results provide evidence demonstrating the role schools can play in public health promotion. Its study design and QC systems and procedures provide useful models for other similar school based multi- or single-site studies.

Drug abuse treatment training in Peru. A social policy experiment

A social policy experiment is presented that was conducted from 1997 to 2000 in a setting with a high level of readiness for implementing a randomized experiment of therapeutic community (TC) drug treatment training in Peru. Seventy-six drug abuse treatment organizations were randomly assigned into three groups, and data were collected at multiple assessment periods. Staff and directors in organizations assigned to the training groups participated in either 6-week basic training or 8-week basic plus booster training sessions, which were theoretically grounded. Small- to medium-size positive effects were found on increased staff empowerment to use actual tools and principles from the training; medium and large positive effects were found on the implementation of TC methods with fidelity after the training. A follow-up with the funding and training organizations 1 year later showed use of the evaluation results in decision making in both organizations.

Evaluation of the institutionalization of the coordinated approach to child health (CATCH) in a U.S./Mexico border community

El Paso Coordinated Approach to Child Health (El Paso CATCH) was evaluated in 24 schools for outcome measures of moderate to vigorous physical activity (MVPA) during physical education (PE), content of PE lessons, content of school meals, and numerous process measures. Chi-square analyses compared frequency data across time for activity during PE and percentage fat in school meals. Descriptive summaries were used for process questionnaire results. Data were also compared to CATCH program goals. For most intervention schools, El Paso CATCH significantly increased MVPA, decreased fat in school meals, and decreased sodium in school breakfasts. However, some schools were not meeting the fat content goals for school lunches, and no schools met the vigorous physical activity (VPA) goals for PE or the sodium goals for school lunches.

Evaluation methods for social intervention

Experimental design is the method of choice for establishing whether social interventions have the intended effects on the populations they are presumed to benefit. Experience with field experiments, however, has revealed significant limitations relating chiefly to (a) practical problems implementing random assignment, (b) important uncontrolled sources of variability occurring after assignment, and (c) a low yield of information for explaining why certain effects were or were not found. In response, it is increasingly common for outcome evaluation to draw on some form of program theory and extend data collection to include descriptive information about program implementation, client characteristics, and patterns of change. These supplements often cannot be readily incorporated into standard experimental design, especially statistical analysis. An important advance in outcome evaluation is the recent development of statistical models that are able to represent individual-level change, correlates of change, and program effects in an integrated and informative manner.