How best to measure implementation of school health curricula: a comparison of three measures

Acceptability, feasibility and fidelity of the culturally adapted version of Unplugged ("Yo Se Lo Que Quiero"), a substance use preventive program among adolescents in Chile: a pilot randomized controlled study

INTRODUCTION

The consumption of alcohol, tobacco, and cannabis is a public health problem that impacts the cognitive, social, and emotional development of adolescents. Prevention strategies such as the "Unplugged" program are effective in delaying the progression of daily smoking and episodes of drunkenness among adolescents. "Yo Se Lo Que Quiero" (YSLQQ) corresponds to the adaptation of this program to the Chilean context. This study assesses the acceptability and feasibility of implementing this program to the local reality.

MATERIAL AND METHODS

This was a cluster-randomized controlled pilot study conducted on six public schools. All consented students attending 6th, 7th, and 8th grades (n = 1,180) participated in the study. The schools were randomly assigned to one of two conditions in a 1:1 ratio: (1) the "YSLQQ" intervention group (n = 526), and (2) the Control group (n = 654). The program consisted of a 12-hour class-based curriculum based on a comprehensive social-influence approach delivered by a trained facilitator. The acceptability and feasibility were assessed in the intervention group at the end of the intervention using questionnaires answered by students and facilitators. The quality and fidelity of the program were evaluated during the implementation using self-ported surveys answered by the facilitators and the assessment of video-recorded sessions rated by external observers. Finally, a pre-test and a post-test survey assessing past and current substance use and risk and protective factors were conducted before and immediately after the program's implementation.

RESULTS

A high proportion of students (49.6%) liked the sessions. 79.2% reported that the YSLQQ helped them learn about the dangers of substances, while 65.8% reported having more skills to avoid substance use in the future. Regarding students' satisfaction with YSLQQ, 62.9% reported being happy or very happy with the program. Facilitators reported implementing the intervention according to the manual in 73.9% of sessions. Regarding substance use, students who participated in the intervention groups reported a significant reduction in drunkenness in the last year and last 30-day prevalence and also a significant reduction in a lifetime and 30-day prevalence of cannabis use when compared with those students in the control group.

CONCLUSIONS

Our results suggest that YSLQQ has adequate acceptability and feasibility to be implemented in the Chilean context, and there were promising results in reducing drunkenness and cannabis use. Future research should confirm these results in a larger RCT study.

TRIAL REGISTRATION

The trial was registered in ClinicalTrials.gov, NCT04566627; registration date: 01/03/2019.

Fidelity of implementation of Reptes, an alcohol and cannabis selective prevention program for youngsters

OBJECTIVE

Implementation fidelity is a key dimension in process evaluation but has been scarcely studied. Reptes is a selective alcohol and cannabis use prevention program performed in vulnerable young people (16-21 years) from different educational and leisure settings. The study aimed to describe the components of this program, its implementation fidelity, and satisfaction among participants and facilitators.

METHOD

A descriptive cross-sectional study was conducted. Implementation fidelity was assessed in terms of adherence, dose, participant responsiveness, and quality of the intervention through self-reports completed by 15 facilitators and self-administered questionnaires by 99 youth from the 15 groups completing the program.

RESULTS

70.7% of the groups performed 30 of the 43 planned activities with high use of various teaching methods, except for Information and Communication Technology resources. Fidelity implementation was above 70%. 2/3 groups completed an acceptable intervention and 1/3 completed a qualified intervention. Satisfaction was higher than 7.

CONCLUSIONS

Our results show high implementation fidelity and satisfaction compared with those from similar studies.

Monitoring of the Village Malaria Workers to conduct activities of Malaria Elimination Demonstration Project in Mandla, Madhya Pradesh

Background

The capacity of the field staff to conduct activities related to disease surveillance, case management, and vector control has been one of the key components for successfully achieving malaria elimination. India has committed to eliminate malaria by 2030, and it has placed significance on monitoring and evaluation at the district level as one of the key strategies in its national framework. To support and guide the country’s malaria elimination objectives, the Malaria Elimination Demonstration Project was conducted in the tribal district of Mandla, Madhya Pradesh. Robust monitoring of human resources received special attention to help the national programme formulate a strategy to plug the gaps in its supply chain and monitoring and evaluation systems.

Methods

A monitoring tool was developed to test the capabilities of field workers to conduct activities related to malaria elimination work. Between November 2018 to February 2021, twenty-five Malaria Field Coordinators (MFCs) of the project utilized this tool everyday during the supervisory visits for their respective Village Malaria Workers (VMWs). The data was analysed and the scores were tested for variations against different blocks, educational status, duration of monitoring, and post-training scores.

Results

During the study period, the VMWs were monitored a total of 8974 times using the monitoring tool. Each VMW was supervised an average of 1.8 times each month. The critical monitoring indicators scored well in all seven quarters of the study as monitored by the MFCs. Monitoring by MFCs remained stable at 97.3% in all quarters. Contrary to expectations, the study observed longer diagnosis to treatment initiation time in urban areas of the district.

Conclusion

This study demonstrated the significance of a robust monitoring tool as an instrument to determine the capacity of the field workers in conducting surveillance, case management, and vector control related work for the malaria elimination programme. Similar tools can be replicated not only for malaria elimination, but other public health interventions as well.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12936-021-04040-2.

Classifying Changes to Preventive Interventions: Applying Adaptation Taxonomies

High-quality implementation is important for preventive intervention effectiveness. Although this implies fidelity to a practice model, some adaptation may be inevitable or even advantageous in routine practice settings. In order to organize the study of adaptation and its effect on intervention outcomes, scholars have proposed various adaptation taxonomies. This paper examines how four published taxonomies retrospectively classify adaptations: the Ecological Validity Framework (EVF; Bernal et al. in J Abnorm Child Psychol 23(1):67-82, 1995), the Hybrid Prevention Program Model (HPPM; Castro et al. in Prev Sci 5(1):41-45, 2004. https://doi.org/10.1023/B:PREV.0000013980.12412.cd ), the Moore et al. (J Prim Prev 34(3):147-161, 2013. https://doi.org/10.1007/s10935-013-0303-6 ) taxonomy, and the Stirman et al. (Implement Sci 8:65, 2013. https://doi.org/10.1186/1748-5908-8-65 ) taxonomy. We used these taxonomies to classify teacher-reported adaptations made during the implementation of TOOLBOX™, a social emotional learning program implemented in 11 elementary schools during the 2014-2015 academic year. Post-implementation, 271 teachers and staff responded to an online survey that included questions about adaptation, yielding 98 adaptation descriptions provided by 42 respondents. Four raters used each taxonomy to try to classify these descriptions. We assessed the extent to which raters agreed they could classify the descriptions using each taxonomy (coverage), as well as the extent to which raters agreed on the subcategory they assigned (clarity). Results indicated variance among taxonomies, and tensions between the ideals of coverage and clarity emerged. Further studies of adaptation taxonomies as coding instruments may improve their performance, helping scholars more consistently assess adaptations and their effects on preventive intervention outcomes.

School district-provided supports to enhance sexual health education among middle and high school health education teachers

Schools support teachers in their professional learning, just as teachers support students in their learning. To accomplish this, schools can provide support systems that enhance teachers' knowledge, comfort, and instructional skills. This study examined the impact of two district-provided supports (curriculum and professional development) on sexual health instruction among middle and high school health education teachers. Data were abstracted and analyzed using inductive coding from 24 teacher interviews (2015-2016). Findings illustrate outcomes from both curriculum and PD on teachers' self-reported knowledge, comfort, and skills. The district-provided supports appeared to contribute to improved teachers' self-efficacy in delivering sexual health education.

Implementation fidelity and adolescent smoking: The X:IT study-A school randomized smoking prevention trial

BACKGROUND

Assessing the actual implementation of multi-component interventions can provide important knowledge for future interventions. Intervention components may be implemented differently, knowledge about this can provide an understanding of which components are essential and therefore must be included. The aim of this study was to examine the implementation of one, two, or all three main intervention components at the individual level and to assess the association to current smoking among 13 year-olds in the X:IT study.

METHODS

Data stems from a cluster-randomized controlled trial in 94 Danish elementary schools (51 intervention; 43 control schools). Implementation was measured by aspects of adherence, dose, quality of delivery, and participant responsiveness based on questionnaire data from 4161 pupils at baseline (mean-age: 12.5 years) and 3764 pupils at first follow-up eight months later. Coordinator responses from 49 intervention schools were also included. Associations between individual level implementation of the three main components and pupil smoking were examined through a 3-level logistic regression model.

RESULTS

Although implementation fidelity for the three main intervention components was good, only one third (38.8%) of pupils in intervention schools were exposed to full implementation of the intervention. Among these pupils odds ratio for smoking was 0.25 (95% CI: 0.15 - 0.42).

CONCLUSIONS

School-based programs can be very effective if carefully implemented. Future school-based smoking preventive initiatives should include multiple components, and seek to enhance implementation quality of all components.

Pathways leading to success and non-success: a process evaluation of a cluster randomized physical activity health promotion program applying fuzzy-set qualitative comparative analysis

BACKGROUND

Health promotion programs can only lead to improvements in health outcomes if they are effectively implemented. However, most studies assessing implementation success focus on only one condition, although more conditions influence this process. Therefore, evidence is scarce on what conditions play a role in successful implementation and how they interact. Hence, we aimed to identify which combinations of teacher and implementation process characteristics affected the emotional and social school experience (SCE) of pupils participating in a school-based health promotion program.

METHODS

This study was part of an effectiveness and process evaluation including 24 intervention and 27 control classes. We used fuzzy-set qualitative comparative analysis (fsQCA) to identify combinations of conditions that were associated with either an increase or no increase in the outcome SCE in comparison to the control group at 20 months post intervention. We deductively selected five conditions based on the Consolidated Framework for Implementation Research: teachers' perceived self-efficacy, teachers' expectations of the benefits of the intervention, teachers' previous knowledge about the intervention, dosage of physical activity breaks, and quality of the implementation.

RESULTS

We identified five different pathways that led to no increase in the pupils' outcome (parameters of fit: consistency 94%, coverage 66%). The combination of an unsatisfying quality of implementing the intervention and a low previous knowledge about the intervention showed the highest empirical relevance. Similarly, fewer physical activity breaks in combination with other conditions impeded the program's success. Furthermore, we identified two different pathways characterizing ways to success (consistency: 81%, coverage: 52%). The most relevant combination was good quality implementation of physical activity breaks, implemented by teachers with a high self-efficacy, and a good previous knowledge about the intervention.

CONCLUSIONS

QCA has potential for an in-depth analysis of complex interventions as it can rely on small to medium sample sizes and analyze pathways to success and non-success separately. The investigated program can be improved by considering the following suggestions: The quality of the implementation process should be monitored during the implementation phase, and regular feedback loops and learning opportunities for teachers should accompany a program. Clear recommendations regarding the dosage should be established.

TRIAL REGISTRATION

German register of clinical studies: DRKS00000622 . Retrospectively registered: December 3, 2010, ( http://www.drks.de/drks_web/setLocale_EN.do ). Approved by the Ethics Committee of Lower Austria (GS4-EK-4/107-2010).

Using mixed methods to assess fidelity of delivery and its influencing factors in a complex self-management intervention for people with osteoarthritis and low back pain

OBJECTIVES AND DESIGN

Despite an increasing awareness of the importance of fidelity of delivery within complex behaviour change interventions, it is often poorly assessed. This mixed methods study aimed to establish the fidelity of delivery of a complex self-management intervention and explore the reasons for these findings using a convergent/triangulation design.

SETTING

Feasibility trial of the Self-management of Osteoarthritis and Low back pain through Activity and Skills (SOLAS) intervention (ISRCTN49875385), delivered in primary care physiotherapy.

METHODS AND OUTCOMES

60 SOLAS sessions were delivered across seven sites by nine physiotherapists. Fidelity of delivery of prespecified intervention components was evaluated using (1) audio-recordings (n=60), direct observations (n=24) and self-report checklists (n=60) and (2) individual interviews with physiotherapists (n=9). Quantitatively, fidelity scores were calculated using percentage means and SD of components delivered. Associations between fidelity scores and physiotherapist variables were analysed using Spearman's correlations. Interviews were analysed using thematic analysis to explore potential reasons for fidelity scores. Integration of quantitative and qualitative data occurred at an interpretation level using triangulation.

RESULTS

Quantitatively, fidelity scores were high for all assessment methods; with self-report (92.7%) consistently higher than direct observations (82.7%) or audio-recordings (81.7%). There was significant variation between physiotherapists' individual scores (69.8% - 100%). Both qualitative and quantitative data (from physiotherapist variables) found that physiotherapists' knowledge (Spearman's association at p=0.003) and previous experience (p=0.008) were factors that influenced their fidelity. The qualitative data also postulated participant-level (eg, individual needs) and programme-level factors (eg, resources) as additional elements that influenced fidelity.

CONCLUSION

The intervention was delivered with high fidelity. This study contributes to the limited evidence regarding fidelity assessment methods within complex behaviour change interventions. The findings suggest a combination of quantitative methods is suitable for the assessment of fidelity of delivery. A mixed methods approach provided a more insightful understanding of fidelity and its influencing factors.

TRIAL REGISTRATION NUMBER

ISRCTN49875385; Pre-results.

Multilevel Interventions Targeting Obesity: Research Recommendations for Vulnerable Populations

INTRODUCTION

The origins of obesity are complex and multifaceted. To be successful, an intervention aiming to prevent or treat obesity may need to address multiple layers of biological, social, and environmental influences.

METHODS

NIH recognizes the importance of identifying effective strategies to combat obesity, particularly in high-risk and disadvantaged populations with heightened susceptibility to obesity and subsequent metabolic sequelae. To move this work forward, the National Heart, Lung, and Blood Institute, in collaboration with the NIH Office of Behavioral and Social Science Research and NIH Office of Disease Prevention convened a working group to inform research on multilevel obesity interventions in vulnerable populations. The working group reviewed relevant aspects of intervention planning, recruitment, retention, implementation, evaluation, and analysis, and then made recommendations.

RESULTS

Recruitment and retention techniques used in multilevel research must be culturally appropriate and suited to both individuals and organizations. Adequate time and resources for preliminary work are essential. Collaborative projects can benefit from complementary areas of expertise and shared investigations rigorously pretesting specific aspects of approaches. Study designs need to accommodate the social and environmental levels under study, and include appropriate attention given to statistical power. Projects should monitor implementation in the multiple venues and include a priori estimation of the magnitude of change expected within and across levels.

CONCLUSIONS

The complexity and challenges of delivering interventions at several levels of the social-ecologic model require careful planning and implementation, but hold promise for successful reduction of obesity in vulnerable populations.

Implementation fidelity for promoting the effectiveness of an adolescent sexual health program

The goal of the present study was to examine COMPAS program (Competencies for adolescents with a healthy sexuality) outcomes based on implementation fidelity: dose, adherence, and acceptance. Participants were 716 adolescents aged 14-16 years (46.5% boys). Two fidelity groups were established: high (n=83) and low (n=312), with the remaining sample serving as a non-program control group (n=321). Knowledge about sexually transmitted infections (STIs), attitudes towards HIV, intention to use condoms, and sexual behavior were evaluated. Results indicated that adolescents receiving the intervention displayed improved STI knowledge (p<0.001) and improved attitudes toward HIV (p<0.05) as compared to the control group. Between the two intervention groups, a high-fidelity group intended to engage more in safe sex behaviors (p=0.05) and displayed greater STI knowledge (p=0.05) as compared to the low-fidelity group. The present study revealed improved efficiency when applying prevention programs with implementation fidelity.

Using a domestic and sexual violence prevention advocate to implement a dating violence prevention program with athletes

'Coaching Boys into Men' is an evidence-based dating violence prevention program for coaches to implement with male athletes. A common adaptation of this program is delivery by domestic violence and sexual violence prevention advocates instead of coaches. We explored how this implementer adaptation may influence athlete uptake of program messages and outcomes. Randomly, one school received the program delivered by an advocate while another school received the program delivered by coaches. Athletes completed baseline and follow-up surveys (n = 148), and a subset who received the advocate-led program participated in focus groups (four groups; n = 26). We compared changes in athlete attitudes and behaviors and conducted thematic analyses with qualitative data. We found no significant differences between athletes who received the program from the advocate versus their coaches. Athletes highlighted the advocate's delivery and role as a non-judgmental adult ally as qualities that influenced their uptake of program messages. The acceptability of the advocate-led program may be related to the implementer type along with specific implementer characteristics and delivery methods. Using advocates together with coaches as implementers could increase the reach of this program. Further study of best practices for Coaching Boys into Men adaptation is needed to guide program dissemination and sustainability.

High impact of implementation on school-based smoking prevention: the X:IT study-a cluster-randomized smoking prevention trial

BACKGROUND

Implementation fidelity describes how well an intervention is implemented in the real-world setting. Assessing implementation fidelity is essential in the understanding of intervention results. In most studies, implementation fidelity is measured insufficiently, though, not taking into account the complexity of the concept nor the intervention. The objective of the present study was to develop an overall quantitative measure of implementation fidelity, to examine the degree of implementation fidelity and the association of implementation and effect of a randomized school-based smoking prevention trial-the X:IT study.

METHODS

A cluster-randomized trial testing is a multi-component intervention to prevent smoking among adolescents in 94 Danish elementary schools (51 intervention, 43 control schools). Participants were grade 7 pupils (mean age 12.5 years). Data was collected by electronic questionnaires among pupils at baseline (n = 4161), the first follow-up (n = 3764), and the second follow-up (n = 3269) and among school coordinators at intervention schools at the first and second follow-up (50 and 39 coordinators).

INTERVENTION

The intervention included three components: (1) smoke-free school grounds, (2) smoke-free curriculum, and (3) parental involvement, contracts, and dialogues. Implementation fidelity was assessed by four domains: adherence, dose, quality of delivery, and participant responsiveness. These were combined into an overall school-wise implementation index. The association of implementation and smoking was examined by logistic regression analyses.

RESULTS

One fourth of the schools was characterized as high implementers of the program (all three components) at both first (12 schools, 24.0 %) and second follow-up (11 schools, 28.2 %). Implementation fidelity was strongly associated with smoking at the first and second follow-up, e.g., the odds for smoking at schools with high implementation both years were OR = 0.44 (95 % CI 0.32 to 0.68).

CONCLUSIONS

Using an overall measure based on several aspects of implementation fidelity, we showed a negative graded association between implementation and smoking. This study suggests that higher degrees of implementation will improve the effect of the X:IT intervention. Studying the association between implementation and effect is extremely important; only by doing so, we can distinguish the quality of the intervention from the success of the implementation.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN77415416.

Unravelling the effect of the Dutch school-based nutrition programme Taste Lessons: the role of dose, appreciation and interpersonal communication

BACKGROUND

To unravel the effect of school-based nutrition education, insight into the implementation process is needed. In this study, process indicators of Taste Lessons (a nutrition education programme for Dutch elementary schools) and their association with changes in behavioural determinants relevant to healthy eating behaviour are studied.

METHODS

The study sample consisted of 392 Dutch primary school children from 12 schools. Data were collected using teacher and child questionnaires at baseline, and at one and six months after the intervention. Multilevel regression analyses were conducted to study the association between dose, appreciation and children's engagement in interpersonal communication (talking about Taste Lessons with others after the lessons), and change in knowledge, awareness, skills, attitude, emotion, subjective norm and intention towards two target behaviours.

RESULTS

With an average implementation of a third of the programme activities, dose positively predicted change in children's subjective norm of the teacher after one month. Teachers and children highly appreciated Taste Lessons. Whereas teacher appreciation was inversely associated, child appreciation was positively associated with children's change in awareness, emotion and subjective norm of teachers after one month and in attitude and subjective norm of parents after six months. Interpersonal communication was positively associated with children's change in five determinants after one month and in attitude and intention after six months.

CONCLUSIONS

The implementation process is related to the programme outcomes of Taste Lessons. Process data provide valuable insights into factors that contribute to the effect of interventions in real-life settings.

A Collaborative Approach to Program Evaluation of Community-Based Teen Pregnancy Prevention Projects

The purpose of this article is to demonstrate a model for collaboration between program providers and program evaluators. The article describes how university-based evaluators, a state health department, and local program providers collaborated to evaluate 12 projects implementing commercially developed teenage pregnancy prevention (TPP) programs in school settings. Approximately 2,200 students participate annually in the programs. Program evaluation staff and local program providers worked together to construct logic models that helped guide the intervention and evaluation design. The local providers also participated in training sessions, conducted by the evaluation team, to increase their understanding and skills related to program evaluation methods. Student-level outcomes related to knowledge, attitudes, skills, behaviors, as well as an assessment of curricula fidelity were included in the evaluation. The result of this collaborative model has been a quality program evaluation for the projects while maintaining community input regarding program improvements that reflect local population needs.

An After-School Physical Activity Program for Obesity Prevention in Children

This article describes the process of setting up a 3-year, school-based after-school physical activity intervention in elementary schools. The primary aim of the study is to determine whether adiposity and fitness will improve in children who are exposed to a fitogenic versus an obesogenic environment. Eighteen schools were randomized to the control (obesogenic) or intervention (fitogenic) group. The study design, program components, and evaluation of the intervention are described in detail. The intervention consists of (a) academic enrichment, (b) a healthy snack, and (c) physical activity in a mastery-oriented environment. Successful implementation would show the feasibility of schools’ being able to provide a fitogenic environment. Significant differences between the groups would provide evidence that a fitogenic environment after school has positive health benefits. If feasibility and efficacy are demonstrated, implementing an after-school program like this one in elementary schools could play a major role in preventing and reducing childhood obesity.

Effect and Process Evaluation of a Cluster Randomized Control Trial on Water Intake and Beverage Consumption in Preschoolers from Six European Countries: The ToyBox-Study

Background

Within the ToyBox-study, a kindergarten-based, family-involved intervention was developed to prevent overweight and obesity in European preschoolers, targeting four key behaviours related to early childhood obesity, including water consumption. The present study aimed to examine the effect of the ToyBox-intervention (cluster randomized controlled trial) on water intake and beverage consumption in European preschoolers and to investigate if the intervention effects differed by implementation score of kindergartens and parents/caregivers.

Method

A sample of 4964 preschoolers (4.7±0.4 years; 51.5% boys) from six European countries (Belgium, Bulgaria, Germany, Greece, Poland, Spain) was included in the data analyses. A standardized protocol was used and parents/caregivers filled in socio-demographic data and a food-frequency questionnaire. To assess intervention effects, multilevel repeated measures analyses were conducted for the total sample and for the six country-specific samples. Based on the process evaluation questionnaire of teachers and parents/caregivers, an implementation score was constructed. To assess differences in water intake and beverage consumption by implementation score in the total sample, multilevel repeated measures analyses were performed.

Results

Limited intervention effects on water intake from beverages and overall beverage consumption were found. However, important results were found on prepacked fruit juice consumption, with a larger decrease in the intervention group compared to the control group. However, also a decline in plain milk consumption was found. Implementation scores were rather low in both kindergartens and parents/caregivers. Nevertheless, more favorable effects on beverage choices were found in preschoolers whose parents/caregivers and kindergarten teachers had higher implementation scores compared to those with lower implementation scores.

Conclusion

The ToyBox-intervention can provide the basis for the development of more tailor-made interventions. However, new strategies to improve implementation of interventions should be created.

Rationale, Design, and Methods for Process Evaluation in the Childhood Obesity Research Demonstration Project

OBJECTIVE

The cross-site process evaluation plan for the Childhood Obesity Research Demonstration (CORD) project is described here.

DESIGN

The CORD project comprises 3 unique demonstration projects designed to integrate multi-level, multi-setting health care and public health interventions over a 4-year funding period.

SETTING

Three different communities in California, Massachusetts, and Texas.

PARTICIPANTS

All CORD demonstration projects targeted 2-12-year-old children whose families are eligible for benefits under Title XXI (CHIP) or Title XIX (Medicaid).

INTERVENTION(S)

The CORD projects were developed independently and consisted of evidence-based interventions that aim to prevent childhood obesity. The interventions promote healthy behaviors in children by applying strategies in 4 key settings (primary care clinics, early care and education centers, public schools, and community institutions).

MAIN OUTCOME MEASURE(S)

The CORD process evaluation outlined 3 main outcome measures: reach, dose, and fidelity, on 2 levels (researcher to provider, and provider to participant).

ANALYSIS

The plan described here provides insight into the complex nature of process evaluation for consortia of independently designed multi-level, multi-setting intervention studies. The process evaluation results will provide contextual information about intervention implementation and delivery with which to interpret other aspects of the program.

Preventing alcohol misuse in young people: an exploratory cluster randomised controlled trial of the Kids, Adults Together (KAT) programme

Background

Involvement of parents/carers may increase the effectiveness of primary school-based alcohol-misuse prevention projects. However, few interventions have been designed for pre-adolescent children, or specifically involve parents/carers. The Kids, Adults Together (KAT) programme in primary schools aimed to reduce alcohol misuse through such an approach.

Objective

To determine the value and feasibility of conducting an effectiveness trial of KAT.

Design

Parallel-group cluster randomised exploratory trial with an embedded process evaluation. Schools were the unit of randomisation.

Setting

Primary schools (n = 9) in south Wales, UK.

Participants

Pupils in Year 5/6 (aged 9–11 years) and their parents/carers; school staff.

Intervention

The Kids, Adults Together programme consisted of (1) classwork addressing the effects of alcohol; (2) a family event for children and parents/carers; and (3) a ‘goody bag’ containing fun items, including a digital versatile disc (DVD) for families to watch together. The intervention comprised KAT plus existing alcohol-related activities and lessons. Control-group schools continued with existing alcohol-related lessons and activities.

Main outcome measures

Key outcomes related to the progression criteria for a potential future effectiveness trial. These included the acceptability, participation equity, feasibility and implementation of KAT; the recruitment and retention of research participants; and the acceptability and feasibility of research processes, including data collection methods and outcome measures.

Results

Nine schools (free school meal entitlement ranging from 1% to 37.2%) participated. Two of five intervention schools withdrew but all four control schools were retained, and these seven schools facilitated all research data collections. Programme acceptability and participation rates were high in all three intervention schools (parent/carer participation rates ranged from 45.1% to 65.7%), although implementation quality varied. At baseline, approximately 75% of eligible children (n = 418) provided data, of whom 257 also provided data at follow-up. Only 27 parents/carers (estimated response rate 6.5%) completed interviews. Most children were willing to complete questionnaires but measures were not appropriate for this age group. Measures of alcohol consumption produced inconsistent responses. Intermediate outcomes on family communication showed no evidence of intervention effectiveness.

Conclusions

In the three schools that received the KAT intervention, it was found to be acceptable to schools and pupils and there were good levels of participation from parents/carers from across a range of socioeconomic groups. However, two intervention schools withdrew from the trial. Findings from intermediate outcomes on family communication did not support programme theory. In addition, the study highlighted challenges in identifying suitable outcome measures for children aged 9–11 years and the feasibility of long-term follow-up via secondary schools.

Future work

It would not be appropriate to proceed to an effectiveness trial of KAT. There are doubts/uncertainties about the potential effects of KAT; suitability of measures; the large number of schools which would be required for an effectiveness trial of KAT, and the cost of this; feasibility of follow-up in secondary schools; and programme implementation and theory. There is a need to develop and validate measures for children aged 9–11 years; to test the feasibility of follow-up data collection methods in secondary schools; and to further consider sample size requirements and feasibility.

Trial registration

Current Controlled Trials ISRCTN80672127.

Funding

The exploratory trial of this project was funded by the National Institute for Health Research Public Health Research programme and the process evaluation was funded by the Economic and Social Research Council. The work was undertaken with the support of The Centre for the Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer), a UK Clinical Research Collaboration Public Health Research Centre of Excellence. Joint funding (MR/KO232331/1) from the British Heart Foundation, Cancer Research UK, Economic and Social Research Council, the Welsh Government and the Wellcome Trust, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged. This project will be published in full inPublic Health Research; Vol. 3, No. 15. See the NIHR Journals Library website for further project information.

The role of curriculum dose for the promotion of fruit and vegetable intake among adolescents: results from the Boost intervention

Background

Multi-component interventions combining educational and environmental strategies have proved effective in increasing children and adolescents’ fruit and vegetable intake. However such interventions are complex and difficult to implement and several studies report poor implementation. There is a need for knowledge on the role of dose for behaviour change and for assessment of intervention dose to avoid conclusions that intervention components which are not implemented are ineffective. This study aimed to examine 1) the association between dose of a class curriculum and adolescents’ fruit and vegetable intake in a school-based multi-component intervention, 2) if gender and socioeconomic position modify this association.

Methods

We carried out secondary analysis of data from intervention schools in the cluster-randomized Boost study targeting 13-year-olds’ fruit and vegetable intake. Teacher- and student data on curriculum dose delivered and received were aggregated to the school-level and class-level (only possible for student data). We analysed the association between curriculum dose and students’ (n 995) self-reported fruit and vegetable intake (24-h recall questionnaire) after finalization of the intervention using multi-level analyses. Potential moderation was examined by analyses stratified by gender and socioeconomic position.

Results

Average dose received at class-level was significantly associated with students’ fruit and vegetable intake (10 g (CI: 0.06, 20.33) per curricular activity received). In stratified analyses the association remained significant among boys only (14 g (CI: 2.84, 26.76) per curricular activity received). The average dose delivered and received at the school-level was not significantly associated with students’ intake.

Conclusions

We found a dose—response relationship between number of curricular activities received and adolescents’ fruit and vegetable intake. The results indicate that curriculum dose received only mattered for promotion of fruit and vegetable intake among boys. Future studies should explore this gender difference in larger samples to guide the planning of school-based curricular interventions with regards to the optimal number of curricular activities required to promote behavioural change in subgroups with low fruit and vegetable intake at baseline.

Trial registration

Current Controlled Trials ISRCTN11666034.

The effects of teacher fidelity of implementation of pathways to health on student outcomes

Previous research has demonstrated the importance of ensuring that programs are implemented as intended by program developers in order to achieve desired program effects. The current study examined implementation fidelity of Pathways to Health (Pathways), a newly developed obesity prevention program for fourth- through sixth-grade children. We explored the associations between self-reported and observed implementation fidelity scores and whether implementation fidelity differed across the first 2 years of program implementation. Additionally, we examined whether implementation fidelity affected program outcomes and whether teacher beliefs were associated with implementation fidelity. The program was better received, and implementation fidelity had more effects on program outcomes in fifth grade than in fourth grade. Findings suggest that implementation in school-based obesity programs may affect junk food intake and intentions to eat healthfully and exercise. School support was associated with implementation fidelity, suggesting that prevention programs may benefit from including a component that boosts school-wide support.

Implementation of strategies to increase adolescents' access to fruit and vegetables at school: process evaluation findings from the Boost study

Background

Access to fruit and vegetables (FV) is associated with adolescents’ FV consumption. However, little is known about implementation of strategies to increase access to FV at schools. We examined the implementation of two environmental components designed to increase access to FV at Danish schools.

Methods

We used data from 20 intervention schools involved in the school-based multicomponent Boost trial targeting 13-year-olds’ FV consumption. The environmental components at school included daily provision of free FV and promotion of a pleasant eating environment.

Questionnaire data was collected by the end of the nine-month intervention period among 1,121 pupils (95%), from all school principals (n = 20) and half way through the intervention period and by the end of the intervention among 114 teachers (44%).

The implementation of the components was examined descriptively using the following process evaluation measures; fidelity, dose delivered, dose received and reach. Schools with stable high implementation levels over time were characterised by context, intervention appreciation and implementation of other components.

Results

For all process evaluation measures, the level of implementation varied by schools, classes and over time. Dose received: 45% of pupils (school range: 13-72%, class range: 7-77%) ate the provided FV daily; 68% of pupils (school range: 40-93%, class range: 24-100%) reported that time was allocated to eating FV in class. Reach: The intake of FV provided did not differ by SEP nor gender, but more girls and low SEP pupils enjoyed eating FV together. Dose delivered: The proportion of teachers offering FV at a daily basis decreased over time, while the proportion of teachers cutting up FV increased over time. Schools in which high proportions of teachers offered FV daily throughout the intervention period were characterized by being: small; having a low proportion of low SEP pupils; having a school food policy; high teacher- and pupil intervention appreciation; having fewer teachers who cut up FV; and having high implementation of educational components.

Conclusions

The appliance of different approaches and levels of analyses to describe data provided comprehension and knowledge of the implementation process. This knowledge is crucial for the interpretation of intervention effect.

Trial registration

Current Controlled Trials ISRCTN11666034

A six-step protocol to systematic process evaluation of multicomponent cluster-randomised health promoting interventions illustrated by the Boost study

BACKGROUND

In multicomponent interventions it is important to examine the implementation of each component to enable valid assessments of the effectiveness of each component. Many studies do not systematically document, evaluate and report the level of implementation and there is a lack of systematic approaches to conduct process evaluation studies to guide researchers and evaluators. The aim of this study was to present a systematic approach to plan process evaluation of the implementation of randomised multicomponent interventions.

METHODS

Building on existing process evaluation frameworks and concepts, we developed a six-step protocol: 1. Brainstorm of processes necessary for full implementation and potential barriers and facilitators to implementation; 2. Application of process evaluation concepts to ensure inclusion of important implementation processes; 3. Measurement of proximal outcomes; 4. Identification of relevant data sources; 5. Selection of methods and timing of data collection of process measures; 6. Development of instruments. The protocol was applied to the Boost study, a multicomponent school-based dietary intervention.

RESULTS AND CONCLUSIONS

The protocol was readily applicable for planning process evaluation of environmental and educational intervention components in a school setting. The protocol ensures systematic assessment of the implementation processes that are crucial for interpretation of intervention effects.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN11666034.

Barriers and facilitators for teachers' implementation of the curricular component of the boost intervention targeting adolescents' fruit and vegetable intake

OBJECTIVE

To examine barriers and facilitators to teachers' implementation of the curricular component of the school-based, multicomponent Boost intervention to promote fruit and vegetable intake among 13-year-olds guided by concepts of Diffusion of Innovations Theory and findings of previous implementation studies.

DESIGN

Five focus group and 2 individual interviews.

SETTING

Denmark.

PARTICIPANTS

Twenty-two seventh-grade teachers from 7 of 20 intervention schools. Four schools (15 teachers) with a high implementation level and 3 (7 teachers) with a low implementation level were selected to obtain maximum variation in teachers' view.

PHENOMENON OF INTEREST

Teacher perceptions of implementation of a curricular component.

ANALYSIS

Situational Analysis including an introductory phase of systematic coding.

RESULTS

Teachers' commitment to the Boost curriculum was hindered by intervention duration and extra workload and motivated by a pre-intervention workshop and the thoroughness of the project. Detailed implementation manuals were helpful for some teachers but a barrier to others because they limited opportunities for adaptation.

CONCLUSIONS AND IMPLICATIONS

Implementation of curricular activities in school-based interventions may be supported by a predefined teaching schedule, detailed teacher manuals, clear learning objectives, and a pre-intervention workshop to enhance motivation. Situational Analysis may contribute to future implementation studies by highlighting the importance of contextual factors.

Does the Royal Horticultural Society Campaign for School Gardening increase intake of fruit and vegetables in children? Results from two randomised controlled trials

Background

Children’s fruit and vegetable intake in the UK is low. Changing intake is challenging. Gardening in schools might be a vehicle for facilitating fruit and vegetable intake.

Objectives

To undertake the first clustered randomised controlled trials (RCTs) of a gardening intervention. To evaluate the impact of a school gardening programme, the Royal Horticultural Society (RHS) Campaign for School Gardening, on children’s fruit and vegetable intake.

Methods

Primary school children aged 8–11 years from eight London boroughs were included in one of two related RCTs. Trial 1 consisted of 23 schools, randomised to receive either a RHS-led or teacher-led intervention. Trial 2 consisted of 31 schools, randomised to either the teacher-led intervention or a comparison group. A 24-hour food diary [the Child And Diet Evaluation Tool (CADET)] collected baseline and follow-up dietary intake. Questionnaires measured children’s knowledge and attitudes towards fruit and vegetables and assessed intervention implementation. Data were collected by fieldworkers who were blind to the original allocation of the school. The primary outcome was change in fruit and vegetable intake analysed using a random effects model, based on intention to treat.

Results

Total sample size at baseline for both trials (2529 children) was lower than the original aim of 2900 children. The final sample size was 1557, with 641 children completing trial 1 (RHS-led,n = 312; teacher-led,n = 329) and 916 children completing trial 2 (teacher-led,n = 488; control,n = 428). The response rate at follow-up for the two combined was 62%.

Baseline analysis of children’s fruit and vegetable intake showed that eating a family meal together, cutting up fruit and vegetables, and parental modelling of fruit and vegetable intakes were all associated with higher intakes of fruit and vegetables in children.

The primary trial outcome, combined fruit and vegetable intake, showed that in trial 1 the teacher-led group had a mean change in intake of 8 g [95% confidence interval (CI) –19 to 36 g], compared with a mean of –32 g (95% CI –60 to –3 g) in the RHS-led group. However, this difference was not significant (intervention effect –43 g, 95% CI –88 to 1 g;p = 0.06). In trial 2, the teacher-led group consumed 15 g (95% CI –36 to 148 g) more fruit and vegetables than the comparison group; this difference was not significant. No change was found in children’s knowledge and attitudes. However, if schools improved their RHS gardening score by three levels, children had a higher intake of 81 g of fruit and vegetables (95% CI 0 to 163 g;p = 0.05) compared with schools with no change in gardening score.

Conclusion

Results from these trials provide little evidence that school gardening alone can improve children’s fruit and vegetable intake. In both trials, gardening levels increased across all groups from baseline to follow-up, with no statistically significant difference between groups in terms of improvement in gardening level. This lack of differentiation between groups is likely to have influenced the primary outcome. However, when the gardening intervention was implemented at the highest intensities there was a suggestion that it could improve children’s fruit and vegetable intake by a portion. Analysis of the baseline data showed that family support for fruit and vegetable intakes was associated with higher intakes of fruit and vegetables in children. This study highlights the need for more sophisticated and accurate tools to evaluate diet in children. Future intervention designs should include a greater level of parental involvement in school interventions, along with related components such as cooking, to substantially improve children’s fruit and vegetable intake. In addition, the home environment has been demonstrated to be an important focus for intervention.

Trial registration

Current Controlled Trials ISRCTN11396528.

Funding

This project was funded by the National Institute for Health Research (NIHR) Public Health Research programme and will be published in full inPublic Health Research; Vol. 2, No. 4. See the NIHR Journals Library website for further project information.

Long Live Love. The implementation of a school-based sex-education program in The Netherlands

Implementation of health education programs is often inadequately considered or not considered at all in planning, developing and evaluating interventions. With the focus being predominantly on the adoption stage, little is known about the factors influencing the implementation and continuation stages of the diffusion process. This study contributes to the understanding of factors that promote or impede each stage of the diffusion process in the school setting using the sex education program Long Live Love (LLL) as an example. A survey integrating different diffusion-related concepts was completed by 130 teachers. Results showed that teacher curriculum-related beliefs were associated with all stages in the diffusion process. Although adoption of LLL was predominantly related to teacher curriculum-related beliefs, implementation completeness and fidelity and continued use of LLL were also enhanced by contextual factors, namely teacher training and interactive context variables (school policy, governing body support and student response), respectively. The results of this study can be used to optimize the adoption, implementation and continuation of school-based (sexual) health promotion programs.

Implementationsforschung: Grundbegriffe und Konzepte

Die Implementationsforschung operationalisiert Schritte einer erfolgreichen Umsetzung von Forschungsergebnissen. Im vorliegenden Beitrag wird das Konzept der Implementationsforschung jenseits von Konzepten der praktischen Umsetzung und der erzielten Evaluationsergebnisse betrachtet. Hierzu wird eine Taxonomie von acht konzeptionell unterscheidbaren Outcomeparametern vorgeschlagen: Akzeptanz, Übernahme, Angemessenheit, Machbarkeit, Wiedergabetreue, Kosten, Durchdringung und Nachhaltigkeit. Das Konzeptualisieren und Ermitteln von Implementationsergebnissen tragen dazu bei, die Effizienz der Implementationsforschung zu erhöhen und den Weg für ein besseres Verständnis von Umsetzungsstrategien zu ebnen.

Implementation of a nurse-led behaviour change intervention to support medication taking in type 2 diabetes: beyond hypothesised active ingredients (SAMS Consultation Study)

Background

Implementation of trial interventions is rarely assessed, despite its effects on findings. We assessed the implementation of a nurse-led intervention to facilitate medication adherence in type 2 diabetes (SAMS) in a trial against standard care in general practice. The intervention increased adherence, but not through the hypothesised psychological mechanism. This study aimed to develop a reliable coding frame for tape-recorded consultations, assessing both a priori hypothesised and potential active ingredients observed during implementation, and to describe the delivery and receipt of intervention and standard care components to understand how the intervention might have worked.

Methods

211 patients were randomised to intervention or comparison groups and 194/211 consultations were tape-recorded. Practice nurses delivered standard care to all patients and motivational and action planning (implementation intention) techniques to intervention patients only. The coding frame was developed and piloted iteratively on selected tape recordings until a priori reliability thresholds were achieved. All tape-recorded consultations were coded and a random subsample double-coded.

Results

Nurse communication, nurse-patient relationship and patient responses were identified as potential active ingredients over and above the a priori hypothesised techniques. The coding frame proved reliable. Intervention and standard care were clearly differentiated. Nurse protocol adherence was good (M (SD) = 3.95 (0.91)) and competence of intervention delivery moderate (M (SD) = 3.15 (1.01)). Nurses frequently reinforced positive beliefs about taking medication (e.g., 65% for advantages) but rarely prompted problem solving of negative beliefs (e.g., 21% for barriers). Patients’ action plans were virtually identical to current routines. Nurses showed significantly less patient-centred communication with the intervention than comparison group.

Conclusions

It is feasible to reliably assess the implementation of behaviour change interventions in clinical practice. The main study results could not be explained by poor delivery of motivational and action planning components, definition of new action plans, improved problem solving or patient-centred communication. Possible mechanisms of increased medication adherence include spending more time discussing it and mental rehearsal of successful performance of current routines, combined with action planning. Delivery of a new behaviour change intervention may lead to less patient-centred communication and possible reduction in overall trial effects.

Trial registration

ISRCTN30522359.

Treatment adherence and facilitator characteristics in a community based pediatric weight control intervention

BACKGROUND

There is a pressing need to develop effective and broadly accessible interventions to address pediatric obesity. An important dimension in translating interventions to community settings is evaluating the fidelity with which the intended treatment is delivered and the level of facilitator needed to deliver the intervention with efficacy.

PURPOSE

The primary objectives of this study were to: 1) provide descriptive information regarding adherence to protocol and non-specific facilitator characteristics (e.g. interpersonal characteristics, group management skills) within the context of a community based pediatric weight control intervention delivered by paraprofessionals; and 2) examine the relationships among facilitator adherence and characteristics and rate of change in percent overweight demonstrated by youth over the course of the 24-week intervention.

METHODS

The intervention was conducted between February and September of 2011. Children (6-16 years) and parents completed primary outcome measures at baseline, 12, and 24 weeks (i.e. end of treatment). A 2-part rating form was developed to assess facilitator adherence to weekly content and general provider characteristics at two different time points during the intervention.

RESULTS

Youth participating in this study were on average 11.3 years old (SD = 2.8), with most being under the age of 13 years (74.2%). Over half were female (54.8%) and over two-thirds were White (68.4%). On average, facilitators adhered to 96.0% (SD = 5.2%) of the session content at Time 1 and 92.6% (SD = 6.8%) at Time 2. Higher Content Adherence at Time 1 and Time 2 were associated with greater loss in percent overweight.

CONCLUSIONS

Our data suggest that paraprofessionals without prior expertise in pediatric weight control can be trained to successfully deliver an intervention that is evidence based and incorporates behavioral and educational components. These findings need to be considered in light of some limitations, including the fact that facilitator domains were assessed with a modification of a standardized tool and we did not obtain inter-rater reliability of observations. These limitations not withstanding, investing time in training facilitators to adhere to a given protocol is critical and may be of higher priority than focusing on more general facilitator characteristics.

Evaluating a Fruit and Vegetable Program: In Eastern Ontario Schools

Purpose: Effectiveness was evaluated for a fruit and vegetable program developed to encourage Canadian elementary school children to eat the recommended number of daily servings. Also examined was whether the program modified children’s personal factors, perceived social environment, and perceived physical environment.

Methods: A prospective, quasi-experimental trial was conducted to compare the eight schools receiving the intervention curriculum (Freggie Friday schools [FFS]) with six control schools (CS). A food frequency questionnaire was used to measure differences in fruit and vegetable consumption. Personal factors, perceived social environment, and perceived physical environment supporting fruit and vegetable consumption were assessed with an adapted version of the validated Pro Children study questionnaire.

Results: Of the 942 children who completed the baseline assessment, 807 also completed the follow-up questionnaire (FFS, 450; CS, 357). A mixed-effects regression model indicated no significant intervention effects on fruit or vegetable consumption, snack food consumption, or knowledge or attitudes related to fruit and vegetable consumption.

Conclusions: The results suggest that an intervention based on a single visit from an external group, followed by teacher-led programming, may be an ineffective method of eliciting dietary behaviour change in this population. Future programs may need to implement multicomponent intervention designs.

Implementation quality of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong: findings from the extension phase

To evaluate the implementation quality of the Tier 1 Program of the Project P.A.T.H.S. (Secondary 1 to Secondary 3), a study based on classroom observations of 597 units in 207 schools was conducted under the Co-Walker Scheme. Results demonstrated that the implementation quality of the program was high in general, with a mean program adherence of 80.8%. The 13 aspects of program delivery under observation were significantly intercorrelated. Multiple regression analyses indicated that: (i) overall implementation quality was significantly predicted by classroom control, interactive delivery method, use of positive and supportive feedback, opportunity for reflection, degree of achievement of the objectives, time management, and lesson preparation; (ii) success of implementation was significantly predicted by student participation and involvement, opportunity for reflection, degree of achievement of the objectives, and time management.

Describing teacher-student interactions: a qualitative assessment of teacher implementation of the 7th grade keepin' it REAL substance use intervention

Variations in the delivery of school-based substance use prevention curricula affect students' acquisition of the lesson content and program outcomes. Although adaptation is sometimes viewed as a lack of fidelity, it is unclear what types of variations actually occur in the classroom. This observational study investigated teacher and student behaviors during implementation of a middle school-based drug prevention curriculum in 25 schools across two Midwestern states. Trained observers coded videos of 276 lessons, reflecting a total of 31 predominantly Caucasian teachers (10 males and 21 females) in 73 different classes. Employing qualitative coding procedures, the study provides a working typology of implementation patterns based on varying levels of teacher control and student participation. These patterns are fairly consistent across lessons and across classes of students, suggesting a teacher-driven delivery model where teachers create a set of constraints within which students vary their engagement. Findings provide a descriptive basis grounded in observation of classroom implementation that can be used to test models of implementation fidelity and quality as well as impact training and other dissemination research.

Intervention dose estimation in health promotion programmes: a framework and a tool. Application to the diet and physical activity promotion PRALIMAP trial

Background

Although the outcomes of health promotion and prevention programmes may depend on the level of intervention, studies and trials often fail to take it into account. The objective of this work was to develop a framework within which to consider the implementation of interventions, and to propose a tool with which to measure the quantity and the quality of activities, whether planned or not, relevant to the intervention under investigation. The framework and the tool were applied to data from the diet and physical activity promotion PRALIMAP trial.

Methods

A framework allowing for calculation of an intervention dose in any health promotion programme was developed. A literature reviews revealed several relevant concepts that were considered in greater detail by a multidisciplinary working group. A method was devised with which to calculate the dose of intervention planned and that is actually received (programme-driven activities dose), as well as the amount of non-planned intervention (non-programme-driven activities dose).

Results

Indicators cover the roles of all those involved (supervisors, anchor personnel as receivers and providers, targets), in each intervention-related groups (IRG: basic setting in which a given intervention is planned by the programme and may differ in implementation level) and for every intervention period. All indicators are described according to two domains (delivery, participation) in two declensions (quantity and quality). Application to PRALIMAP data revealed important inter- and intra-IRG variability in intervention dose.

Conclusions

A literature analysis shows that the terminology in this area is not yet consolidated and that research is ongoing. The present work provides a methodological framework by specifying concepts, by defining new constructs and by developing multiple information synthesis methods which must be introduced from the programme's conception. Application to PRALIMAP underlined the feasibility of measuring the implementation level. The framework and the tool can be used in any complex programme evaluation. The intervention doses obtained could be particularly useful in comparative trials.

Trial registration

PRALIMAP is registered at ClinicalTrials.gov under NCT00814554

Early origins of child obesity: bridging disciplines and phases of development -- September 30--October 1, 2010

This report summarizes a conference: “Early Origins of Child Obesity: Bridging Disciplines and Phases of Development”, held in Chicago on September 30–October 1, 2010. The conference was funded in part by the National Institutes of Health and the Williams Heart Foundation, to achieve the conference objective: forging a next-step research agenda related to the early origins of childhood obesity. This research agenda was to include working with an array of factors (from genetic determinants to societal ones) along a continuum from prenatal life to age 7, with an emphasis on how the developing child deals with the challenges presented by his/her environment (prenatal, parental, nutritional, etc.). The conference offered a unique opportunity to facilitate communication and planning of future work among a variety of researchers whose work separately addresses different periods in early life. Over the span of two days, speakers addressed existing, critical research topics within each of the most-studied age ranges. On the final day, workshops fostered the discussion needed to identify the highest priority research topics related to linking varied early factor domains. These are presented for use in planning future research and research funding.

[Process evaluation in relation to effectiveness assessment: experiences with school-based programs]

Evaluation of public health interventions usually focus on the quality of design and research methods, and less on the quality of the intervention or process evaluation. In process evaluation of school-based interventions, key issues are how completely the intervention is carried out and adherence to the protocol. In addition, exploration of intermediate variables, such as those that influence (and often predict) preventable behavior, is highly useful. This article describes the basic concepts in this topic, using examples of the effectiveness of some preventive interventions carried out in schools. The interventions discussed were mainly quasi-experimental studies, based on data from programs promoted by public health teams in the city of Barcelona. Data from process evaluation of preventive programs in secondary schools that underwent formal assessment of their effectiveness is provided. The examples are drawn from preventive programs of HIV infection or unprotected sexual intercourse (PRESSEC program) and drug consumption prevention (the PASE, PASE.bcn and x kpts programs). These examples show why the intervention process influences the impact of the programs and their results. Thorough planning of process evaluation is essential to obtain valid indicators that will identify, in the effectiveness evaluation of the intervention, the most efficacious strategies to obtain positive outcomes.

Examining the Sustainability of an Evidence-Based Preschool Curriculum: The REDI Program

This study examined the extent to which an evidence-based preschool curriculum (Head Start REDI) was sustained by 20 teachers during the year following a randomized controlled efficacy trial, when teachers were no longer required by the research project to implement the curriculum. Two quantitative measures of sustainability (teacher ratings, REDI coach ratings) and a qualitative measure (teacher interview) were collected and compared. Sustainability varied by the specific curriculum component, with higher rates of sustainability for the social-emotional component (Preschool PATHS) than for the language and literacy components. Estimates of sustainability were affected by the method of measurement, with REDI coach ratings and qualitative teacher interviews more closely aligned than teacher ratings. Responses from qualitative interviews identified the main factors that teachers thought affected sustainability. Teacher responses suggest that efforts to promote sustainability are best targeted at reducing barriers, such as competing demands, rather than simply highlighting the benefits of the new curriculum.

Putting the pieces together: an integrated model of program implementation

Considerable evidence indicates that variability in implementation of prevention programs is related to the outcomes achieved by these programs. However, while implementation has been conceptualized as a multidimensional construct, few studies examine more than a single dimension, and no theoretical framework exists to guide research on the effects of implementation. We seek to address this need by proposing a theoretical model of the relations between the dimensions of implementation and outcomes of prevention programs that can serve to guide future implementation research. In this article, we focus on four dimensions of implementation, which we conceptualize as behaviors of program facilitators (fidelity, quality of delivery, and adaptation) and behaviors of participants (responsiveness) and present the evidence supporting these as predictors of program outcomes. We then propose a theoretical model by which facilitator and participant dimensions of implementation influence participant outcomes. Finally, we provide recommendations and directions for future implementation research.

Evidence-based practice in school substance use prevention: fidelity of implementation under real-world conditions

Fidelity of program implementation under real-world conditions is a critical issue in the dissemination of evidence-based school substance use prevention curricula. Program effects are diminished when programs are implemented with poor fidelity. We assessed five domains of fidelity--adherence, exposure (dosage), quality of delivery, participant responsiveness and program differentiation (lack of contamination from other programs)--in a subset of respondents (N = 342) from a national random sample of public schools with middle school grades (N = 1721). Respondents taught 1 of 10 evidence-based universal substance use prevention programs as their primary program during the 2004-05 school year. Their responses to survey questions about their recent implementation practices indicated that fidelity was high for quality of delivery and participant responsiveness, low for program differentiation and modest for adherence and exposure--the two core domains of fidelity. Results suggest the need for continued emphasis on fidelity in program materials, trainings and on-going technical support. Particular attention should be paid to supporting use of interactive delivery strategies.

Process evaluation of HIV prevention peer groups in Malawi: a look inside the black box

This paper reports the process evaluation of a peer group intervention for human immunodeficiency virus (HIV) prevention which had positive outcomes for three target groups in Malawi: rural adults, adolescents and urban hospital workers. The six-session intervention was delivered to small groups of 10-12 participants by 85 trained volunteer peer leaders working in pairs. A descriptive, observational mixed methods design was used with a convenience sample of 294 intervention sessions. Using project records and a conceptually based observation guide, we examined five aspects of the implementation process. The context was favorable, but privacy to discuss sensitive issues was a concern for some groups. In study communities, program reach was 58% of rural adults, 70% of adolescents and nearly all hospital workers. Session records confirmed that all peer groups received the intended six sessions (dose delivered). The dose received was high, as evidenced by high participant engagement in peer group activities. Peer leaders were rated above the median for three indicators of peer group content and process fidelity: session management skills, interpersonal facilitation skills and whether more like a peer group than classroom. Documenting that this HIV prevention peer group intervention was delivered as intended by trained peer volunteers supports widespread dissemination of the intervention.

Process evaluation of a tobacco prevention program in Indian schools--methods, results and lessons learnt

In India, 57% of men between 15 and 54 years and 10.8% of women between 15 and 49 years use tobacco. A wide variety of tobacco gets used and the poor and the underprivileged are the dominant victims of tobacco and its adverse consequences. Project MYTRI (Mobilizing Youth for Tobacco-Related Initiatives in India) was a tobacco prevention intervention program, a cluster-randomized trial in 32 Indian schools which aimed to decrease susceptibility to tobacco use among sixth- to ninth-grade students in urban settings in India. This culture-specific intervention, which addressed both smokeless and smoked forms of tobacco, was Indian in content and communication. We qualitatively developed indicators which would help accurately measure the dose of the intervention given, received and reached. A multi-staged process evaluation was done through both subjective and objective measures. Training the teachers critically contributed toward a rigorous implementation and also correlated with the outcomes, as did a higher proportion of students participating in the classroom discussions and better peer-leader-student communication. A sizeable proportion of subjective responses were 'socially desirable', making objective assessment a preferred methodology even for 'dose received'. The peer-led health activism was successful. Teachers' manuals need to be concise.

Process evaluation of a school based physical activity related injury prevention programme using the RE-AIM framework

Background

In general, only information regarding the effectiveness of an intervention programme is ever published. However, in recent years evaluating the translatability and feasibility of an intervention programme has become more important. Therefore, this paper presents the results of the evaluation of the iPlay programme aimed at preventing physical activity related injuries in primary school children.

Methods

The iPlay programme targeted injuries gained through physical activity, and consisted of a teacher's manual, informative newsletters and posters, a website, and set exercises to be carried out during physical education (PE) classes. In order to evaluate the iPlay programme for translatability and feasibility, teachers, children and parents who participated in the iPlay programme filled out a questionnaire

The objective of this study is to describe the results of the process-evaluation of the iPlay programme based on the five dimensions of the RE-AIM framework.

Results

The results showed that the participation rate of the children was 100% (reach). Nine percent of the schools who were invited to take part were willing to participate in the study (adoption rate). Teachers stated that they implemented the different elements of the programme partly as intended (implementation). The percentage of children and parents who followed the programme was less than expected. In addition, 52% of the teachers indicated that the current iPlay programme could become standard practice in their teaching routine (maintenance).

Conclusion

The iPlay programme is a first start in the prevention of physical activity related injuries in children, but further improvements need to be made to the programme on the basis of this process evaluation.

Trial registration

ISRCTN78846684; http://www.controlled-trials.com

Putting the pieces together: an integrated model of program implementation

Considerable evidence indicates that variability in implementation of prevention programs is related to the outcomes achieved by these programs. However, while implementation has been conceptualized as a multidimensional construct, few studies examine more than a single dimension, and no theoretical framework exists to guide research on the effects of implementation. We seek to address this need by proposing a theoretical model of the relations between the dimensions of implementation and outcomes of prevention programs that can serve to guide future implementation research. In this article, we focus on four dimensions of implementation, which we conceptualize as behaviors of program facilitators (fidelity, quality of delivery, and adaptation) and behaviors of participants (responsiveness) and present the evidence supporting these as predictors of program outcomes. We then propose a theoretical model by which facilitator and participant dimensions of implementation influence participant outcomes. Finally, we provide recommendations and directions for future implementation research.

Rationale, design and methods for process evaluation in the HEALTHY study

The HEALTHY study was a multi-site randomized trial designed to determine whether a 3-year school-based intervention targeting nutrition and physical activity behaviors could effectively reduce risk factors associated with type 2 diabetes in middle school children. Pilot and formative studies were conducted to inform the development of the intervention components and the process evaluation methods for the main trial. During the main trial, both qualitative and quantitative assessments monitored the fidelity of the intervention and motivated modifications to improve intervention delivery. Structured observations of physical education classes, total school food environments, classroom-based educational modules, and communications and promotional campaigns provided verification that the intervention was delivered as intended. Interviews and focus groups yielded a multidimensional assessment of how the intervention was delivered and received, as well as identifying the barriers to and facilitators of the intervention across and within participating schools. Interim summaries of process evaluation data were presented to the study group as a means of ensuring standardization and quality of the intervention across the seven participating centers. Process evaluation methods and procedures documented the fidelity with which the HEALTHY study was implemented across 21 intervention schools and identified ways in which the intervention delivery might be enhanced throughout the study.

Implementation Quality: Lessons Learned in the Context of the Head Start REDI Trial

This study uses data collected in the intervention classrooms (N = 22) of Head Start REDI (Research-based, Developmentally Informed), a randomized clinical trial testing the efficacy of a comprehensive preschool curriculum targeting children's social-emotional competence, language, and emergent literacy skills delivered by teachers who received weekly coaching support. Multiple dimensions of implementation (Dosage, Fidelity, Generalization, and Child Engagement) were assessed across curriculum components. Results indicated that REDI Trainers perceived significant growth in teacher implementation quality over time but that patterns differed by implementation dimension. Dosage and Fidelity of all intervention components were high at the beginning of the year and remained so over time while Generalization was low at baseline but increased significantly across the year. Variation in implementation was associated with variation on several child outcome measures in the social-emotional domain but not in the language and literacy domains.