Using process evaluation for program improvement in dose, fidelity and reach: the ACT trial experience

Theory-guided process evaluation of a multicomponent, technology-based 'SMART Eating' trial among Indian adults: an embedded mixed-methods study

We report process evaluation findings from the 'SMART Eating' intervention trial, which significantly improved fat, sugar and salt (FSS), and fruits and vegetables (FVs) intake among adults. Intervention used information technology [short message service (SMS), WhatsApp and website] and interpersonal communication (distribution of SMART Eating kit) and pamphlet for comparison group. Guided by UK Medical Research Council's framework, using embedded mixed-methods design, continuous process evaluation documented fidelity, dose, reach, acceptability and mechanisms. Intervention was implemented as intended, with high reach (91%) in both groups: 'comparison group' (n = 366): inadequate use of pamphlets (46%); 'intervention group' (n = 366): with timely remedial measures to remove implementation/usage barriers, dose of SMS (93%), WhatsApp (89%) and 'SMART Eating' kit (100%) was adequate, but website usage was low (50%); compliance was evident from participants' interactions with the implementor and observations on kit usage. All these might have improved attitude, social influence, self-efficacy and household behaviours, which, in turn, mediated intervention's effect on improving FSS and FV intake. Among poor performers, lack of effect on FV intake was perceived to be related to high cost/pesticides use and FSS intake was related to lack of family support. Low website usage, challenges with WhatsApp messaging and contextual factors (cost, pesticides abuse and family support) need to be considered while designing similar future interventions.

Implementation models and frameworks used to guide community-based physical activity programs for children: a scoping review

BACKGROUND

The implementation of community-based programs is key to effective, sustainable initiatives that can support population-level changes in children's physical activity. The purpose of this scoping review was to explore the implementation models and frameworks used to develop (process models), explore (determinant frameworks), and/or evaluate (evaluation frameworks) community-based physical activity programs for children. Also, the foundational components of the implementation models and frameworks and practical application in real-world settings were described.

METHODS

The methodological framework developed by Arksey and O'Malley (2005) and the updated recommendations from Levac, Colquhoun and O'Brien (2010) were used to search, identify, and summarize applicable studies. This review also met the requirements in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Scoping Reviews Checklist (PRISMA-ScR). A detailed search of six databases and three academic journals was conducted. Information about the article, the program, and the implementation model/framework were extracted and summarized.

RESULTS

The search retrieved 42,202 articles, of which 27 met the inclusion criteria. Eleven process models, one determinant framework, and two evaluation frameworks were identified. Nineteen components were developed from the models and frameworks. Tailoring, situational analysis, and element identification were common components among the identified models and frameworks.

CONCLUSIONS

Since the execution of interventions is vital for creating successful health-promoting initiatives, researchers and program developers should consider using implementation models and frameworks to guide their community-based physical activity programs. Further research examining the application of new and existing implementation models and frameworks in developing, exploring, and evaluating community-level programs is warranted.

What do we want to get out of this? a critical interpretive synthesis of the value of process evaluations, with a practical planning framework

BACKGROUND

Process evaluations aim to understand how complex interventions bring about outcomes by examining intervention mechanisms, implementation, and context. While much attention has been paid to the methodology of process evaluations in health research, the value of process evaluations has received less critical attention. We aimed to unpack how value is conceptualised in process evaluations by identifying and critically analysing 1) how process evaluations may create value and 2) what kind of value they may create.

METHODS

We systematically searched for and identified published literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues. We conducted a critical interpretive synthesis and developed a practical planning framework.

RESULTS

We identified and included 147 literature items. From these we determined three ways in which process evaluations may create value or negative consequences: 1) through the socio-technical processes of 'doing' the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge. We identified 15 value themes. We also found that value varies according to the characteristics of individual process evaluations, and is subjective and context dependent.

CONCLUSION

The concept of value in process evaluations is complex and multi-faceted. Stakeholders in different contexts may have very different expectations of process evaluations and the value that can and should be obtained from them. We propose a planning framework to support an open and transparent process to plan and create value from process evaluations and negotiate trade-offs. This will support the development of joint solutions and, ultimately, generate more value from process evaluations to all.

Assessing the fidelity of delivery style of a mental skills training programme for young people experiencing homelessness

There is a need for positive youth development/strengths-based approaches to support the wellbeing and social inclusion of young people experiencing or at risk of homelessness. My Strengths Training for Life™ (MST4Life™) uses a strengths-based approach with the aim to improve young people's resilience, self-worth, wellbeing and engagement in education, employment, and training. This mixed methods study assessed the fidelity of delivery style of the MST4Life™ programme, the extent to which frontline service staff can delivery psychologically informed programmes to service users and identified barriers and enablers to delivering with fidelity to the intended style. Observations of programme delivery (two facilitators per session) took place across early, middle, and late phases of the programme across a pilot phase (n = 18) and main study (n = 45). Facilitators also completed self-reflection forms following each session. The mean observation score was 82.2 ± 15.7 %, and facilitator self-report mean adherence score was 89.3 ± 6.2 % which indicate that the programme was delivered with high fidelity. Quantitative data was also analysed using non-parametric statistical test (Mann-Whitney U Test). There was a significant difference between observation scores for deliverers with postgraduate psychology training compared to deliverers without postgraduate psychology training (p = .029). Qualitative data were analysed using inductive thematic analysis. Barriers and enablers included communication, frontline staff support, logistics, and participant behaviours. Overall, this study highlights that despite the challenges of delivering complex community programmes to young people experiencing homelessness, it was possible for frontline service staff to deliver MST4Life™ with high fidelity.

Evaluation of the Special Olympics Team Wellness health promotion program for individuals with intellectual disabilities

INTRODUCTION

Individuals with intellectual disabilities are a known health disparities group. The Team Wellness (TW) health promotion program was developed to provide healthy lifestyle information to target this population. The purpose of this study was to conduct a qualitative program evaluation on Team Wellness.

METHODS

Interviews were conducted via one-on-one format with program coaches (n = 4, mean age = 25, female = 2, male = 2) and focus group with program athletes (n = 6, mean age = 45, female = 4, male = 2) who completed the TW program.

RESULTS

Themes that emerged included: (1) Program Perceptions; (2) TW Support; and (3) Program Feedback. Program perceptions were positive and continued athlete participation was anticipated. Coaches felt prepared to facilitate based on previous experience though program fidelity varied.

CONCLUSION

Study participants enjoyed TW and identified valuable recommendations. Program fidelity and marketing efforts are important for future consideration. Further quantitative assessment is needed to determine the effectiveness the program.

George SM, Prinz R, Resnicow K. The Results of the Families Improving Together (FIT) for Weight Loss Randomized Trial in Overweight African American Adolescents

BACKGROUND

Few intervention studies have integrated cultural tailoring, parenting, behavioral, and motivational strategies to address African American adolescent weight loss.

PURPOSE

The Families Improving Together (FIT) for Weight Loss trial was a randomized group cohort study testing the efficacy of a cultural tailoring, positive parenting, and motivational intervention for weight loss in overweight African American adolescents (N = 241 adolescent/caregiver dyads).

METHODS

The trial tested an 8-week face-to-face group motivational plus family weight loss program (M + FWL) compared with a comprehensive health education control program. Participants were then rerandomized to an 8-week tailored or control online program to test the added effects of the online intervention on reducing body mass index and improving physical activity (moderate-to-vigorous physical activity [MVPA], light physical activity [LPA]), and diet.

RESULTS

There were no significant intervention effects for body mass index or diet. There was a significant effect of the group M + FWL intervention on parent LPA at 16 weeks (B = 33.017, SE = 13.115, p = .012). Parents in the group M + FWL intervention showed an increase in LPA, whereas parents in the comprehensive health education group showed a decrease in LPA. Secondary analyses using complier average causal effects showed a significant intervention effect at 16 weeks for parents on MVPA and a similar trend for adolescents.

CONCLUSIONS

While the intervention showed some impact on physical activity, additional strategies are needed to impact weight loss among overweight African American adolescents.

Process Evaluation of an Application-Based Salt Reduction Intervention in School Children and Their Families (AppSalt) in China: A Mixed-Methods Study

Background

Salt reduction is a cost-effective, and rather challenging public health strategy for controlling chronic diseases. The AppSalt program is a school-based multi-component mobile health (mhealth) salt reduction program designed to tackle the high salt intake in China. This mixed-methods process evaluation was conducted to investigate the implementation of this program across sites, identify factors associated with the implementation, and collect evidence to optimize the intervention design for future scale-up.

Methods

Mixed methods were used sequentially to collect data regarding five process evaluation dimensions: fidelity, dose delivered, dose received, reach, and context. Quantitative data were collected during the intervention process. Participation rate of intervention activities was calculated and compared across cities. The quantitative data was used for the selection of representative intervention participants for the qualitative interviews. Qualitative data were collected in face-to-face semi-structured interviews with purposively selected students (n = 33), adult family members (n = 33), teachers (n = 9), heads of schools (n = 9), key informants from local health, and education departments (n = 8). Thematic analysis technique was applied to analyze the interview transcripts using NVivo. The qualitative data were triangulated with the quantitative data during the interpretation phase.

Results

The total number of families recruited for the intervention was 1,124. The overall retention rate of the AppSalt program was 97%. The intervention was implemented to a high level of fidelity against the protocol. About 80% of intervention participants completed all the app-based salt reduction courses, with a significant difference across the three cities (Shijiazhuang: 95%; Luzhou: 73%; Yueyang: 64%). The smartphone app in this program was perceived as a feasible and engaging health education tool by most intervention participants and key stakeholders. Through the interviews with participants and key stakeholders, we identified some barriers to implementing this program at primary schools, including the left-behind children who usually live with their grandparents and have limited access of smartphones; perceived adverse effects of smartphones on children (e.g., eyesight damage); and overlooked health education curriculum at Chinese primary schools.

Conclusion

This process evaluation demonstrated the feasibility and acceptability of using smartphone applications delivered through the education system to engage families in China to reduce excessive salt intake.

Clinical Trial Registration

The AppSalt study was registered at www.chictr.org.cn, identifier: ChiCTR1800017553. The date of registration is August 3, 2018.

Process Evaluation in Health Communication Media Campaigns: A Systematic Review

Objective. The objective is to examine the scope of health communication media campaign process evaluation methods, findings, and dissemination practices. Data Source. A systematic review of peer-reviewed literature was conducted using database searches. Study Inclusion and Exclusion Criteria. Published studies on process and implementation evaluation of health campaigns with a media component were included. Exclusion criteria included not health, non-empirical, no media campaign, or a focus on other evaluation types. Data Extraction. Articles were assessed for general campaign information, theory use, and details about process evaluation plan and procedures. Data Synthesis. A coding scheme based on 9 process evaluation best practice elements (e.g., fidelity and context) was applied. Process evaluation methods, measures, and reporting themes were synthesized. Results. Among 691 unique records, 46 articles were included. Process evaluation was the main focus for 71.7% of articles, yet only 39.1% reported how process evaluation informed campaign implementation strategy. Articles reported 4.39 elements on average (SD = 1.99; range 1-9), with reach (87.0%) and recruitment (73.9%) described most frequently, yet reporting was inconsistent. Further, the level of detail in reporting methods, theory, and analysis varied. Conclusions. Process evaluation provides insight about mechanisms and intervening variables that could meaningfully impact interpretations of outcome evaluations; however, process evaluations are less often included in literature. Recommendations for evidence-based process evaluation components to guide evaluation are discussed.

Fidelity of implementation of the building bridges mentoring intervention to prevent violence among youth in low income settings

Violence among youth is a major health and safety burden globally. There is a dearth in the development and evaluation of targeted interventions that addresses the multi-faceted nature of youth violence to ensure effectiveness and replicability. This study aims to systematically evaluate the fidelity of implementation of the Building Bridges Mentoring intervention that focuses on the prevention of youth violence in two low-income communities in South Africa. This study employed a mixed methods concurrent triangulation design. Qualitative data were analysed using deductive thematic analysis, and quantitative data using descriptive statistics and T-tests. The results indicate that for all the intervention components, implementation fidelity (i.e. adherence, exposure, quality of programme delivery, and participant responsiveness) was generally moderate to high, suggesting a 'good' implementation of the programme in the real world. Evaluating the fidelity of implementation is vital to obtain a comprehensive insight into whether an intervention was implemented according to its design, and to verify and validate the findings and outcomes, and accord credibility and integrity of the study.

School-based physical activity programs for promoting physical activity and fitness in children and adolescents aged 6 to 18

BACKGROUND

Physical activity among children and adolescents is associated with lower adiposity, improved cardio-metabolic health, and improved fitness. Worldwide, fewer than 30% of children and adolescents meet global physical activity recommendations of at least 60 minutes of moderate to vigorous physical activity per day. Schools may be ideal sites for interventions given that children and adolescents in most parts of the world spend a substantial amount of time in transit to and from school or attending school.

OBJECTIVES

The purpose of this review update is to summarise the evidence on effectiveness of school-based interventions in increasing moderate to vigorous physical activity and improving fitness among children and adolescents 6 to 18 years of age. Specific objectives are: • to evaluate the effects of school-based interventions on increasing physical activity and improving fitness among children and adolescents; • to evaluate the effects of school-based interventions on improving body composition; and • to determine whether certain combinations or components (or both) of school-based interventions are more effective than others in promoting physical activity and fitness in this target population.

SEARCH METHODS

We searched CENTRAL, MEDLINE, Embase, CINAHL, PsycINFO, BIOSIS, SPORTDiscus, and Sociological Abstracts to 1 June 2020, without language restrictions. We screened reference lists of included articles and relevant systematic reviews. We contacted primary authors of studies to ask for additional information.

SELECTION CRITERIA

Eligible interventions were relevant to public health practice (i.e. were not delivered by a clinician), were implemented in the school setting, and aimed to increase physical activity among all school-attending children and adolescents (aged 6 to 18) for at least 12 weeks. The review was limited to randomised controlled trials. For this update, we have added two new criteria: the primary aim of the study was to increase physical activity or fitness, and the study used an objective measure of physical activity or fitness. Primary outcomes included proportion of participants meeting physical activity guidelines and duration of moderate to vigorous physical activity and sedentary time (new to this update). Secondary outcomes included measured body mass index (BMI), physical fitness, health-related quality of life (new to this update), and adverse events (new to this update). Television viewing time, blood cholesterol, and blood pressure have been removed from this update.  DATA COLLECTION AND ANALYSIS: Two independent review authors used standardised forms to assess each study for relevance, to extract data, and to assess risk of bias. When discrepancies existed, discussion occurred until consensus was reached. Certainty of evidence was assessed according to GRADE. A random-effects meta-analysis based on the inverse variance method was conducted with participants stratified by age (children versus adolescents) when sufficient data were reported. Subgroup analyses explored effects by intervention type.

MAIN RESULTS

Based on the three new inclusion criteria, we excluded 16 of the 44 studies included in the previous version of this review. We screened an additional 9968 titles (search October 2011 to June 2020), of which 978 unique studies were potentially relevant and 61 met all criteria for this update. We included a total of 89 studies representing complete data for 66,752 study participants. Most studies included children only (n = 56), followed by adolescents only (n = 22), and both (n = 10); one study did not report student age. Multi-component interventions were most common (n = 40), followed by schooltime physical activity (n = 19), enhanced physical education (n = 15), and before and after school programmes (n = 14); one study explored both enhanced physical education and an after school programme. Lack of blinding of participants, personnel, and outcome assessors and loss to follow-up were the most common sources of bias.  Results show that school-based physical activity interventions probably result in little to no increase in time engaged in moderate to vigorous physical activity (mean difference (MD) 0.73 minutes/d, 95% confidence interval (CI) 0.16 to 1.30; 33 studies; moderate-certainty evidence) and may lead to little to no decrease in sedentary time (MD -3.78 minutes/d, 95% CI -7.80 to 0.24; 16 studies; low-certainty evidence). School-based physical activity interventions may improve physical fitness reported as maximal oxygen uptake (VO₂max) (MD 1.19 mL/kg/min, 95% CI 0.57 to 1.82; 13 studies; low-certainty evidence). School-based physical activity interventions may result in a very small decrease in BMI z-scores (MD -0.06, 95% CI -0.09 to -0.02; 21 studies; low-certainty evidence) and may not impact BMI expressed as kg/m² (MD -0.07, 95% CI -0.15 to 0.01; 50 studies; low-certainty evidence). We are very uncertain whether school-based physical activity interventions impact health-related quality of life or adverse events.

AUTHORS' CONCLUSIONS

Given the variability of results and the overall small effects, school staff and public health professionals must give the matter considerable thought before implementing school-based physical activity interventions. Given the heterogeneity of effects, the risk of bias, and findings that the magnitude of effect is generally small, results should be interpreted cautiously.

Connect through PLAY: a randomized-controlled trial in afterschool programs to increase adolescents' physical activity

The current study is a randomized controlled trial to test a novel 10-week climate-based intervention within pre-existing afterschool programs, designed to increase moderate-to-vigorous physical activity (MVPA) in underserved (low-income, minority status) middle school youth by addressing youth social developmental needs. Participants (n = 167; 56% female; 62% Black; 50% overweight/obese) enrolled in 6 middle schools were randomized to either the Connect through PLAY intervention or a wait-list control. Process evaluation measures (i.e., observations of external evaluators; staff surveys) indicated that essential elements were implemented with fidelity, and staff endorsed implementation ease/feasibility and acceptability. Regression analysis demonstrated that participation in the intervention (vs. control) was associated with an increase of 8.17 min of daily accelerometry-measured MVPA (56 min of additional weekly MVPA) at post-intervention controlling for baseline MVPA, school, gender, and weight status. The results provide support for social-motivational climate-based interventions for increasing MVPA in underserved youth that can inform future school-based health initiatives.Trial Registration: NCT03850821: https://clinicaltrials.gov/ct2/show/study/NCT03850821?term=NCT03850821&rank=1.

Feasibility of a multicomponent cognitive behavioral intervention for fear of falling after hip fracture: process evaluation of the FIT-HIP intervention

BACKGROUND

This study describes the process evaluation of an intervention developed to reduce fear of falling (FoF) after hip fracture, within an inpatient geriatric rehabilitation setting. This 'FIT-HIP intervention' is a multicomponent cognitive behavioral intervention, conducted by physiotherapists and embedded in usual care in geriatric rehabilitation in the Netherlands. A previous study (cluster randomized controlled trial) showed no beneficial effects of this intervention when compared to usual care. The aim of this study was to gain insight into factors related to the intervention process that may have influenced the effectiveness of the intervention.

METHODS

This process evaluation was conducted using an observational prospective study design. Based on quantitative and qualitative data derived from session logs, evaluation questionnaires and interviews, we addressed: 1] recruitment and reach; 2] performance according to protocol; 3] patients' adherence; and 4] opinions of patients and facilitators on the intervention. Participants in this study were: a) patients from 6 geriatric rehabilitation units, who were invited to participate in the intervention (39 adults aged ≥65 years with hip fracture and FoF) and; b) intervention facilitators (14 physiotherapists and 8 psychologists who provide coaching to the physiotherapists).

RESULTS

Thirty-six patients completed the intervention during inpatient geriatric rehabilitation. Apart from cognitive restructuring and telephonic booster (which was not provided to all patients), the intervention was performed to a fair degree in accordance with protocol. Patients' adherence to the intervention was very good, and patients rated the intervention positively (average 8.1 on a scale 0-10). Although most facilitators considered the intervention feasible, a limited level of FoF (possibly related to timing of intervention), and physiotherapists' limited experience with cognitive restructuring were identified as important barriers to performing the intervention according to protocol.

CONCLUSIONS

The FIT-HIP intervention was only partly feasible, which may explain the lack of effectiveness in reducing FoF. To improve the intervention's feasibility, we recommend selecting patients with maladaptive FoF (i.e. leading to activity restriction), being more flexible in the timing of the intervention, and providing more support to the physiotherapists in conducting cognitive restructuring.

TRIAL REGISTRATION

Netherlands Trial Register: NTR5695 (7 March 2016).

Implementing a systems approach to suicide prevention in a mental health service using the Zero Suicide Framework

OBJECTIVE

The Zero Suicide Framework, a systems approach to suicide prevention within a health service, is being implemented across a number of states in Australia, and internationally, although there is limited published evidence for its effectiveness. This paper aims to provide a description of the implementation process within a large health service in Australia and describes some of the outcomes to date and learnings from this process.

METHOD

Gold Coast Mental Health and Specialist Services has undertaken an implementation of the Zero Suicide Framework commencing in late 2015, aiming for high fidelity to the seven key elements. This paper describes the practical steps undertaken by the service, the new practices embedded, emphasis on supporting staff following the principles of restorative just culture and the development of an evaluation framework to support a continuous quality improvement approach.

RESULTS

Improvements have been demonstrated in terms of processes implementation, enhanced staff skills and confidence, positive cultural change and innovations in areas such as the use of machine learning for identification of suicide presentations. A change to 'business as usual' has benefited thousands of consumers since the implementation of a Suicide Prevention Pathway in late 2016 and achieved reductions in rates of repeated suicide attempts and deaths by suicide in Gold Coast Mental Health and Specialist Services consumers.

CONCLUSION

An all-of-service, systems approach to suicide prevention with a strong focus on cultural shifts and aspirational goals can be successfully implemented within a mental health service with only modest additional resources when supported by engaged leadership across the organisation. A continuous quality improvement approach is vital in the relentless pursuit of zero suicides in healthcare.

The Cardiac Care Bridge randomized trial in high-risk older cardiac patients: A mixed-methods process evaluation

Aim

To evaluate healthcare professionals' performance and treatment fidelity in the Cardiac Care Bridge (CCB) nurse‐coordinated transitional care intervention in older cardiac patients to understand and interpret the study results.

Design

A mixed‐methods process evaluation based on the Medical Research Council Process Evaluation framework.

Methods

Quantitative data on intervention key elements were collected from 153 logbooks of all intervention patients. Qualitative data were collected using semi‐structured interviews with 19 CCB professionals (cardiac nurses, community nurses and primary care physical therapists), from June 2017 until October 2018. Qualitative data‐analysis is based on thematic analysis and integrated with quantitative key element outcomes. The analysis was blinded to trial outcomes. Fidelity was defined as the level of intervention adherence.

Results

The overall intervention fidelity was 67%, ranging from severely low fidelity in the consultation of in‐hospital geriatric teams (17%) to maximum fidelity in the comprehensive geriatric assessment (100%). Main themes of influence in the intervention performance that emerged from the interviews are interdisciplinary collaboration, organizational preconditions, confidence in the programme, time management and patient characteristics. In addition to practical issues, the patient's frailty status and limited motivation were barriers to the intervention.

Conclusion

Although involved healthcare professionals expressed their confidence in the intervention, the fidelity rate was suboptimal. This could have influenced the non‐significant effect of the CCB intervention on the primary composite outcome of readmission and mortality 6 months after randomization. Feasibility of intervention key elements should be reconsidered in relation to experienced barriers and the population.

Impact

In addition to insight in effectiveness, insight in intervention fidelity and performance is necessary to understand the mechanism of impact. This study demonstrates that the suboptimal fidelity was subject to a complex interplay of organizational, professionals' and patients' issues. The results support intervention redesign and inform future development of transitional care interventions in older cardiac patients.

Formative Evaluation of Open Goals: A UK Community-Based Multi-Sport Family Programme

Community parks provide opportunities for physical activity (PA) and facilitate social interactions. This formative evaluation assesses the implementation of ‘Open Goals’ (OG), a novel multi-sport programme aiming to increase family PA and community cohesion, delivered weekly by Liverpool Football Club’s charitable foundation to local parks in Liverpool, North West England. Three Open Goals parks were chosen for the evaluation settings. Formative evaluation measures included: System for Observing Play and Recreation in Communities (SOPARC) observations (n = 10), direct session observations (n = 8), semi-structured interviews with Open Goals coaching staff (n = 3), and informal feedback from families (n = 5) about their experiences of Open Goals. Descriptive statistics and thematic analysis were applied to quantitative and qualitative data, respectively. Within the three evaluation parks, Open Goals reached 107 participants from May–July 2019, through 423 session attendances. Fidelity of the programme was high (M = 69% of session content delivered as intended). Overall park use when OG was offered compared to when it was not offered was not statistically significant (p = 0.051), however, target area use was significantly increased (p = 0.001). Overall physical activity levels in parks were significantly (p = 0.002) higher when Open Goals was being offered, compared to when it was not. Coaches reported that engagement in OG positively affected family co-participation and children’s behavioural development. Contextual issues included environmental and social barriers to programme engagement, including the co-participation element of the programme and criticism of the marketing of OG. It is evident that community-based multi-sport PA programmes endorsed by professional football clubs are well positioned to connect with local communities in deprived areas and to encourage PA and community engagement. This study suggests that such programmes may have the ability to improve park usage in specific areas, along with improving physical activity levels among families, although further research is required. Effective marketing strategies are needed for promotional purposes. Upskilling of coaches in the encouragement of family co-participation may support regular family engagement in PA in local parks.

Formative Process Evaluation of the "Connect" Physical Activity Feasibility Trial for Adolescents

Most interventions do not reach full implementation in real-world settings. Due to this issue, formative process evaluation during pilot programs can be especially useful to understand implementation strengths and areas for improvement so that full implementation can be reached in future iterations. This study demonstrated how a formative process evaluation of the Connect through Positive Leisure Activities for Youth (Connect) pilot informed course corrections for year 2 implementation. Connect is an intervention to promote a positive social motivational climate for physical activity (PA) in pre-existing after school programs. Connect ran 3 days a week for 8 weeks and had 2 components: a 30-minute “Get-to-Know-You” (GTKY) session and a 60-minute PA session. Formative evaluation was assessed using an observational tool and staff surveys. Changes in youth PA during program hours was assessed as a process outcome using the System for Observing Children’s Activity and Relationships during Play (SOCARP). All Connect essential elements were assessed with the observational tool including (a) social goal-oriented support; (b) collaborative, cooperative play; (c) equal treatment/access; and (d) an inclusive and engaging climate. Adequate dose was achieved on all items in all sessions. Although GTKY and PA sessions both reached high fidelity in promoting equal treatment and access, success in reaching fidelity varied for the 3 remaining essential elements. Post-intervention staff surveys indicated acceptability/adoptability of the Connect program and SOCARP observations indicated significant increases in PA from baseline to post-intervention. Changes for year 2 implementation based on the findings are discussed.

Protocol for a mixed methods process evaluation of a hybrid implementation-effectiveness trial of a scaled-up whole-school physical activity program for adolescents: Physical Activity 4 Everyone (PA4E1)

Background

Physical Activity 4 Everyone (PA4E1) is a physical activity program for secondary schools located in low-socioeconomic areas. Over a 24-month period, schools in the program arm of a cluster randomised controlled trial (n = up to 38 schools) will receive a multi-component implementation support strategy to embed the seven school physical activity practices of the PA4E1 program. This article describes the process evaluation of the PA4E1 hybrid implementation-effectiveness trial. The process evaluation aims to describe the fidelity and reach of the implementation support strategies using quantitative data; and to describe the acceptability, appropriateness and feasibility of the implementation support strategies and physical activity practices to school stakeholders using mixed methods.

Methods

Quantitative and qualitative data will be collected from participants (Physical Education teachers, in-School Champions, students) in the program arm. Data collection will involve semi-structured interviews, focus groups, a fidelity monitoring log, a fidelity checklist, surveys, and routinely collected administrative and website data. Quantitative data will be analysed descriptively and qualitative data will be analysed thematically within and across data sets. Triangulation between data sources will be used to synthesise findings regarding the implementation and potential mechanisms of impact of PA4E1 on school physical activity practice adoption, with respect to context.

Discussion

Results of the process evaluation will facilitate the interpretation of the findings of the trial outcomes. It will comprehensively describe what was actually implemented and identify the potential contribution of the various components of the implementation support strategy to the school physical activity practice adoption outcomes. Findings will inform future improvement and scale-up of PA4E1 and approaches to implementing secondary school-based physical activity programs more broadly.

Trial registration

Australian New Zealand Clinical Trials Registry ACTRN12617000681358 registered 12 May 2017.

Effects of the Girls on the Move randomized trial on adiposity and aerobic performance (secondary outcomes) in low-income adolescent girls

BACKGROUND

Limited, mixed evidence exists regarding the effectiveness of physical activity interventions on adiposity and aerobic performance in adolescent underrepresented populations.

OBJECTIVE

To examine effects of Girls on the Move on body mass index z-scores (BMI-z), percent (%) body fat, and aerobic performance in fifth- to eighth-grade underrepresented girls.

METHODS

A group randomized trial, involving 12 intervention and 12 control schools in low-income areas, was conducted. Participants (n = 1519) were low-active girls. The 17-week intervention included (a) a physical activity club, (b) two motivational interviewing sessions, and (c) one Internet-based session. BMI-z was determined from measured height and weight; % body fat was assessed using bioelectric impedance. Aerobic performance was assessed using a shuttle run. Demographics, physical activity (accelerometer), and pubertal development were assessed. Linear mixed models, adjusting for baseline, were used to examine group differences in postintervention.

RESULTS

No significant between-group differences in BMI-z existed at postintervention, but % body fat increased less among intervention than control group girls (M_change = 0.43% vs 0.73%). Aerobic performance decreased less in intervention vs control (M_change = -0.39 vs -0.57).

CONCLUSIONS

Although the intervention positively impacted % body fat and aerobic performance in underrepresented girls, more research is necessary to determine optimal implementation for yielding greater effects.

Are school-based physical activity interventions effective and equitable? A meta-analysis of cluster randomized controlled trials with accelerometer-assessed activity

The prevalence of childhood obesity is increasing at epidemic rates globally, with widening inequalities between advantaged and disadvantaged groups. Despite the promise of schools as a universal context to access and influence all children, the potential of school-based interventions to positively impact children's physical activity behaviour, and obesity risk, remains uncertain. We searched six electronic databases to February 2017 for cluster randomized trials of school-based physical activity interventions. Following data extraction, authors were sent re-analysis requests. For each trial, a mean change score from baseline to follow-up was calculated for daily minutes of accelerometer-assessed moderate-to-vigorous physical activity (MVPA), for the main effect, by gender, and by socio-economic position (SEP). Twenty-five trials met the inclusion criteria; 17 trials provided relevant data for inclusion in the meta-analyses. The pooled main effect for daily minutes of MVPA was nonexistent and nonsignificant. There was no evidence of differential effectiveness by gender or SEP. This review provides the strongest evidence to date that current school-based efforts do not positively impact young people's physical activity across the full day, with no difference in effect across gender and SEP. Further assessment and maximization of implementation fidelity is required before it can be concluded that these interventions have no contribution to make.

Assessing fidelity: balancing methodology and reality in jail interventions

BACKGROUND

While fidelity to research protocols is important to ensure generalizable outcomes, interventions in criminal justice settings present unique challenges to uniform implementation. The goal of this paper is to describe the fidelity methods and outcomes for a sexual health intervention implemented in three local county jails.

METHODS

As part of a longitudinal cohort study, four trained fidelity assessors observed 25 of the 230 sessions presented (including both intervention and comparison groups) at three separate times during the 29 months of the intervention. Assessment methods included the assessors' field notes, a nine-item facilitator quality scale and a content inclusion scale with 6-13 items specific for each of the five sessions.

RESULTS

Facilitator quality score ranged from 87.6 to 99.2%. Content inclusion scores ranged from 77.3 to 88%. Specific challenges to fidelity were found in two areas: the jail environment and the participants' response to content.

CONCLUSIONS

The realities of conducting research in jails and prisons must be addressed in real time by adjusting program content to fit both unexpected facility and participant situations. Skilled facilitators are essential to this effort.

Exposure to a multi-level multi-component childhood obesity prevention community-randomized controlled trial: patterns, determinants, and implications

BACKGROUND

For community interventions to be effective in real-world conditions, participants need to have sufficient exposure to the intervention. It is unclear how the dose and intensity of the intervention differ among study participants in low-income areas. We aimed to understand patterns of exposure to different components of a multi-level multi-component obesity prevention program to inform our future impact analyses.

METHODS

B'more Healthy Communities for Kids (BHCK) was a community-randomized controlled trial implemented in 28 low-income zones in Baltimore in two rounds (waves). Exposure to three different intervention components (corner store/carryout restaurants, social media/text messaging, and youth-led nutrition education) was assessed via post-intervention interviews with 385 low-income urban youths and their caregivers. Exposure scores were generated based on self-reported viewing of BHCK materials (posters, handouts, educational displays, and social media posts) and participating in activities, including taste tests during the intervention. For each intervention component, points were assigned for exposure to study materials and activities, then scaled (0-1 range), yielding an overall BHCK exposure score [youths: mean 1.1 (range 0-7.6 points); caregivers: 1.1 (0-6.7), possible highest score: 13]. Ordered logit regression analyses were used to investigate correlates of youths' and caregivers' exposure level (quartile of exposure).

RESULTS

Mean intervention exposure scores were significantly higher for intervention than comparison youths (mean 1.6 vs 0.5, p < 0.001) and caregivers (mean 1.6 vs 0.6, p < 0.001). However, exposure scores were low in both groups and 10% of the comparison group was moderately exposed to the intervention. For each 1-year increase in age, there was a 33% lower odds of being highly exposed to the intervention (odds ratio 0.77, 95% confidence interval 0.69; 0.88) in the unadjusted and adjusted model controlling for youths' sex and household income.

CONCLUSION

Treatment effects may be attenuated in community-based trials, as participants may be differentially exposed to intervention components and the comparison group may also be exposed. Exposure should be measured to provide context to impact evaluations in multi-level trials. Future analyses linking exposure scores to the outcome should control for potential confounders in the treatment-on-the-treated approach, while recognizing that confounding and selection bias may exist affecting causal inference.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT02181010 . Retrospectively registered on 2 July 2014.

Promoting Social Nurturance and Positive Social Environments to Reduce Obesity in High-Risk Youth

Nurturing environments within the context of families, schools, and communities all play an important role in enhancing youth's behavioral choices and health outcomes. The increasing prevalence rates of obesity among youth, especially among low income and ethnic minorities, highlight the need to develop effective and innovative intervention approaches that promote positive supportive environments across different contexts for at-risk youth. We propose that the integration of Social Cognitive Theory, Family Systems Theory, and Self-Determination Theory offers a useful framework for understanding how individual, family, and social-environmental-level factors contribute to the development of nurturing environments. In this paper, we summarize evidence-based randomized controlled trials that integrate positive parenting, motivational, and behavioral skills strategies in different contexts, including primary care, home, community, and school-based settings. Taken together, these studies suggest that youth and parents are most likely to benefit when youth receive individual-level behavioral skills, family-level support and communication, and autonomous motivational support from the broader social environment. Future investigators and healthcare providers should consider integrating these evidence-based approaches that support the effects of positive social climate-based interventions on promoting healthy eating, physical activity, and weight management in youth.

Physical activity and nutrition program for adults with metabolic syndrome: Process evaluation

INTRODUCTION

The Vietnam Physical Activity and Nutrition (VPAN) program aimed to improve physical activity and nutrition for adults aged 50-65 years with Metabolic Syndrome in Vietnam. The VPAN program consisted of a range of resources and strategies, including an information booklet, resistance band, face-to-face education sessions, and walking groups. This process evaluation assessed the participation, fidelity, satisfaction, and reasons for completing and not-completing the VPAN.

METHODS

Data were collected by mixed-methods from a sample of 214 intervention participants. Quantitative data were collected via surveys (n=163); qualitative data via face-to-face exit interviews with intervention program completers (n=10) and non-completers (n=10), and brief post education session discussions.

RESULTS

Most participants (87%-96%) reported the program resources and strategies useful, assisting them to increase their physical activity level and improving their diet. The education sessions were the most preferred strategy (97%) with high attendance (>78% of participants). The main reasons for withdrawal were work commitments and being too busy.

CONCLUSION

The evaluation indicated that the program reached and engaged the majority of participants throughout the six-month intervention. The combination of printed resources and face-to-face intervention components was a suitable approach to support lifestyle behavioural change in the Vietnamese population.

Process evaluation of a preschool physical activity intervention using web-based delivery

BACKGROUND AND PURPOSE

Preschool/childcare settings offer a practical target for physical activity interventions. Online learning programs have the potential for greater public health reach and impact. The SHAPES-Dissemination (SHAPES-D) project adapted the original SHAPES in-person intervention for online delivery to teachers. The purpose of this paper is to describe the implementation monitoring and process evaluation for the SHAPES-D project.

METHODS

Nine preschools with 26 classrooms participated. A total of 41 teachers were trained via online learning to implement the SHAPES-D program in their classrooms. The dose received, completeness, and fidelity of implementation were assessed through website metrics, teacher surveys and interviews, and classroom observations.

RESULTS

Dose received was adequate (73%). Observed completeness and physical activity enjoyment fidelity were high (100%), although moderate-to-vigorous physical activity fidelity and social environment fidelity were low (25% each). Overall implementation was high (91%).

DISCUSSION

Results indicate that the online method of delivery is viable for dissemination. The online delivery system provides an easy method of monitoring dose received. This may be the first structural intervention to monitor dose received through web metrics.

CONCLUSION

The adaptation of an in-person intervention to an online delivery system increases the potential for dissemination of a successful program to increase physical activity in preschool settings.

A self-determination theory and motivational interviewing intervention to decrease racial/ethnic disparities in physical activity: rationale and design

Background

Although the mental and physical benefits of physical activity are well-established, there is a racial/ethnic disparity in activity such that minorities are much less likely to engage in physical activity than are White individuals. Research suggests that a lack of motivation may be an important barrier to physical activity for racial/ethnic minorities. Therefore, interventions that increase participants’ motivation may be especially useful in promoting physical activity within these groups. Physical activity interventions that utilized the clinical technique of motivational interviewing (MI) in conjunction with the theoretical background of self-determination theory (SDT) have been effective in increasing White individuals’ physical activity. Nevertheless, it remains unclear the extent to which these results apply to minority populations.

Methods/Design

The current study involves conducting a 12-week physical activity intervention based on SDT and MI to promote physical activity in a racially/ethnically-diverse sample. It is hypothesized that this intervention will successfully increase physical activity in participants. Specifically, it is expected that minorities will experience a greater relative increase in physical activity than Whites within the intervention group because minorities are expected to have lower baseline levels of activity.

Discussion

Results from this study will give us a greater understanding of the generalizability of SDT interventions designed to improve motivation for physical activity and level of physical activity.

Trial registration

Clinical Trials Gov. Identifier NCT02250950 Registered 24 September 2014.

Examining reach, dose, and fidelity of the "Girls on the Move" after-school physical activity club: a process evaluation

Background

After-school programs represent a promising opportunity to assist adolescent girls’ in attaining adequate physical activity. Although evaluating the process of intervention implementation is important for determining if an intervention was delivered and received as intended, comprehensive information about process evaluation methods and results are rarely reported. The purpose of this article was to evaluate the reach, dose, and fidelity of a 90-minute after-school physical activity club offered 3 days a week. The club is 1 of 3 components included in a 17-week intervention designed for 5th-8th grade girls, the majority of whom were of minority and/or low socioeconomic status.

Methods

A total of 24 schools (12 intervention; 12 control) and 56–67 girls per school (total N = 1519 girls) were included in the Girls on the Move group randomized controlled trial. At the beginning of each of 3 academic years (2012–2015), 8 schools per year were randomized to receive either the intervention (n = 4) or control condition (n = 4). To evaluate the club, data collected via surveys from girls, club coaches and managers, and process evaluators were analyzed. To evaluate the opportunity for physical activity provided by the coaches and managers, process evaluators used an observation tool based on the System for Observing Fitness Instruction Time and Academic Learning Time - Physical Education. Girls wore accelerometers every other week during the club time.

Results

Mean attendance was 41 % with the average attendance in year 3 being higher than rates for years 1 or 2. Mean moderate-to-vigorous physical activity time was 21.85 minutes measured via accelerometry and 21.81 minutes observed by process evaluators. Satisfaction with the intervention was high. For the most part, process evaluators perceived the club was delivered as planned and reflected constructs of the Health Promotion Model and Self-Determination Theory. Areas contributing to success included using incentives and offering a variety of activities. Issues negatively impacting implementation included managing behavioral problems, having limited space for moderate-to-vigorous physical activity, dealing with inclement weather, and getting coaches to actively participate in all physical activities with the girls.

Conclusions

This process evaluation provides important information to guide future school-based physical activity intervention delivery. Barriers to implementation have been identified. Ways to overcome them warrant consideration when designing physical activity interventions. Research is needed to test innovative approaches for enhancing attendance and increasing girls’ moderate-to-vigorous physical activity in after-school programs.

Trial registration

ClinicalTrials.gov Identifier NCT01503333. Registered 23 December, 2011.

Integrating care for frequent users of emergency departments: implementation evaluation of a brief multi-organizational intensive case management intervention

Background

Addressing the needs of frequent users of emergency departments (EDs) is a health system priority in many jurisdictions. This study describes stakeholder perspectives on the implementation of a multi-organizational brief intervention designed to support integration and continuity of care for frequent ED users with mental health and addictions problems, focusing on perceived barriers and facilitators to early implementation in a large urban centre.

Methods

Coordinating Access to Care from Hospital Emergency Departments (CATCH-ED) is a brief case management intervention bridging hospital, primary and community care for frequent ED users experiencing mental illness and addictions. To examine barriers and facilitators to early implementation of this multi-organizational intervention, between July and October 2012, 47 stakeholders, including direct service providers, managers and administrators participated in 32 semi-structured qualitative interviews and one focus group exploring their experience with the intervention and factors that helped or hindered successful early implementation. Qualitative data were analyzed using thematic analysis.

Results

Stakeholders valued the intervention and its potential to support continuity of care for this population. Service delivery system factors, including organizational capacity and a history of collaborative relationships across the healthcare continuum, and support system factors, such as training and supervision, emerged as key facilitators of program implementation. Operational challenges included early low program referral rates, management of a multi-organizational initiative, variable adherence to the model among participating organizations, and scant access to specialty psychiatric resources. Factors contributing to these challenges included lack of dedicated staff in the ED and limited local system capacity to support this population, and insufficient training and technical assistance available to participating organizations.

Conclusions

A multi-organizational brief intervention is an acceptable model to support integration of hospital, primary and community care for frequent ED users. The study highlights the importance of early implementation evaluation to identify potential solutions to implementation barriers that may be applicable to many jurisdictions.

Process evaluation of the Bristol girls dance project

Background

The Bristol Girls Dance Project was a cluster randomised controlled trial that aimed to increase objectively measured moderate-to-vigorous physical activity (MVPA) levels of Year 7 (age 11–12) girls through a dance-based after-school intervention. The intervention was delivered in nine schools and consisted of up to forty after-school dance sessions. This paper reports on the main findings from the detailed process evaluation that was conducted.

Methods

Quantitative and qualitative data were collected from intervention schools. Dose and fidelity were reported by dance instructors at every session. Intervention dose was defined as attending two thirds of sessions and was measured by attendance registers. Fidelity to the intervention manual was reported by dance instructors. On four randomly-selected occasions, participants reported their perceived level of exertion and enjoyment. Reasons for non-attendance were self-reported at the end of the intervention. Semi-structured interviews were conducted with all dance instructors who delivered the intervention (n = 10) and school contacts (n = 9) in intervention schools. A focus group was conducted with girls who participated in each intervention school (n = 9).

Results

The study did not affect girls’ MVPA. An average of 31.7 girls participated in each school, with 9.1 per school receiving the intervention dose. Mean attendance and instructors’ fidelity to the intervention manual decreased over time. The decline in attendance was largely attributed to extraneous factors common to after-school activities. Qualitative data suggest that the training and intervention manual were helpful to most instructors. Participant ratings of session enjoyment were high but perceived exertion was low, however, girls found parts of the intervention challenging.

Conclusions

The intervention was enjoyed by participants. Attendance at the intervention sessions was low but typical of after-school activities. Participants reported that the intervention brought about numerous health and social benefits and improved their dance-based knowledge and skills. The intervention could be improved by reducing the number of girls allowed to participate in each school and providing longer and more in-depth training to those delivering the intervention.

Trial registration

ISRCTN52882523 Registered 25th April 2013.

Process Evaluation of the Project SHINE Intervention for African American Families: An Integrated Positive Parenting and Peer Monitoring Approach to Health Promotion

This study describes the process evaluation of Project SHINE, a randomized family-based health promotion intervention that integrated parenting and peer monitoring for improving sedentary behavior, physical activity, and diet in African American families. Adolescent-parent dyads (n = 89) were randomized to a 6-week behavioral, positive parenting, and peer monitoring skills intervention or a general health education comparison condition. Process evaluation included observational ratings of fidelity, attendance records, psychosocial measures, and qualitative interviews. Results indicated that the intervention was delivered with high fidelity based on facilitator adherence (>98% of content delivered) and competent use of theoretically based behavior change and positive parenting skills (100% of ratings >3 on a 1-4 scale). Although only 43% of peers attended the "bring a friend" session, overall attendance was high (4.39 ± 1.51 sessions) as was the retention rate (88%). Parents in the intervention condition reported significant improvements in communication related to adolescents' engagement in health behaviors both on their own and with peers. These findings were supported by qualitative themes related to improvements in family communication and connectedness. This study provides an innovative example of how future family-based health promotion trials can expand their process evaluation approaches by assessing theoretically relevant positive parenting variables as part of ongoing monitoring.

Figuring out fidelity: a worked example of the methods used to identify, critique and revise the essential elements of a contextualised intervention in health policy agencies

BACKGROUND

In this paper, we identify and respond to the fidelity assessment challenges posed by novel contextualised interventions (i.e. interventions that are informed by composite social and psychological theories and which incorporate standardised and flexible components in order to maximise effectiveness in complex settings). We (a) describe the difficulties of, and propose a method for, identifying the essential elements of a contextualised intervention; (b) provide a worked example of an approach for critiquing the validity of putative essential elements; and (c) demonstrate how essential elements can be refined during a trial without compromising the fidelity assessment. We used an exploratory test-and-refine process, drawing on empirical evidence from the process evaluation of Supporting Policy In health with Research: an Intervention Trial (SPIRIT). Mixed methods data was triangulated to identify, critique and revise how the intervention's essential elements should be articulated and scored.

RESULTS

Over 50 provisional elements were refined to a final list of 20 and the scoring rationalised. Six (often overlapping) challenges to the validity of the essential elements were identified. They were (1) redundant-the element was not essential; (2) poorly articulated-unclear, too specific or not specific enough; (3) infeasible-it was not possible to implement the essential element as intended; (4) ineffective-the element did not effectively deliver the change principles; (5) paradoxical-counteracting vital goals or change principles; or (6) absent or suboptimal-additional or more effective ways of operationalising the theory were identified. We also identified potentially valuable 'prohibited' elements that could be used to help reduce threats to validity.

CONCLUSIONS

We devised a method for critiquing the construct validity of our intervention's essential elements and modifying how they were articulated and measured, while simultaneously using them as fidelity indicators. This process could be used or adapted for other contextualised interventions, taking evaluators closer to making theoretically and contextually sensitive decisions upon which to base fidelity assessments.

Behavior matters: the relevance, impact, and reach of behavioral medicine

BACKGROUND/PURPOSE

Growing evidence suggests behavioral interventions that target a few key behaviors may be effective at improving population-level health outcomes; health status indicators; social, economic, and physical environments; personal capacity; and biological outcomes. A theoretical framework that targets both social and cognitive mechanisms of behavioral interventions is outlined as critical for understanding "ripple effects" of behavioral interventions on influencing a broad range of outcomes associated with improved health and well-being.

METHODS/RESULTS

Evidence from randomized controlled trials is reviewed and demonstrates support for ripple effects-the effects that behavioral interventions have on multiple outcomes beyond the intended primary target of the interventions. These outcomes include physical, psychological, and social health domains across the lifespan.

CONCLUSIONS

Cascading effects of behavioral interventions have important implications for policy that argue for a broader conceptualization of health that integrates physical, mental, and social well-being outcomes into future research to show the greater return on investment.

A multilevel approach to examining time-specific effects in accelerometer-assessed physical activity

OBJECTIVES

Popular methods for analyzing accelerometer data often use a single physical activity outcome variable such as average-weekly or total physical activity. These approaches limit the types of research questions that can be answered and fail to utilize the detailed, time-specific information available from accelerometers. This study proposes the use of multilevel modeling, which tested intervention effects at specific time periods.

DESIGN

The motivating example was the Active by Choice Today trial. Simulations were used to test whether the application of time-specific hypotheses about when physical activity intervention treatment effects were expected to occur (e.g., after-school hours) increased power to detect effects compared to traditional methods.

METHODS

Six simulation conditions were tested: (1) no treatment effects (to test the type 1 error rate), (2) time-specific effects, but no traditionally-tested effects, (3) traditionally-tested effects, but no time-specific effects, and (4) combinations of traditional and time-specific effects in 3 proportions.

RESULTS

Results showed the proposed multilevel approach demonstrated appropriate type 1 error rates and increased power to detect treatment effects during hypothesized times by 31-38 percentage points compared to traditional approaches. This was consistent across varying proportions of traditional versus time-specific effects, and there was no loss of power using the multilevel approach when only traditional effects were present.

CONCLUSIONS

The current study showed potential advantages of testing time-specific hypotheses about intervention effects using a multilevel time-specific approach. This approach may show intervention effects when traditional approaches do not. Future research should explore the application of this additional analytic tool for accelerometer physical activity estimates.

Development of an innovative process evaluation approach for the Families Improving Together (FIT) for weight loss trial in African American adolescents

This study demonstrates how a multi-theoretical, multilevel process evaluation was used to assess implementation of the Families Improving Together (FIT) for weight loss intervention. FIT is a randomized controlled trial evaluating a culturally tailored, motivational plus family-based program on weight loss in African American adolescents and their parents. Social Cognitive, Self Determination, Family Systems theories and cultural tailoring principles guided the conceptualization of essential elements across individual/family, facilitator, and group levels. Data collection included an observational rating tool, attendance records, and a validated psychosocial measure.

RESULTS

Attendance records (0=absent, 1=present, criteria=≥70%) indicated that 71.5% of families attended each session. The survey (1=false, 6=true, criteria=≥4.5) indicated that participants perceived a positive group climate (M=5.16, SD=0.69). A trained evaluator reported that facilitator dose delivered (0=no, 1=yes, criteria=≥75%) was high (99.6%), and fidelity (1=none to 4=all, criteria=≥3) was adequate at facilitator (M=3.63, SD=0.41) and group levels (M=3.35, SD=0.49). Five cultural topics were raised by participants related to eating (n=3) and physical activity (n=2) behaviors and were integrated as part of the final curriculum.

DISCUSSION

Results identify areas for program improvement related to delivery of multi-theoretical and cultural tailoring elements. Findings may inform future strategies for implementing effective weight loss programs for ethnic minority families.

An overview of the Families Improving Together (FIT) for weight loss randomized controlled trial in African American families

BACKGROUND

The Families Improving Together (FIT) randomized controlled trial tests the efficacy of integrating cultural tailoring, positive parenting, and motivational strategies into a comprehensive curriculum for weight loss in African American adolescents. The overall goal of the FIT trial is to test the effects of an integrated intervention curriculum and the added effects of a tailored web-based intervention on reducing z-BMI in overweight African American adolescents.

DESIGN AND SETTING

The FIT trial is a randomized group cohort design the will involve 520 African American families with an overweight adolescent between the ages of 11-16 years. The trial tests the efficacy of an 8-week face-to-face group randomized program comparing M + FWL (Motivational Plus Family Weight Loss) to a comprehensive health education program (CHE) and re-randomizes participants to either an 8-week on-line tailored intervention or control on-line program resulting in a 2 (M + FWL vs. CHE group) × 2 (on-line intervention vs. control on-line program) factorial design to test the effects of the intervention on reducing z-BMI at post-treatment and at 6-month follow-up.

INTERVENTION

The interventions for this trial are based on a theoretical framework that is novel and integrates elements from cultural tailoring, Family Systems Theory, Self-Determination Theory and Social Cognitive Theory. The intervention targets positive parenting skills (parenting style, monitoring, communication); cultural values; teaching parents to increase youth motivation by encouraging youth to have input and choice (autonomy-support); and provides a framework for building skills and self-efficacy through developing weight loss action plans that target goal setting, monitoring, and positive feedback.

Increasing children's physical activity through a teaching-assistant led extracurricular intervention: process evaluation of the action 3:30 randomised feasibility trial

BACKGROUND

Many children do not engage in recommended levels of physical activity (PA), highlighting the need to find ways to increase children's PA. Process evaluations play an important role in improving the science of randomised controlled trials. We recently reported the results of the Action 3:30 cluster randomised feasibility trial illustrating higher levels of moderate to vigorous intensity PA among boys but not girls. The aim of this paper is to report the process evaluation results including intervention fidelity, implementation, context and how intervention components and trial design could be improved before proceeding to a definitive RCT.

METHODS

Children's session enjoyment was assessed every two weeks. Reasons for non-attendance were provided by questionnaire at the end of the intervention. Post intervention interviews were held with participating teaching assistants (TAs) and school key contacts (KCs), and focus groups were conducted with children in all 10 intervention schools. Interviews and focus groups examined how recruitment and session attendance might be improved and established which elements of the programme that were and were not well received.

RESULTS

Data indicated good intervention fidelity with TA's adopting enjoyment-focussed teaching styles and the sessions improving children's skills and self-esteem. Several positive aspects of implementation were identified, including high session variety, the opportunity to work in teams, the child-led sessions and the engaging leader style. In terms of context there was evidence that TA's faced difficulties managing challenging behaviour and that further training in this area was needed. TAs and KCs felt that recruitment could be improved by providing taster sessions during PE lessons and clarifying the days that the clubs would run at the point of recruitment. The programme could be improved to enhance interest for girls, by including training for managing disruptive behaviour and making some activities more age-group appropriate.

CONCLUSIONS

Action 3.30 showed promise but could be improved by ensuring age appropriate activities, providing more appeal to girls and improving recruitment through taster sessions and early establishment of days of the week it is to be offered on.

TRIAL REGISTRATION

ISRCTN58502739.

Ethnographic process evaluation in primary care: explaining the complexity of implementation

Background

The recent growth of implementation research in care delivery systems has led to a renewed interest in methodological approaches that deliver not only intervention outcome data but also deep understanding of the complex dynamics underlying the implementation process. We suggest that an ethnographic approach to process evaluation, when informed by and integrated with quantitative data, can provide this nuanced insight into intervention outcomes. The specific methods used in such ethnographic process evaluations are rarely presented in detail; our objective is to stimulate a conversation around the successes and challenges of specific data collection methods in health care settings. We use the example of a translational clinical trial among 11 community clinics in Portland, OR that are implementing an evidence-based, health-information technology (HIT)-based intervention focused on patients with diabetes.

Discussion

Our ethnographic process evaluation employed weekly diaries by clinic-based study employees, observation, informal and formal interviews, document review, surveys, and group discussions to identify barriers and facilitators to implementation success, provide insight into the quantitative study outcomes, and uncover lessons potentially transferable to other implementation projects. These methods captured the depth and breadth of factors contributing to intervention uptake, while minimizing disruption to clinic work and supporting mid-stream shifts in implementation strategies. A major challenge is the amount of dedicated researcher time required.

Summary

The deep understanding of the ‘how’ and ‘why’ behind intervention outcomes that can be gained through an ethnographic approach improves the credibility and transferability of study findings. We encourage others to share their own experiences with ethnography in implementation evaluation and health services research, and to consider adapting the methods and tools described here for their own research.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-014-0607-0) contains supplementary material, which is available to authorized users.

What are successful recruitment and retention strategies for underserved populations? Examining physical activity interventions in primary care and community settings

The purposes of this review are to (1) describe recruitment and retention strategies for physical activity interventions focusing on underserved populations and (2) identify successful strategies which show the most promise for "best practices" recommendations to guide future research. The method used was systematic review. Data on recruitment and retention strategies were abstracted and analyzed according to participant characteristics, types of strategies used, and effectiveness using an ecological framework. Thirty-eight studies were identified. Populations included African American (n = 25), Hispanic (n = 8), or Asian (n = 3) groups. Successful recruitment strategies consisted of partnering with respected community stakeholders and organizations, well-trained study staff ethnically, linguistically, and culturally matched to the population of interest, and use of multiple advertising channels. Successful retention strategies included efficient administrative tracking of participants, persistence, skillful teamwork, and demonstrating a positive, caring attitude towards participants. Promising recruitment and retention strategies correspond to all levels of ecological influence: individual, interpersonal, organizational, and societal.

Improving contraceptive choice: fidelity of implementation and the gap between effectiveness and efficacy

Interventions to reduce unintended pregnancy through improved contraceptive use are a public health priority. A comprehensive process evaluation of a contraceptive assessment module intervention with demonstrated efficacy was undertaken. The 12-month process evaluation goal was to describe the extent to which the intervention was implemented as intended over time, and to identify programmatic adjustments to improve implementation fidelity. Quantitative and qualitative methods included staff surveys, electronic health record data, usage monitoring, and observations. Fidelity of implementation was low overall (<10% of eligible patients completed the entire module [dose received]). Although a midcourse correction making the module available in clinical areas led to increased dose delivered (23% vs. 30%, chi-square test p = .006), dose received did not increase significantly after this adjustment. Contextual factors including competing organizational and staff priorities and staff buy-in limited the level of implementation and precluded adoption of some strategies such as adjusting patient flow. Using a process evaluation framework enabled the research team to identify and address complexities inherent in effectiveness studies and facilitated the alignment of program and context.

The tug-of-war: fidelity versus adaptation throughout the health promotion program life cycle

Researchers across multiple fields have described the iterative and nonlinear phases of the translational research process from program development to dissemination. This process can be conceptualized within a "program life cycle" framework that includes overlapping and nonlinear phases: development, adoption, implementation, maintenance, sustainability or termination, and dissemination or diffusion, characterized by tensions between fidelity to the original plan and adaptation for the setting and population. In this article, we describe the life cycle (phases) for research-based health promotion programs, the key influences at each phase, and the issues related to the tug-of-war between fidelity and adaptation throughout the process using a fictionalized case study based on our previous research. This article suggests the importance of reconceptualizing intervention design, involving stakeholders, and monitoring fidelity and adaptation throughout all phases to maintain implementation fidelity and completeness. Intervention fidelity should be based on causal mechanisms to ensure effectiveness, while allowing for appropriate adaption to ensure maximum implementation and sustainability. Recommendations for future interventions include considering the determinants of implementation including contextual factors at each phase, the roles of stakeholders, and the importance of developing a rigorous, adaptive, and flexible definition of implementation fidelity and completeness.

Bringing rigor to evaluations of large-scale programs to improve infant and young child feeding and nutrition: the evaluation designs for the Alive & Thrive initiative

BACKGROUND

The evidence base on the impact of large-scale infant and young child feeding (IYCF) and nutrition programs is limited, partly due to the challenges of rigorously evaluating complex programs including multiple interventions.

OBJECTIVES

To describe the process used to design Alive & Thrive's impact evaluations in the three target countries and discuss the feasibility of developing contextually relevant designs adapted to the country-specific programmatic context.

METHODS

The evaluation designs for Alive & Thrive needed to address several challenges. These included the selection of intervention components to evaluate rigorously; the identification of appropriate comparison groups in the context of rapidly scaling-up programs; the choice of impact indicators; addressing measurement challenges related to evaluating the impact of interventions targeted during the first 2 years of life on stunting; and developing methods and tools to assess implementation, utilization, and program impact pathways within evolving program portfolios.

RESULTS

In Bangladesh and Vietnam, cluster-randomized probability designs are used for the impact evaluations; in Ethiopia, the impact evaluation uses an adequacy design. In all three countries, repeated cross-sectional surveys, 4 years apart, are used to measure impact, and appropriate age groups are sampled separately to capture change in the main impact indicators. In addition, theory-driven process evaluations are used to study factors that facilitate or prevent achievement of impact and scale.

CONCLUSIONS

We conclude that robust impact and process evaluations of complex, large-scale nutrition programs are feasible, but that early implementer-evaluator engagement and shared vision and motivation to establishing meaningful evaluations are essential.

Implementation of an educational intervention to improve hand washing in primary schools: process evaluation within a randomised controlled trial

Background

Process evaluations are useful for understanding how interventions are implemented in trial settings. This is important for interpreting main trial results and indicating how the intervention might function beyond the trial. The purpose of this study was to examine the reach, dose, fidelity, acceptability, and sustainability of the implementation of an educational hand washing intervention in primary schools, and to explore views regarding acceptability and sustainability of the intervention.

Methods

Process evaluation within a cluster randomised controlled trial, including focus groups with pupils aged 6 to 11, semi-structured interviews with teachers and external staff who coordinated the intervention delivery, and school reports and direct observations of the intervention delivery.

Results

The educational package was delivered in 61.4% of schools (85.2% of intervention schools, 37.8% of control schools following completion of the trial). Teachers and pupils reacted positively to the intervention, although concerns were raised about the age-appropriateness of the resources. Teachers adapted the resources to suit their school setting and pupils. Staff coordinating the intervention delivery had limited capacity to follow up and respond to schools.

Conclusions

The hand washing intervention was acceptable to schools, but its reach outside of a randomised trial, evidenced in the low proportion of schools in the control arm who received it after the trial had ended, suggests that the model of delivery may not be sustainable.

Trial registration

ISRCTN: ISRCTN93576146

Environmental and social-motivational contextual factors related to youth physical activity: systematic observations of summer day camps

BACKGROUND

Youth risk of obesity is high during the summer months. Summer day camps can be ideal settings for preventing obesity through reducing youth summer sedentary behaviors. However, with limited research on camp settings, the mechanisms by which these programs promote children's physical activity (PA) remains largely unknown. The current study was designed to take a first step in addressing this gap in research through systematic observations of 4 summer day camps.

METHODS

Systematic observations of 4 summer day camps was conducted using the System for Observing Play and Leisure Activity in Youth (SOPLAY) and a social-motivational climate supplemental observation tool founded on Self-Determination Theory and previous research developed by the authors. Teams of two coders observed daily activities for four days across two-week periods at each camp. On 15 minute intervals throughout each day, camps were assessed on level of youth PA (e.g., sedentary, moderate, vigorous), five physical features (e.g., equipment), eight staff interactions (e.g., encourage PA), and six social climate components (e.g., inclusive game).

RESULTS

Across the sample, highly engaging games [F(1,329) = 17.68, p < .001], positive peer interactions [F(1,329) = 8.43, p < .01], and bullying [F(1,329) = 9.39, p < .01] were significantly related to higher PA participation rates, and clarity of rules [F(1,329) = 11.12, p < .001] was related to fewer youth participating in PA. Separate analyses for males and females indicated some sex differences with highly engaging games [F(1,329) = 23.10, p < .001] and bullying [F(1,329) = 10.00, p < .01] related to males' but not females' PA, and positive peer interactions related to only females' PA [F(1,329) = 9.58, p < .01]. Small, yet significant physical-environmental effects of temperature [F(1,328) = 1.54, p < .05] and equipment [F(1,328) = 4.34, p = .05] for girls also suggests that activities offered indoors (which was most common during high temperatures), and provision of equipment may also be important considerations for promoting girls' PA. Staff behaviors were minimally predictive of youth PA.

CONCLUSIONS

This is the first study to conduct systematic observations of the physical and social resources of summer day camps and contributes to our understanding of the strengths and needs of camps to effectively promote PA in both boys and girls during the summer months when risks for obesity are high.

Using a systematic conceptual model for a process evaluation of a middle school obesity risk-reduction nutrition curriculum intervention: choice, control & change

OBJECTIVE

To use and review a conceptual model of process evaluation and to examine the implementation of a nutrition education curriculum, Choice, Control & Change, designed to promote dietary and physical activity behaviors that reduce obesity risk.

DESIGN

A process evaluation study based on a systematic conceptual model.

SETTING

Five middle schools in New York City.

PARTICIPANTS

Five hundred sixty-two students in 20 classes and their science teachers (n = 8).

MAIN OUTCOME MEASURES

Based on the model, teacher professional development, teacher implementation, and student reception were evaluated. Also measured were teacher characteristics, teachers' curriculum evaluation, and satisfaction with teaching the curriculum.

ANALYSIS

Descriptive statistics and Spearman ρ correlation for quantitative analysis and content analysis for qualitative data were used.

RESULTS

Mean score of the teacher professional development evaluation was 4.75 on a 5-point scale. Average teacher implementation rate was 73%, and the student reception rate was 69%. Ongoing teacher support was highly valued by teachers. Teacher satisfaction with teaching the curriculum was highly correlated with student satisfaction (P < .05). Teacher perception of amount of student work was negatively correlated with implementation and with student satisfaction (P < .05).

CONCLUSIONS AND IMPLICATIONS

Use of a systematic conceptual model and comprehensive process measures improves understanding of the implementation process and helps educators to better implement interventions as designed.

The role of motivation in understanding social contextual influences on physical activity in underserved adolescents in the ACT Trial: a cross-sectional study

BACKGROUND

Previous research has shown that social contextual factors are important in understanding physical activity (PA) behavior, although little is known about how these factors may relate to PA, especially in underserved adolescents (low income, minorities). This study examined how motivation may differentially mediate the relationship of two social contextual variables (i.e., peer and parent social support) and moderate-to-vigorous PA (MVPA).

METHODS

Baseline data (n = 1421 sixth graders, 54% female, 72% African American) from the Active by Choice Today (ACT) trial in underserved adolescents were analyzed. Motivation was examined as a mediator of the relationships between peer social support, parent social support, and MVPA (measured by 7-day accelerometer estimates).

RESULTS

Motivation and peer but not parent support were significantly related to MVPA overall. Significant mediation effects were found indicating motivation partially mediated the relation between peer social support and MVPA and to a lesser degree parent support and MVPA.

CONCLUSIONS

These findings provide support for the importance of social contextual influences, especially peer social support, on underserved adolescents' PA and motivation for PA.

Formative process evaluation for implementing a social marketing intervention to increase walking among African Americans in the Positive Action for Today's Health trial

OBJECTIVES

Evaluating programs targeting physical activity may help to reduce disparate rates of obesity among African Americans. We report formative process evaluation methods and implementation dose, fidelity, and reach in the Positive Action for Today's Health trial.

METHODS

We applied evaluation methods based on an ecological framework in 2 community-based police-patrolled walking programs targeting access and safety in underserved African American communities. One program also targeted social connectedness and motivation to walk using a social marketing approach. Process data were systematically collected from baseline to 12 months.

RESULTS

Adequate implementation dose was achieved, with fidelity achieved but less stable in both programs. Monthly walkers increased to 424 in the walking-plus-social marketing program, indicating expanding program reach, in contrast to no increase in the walking-only program. Increased reach was correlated with peer-led Pride Strides (r = .92; P < .001), a key social marketing component, and program social interaction was the primary reason for which walkers reported participating.

CONCLUSIONS

Formative process evaluation demonstrated that the walking programs were effectively implemented and that social marketing increased walking and perceived social connectedness in African American communities.

Process evaluation of a community-based mental health promotion intervention for refugee children

Public health interventions are complex in nature and composed of multiple components. Evaluation of process and impact is necessary to build evidence of effectiveness. Process evaluation involves monitoring extent of implementation and comparison against the program plan. This article describes the process evaluation of the 'Qaderoon' (We are Capable) intervention; a community-based mental health promotion intervention for children living in a Palestinian refugee camp of Beirut, Lebanon. The manuscript describes the context of Palestinian refugees in Lebanon, the intervention, the process evaluation plan and results. The process evaluation was guided by the literature and by a Community Youth Committee. Findings indicated that attendance was 54 and 38% for summer and fall sessions, respectively. Session objectives and activities were commonly achieved. Over 78.4% of activities were reported to be implemented fully as planned. Over 90% of the children indicated high satisfaction with the sessions. Contextual facilitators and challenges to implementing the intervention are discussed. The most challenging were maintaining attendance and the actual implementation of the process evaluation plan. Findings from process evaluation will strengthen interpretation of impact evaluation results.

The effect of a translating research into practice (TRIP)--cancer intervention on cancer pain management in older adults in hospice

BACKGROUND

Pain is a major concern for individuals with cancer, particularly older adults who make up the largest segment of individuals with cancer and who have some of the most unique pain challenges. One of the priorities of hospice is to provide a pain-free death, and while outcomes are better in hospice, patients still die with poorly controlled pain.

OBJECTIVE

This article reports on the results of a Translating Research into Practice intervention designed to promote the adoption of evidence-based pain practices for older adults with cancer in community-based hospices.

SETTING

  This Institutional Human Subjects Review Board-approved study was a cluster randomized controlled trial implemented in 16 Midwestern hospices.

METHODS

Retrospective medical records from newly admitted patients were used to determine the intervention effect. Additionally, survey and focus group data gathered from hospice staff at the completion of the intervention phase were analyzed.

RESULTS

Improvement on the Cancer Pain Practice Index, an overall composite outcome measure of evidence-based practices for the experimental sites, was not significantly greater than control sites. Decrease in patient pain severity from baseline to post-intervention in the experimental group was greater; however, the result was not statistically significant (P = 0.1032).

CONCLUSIONS

Findings indicate a number of factors that may impact implementation of multicomponent interventions, including unique characteristics and culture of the setting, the level of involvement with the change processes, competing priorities and confounding factors, and complexity of the innovation (practice change). Our results suggest that future study is needed on specific factors to target when implementing a community-based hospice intervention, including determining and measuring intervention fidelity prospectively.

A qualitative study of staff's perspectives on implementing an after school program promoting youth physical activity

Minimal effects found across youth physical activity (PA) interventions, and increased attention to circumstances that impede adequate delivery of program components, has highlighted the importance of learning from staff what is needed to foster staff comprehension and engagement for developing, adopting, and successfully implementing PA-based youth interventions. The purpose of this study is to address this knowledge gap by conducting a qualitative assessment of school staff perspectives on the positive aspects and challenges of implementing the 17-week ACT program, an after-school intervention that integrated motivational and behavioral components to promote PA in underserved adolescents. Interviews were conducted with one school staff member from each participating school for all four trial cohorts (N=12). Transcripts were coded by independent coders (r=.84) and content analyses of themes was performed using QSR NVivo. Themes were organized into five meta-themes: (1) Logistics; (2) Essential Elements; (3) Staff and Child Challenges; (4) Staff Comprehension, Value, and Enjoyment; (5) Spill-Over Effects. Findings indicate that staff can be successful at understanding, valuing, and reaching fidelity in implementing climate-based mediation components. The insight gained from this study lays the foundation for understanding the components needed for establishing well-implemented, effective, and generalizable interventions for increasing youth PA.

From theory to practice: intervention fidelity in a randomized controlled trial aiming to optimize weight development during pregnancy

BACKGROUND

Intervention fidelity is an increasingly important methodological concept in process evaluations. In this article, the authors investigated the intervention fidelity in a randomized controlled trial on excessive weight gain prevention in pregnancy.

METHOD

A sample of 109 audiotaped counseling sessions, linked to 65 women in the intervention group of the New Life(style) trial, was drawn. The following criteria were quantitatively evaluated using a fidelity checklist: (a) reach, (b) dose, (c) adherence to study objectives, (d) adherence to underlying problem-solving treatment (PST) theory, and (e) counselor competence.

RESULTS

A total of 60.4% received all counseling sessions. The dose of intervention components was generally moderate (50.9% to 60.4%), and the dose of PST components was low (17.3%). Adherence to study objectives was moderate (64.2%) and adherence to PST theory was low (43.2%). The counselors sufficiently stimulated the participant to optimize lifestyle (54.2% of the sessions), provided positive feedback (50.5%), and left the initiative regarding problem solving to the participant (71%). One of the two counselors performed significantly better on all measured criteria (p < .001).

CONCLUSIONS

Intervention fidelity in the New Life(style) trial was generally low to moderate. In future interventions, it is recommended to put more emphasis on counselor recruitment, training, and intervention protocol contents. Fellow researchers are encouraged to embed a process evaluation into all study stages, taking into account all essential process elements, and to link process outcomes to more distal, health outcomes.