Protocol for a mixed methods process evaluation of a hybrid implementation-effectiveness trial of a scaled-up whole-school physical activity program for adolescents: Physical Activity 4 Everyone (PA4E1)

Process Evaluation of a Scaled-Up School-Based Physical Activity Program for Adolescents: Physical Activity 4 Everyone

BACKGROUND

Physical Activity 4 Everyone (PA4E1) is a whole-school physical activity program, with demonstrated efficacy (2012-2014). PA4E1 was adapted (scaled-up) and tested in a scale-up trial (2017-2020). This process evaluation study of the scale-up trial had 2 aims. First, to describe the acceptability, appropriateness, and feasibility of PA4E1 in the scale-up trial, from the perspective of school staff involved in the program management and delivery. Second, to generate themes that may explain school staff assessments of acceptability, appropriateness, and feasibility.

METHODS

Data were collected at various time points throughout the 2-year implementation phase. Online surveys were collected from In-School Champions, Head Physical Education teachers, Principals, and Physical Education teachers (quantitative data). Focus groups and interviews were conducted with In-School Champions, Principals, and Physical Education teachers (qualitative data). Existing published data on website engagement, adaptations, modifications, and the scale-up trial primary outcome (implementation of physical activity practices) were triangulated with the quantitative and qualitative during analysis, to generate themes.

RESULTS

School staff delivering PA4E1 reported it was highly acceptable, appropriate, and feasible. Seven themes were generated relating to acceptability, appropriateness, and feasibility. The themes related to how the program was funded, the delivery modes of implementation support, the identification of easy-wins, the recruitment of the right in-school champion, facilitating principal buy-in, mitigating the impact of school staff turnover, and engaging the whole school.

CONCLUSIONS

Recommendations are made to inform future adaptations for PA4E1 and potentially school-based physical activity programs more generally. The findings may inform future scalability assessments of the suitability of programs for scale-up.

School-based physical activity interventions: which intervention characteristics are associated with participation and retention? A meta-analysis

OBJECTIVE

Many school-based intervention studies are conducted to increase students' physical activity (PA). Recruitment and retention problems potentially impact the robustness of RCT findings. We conducted a meta-analysis to summarize recruitment and retention rates in long-term secondary school-based PA intervention studies and examined associated participant and intervention characteristics.

METHODS

Web of Science, Pubmed, Medline, and PsychInfo were searched until March 20th 2023. We included studies on secondary school-based PA interventions ≥12 weeks, aimed at typically developing adolescents. We abstracted number of schools and students invited, randomized, and participating at follow-up to calculate pooled recruitment and retention rates; participant and intervention characteristics were abstracted to execute subgroup or meta-regression analyses.

RESULTS

Recruitment rates were 51% for invited schools and 80% for invited students, the retention for schools was almost 100% and for students 91%. Interventions with fixed and flexible components, executed in Asia and South America, and from later publication years had higher student recruitment rates. Students' retention rates were lower for interventions which had flexible components, were theory/model-based, used an accelerometer, had a longer intervention duration, and included more females.

CONCLUSION

Recruitment and retention rates in school-based PA interventions are high. Some participant and intervention characteristics influence these rates: flexibility of the intervention, theory/model-based intervention, accelerometer use, intervention duration, continent, and number of females. Researchers should consider these characteristics in intervention development to achieve optimal balance between intervention effectiveness, recruitment, and retention.

Functional independence and agitation outcomes following inpatient rehabilitation after structural brain injury: A retrospective cohort study

The study's purpose was to analyze outcomes of patients with severe behavioural disturbances after acquired brain injury (ABI) in order to identify predictors of discharge destination from a specialized unit (SU) of a Swiss neurorehabilitation facility. Retrospective analysis of 85 patients. Potential predictors of the main outcome discharge destination were assessed with a correlation analysis. The Agitation Behaviour Scale (ABS), Functional Independence Measure (FIM), length of stay (LOS) and pre-living situation were tested in a logistic regression analysis. Twenty-nine patients were institutionalized and 56 patients went home. Discharge destination was significantly correlated to ABS score at discharge from SU (rs = -.33, p = .002), total FIM score on admission to the SU (rs = .25, p = .022), total FIM score at discharge from the SU (rs = .37, p < .001), LOS at the SU (rs = -.36, p = .001), and LOS after discharge of the SU (rs = .36, p = .001). Multivariate analysis showed that FIM scores at discharge (OR = 1.03, p = .008), and LOS at SU (OR = 0.98, p = .017) predicted discharge destination significantly. FIM or LOS may be important predictors in planning discharge destination in patients with severe behavioural disturbances. Further prospective studies are critically needed to better understand the complexity of interactions amongst important predictive factors.

A Mixed-Method Study Examining Solutions to Common Barriers to Teachers' Adoption of Evidence-Based Classroom Practices

Objectives

We conducted a mixed-method focus group study to (a) assess the appropriateness and likely effectiveness of strategies that target individual behavior change mechanisms associated with perceived barriers of lack of time and unsupportive leadership and (b) identify recommendations regarding strategies for overcoming the barriers.

Method

Sample included 39 school-based staff (80% female, 77% White) across two districts in the Midwest. Mixed methods included a simultaneous approach.

Results

Lack of time and supportive leadership continue to pervade school-based implementation efforts. Recommendations centered around the need for school leaders to give teachers the power to re-prioritize how they spend their time as well as providing protected, facilitated time for teachers to collaborate and learn practical skills targeting self-advocacy.

Conclusion

Our findings provide compelling evidence for the use of implementation methodology to strategically target mechanisms of individual behavior change during the process of incorporating new and innovative practices in schools.

Strategies for enhancing the implementation of school-based policies or practices targeting diet, physical activity, obesity, tobacco or alcohol use

BACKGROUND

Several school-based interventions are effective in improving child diet and physical activity, and preventing excessive weight gain, and tobacco or harmful alcohol use. However, schools are frequently unsuccessful in implementing such evidence-based interventions.

OBJECTIVES

1. To evaluate the benefits and harms of strategies aiming to improve school implementation of interventions to address student diet, physical activity, tobacco or alcohol use, and obesity. 2. To evaluate the benefits and harms of strategies to improve intervention implementation on measures of student diet, physical activity, obesity, tobacco use or alcohol use; describe their cost or cost-effectiveness; and any harms of strategies on schools, school staff or students.

SEARCH METHODS

We used standard, extensive Cochrane search methods. The latest search was between 1 September 2016 and 30 April 2021 to identify any relevant trials published since the last published review.

SELECTION CRITERIA

We defined 'Implementation' as the use of strategies to adopt and integrate evidence-based health interventions and to change practice patterns within specific settings. We included any trial (randomised controlled trial (RCT) or non-randomised controlled trial (non-RCT)) conducted at any scale, with a parallel control group that compared a strategy to implement policies or practices to address diet, physical activity, overweight or obesity, tobacco or alcohol use by students to 'no intervention', 'usual' practice or a different implementation strategy.

DATA COLLECTION AND ANALYSIS

We used standard Cochrane methods. Given the large number of outcomes reported, we selected and included the effects of a single outcome measure for each trial for the primary (implementation) and secondary (student health behaviour and obesity) outcomes using a decision hierarchy. Where possible, we calculated standardised mean differences (SMDs) to account for variable outcome measures with 95% confidence intervals (CI). For RCTs, we conducted meta-analyses of primary and secondary outcomes using a random-effects model, or in instances where there were between two and five studies, a fixed-effect model. The synthesis of the effects for non-randomised studies followed the 'Synthesis without meta-analysis' (SWiM) guidelines.

MAIN RESULTS

We included an additional 11 trials in this update bringing the total number of included studies in the review to 38. Of these, 22 were conducted in the USA. Twenty-six studies used RCT designs. Seventeen trials tested strategies to implement healthy eating, 12 physical activity and six a combination of risk factors. Just one trial sought to increase the implementation of interventions to delay initiation or reduce the consumption of alcohol. All trials used multiple implementation strategies, the most common being educational materials, educational outreach and educational meetings. The overall certainty of evidence was low and ranged from very low to moderate for secondary review outcomes. Pooled analyses of RCTs found, relative to a control, the use of implementation strategies may result in a large increase in the implementation of interventions in schools (SMD 1.04, 95% CI 0.74 to 1.34; 22 RCTs, 1917 participants; low-certainty evidence). For secondary outcomes we found, relative to control, the use of implementation strategies to support intervention implementation may result in a slight improvement on measures of student diet (SMD 0.08, 95% CI 0.02 to 0.15; 11 RCTs, 16,649 participants; low-certainty evidence) and physical activity (SMD 0.09, 95% CI -0.02 to 0.19; 9 RCTs, 16,389 participants; low-certainty evidence). The effects on obesity probably suggest little to no difference (SMD -0.02, 95% CI -0.05 to 0.02; 8 RCTs, 18,618 participants; moderate-certainty evidence). The effects on tobacco use are very uncertain (SMD -0.03, 95% CIs -0.23 to 0.18; 3 RCTs, 3635 participants; very low-certainty evidence). One RCT assessed measures of student alcohol use and found strategies to support implementation may result in a slight increase in use (odds ratio 1.10, 95% CI 0.77 to 1.56; P = 0.60; 2105 participants). Few trials reported the economic evaluations of implementation strategies, the methods of which were heterogeneous and evidence graded as very uncertain. A lack of consistent terminology describing implementation strategies was an important limitation of the review.

AUTHORS' CONCLUSIONS

The use of implementation strategies may result in large increases in implementation of interventions, and slight improvements in measures of student diet, and physical activity. Further research is required to assess the impact of implementation strategies on such behavioural- and obesity-related outcomes, including on measures of alcohol use, where the findings of one trial suggest it may slightly increase student risk. Given the low certainty of the available evidence for most measures further research is required to guide efforts to facilitate the translation of evidence into practice in this setting.

A whole system approach to increasing children's physical activity in a multi-ethnic UK city: a process evaluation protocol

BACKGROUND

Engaging in regular physical activity requires continued complex decision-making in varied and dynamic individual, social and structural contexts. Widespread shortfalls of physical activity interventions suggests the complex underlying mechanisms of change are not yet fully understood. More insightful process evaluations are needed to design and implement more effective approaches. This paper describes the protocol for a process evaluation of the JU:MP programme, a whole systems approach to increasing physical activity in children and young people aged 5-14 years in North Bradford, UK.

METHODS

This process evaluation, underpinned by realist philosophy, aims to understand the development and implementation of the JU:MP programme and the mechanisms by which JU:MP influences physical activity in children and young people. It also aims to explore behaviour change across wider policy, strategy and neighbourhood systems. A mixed method data collection approach will include semi-structured interview, observation, documentary analysis, surveys, and participatory evaluation methods including reflections and ripple effect mapping.

DISCUSSION

This protocol offers an innovative approach on the use of process evaluation feeding into an iterative programme intended to generate evidence-based practice and deliver practice-based evidence. This paper advances knowledge regarding the development of process evaluations for evaluating systems interventions, and emphasises the importance of process evaluation.

Extent, Type and Reasons for Adaptation and Modification When Scaling-Up an Effective Physical Activity Program: Physical Activity 4 Everyone (PA4E1)

Background: Few studies have described the extent, type and reasons for making changes to a program prior to and during its delivery using a consistent taxonomy. Physical Activity 4 Everyone (PA4E1) is a secondary school physical activity program that was scaled-up for delivery to a greater number of schools. We aimed to describe the extent, type and reasons for changes to the PA4E1 program (the evidence-based physical activity practices, implementation support strategies and evaluation methods) made before its delivery at scale (adaptations) and during its delivery in a scale-up trial (modifications). Methods: The Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) was used to describe adaptations (planned and made prior to the scale-up trial) and modifications (made during the conduct of the trial). A list of adaptations was generated from a comparison of the efficacy and scale-up trials via published PA4E1 protocols, trial registrations and information provided by trial investigators. Monthly trial team meetings tracked and coded modifications in "real-time" during the conduct of the scale-up trial. The extent, type and reasons for both adaptations and modifications were summarized descriptively. Results: In total, 20 adaptations and 20 modifications were identified, these were to physical activity practices (n = 8; n = 3), implementation support strategies (n = 6; n = 16) and evaluation methods (n = 6, n = 1), respectively. Few adaptations were "fidelity inconsistent" (n = 2), made "unsystematically" (n = 1) and proposed to have a "negative" impact on the effectiveness of the program (n = 1). Reasons for the adaptations varied. Of the 20 modifications, all were "fidelity consistent" and the majority were made "proactively" (n = 12), though most were "unsystematic" (n = 18). Fifteen of the modifications were thought to have a "positive" impact on program effectiveness. The main reason for modification was the "available resources" (n = 14) of the PA4E1 Implementation Team. Conclusions: Adaptations and modifications to public health programs are common. Modifications have the potential to impact the implementation and effectiveness of programs. Our findings underscore the importance of comprehensive reporting of the extent, type and reasons for modifications as part of process evaluations, as this data may be important to the interpretation of trial findings. Clinical Trial Registration: https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=372870, Identifier ACTRN12617000681358.

Scale-up of the Physical Activity 4 Everyone (PA4E1) intervention in secondary schools: 24-month implementation and cost outcomes from a cluster randomised controlled trial

Background

Physical Activity 4 Everyone (PA4E1) is an evidence-based program effective at increasing adolescent physical activity (PA) and improving weight status. This study aimed to determine a) the effectiveness of an adapted implementation intervention to scale-up PA4E1 at 24-month follow-up, b) fidelity and reach, and c) the cost and cost-effectiveness of the implementation support intervention.

Methods

A cluster randomised controlled trial using a type III hybrid implementation-effectiveness design in 49 lower socio-economic secondary schools, randomised to a program (n = 24) or control group (n = 25). An adapted implementation intervention consisting of seven strategies was developed to support schools to implement PA4E1 over 24-months. The primary outcome was the proportion of schools implementing at least four of the 7 PA practices, assessed via computer assisted telephone interviews (CATI) with Head Physical Education Teachers. Secondary outcomes included the mean number of PA practices implemented, fidelity and reach, cost and cost-effectiveness. Logistic regression models assessed program effects.

Results

At baseline, no schools implemented four of the 7 PA practices. At 24-months, significantly more schools in the program group (16/23, 69.6%) implemented at least four of the 7 PA practices than the control group (0/25, 0%) (p < 0.001). At 24-months, program schools were implementing an average of 3.6 more practices than control schools (4.1 (1.7) vs. 0.5 (0.8), respectively) (P < 0.001). Fidelity and reach of the implementation intervention were high (> 75%). The total cost of the program was $415,112 AUD (2018) ($17,296 per school; $117.30 per student).

Conclusions

The adapted implementation intervention provides policy makers and researchers with an effective and potentially cost-effective model for scaling-up the delivery of PA4E1 in secondary schools. Further assessment of sustainability is warranted.

Trial registration

Australian New Zealand Clinical Trials Registry ACTRN12617000681358 prospectively registered 12th May 2017.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12966-021-01206-8.

School-based physical activity programs for promoting physical activity and fitness in children and adolescents aged 6 to 18

BACKGROUND

Physical activity among children and adolescents is associated with lower adiposity, improved cardio-metabolic health, and improved fitness. Worldwide, fewer than 30% of children and adolescents meet global physical activity recommendations of at least 60 minutes of moderate to vigorous physical activity per day. Schools may be ideal sites for interventions given that children and adolescents in most parts of the world spend a substantial amount of time in transit to and from school or attending school.

OBJECTIVES

The purpose of this review update is to summarise the evidence on effectiveness of school-based interventions in increasing moderate to vigorous physical activity and improving fitness among children and adolescents 6 to 18 years of age. Specific objectives are: • to evaluate the effects of school-based interventions on increasing physical activity and improving fitness among children and adolescents; • to evaluate the effects of school-based interventions on improving body composition; and • to determine whether certain combinations or components (or both) of school-based interventions are more effective than others in promoting physical activity and fitness in this target population.

SEARCH METHODS

We searched CENTRAL, MEDLINE, Embase, CINAHL, PsycINFO, BIOSIS, SPORTDiscus, and Sociological Abstracts to 1 June 2020, without language restrictions. We screened reference lists of included articles and relevant systematic reviews. We contacted primary authors of studies to ask for additional information.

SELECTION CRITERIA

Eligible interventions were relevant to public health practice (i.e. were not delivered by a clinician), were implemented in the school setting, and aimed to increase physical activity among all school-attending children and adolescents (aged 6 to 18) for at least 12 weeks. The review was limited to randomised controlled trials. For this update, we have added two new criteria: the primary aim of the study was to increase physical activity or fitness, and the study used an objective measure of physical activity or fitness. Primary outcomes included proportion of participants meeting physical activity guidelines and duration of moderate to vigorous physical activity and sedentary time (new to this update). Secondary outcomes included measured body mass index (BMI), physical fitness, health-related quality of life (new to this update), and adverse events (new to this update). Television viewing time, blood cholesterol, and blood pressure have been removed from this update.  DATA COLLECTION AND ANALYSIS: Two independent review authors used standardised forms to assess each study for relevance, to extract data, and to assess risk of bias. When discrepancies existed, discussion occurred until consensus was reached. Certainty of evidence was assessed according to GRADE. A random-effects meta-analysis based on the inverse variance method was conducted with participants stratified by age (children versus adolescents) when sufficient data were reported. Subgroup analyses explored effects by intervention type.

MAIN RESULTS

Based on the three new inclusion criteria, we excluded 16 of the 44 studies included in the previous version of this review. We screened an additional 9968 titles (search October 2011 to June 2020), of which 978 unique studies were potentially relevant and 61 met all criteria for this update. We included a total of 89 studies representing complete data for 66,752 study participants. Most studies included children only (n = 56), followed by adolescents only (n = 22), and both (n = 10); one study did not report student age. Multi-component interventions were most common (n = 40), followed by schooltime physical activity (n = 19), enhanced physical education (n = 15), and before and after school programmes (n = 14); one study explored both enhanced physical education and an after school programme. Lack of blinding of participants, personnel, and outcome assessors and loss to follow-up were the most common sources of bias.  Results show that school-based physical activity interventions probably result in little to no increase in time engaged in moderate to vigorous physical activity (mean difference (MD) 0.73 minutes/d, 95% confidence interval (CI) 0.16 to 1.30; 33 studies; moderate-certainty evidence) and may lead to little to no decrease in sedentary time (MD -3.78 minutes/d, 95% CI -7.80 to 0.24; 16 studies; low-certainty evidence). School-based physical activity interventions may improve physical fitness reported as maximal oxygen uptake (VO₂max) (MD 1.19 mL/kg/min, 95% CI 0.57 to 1.82; 13 studies; low-certainty evidence). School-based physical activity interventions may result in a very small decrease in BMI z-scores (MD -0.06, 95% CI -0.09 to -0.02; 21 studies; low-certainty evidence) and may not impact BMI expressed as kg/m² (MD -0.07, 95% CI -0.15 to 0.01; 50 studies; low-certainty evidence). We are very uncertain whether school-based physical activity interventions impact health-related quality of life or adverse events.

AUTHORS' CONCLUSIONS

Given the variability of results and the overall small effects, school staff and public health professionals must give the matter considerable thought before implementing school-based physical activity interventions. Given the heterogeneity of effects, the risk of bias, and findings that the magnitude of effect is generally small, results should be interpreted cautiously.

Evaluating Digital Program Support for the Physical Activity 4 Everyone (PA4E1) School Program: Mixed Methods Study

BACKGROUND

Effectively scaled-up physical activity interventions are urgently needed to address the high prevalence of physical inactivity. To facilitate scale-up of an efficacious school-based physical activity program (Physical Activity 4 Everyone [PA4E1]), provision of implementation support to physical education (PE) teachers was adapted from face-to-face and paper-based delivery modes to partial delivery via a website. A lack of engagement (usage and subjective experience) with digital delivery modes, including websites, may in part explain the typical reduction in effectiveness of scaled-up interventions that use digital delivery modes. A process evaluation focused on the PA4E1 website was undertaken.

OBJECTIVE

The 2 objectives were to (1) describe the usage of the PA4E1 program website by in-school champions (PE teachers leading the program within their schools) and PE teachers using quantitative methods; (2) examine the usage, subjective experience, and usability of the PA4E1 program website from the perspective of in-school champions using mixed methods.

METHODS

The first objective used website usage data collected across all users (n=273) throughout the 9 school terms of the PA4E1 implementation support. The 4 usage measures were sessions, page views, average session duration, and downloads. Descriptive statistics were calculated and explored across the duration of the 26-month program. The second objective used mixed methods, triangulating data from the first objective with data from a think-aloud survey and usability test completed by in-school champions (n=13) at 12 months. Qualitative data were analyzed thematically alongside descriptive statistics from the quantitative data in a triangulation matrix, generating cross-cutting themes using the "following a thread" approach.

RESULTS

For the first objective, in-school champions averaged 48.0 sessions per user, PE teachers 5.8 sessions. PE teacher sessions were of longer duration (10.5 vs 7.6 minutes) and included more page views (5.4 vs 3.4). The results from the mixed methods analysis for the second objective found 9 themes and 2 meta-themes. The first meta-theme indicated that the website was an acceptable and appropriate delivery mode, and usability of the website was high. The second meta-theme found that the website content was acceptable and appropriate, and identified specific suggestions for improvement.

CONCLUSIONS

Digital health interventions targeting physical activity often experience issues of lack of user engagement. By contrast, the findings from both the quantitative and mixed methods analyses indicate high usage and overall acceptability and appropriateness of the PA4E1 website to school teachers. The findings support the value of the website within a multidelivery mode implementation intervention to support schools to implement physical activity promoting practices. The analysis identified suggested intervention refinements, which may be adopted for future iterations and further scale-up of the PA4E1 program.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry ACTRN12617000681358; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=372870.

Scale-up of the Physical Activity 4 Everyone (PA4E1) intervention in secondary schools: 12-month implementation outcomes from a cluster randomized controlled trial

BACKGROUND

'Physical Activity 4 Everyone' (PA4E1) was an efficacious multi-component school-based physical activity (PA) program targeting adolescents. PA4E1 has seven PA practices. It is essential to scale-up, evaluate effectiveness and assess implementation of such programs. Therefore, the aim is to assess the impact of implementation support on school practice uptake of the PA4E1 program at 12 and 24 months.

METHODS

A cluster randomised controlled trial, utilising a type III hybrid implementation-effectiveness design, was conducted in 49 randomly selected disadvantaged Australian Government and Catholic secondary schools. A blinded statistician randomly allocated schools to a usual practice control (n = 25) or the PA4E1 program group (n = 24), with the latter receiving seven implementation support strategies to support school PA practice uptake of the seven practices retained from the efficacy trial. The primary outcome was the proportion of schools adopting at least four of the seven practices, assessed via telephone surveys with Head Physical Education Teachers and analysed using exact logistic regression modelling. This paper reports the 12-month outcomes.

RESULTS

Schools were recruited from May to November 2017. At baseline, no schools implemented four of the seven practices. At 12 months significantly more schools in the program group had implemented four of the seven practices (16/24, 66.7%) than the control group (1/25, 4%) (OR = 33.0[4.15-1556.4], p < 0.001). The program group implemented on average 3.2 (2.5-3.9) more practices than the control group (p < 0.001, mean 3.9 (SD 1.5) vs 0.7 (1.0)). Fidelity and reach of the implementation support intervention were high (both > 80%).

CONCLUSIONS

Through the application of multiple implementation support strategies, secondary schools were able to overcome commonly known barriers to implement evidence based school PA practices. As such practices have been shown to result in an increase in adolescent PA and improvements in weight status, policy makers and practitioners responsible for advocating PA in schools should consider this implementation approach more broadly when working with schools. Follow-up is required to determine whether practice implementation is sustained.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry ACTRN12617000681358 registered 12th May 2017.