Mixed method evaluation of the Virtual Traveller physically active lesson intervention: An analysis using the RE-AIM framework

Teaching LGBTQ+ Health, a Web-Based Faculty Development Course: Program Evaluation Study Using the RE-AIM Framework

BACKGROUND

Many health professions faculty members lack training on fundamental lesbian, gay, bisexual, transgender, and queer (LGBTQ+) health topics. Faculty development is needed to address knowledge gaps, improve teaching, and prepare students to competently care for the growing LGBTQ+ population.

OBJECTIVE

We conducted a program evaluation of the massive open online course Teaching LGBTQ+ Health: A Faculty Development Course for Health Professions Educators from the Stanford School of Medicine. Our goal was to understand participant demographics, impact, and ongoing maintenance needs to inform decisions about updating the course.

METHODS

We evaluated the course for the period from March 27, 2021, to February 24, 2023, guided by the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework. We assessed impact using participation numbers, evidence of learning, and likelihood of practice change. Data included participant demographics, performance on a pre- and postcourse quiz, open-text entries throughout the course, continuing medical education (CME) credits awarded, and CME course evaluations. We analyzed demographics using descriptive statistics and pre- and postcourse quiz scores using a paired 2-tailed t test. We conducted a qualitative thematic analysis of open-text responses to prompts within the course and CME evaluation questions.

RESULTS

Results were reported using the 5 framework domains. Regarding Reach, 1782 learners participated in the course, and 1516 (85.07%) accessed it through a main course website. Of the different types of participants, most were physicians (423/1516, 27.9%) and from outside the sponsoring institution and target audience (1452/1516, 95.78%). Regarding Effectiveness, the median change in test scores for the 38.1% (679/1782) of participants who completed both the pre- and postcourse tests was 3 out of 10 points, or a 30% improvement (P<.001). Themes identified from CME evaluations included LGBTQ+ health as a distinct domain, inclusivity in practices, and teaching LGBTQ+ health strategies. A minority of participants (237/1782, 13.3%) earned CME credits. Regarding Adoption, themes identified among responses to prompts in the course included LGBTQ+ health concepts and instructional strategies. Most participants strongly agreed with numerous positive statements about the course content, presentation, and likelihood of practice change. Regarding Implementation, the course cost US $57,000 to build and was intramurally funded through grants and subsidies. The course faculty spent an estimated 600 hours on the project, and educational technologists spent another 712 hours. Regarding Maintenance, much of the course is evergreen, and ongoing oversight and quality assurance require minimal faculty time. New content will likely include modules on transgender health and gender-affirming care.

CONCLUSIONS

Teaching LGBTQ+ Health improved participants' knowledge of fundamental queer health topics. Overall participation has been modest to date. Most participants indicated an intention to change clinical or teaching practices. Maintenance costs are minimal. The web-based course will continue to be offered, and new content will likely be added.

Evaluative criteria in practice: Findings from an analysis of evaluations published in Evaluation and Program Planning

BACKGROUND

Evaluative criteria represent values about characteristics that define a successful evaluand. Criteria direct evaluators' lines of inquiry, including how aims are framed, which questions are asked, and what conclusions are reached. Yet, criteria often remain unstated, and little is known about criteria employed in practice.

METHODS

We analyzed 141 evaluations published in Evaluation and Program Planning over four years (2016-2019). Applying a model of criteria domains, we utilized collaborative, interpretive coding to surface and examine the criteria embedded in evaluation aims, questions, and conclusions. We also identified methods used to gather evaluation data.

FINDINGS

Analysis illuminated eight criteria domains evident in the sample and revealed how multiple domains were combined within individual evaluations. We found overlap among data collection methods used to investigate different domains.

CONCLUSIONS

Findings reveal the variety of criteria employed in practice and how evaluators combine criteria to examine multifaceted evaluands. Findings also highlight the need for evaluators to understand how to adapt methods to investigate different criteria domains. This article contributes to the empirical knowledge base about evaluative criteria and identifies revisions to the criteria model.

Features of effective staff training programmes within school-based interventions targeting student activity behaviour: a systematic review and meta-analysis

BACKGROUND

Evaluations of school-based activity behaviour interventions suggest limited effectiveness on students' device-measured outcomes. Teacher-led implementation is common but the training provided is poorly understood and may affect implementation and student outcomes. We systematically reviewed staff training delivered within interventions and explored if specific features are associated with intervention fidelity and student activity behaviour outcomes.

METHODS

We searched seven databases (January 2015-May 2020) for randomised controlled trials of teacher-led school-based activity behaviour interventions reporting on teacher fidelity and/or students' device-measured activity behaviour. Pilot, feasibility and small-scale trials were excluded. Study authors were contacted if staff training was not described using all items from the Template for Intervention Description and Replication reporting guideline. Training programmes were coded using the Behaviour Change Technique (BCT) Taxonomy v1. The Effective Public Health Practice Project tool was used for quality assessment. Promise ratios were used to explore associations between BCTs and fidelity outcomes (e.g. % of intended sessions delivered). Differences between fidelity outcomes and other training features were explored using chi-square and Wilcoxon rank-sum tests. Random-effects meta-regressions were performed to explore associations between training features and changes in students' activity behaviour.

RESULTS

We identified 68 articles reporting on 53 eligible training programmes and found evidence that 37 unique teacher-targeted BCTs have been used (mean per programme = 5.1 BCTs; standard deviation = 3.2). The only frequently identified BCTs were 'Instruction on how to perform the behaviour' (identified in 98.1% of programmes) and 'Social support (unspecified)' (50.9%). We found moderate/high fidelity studies were significantly more likely to include shorter (≤6 months) and theory-informed programmes than low fidelity studies, and 19 BCTs were independently associated with moderate/high fidelity outcomes. Programmes that used more BCTs (estimated increase per additional BCT, d: 0.18; 95% CI: 0.05, 0.31) and BCTs 'Action planning' (1.40; 0.70, 2.10) and 'Feedback on the behaviour' (1.19; 0.36, 2.02) were independently associated with positive physical activity outcomes (N = 15). No training features associated with sedentary behaviour were identified (N = 11).

CONCLUSIONS

Few evidence-based BCTs have been used to promote sustained behaviour change amongst teachers in school-based activity behaviour interventions. Our findings provide insights into why interventions may be failing to effect student outcomes.

TRIAL REGISTRATION

PROSPERO registration number: CRD42020180624.

A-REST (Activity to Reduce Excessive Sitting Time): A Feasibility Trial to Reduce Prolonged Sitting in Police Staff

The aim of this study was to evaluate the acceptability and feasibility of a theory-derived sedentary workplace intervention for police office staff. Twenty-four staff participated in an 8-week intervention (single arm, pre-post design) incorporating an education session, team competition with quick response (QR) codes, team trophy, weekly leaderboard newsletters, a self-monitoring phone app, and electronic prompt tools. The intervention supported participants to reduce and break up their sitting time with three minutes of incidental movement every 30 min at work. Feasibility and acceptability were assessed using mixed methods via the RE-AIM QuEST and PRECIS-2 frameworks. The intervention was highly pragmatic in terms of eligibility, organisation, adherence, outcome, and analysis. It was slightly less pragmatic on recruitment and setting. Delivery and follow-up were more explanatory. Reach and adoption indicators demonstrated feasibility among police staff, across a range of departments, who were demographically similar to participants in previous office-based multi-component interventions. The intervention was delivered mostly as planned with minor deviations from protocol (implementation fidelity). Participants perceived the intervention components as highly acceptable. Results showed improvements in workplace sitting and standing, as well as small improvements in weight and positive affect. Evaluation of the intervention in a fully powered randomised controlled trial to assess behaviour and health outcomes is recommended.

A Scoping Review Exploring Whether a Free "Offer" Devalues or Widens Sport and Physical Activity Participation Amongst Children and Young Adults Aged 0-25?

Background

Socio-economic status continues to mediate physical activity engagement, despite a range of interventions aimed at reducing inequalities and widening sport and physical activity participation. As a result there has been increasing interest amongst policy makers, national governing bodies (NGB), county sports partnerships (CPS) and the sport and physical activity sector more broadly, in understanding how best to reduce inequalities and widen participation. The "price point" of offers and whether a "free offer" enables or devalues participation, has been a key area of interest. This scoping review aimed to explore this topic further by investigating whether "a free "offer" devalues or widens sport and physical activity participation amongst children and young adults aged 0-25?".

Methods

This scoping review searched three electronic bibliographic databases (MEDLINE, PsycINFO, SPORTDiscus) using a structured search strategy to identify articles published between 2017 and January 2022. Studies were included using the PICO criteria of; Population: children and young adults aged 0-25; Intervention: free "offer" relating to physical activity; Context: areas of deprivation in the UK; Outcome: engagement, involvement, participation in sport and physical activity.

Results and Discussion

Five studies were eligible after screening 1301 titles and reviewing 14 full-text studies. Features reported included intervention design, outcomes, potential challenges and wider implications / future recommendations. Specifically, a narrative synthesis of the key themes of participation deprivation and cost effectiveness were outlined in more detail. A subsidized cost or free offer can improve participation generally and in attracting those from lower socio-economic backgrounds. However, the impact of such initiatives decrease with increasing deprivation highlighting that groups with the highest levels of deprivation have wider complexities affecting their participation. Competing priorities and potentially unrealistic expectations at stakeholders level was also identified.

Conclusion

Despite the paucity of current research exploring the impact of a "free offer" in children and young adults, recommendations for future research, practice and policy included the need for longitudinal, more holistic and participant centric approaches. Further research is required to explore the impact of a "free offer" from an individual, societal and policy-level perspective, in widening and increasing participation in sport and physical activity.

Evaluation of a Proposal for Movement Integration in the Teaching-Learning Process in Early Childhood Education

Physical activity is essential to child development, but studies show that children are increasingly inactive. Due to schools being considered privileged environments to promote physical activity, the aim of this study was to increase the physical activity performed by early childhood education children during the school day by integrating movement into academic content and analyze this process. The amount and intensity of physical activity performed by a group of 24 3-4-year-old children in three different weeks were measured by accelerometry: one week with the methodology they had been following (week 1); and two weeks in which movement was integrated into the content through a specific proposal (week 2) and the same improved proposal (week 3). The results reveal that the application of a movement integration program not only allowed students to work on academic content in a physically active way, but also significantly increased the amount of physical activity that children performed during the school day. However, it was necessary to carry out several interventions the same day, or make movement integration the reference methodology, to meet the minimum recommended physical activity levels. In addition, to increase their effectiveness, interventions should be continually reviewed and improved to increase the motor engagement time.

Does Learning Through Movement Improve Academic Performance in Primary Schoolchildren? A Systematic Review

Physically active children have greater motor competence and a faster maturation compared with their sedentary peers. Recent research also suggests that physical activity during childhood may also promote cognitive development and therefore improve academic performance. The aim of this study was to understand if physically active academic lessons may improve academic achievement in primary schoolchildren. A systematic review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines was conducted. The search was performed on the following database: PubMed, Web of Science, Scopus, Education Resources Information Center (ERIC), and PsycINFO (APA). Studies evaluating schoolchildren aged between 3 and 11 years taking part in educational contexts that include physical activity and natural environments evaluating physical fitness and/or educational outcomes were included. A total of 54 studies (for a total sample of 29,460 schoolchildren) were considered eligible and included in the qualitative synthesis. The Effective Public Health Practice Project risk-of-bias assessment revealed a moderate quality of the included studies with only two considered weeks. Despite differences in the retrieved protocols, physically active academic lessons improve the total time engaged in physical activity, motor skills, and/or academic performance. The results of this review suggest that learning through movement is an effective, low-cost, and enjoyable strategy for elementary schoolchildren.

Teaching Methodologies and School Organization in Early Childhood Education and Its Association with Physical Activity

Early childhood represents a crucial period for child development. Physical activity is essential in this process, but studies show that children are very inactive and do not meet the recommended minimums. Due to the large proportion of time they spend at school, it is necessary to examine active and sedentary behaviors in these environments. The aim of the study is to analyze the amount and intensity of physical activity in preschool children during the school day according to the methodology used. Using accelerometry, the amount and intensity of physical activity and sedentary behavior of 156 children aged 4-6 years at different times of the school day were evaluated. The results revealed that preschoolers spend most of their class time sedentary, with children participating in active methodologies registering the highest amount and intensity of physical activity. Recess and specific motor sessions are the most active times, although the latter should increase the time of intense activity that they imply. To increase physical activity during the school day, it is necessary to establish movement integration methodologies, while increasing the number and adjusting the duration of specific motor sessions and of recesses, so that the maximum possible use is made of them.

Physically Active Lessons Improve Lesson Activity and On-Task Behavior: A Cluster-Randomized Controlled Trial of the "Virtual Traveller" Intervention

BACKGROUND

Physically active lessons have not often been assessed with randomized controlled trials.

AIMS

Evaluate the effects of the "Virtual Traveller" (VT) intervention delivered using classroom interactive whiteboards on physical activity, on-task behavior, and student engagement.

METHODS

Participants were 219 children aged 8 to 9 years from 10 schools in Greater London, assessed in a cluster-randomized controlled trial between March 2015 and May 2016. For 6 weeks, intervention children received 10-minute VT sessions three times a week during math and English lessons (VT group: n = 113). Children in control schools received regular teaching (COM group: n = 106). Outcomes were school-day, weekend-day, and lesson-time sedentary behavior (SB), light physical activity (LPA) and moderate-to-vigorous physical activity (MVPA), and on-task behavior and student engagement, assessed at baseline (T0), 2 weeks (T1), and 4 weeks (T2) during the VT intervention and 1 week (T3) and 3 months (T4) postintervention using multilevel modeling.

RESULTS

VT pupils engaged in significantly more school-day MVPA at T1 only, with no other significant differences between groups in overall school-day or weekend-day activity. VT pupils engaged in significantly less SB and more MVPA during lesson time than COM pupils. More on-task behavior was shown in VT pupils than COM pupils but there was no difference in student engagement.

DISCUSSION

VT reduced sedentary behavior and increased physical activity during lesson time but not across overall school or weekend days. VT improved on-task behavior but had no effect on student engagement.

CONCLUSION

Physical activity can be integrated into teaching using interactive whiteboards with no detriment to educational outcomes.