Maintaining Program Fidelity in a Changing World: National Implementation of a School-Based HIV Prevention Program

Large-scale, evidence-based interventions face challenges to program fidelity of implementation. We developed implementation strategies to support teachers implementing an evidence-based HIV prevention program in schools, Focus on Youth in The Caribbean (FOYC) and Caribbean Informed Parents and Children Together (CImPACT) in The Bahamas. We examined the effects of these implementation strategies on teachers' implementation in the subsequent year after the initial implementation during the COVID-19 pandemic. Data were collected from 79 Grade 6 teachers in 24 government elementary schools. Teachers completed training workshops and a pre-implementation questionnaire to record their characteristics and perceptions that might affect their program fidelity. School coordinators and peer mentors provided teachers with monitoring, feedback, and mentoring. In Year 1, teachers on average taught 79.3% of the sessions and 80.8% of core activities; teachers in Year 2 covered 84.2% of sessions and 72.9% of the core activities. Teachers with "good" or "excellent" school coordinators in the second year taught significantly more sessions on average (7.8 vs. 7.0, t = 2.04, P < 0.05) and more core activities (26.3 vs. 23.0, t = 2.41, P < 0.05) than teachers with "satisfactory" coordinators. Teachers who had a "good" or "satisfactory" mentor taught more sessions than teachers who did not have a mentor (7.9 vs. 7.3; t = 2.22; P = 0.03). Two-level mixed-effects model analysis indicated that teachers' program fidelity in Year 1, confidence in the execution of core activities, and school coordinators' performance were significantly associated with Year 2 implementation dose. Implementation of FOYC + CImPACT was significantly associated with improved student outcomes. Teachers maintained high fidelity to a comprehensive HIV prevention program over 2 years during the COVID-19 pandemic. Future program implementers should consider additional implementation support to improve the implementation of school-based programs.

Replicating or franchising a STEM afterschool program model: core elements of programmatic integrity

BACKGROUND

Designed in 2012 with a first implementation in 2013, NE STEM 4U is a professional development program for post-secondary students/undergraduates, and serves as a source of outreach, content knowledge generation, and STEM literacy for youth in grades kindergarten through 8th grade (ages 5-14). The model empowers post-secondary students as facilitators of inquiry-based learning within the context of an out-of-school time program. This study investigated the potential for replicating or 'franchising' this model by evaluating on the following: (1) Is the model replicable? And, if so, (2) what core elements are necessary for program fidelity? And (3) is there a dependency on a particular setting/participant type (e.g., a more rural or urban setting)?

RESULTS

Strategic expansion of the program to different institutional types (i.e., Research 1, Research II, and a predominantly undergraduate institution), different geographical locations (i.e., rural and urban), and with various school district partners (i.e., large and small) determined that program fidelity and replicability required 4 core elements or criteria: (i) intentional programming, (ii) staff quality, (iii) effective partnerships, and (iv) program evaluation and continuous improvement. Importantly, we examined emergent themes by each site, as well as in combination (n = 16 focus group participants, n = 12 reflection surveys). These data indicated that Flexibility (21.22%), Student Engagement (i.e., Youth) (19.53%), Classroom Management (i.e., also pertaining to youth) (19.31%), and Communication (15.71%) were the themes most referenced by the post-secondary student mentors in the NE STEM 4U program, regardless of site. Finally, the YPQA results demonstrate general replication of program quality in a "franchise" location.

CONCLUSIONS

These results highlight the core elements of the NE STEM 4U program for consideration of expansion (through strategic replication or 'franchising') as a possible international model. The findings and voices highlight the program's trajectory toward success into environments that expand professional development for post-secondary students, and for delivering STEM opportunities for youth.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1186/s40594-021-00320-0.

A comparison of sampling approaches for monitoring schoolwide inclusion program fidelity

Including students with disabilities requires schoolwide interventions that are implemented with fidelity (adherence). Collection of fidelity data may become problematic when multiple evidence-based treatments exist in one setting. To address concerns around efficiency of data collection, this study hypothesized that the three sampling approaches (proportional, consensus, stratified random) provided similar levels of agreement with an expert rater, thus warranting the differentiation in their applications based on the needs of the evaluators. Three high schools were randomly assigned to one of three sampling approaches (i.e. proportional, consensus, stratified random) to complete a fidelity measure for the implementation of a schoolwide inclusion approach. Based on the Median Test, each sampling approach did not differ significantly from the scores of an expert rater. The results indicate that schools may have some choice in the methods they use to sample their staff around the measurement of fidelity of implementation.

Lessons learned: Evaluating the program fidelity of UNWomen Partnership for Peace domestic violence diversion program in the Eastern Caribbean

To date, there have been a plethora of punitive and diversion programs to address domestic violence around the world. However, the evaluative scholarship of such programs overwhelmingly reflects studies in developed countries while barely showcasing the realities of addressing domestic violence in developing countries. This paper features a multi-year (2008-2011) evaluation study that measured the fidelity of the United Nations Partnership for Peace (PfP) domestic violence diversion program in the Eastern Caribbean country of Grenada. Our findings illuminate organic engagement strategies that were built within existing multi-sectoral partnerships that included magistrate court judges, law enforcement officials, and social service agencies. Furthermore, we documented how the locally-devised implementation strategies ensured the program's fidelity within a resource-limited context. This paper contributes to the global evaluative scholarship, highlighting the lessons learned about implementing culturally-adapted and theoretically-driven domestic violence diversion within a developing country.

Higher compliance to a neuromuscular injury prevention program improves overall injury rate in male football players

PURPOSE

The 11+ injury prevention program has been shown to decrease injury rate. However, few studies have investigated compliance and if it is correlated to time loss. The purpose of this study was to (1) analyze how differences in compliance may impact injury rate and (2) if compliance may impact time loss due to injury.

METHODS

This study was a Level 1 prospective cluster randomized controlled trial conducted in NCAA men's football (soccer) teams that examined the efficacy of the 11+ injury prevention program. The two outcome variables examined were number of injuries and number of days missed from competition. Twenty-seven teams (n = 675 players) used the 11+ program. Compliance, injuries and time loss were recorded. There were three compliance categories, low (LC, 1-19 doses/season), moderate (MC, 20-39 doses/season), and high (HC, > 40 doses/season).

RESULTS

There was a significant difference among the groups for injuries, p = 0.04, _pη² = 0.23. The LC group [mean (M) = 13.25, 95% confidence interval (CI) 9.82-16.68, injury rate (IR) = 10.35 ± 2.21] had a significantly higher injury rate than the HC group (M = 8.33, 95%CI 6.05-10.62, IR = 10.35 ± 2.21), p = 0.02. The MC group (M = 11.21, 95%CI 9.38-13.05, IR = 8.55 ± 2.46) was not significantly different than the LC group, p = 0.29, but was significantly greater than the HC group, p = 0.05. When examined as a continuous variable, compliance was significantly negatively related to injury rate (p = 0.004). It was also significantly negatively related to number of days missed (p = 0.012).

CONCLUSIONS

When compliance was high, there was a significant reduction in injury and time loss. This evidence reinforces the importance of consistent injury prevention program utilization. Clinically, these findings have important implications when discussing the importance of consistent utilization of an injury prevention protocol in sport.

LEVEL OF EVIDENCE

Level 1-Randomized controlled trial (RCT).

Pilot Study of a Group-Based Psychosocial Trauma Recovery Program in Secure Accommodation in Scotland

The current study is the first to implement and evaluate a group-based trauma-specific program for adolescents in a secure accommodation facility in Scotland. A randomized control and qualitative pilot study compared an intervention group (n = 10), who received Teaching Recovery Techniques, to a waitlist control group (n = 7). Measures included subjective units of disturbance (SUDs), standardized trauma symptom questionnaires, and analysis of behavior monitoring logs. Adolescent interviews (n = 10) and a presenter focus group (n = 4) assessed program experience and views on future development. Sessions were videoed and analyzed for program adherence. Analysis involved MANOVA, and a quasi-qualitative thematic approach for participant views. Adolescents reported high SUDs and a range of trauma symptoms. A large effect size was found for reduced SUDs (d = 1.10) and positive trends were identified for symptoms and behavior change in the intervention group. Program adaptations included smaller groups, the use of visual materials and liaison with care staff to facilitate generalization. Recommendations are made for program development and large scale evaluation.

Sustaining program effectiveness after implementation: The case of the self-management of well-being group intervention for older adults

OBJECTIVE

The Self-Management of Well-being (SMW) group intervention for older women was implemented in health and social care. Our aim was to assess whether effects of the SMW intervention were comparable with the original randomized controlled trial (RCT). Furthermore, we investigated threats to effectiveness, such as participant adherence, group reached, and program fidelity.

METHODS

In the implementation study (IMP) 287 and RCT 142 women participated. We compared scores on self-management ability and well-being of the IMP and RCT. For adherence, drop-out rates and session attendance were compared. Regarding reach, we compared participants' baseline characteristics. Professionals completed questions regarding program fidelity.

RESULTS

No significant differences were found on effect outcomes and adherence between IMP and RCT (all p≥0.135). Intervention effect sizes were equal (0.47-0.59). IMP participants were significantly less lonely and more likely to be married, but had lower well-being. Most professionals followed the protocol, with only minimal deviations.

CONCLUSION

The effectiveness of the SMW group intervention was reproduced after implementation, with similar participant adherence, minimal changes in the group reached, and high program fidelity.

PRACTICE IMPLICATIONS

The SMW group intervention can be transferred to health and social care without loss of effectiveness. Implementation at a larger scale is warranted.