Multi-year school-based implementation and student outcomes of an evidence-based risk reduction intervention

Teachers' Perceptions of the Impact of the COVID-19 Pandemic and Their Implementation of an Evidence-based HIV Prevention Program in the Bahamas

Information on how school-based programs is implemented and sustained during crises is limited. In this study, we assessed the impact of the COVID-19 pandemic on the implementation of a HIV prevention intervention in The Bahamas. Data were collected from 139 Grade 6 teachers in 2021-2022. Teachers attended virtual training and received implementation monitoring from coordinators. On average, teachers taught 26.4 (SD = 9.2) of the 35 core activities, and 7.4 (SD = 2.4) out of 9 sessions. More than half (58.3%) of teachers completed 28 or more core activities; 69.1% covered eight or all nine sessions, which is equivalent to 80% of the HIV intervention curriculum. Almost half of the teachers (43%) reported that the pandemic negatively impacted their ability to teach the program; 72% of teachers maintained that the program remained "very important" during times of crisis. Greater self-efficacy and supports increased implementation fidelity.

Maintaining Program Fidelity in a Changing World: National Implementation of a School-Based HIV Prevention Program

Large-scale, evidence-based interventions face challenges to program fidelity of implementation. We developed implementation strategies to support teachers implementing an evidence-based HIV prevention program in schools, Focus on Youth in The Caribbean (FOYC) and Caribbean Informed Parents and Children Together (CImPACT) in The Bahamas. We examined the effects of these implementation strategies on teachers' implementation in the subsequent year after the initial implementation during the COVID-19 pandemic. Data were collected from 79 Grade 6 teachers in 24 government elementary schools. Teachers completed training workshops and a pre-implementation questionnaire to record their characteristics and perceptions that might affect their program fidelity. School coordinators and peer mentors provided teachers with monitoring, feedback, and mentoring. In Year 1, teachers on average taught 79.3% of the sessions and 80.8% of core activities; teachers in Year 2 covered 84.2% of sessions and 72.9% of the core activities. Teachers with "good" or "excellent" school coordinators in the second year taught significantly more sessions on average (7.8 vs. 7.0, t = 2.04, P < 0.05) and more core activities (26.3 vs. 23.0, t = 2.41, P < 0.05) than teachers with "satisfactory" coordinators. Teachers who had a "good" or "satisfactory" mentor taught more sessions than teachers who did not have a mentor (7.9 vs. 7.3; t = 2.22; P = 0.03). Two-level mixed-effects model analysis indicated that teachers' program fidelity in Year 1, confidence in the execution of core activities, and school coordinators' performance were significantly associated with Year 2 implementation dose. Implementation of FOYC + CImPACT was significantly associated with improved student outcomes. Teachers maintained high fidelity to a comprehensive HIV prevention program over 2 years during the COVID-19 pandemic. Future program implementers should consider additional implementation support to improve the implementation of school-based programs.

A Mixed Methods Examination of Decision-Making During Program Exploration and Implementation in Schools

Schools are a critical setting to promote healthy youth development through the provision of evidence-based programs (EBPs), yet preventive EBPs in schools are underutilized. The Exploration, Preparation, Implementation, Sustainment (EPIS) framework highlights numerous factors that may influence program adoption during the Exploration phase and progress monitoring during the Implementation phase. However, no research has systematically and simultaneously identified the factors that influence school administrators' decision-making during these important processes. We conducted semi-structured interviews with 24 school administrators in the Midwestern region of the U.S. to understand how they weigh various considerations that inform their adoption and progress monitoring of prevention programs. Results indicated that school administrators consider five separate factors during the adoption decision, prioritized in the following order: need for the program, school community buy-in, contextual fit, resources, and program characteristics (including the evidence-base). Further, administrators consider five indicators to monitor program performance, prioritized as follows: intervention fidelity, quantitative and qualitative data that determine if the identified need was met, school community buy-in, resource consumption, and program characteristics. Implications for prevention scientists and suggestions for future research are discussed.

Exploring effects of multi-level factors on transitions of risk-taking behaviors among middle-to-late adolescents

Adolescents experiment with risk behaviors, including delinquency, substance use, and sexual activity. Multi-level social factors, such as having high-risk peers, neighborhood risks, and parental monitoring, influence adolescents' behaviors. We modeled transition patterns in Bahamian adolescents' risk behaviors across three high school years and examined the effects of multi-level factors. We collected data from 2,564 Bahamian adolescents in Grade 10 and follow-ups through Grade 12. We used latent transition model to identify adolescents' risk statuses. Further analyses used multinomial logistic regression to explore the effects of multi-level factors on assignment to those latent statuses and transitions. We identified four distinct statuses: "low risk" (47.9% of the sample at baseline), "alcohol use" (36.8%), "alcohol use and sexual activity" (5.5%), and "high risk" (9.8%). Males were more likely to be in higher-risk statuses at baseline and to transition from a lower-risk status in Grade 10 to a higher-risk status in Grade 11. Social risk factors were significantly associated with higher-risk statuses at baseline. Neighborhood risk and peer risk involvement continued to affect transitions from lower to higher risk; parental monitoring did not have a significant effect in later years. Our findings have important implications for developing targeted and developmentally appropriate interventions to prevent and reduce risk behaviors among middle-to-late adolescents.

Implementing a school-based HIV prevention program during public health emergencies: lessons learned in The Bahamas

Background

Natural disasters and public health crises can disrupt communities’ capacities to implement important public health programs. A nationwide implementation of an evidence-based HIV prevention program, Focus on Youth in The Caribbean (FOYC) and Caribbean Informed Parents and Children Together (CImPACT), in The Bahamas was disrupted by Hurricane Dorian and the COVID-19 pandemic, especially in its more remote, Family Islands. We explored the teacher- and school-level factors that affected implementation of the program in these islands during those disruptions.

Methods

Data were collected from 47 Grade 6 teachers and 984 students in 34 government elementary schools during the 2020–2021 school year. Teachers completed a pre-implementation questionnaire to record their characteristics and perceptions that might affect their implementation fidelity and an annual program training workshop. School coordinators and high-performing teachers acting as mentors received additional training to provide teachers with monitoring, feedback, and additional support. Teachers submitted data on their completion of the 9 sessions and 35 core activities of FOYC + CImPACT. The fidelity outcomes were the number of sessions and core activities taught by teachers.

Results

On average, teachers taught 60% of sessions and 53% of core activities. Teachers with “very good” school coordinators (34% of teachers) taught more activities than those with “satisfactory” (43%) or no (34%) school coordinator (27.5 vs. 16.8 vs. 14.8, F = 12.86, P < 0.001). Teachers who had attended online training or both online and in-person training taught more sessions (6.1 vs. 6.2 vs. 3.6, F = 4.76, P < 0.01) and more core activities (21.1 vs. 20.8 vs. 12.6, F = 3.35, P < 0.05) than those who received no training. Teachers’ implementation was associated with improved student outcomes (preventive reproductive health skills, self-efficacy, and intention).

Conclusions

The Hurricane Dorian and the COVID-19 pandemic greatly disrupted education in The Bahamas Family Islands and affected implementation of FOYC + CImPACT. However, we identified several strategies that supported teachers’ implementation following these events. Teacher training and implementation monitoring increased implementation fidelity despite external challenges, and students achieved the desired learning outcomes. These strategies can better support teachers’ implementation of school-based interventions during future crises.

Influence of Classroom-Level Factors on Implementation Fidelity During Scale-up of Evidence-Based Interventions

As evidence-based interventions (EBIs) become more widely disseminated, fidelity of implementation (FOI) often wanes. This study explores the association between FOI and malleable variables within classrooms that could be targeted to optimize resources without compromising FOI as school-based EBIs are disseminated across real-world settings. We utilized process evaluation data from a national dissemination project of the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention shown to reduce substance use. The sample included 1,626 teachers in 371 schools across 14 states. Hierarchical linear models examined the relationship between observational measures of implementation factors and three domains of fidelity (e.g., adherence, student responsiveness, and quality of delivery). Findings suggest that curriculum modifications, student misbehavior, and shortage of time to implement the LST middle school program were factors most associated with lower FOI. Class size, access to program materials, and whether LST was delivered in a traditional classroom setting that is well-suited for instruction (versus in a less structured environment such as the school cafeteria) are less predictive. In scale-up of classroom-based universal interventions targeting behavioral health outcomes, our findings indicate that carefully vetting modifications, supporting classroom management strategies, and ensuring sufficient class time for implementation of highly interactive EBIs such as LST are important considerations. Since changes to EBIs are inevitable, efforts are needed to guide facilitators in making adjustments that improve program fit without compromising the essential intervention activities deemed necessary to produce desired outcomes.

Understanding key implementation determinants for a school-based universal prevention intervention: a qualitative study

This study examined how teachers discuss various factors as impacting their ability to execute with fidelity the Michigan Model for Health (MMH), an evidence-based health universal prevention curriculum widely adopted throughout Michigan. Researchers have found a robust relationship between fidelity and participant outcomes, including in schools. While previous studies have identified barriers that inhibit fidelity, few have focused on identifying key barriers and deepening our understanding of how these factors influence intervention fidelity. We conducted a thematic analysis using the reflexive thematic approach to identify key barriers and facilitators and deepen our understanding of how these factors influence MMH implementation. Guided by the Consolidated Framework for Implementation Research (CFIR) and the Implementation Outcomes Framework, we conducted semistructured interviews with 23 high school health teachers across Michigan. Teachers identified intervention characteristics (e.g., design quality, packaging, and program adaptability), student needs (e.g., trauma exposure, substances), and the fit between the intervention and the context as factors that contributed to acceptability. They also discussed the curriculum and its alignment with their teaching style and/or experiences as contributing to fidelity. Teachers shared how they would often go "off protocol" to improve intervention-context fit and meet students' needs. Our results identified acceptability, a perceptual implementation outcome, as demonstrating an important role in shaping the relationship between CFIR factors and fidelity. Results provide guidance for systematically designing implementation strategies that address key barriers to improve acceptability, enhance fidelity, and ultimately achieve desired public health objectives.

Process evaluation of a pre-adolescent transdisciplinary health intervention for inter-generational outcomes

Background

The CIrCLE of Life Initiative was implemented to 537 grade 6 learners and their parents, in five Government-run South African primary schools. The transdisciplinary intervention was intended to increase knowledge and skills on HIV and obesity. The study aim was to assess and report on the implementation process.

Methods

Data was collected on an adapted Proctor’s taxonomy of implementation outcomes, and to assess participants’ experiences. Qualitative and quantitative data were collected through educator logbooks, researcher observations, and learner-parent workbooks.

Results

Differentiations between the various school contexts were observed. The process evaluation revealed high learner penetration (97.2%), but lower learner and parent exposure (44.3% and 55.5%, respectively). All educators thought that the intervention was a fit for both rural and urban schools, different socio-economic groups, and people of different ethnic and cultural backgrounds. The intervention was perceived to be sustainable, and there were recommendations for adoption into the school curriculum and scale-up if found to be effective.

Conclusion

The process evaluation facilitated the assessment of the implementation outcomes, described its processes, and acknowledged fundamental characteristics that could justify variability in the intervention impact and outcomes. The value of process evaluations and their benefit to the science of implementation were demonstrated.

Applying the Exploration, Preparation, Implementation, Sustainment (EPIS) Framework to Safe Surgery 2020 Implementation in Tanzania's Lake Zone

BACKGROUND

Access to safe, high-quality surgical care in sub-Saharan Africa is a critical gap. Interventions to improve surgical quality have been developed, but research on their implementation is still at a nascent stage. We retrospectively applied the Exploration, Preparation, Implementation, Sustainment framework to characterize the implementation of Safe Surgery 2020, a multicomponent intervention to improve surgical quality.

METHODS

We used a longitudinal, qualitative research design to examine Safe Surgery 2020 in 10 health facilities in Tanzania's Lake Zone. We used documentation analysis with confirmatory key informant interviews (n = 6) to describe the exploration and preparation phases. We conducted interviews with health facility leaders and surgical team members at 1, 6, and 12 months (n = 101) post initiation to characterize the implementation phase. Data were analyzed using the constant comparison method.

RESULTS

In the exploration phase, research, expert consultation, and scoping activities revealed the need for a multicomponent intervention to improve surgical quality. In the preparation phase, onsite visits identified priorities and barriers to implementation to adapt the intervention components and curriculum. In the active implementation phase, 4 themes related to the inner organizational context-vision for safe surgery, existing surgical practices, leadership support, and resilience-and 3 themes related to the intervention-innovation-value fit, holistic approach, and buy-in-facilitated or hindered implementation. Interviewees perceived improvements in teamwork and communication and intra- and inter-facility learning, and their need to deliver safe surgery evolved during the implementation period.

CONCLUSIONS

Examining implementation through the exploration, preparation, implementation, and sustainment phases offers insights into the implementation of interventions to improve surgical quality and promote sustainability.

Does Training Modality Predict Fidelity of an Evidence-based Intervention Delivered in Schools?

Training prior to implementing evidence-based interventions (EBIs) is essential to reach high levels of fidelity. However, the time and cost of in-person training are often barriers to implementation. Online learning offers a potential solution, though few studies examine the relationship between online training and fidelity of implementation. This study explored whether teachers trained online have similar levels of adherence, dosage, quality of delivery, and student responsiveness compared to teachers trained in-person on the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention proven to reduce substance use and violence, as part of a national dissemination project. This study involved a sample of 989 LST teachers across 114 school districts, representing 296 schools in 14 states. All teachers were first trained in LST implementation between 2016 and 2019. Hierarchical linear models were used to assess relationships between training modality and the four fidelity outcomes. Online training was associated with lower ratings of quality of delivery compared to in-person training, but no significant associations existed between online training and adherence to the curriculum, dosage, or student responsiveness. Findings from this study generally indicate that online training builds competencies important for school-based EBI implementation, while also highlighting potential shortcomings related to quality of delivery. Ensuring the inclusion of experiential learning activities (e.g., practice delivering content, receiving feedback on delivery) may be key to quality of delivery as online trainings for facilitators of school-based EBIs evolve.

Condom application skills and self-efficacy in youth: A systematic review and meta-analysis

Globally, and in the United States (U.S.) specifically, rates of reported sexually transmitted infections (STIs) have been steadily increasing and are especially high among youth aged 13-25 years. Using condoms correctly and consistently is an effective STI prevention measure for sexually active youth, yet public health endeavors tend to focus only on condom use consistency. Directly measuring condom application is challenging and expensive. Alternative tools evaluate this behaviour, but little evidence exists on the appropriateness of these instruments in measuring application skills. This systematic review and meta-analysis examined the association between condom application skills and self-efficacy. We conducted a search of several databases as well as unpublished works. Studies were included if they were in English, examined youth aged 13-25 years, and were available between 1992 and 2019. The authors screened 630 titles and abstracts for initial inclusion criteria. A full-text review of 30 studies was conducted. The authors included 19 studies in the systematic review and 5 studies were included in the meta-analysis. Both a fixed- and random-effects model (Q = .2321, I2 = 0%) yielded a medium-sized statistically non-significant association (r = 0.217) between skills and self-efficacy. Despite the small sample size, findings suggest that skills and self-efficacy may not be as interchangeable as previously assumed when assessing condom application. Implications for future research are discussed.

Evaluating the implementation of the SWITCH® school wellness intervention and capacity-building process through multiple methods

Background

School wellness programming is important for promoting healthy lifestyles and academic achievement in youth; however, research is needed on methods that can help schools implement and sustain such programs on their own. The purpose of this study was to investigate factors within and outside the school environment that influenced school capacity for implementation and potential sustainability of wellness programming.

Methods

As part of the School Wellness Integration Targeting Child Health (SWITCH®) intervention, elementary school wellness teams (N = 30) were guided through a capacity-building process focused on promoting the adoption of healthy lifestyle behaviors in students. Data on implementation were collected through three standardized surveys and interviews (pre-mid-post) and a post-implementation interview. Indicators of organizational capacity were assessed using the School Wellness Readiness Assessment (SWRA). Paired t-tests were run to assess changes in implementation (classroom, physical education, and lunchroom settings), capacity, and stakeholder engagement over time. One-way analysis of variance (ANOVA) tests were run to examine how implementation of best practices (low, moderate, high) explained differences in capacity gains. Qualitative data were analyzed through inductive and deductive analysis, following the Consolidated Framework for Implementation Research (CFIR).

Results

Paired t-tests showed non-significant increases in school and setting-specific capacity and implementation of SWITCH best practices over time, in addition to a consistent level of engagement from key stakeholders. ANOVA results revealed non-significant associations between implementation group and gains in school capacity (F [2, 24] = 1.63; p = .21), class capacity (F [2, 24]=0.20 p = .82), lunchroom capacity (F [2, 24]=0.29; p = .78), and physical education (F [2, 24]=1.45; p = .25). Qualitative data demonstrated that factors within the outer setting (i.e., engaging community partners) facilitated programming. Inner-setting factors (i.e., relationships with administration and staff) influenced implementation. Implementation process themes (e.g., planning, adaptation of resources to meet school capacity/needs, and engaging students as leaders) were cited as key facilitators. Schools discussed factors affecting sustainability, such as school culture and knowledge of school wellness policy.

Conclusions

The results from this implementation study document the importance of allowing schools to adapt programming to meet their local needs, and highlight the strengths of measuring multiple implementation outcomes. Increased support is needed for schools regarding the formation and improvement of wellness policies as a means to enhance sustainability over time.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12966-020-01070-y.

Using ADAPT-ITT to Modify a Telephone-Based HIV Prevention Intervention for SMS Delivery: Formative Study

BACKGROUND

African American adolescent females are disproportionately affected by sexually transmitted infections (STIs) and HIV. Given the elevated risk of STIs and HIV in African American women, there is an urgent need to identify innovative strategies to enhance the adoption and maintenance of STI and HIV preventive behaviors. Texting is a promising technology for creating preventive maintenance interventions (PMIs) that extend the efficacy of the original intervention. However, little guidance in public health literature is available for developing this type of application.

OBJECTIVE

This paper describes a formative pilot study that incorporates user experience methods to design and test PMI texts for Afiya, an original evidence-based intervention (EBI) specifically designed for African American adolescent females. This study aims to describe the adaptation process of health educator-led phone calling to text-based communication.

METHODS

The formative process followed the assessment, decision, adaptation, production, topical experts-integration, training, testing (ADAPT-ITT) framework for adapting EBIs and using them in a new setting, for a new target population or a modified intervention strategy. This study presents the details of how the phases of the ADAPT-ITT framework were applied to the design of the adaptation. An advisory board was constituted from the target population, consisting of 6 African American women aged 18-24 years, participating in formative activities for 12 weeks, and involving components of the PMI design. As Afiya included a telephone-based PMI, developers of the original Afiya phone scripts crafted the initial design of the SMS-based texts and texting protocol. The advisory board participated in the 1-day Afiya workshop, followed by 4 weeks of texting PMI messages and a midcourse focus group, followed by 4 more weeks of texting PMI messages, ultimately ending with a final focus group. At the advisory board's request, this phase included an optional, additional week of text-based PMI messages.

RESULTS

The methods provided a rich source of data and insights into the fundamental issues involved when constructing SMS-based PMI for this target population and for this EBI. Prior contact and context are essential as the health educator was identified as a key persona in the process and the messages were situated in the original (workshop) context. Narrative adaptations for personas emerged from advisory board discussions. Suggestions on how to expand the PMI to current, specific social contexts indicated that the use of narrative analysis is warranted.

CONCLUSIONS

The use of existing EBIs incorporating telephone-based PMI scripts facilitated the initial design of the texts, with a subsequent narrative analysis of the advisory board data providing additional adjustments given the actual context. Additional examination of the advisory board feedback revealed that personas would offer insight into and opportunities for a persona-specific modification of texting narratives.

Teachers' quality of work life and attitudes toward implementing a psychosocial intervention for children affected by parental HIV/AIDS: roles of self-efficacy and burnout

ABSTRACT Several studies have highlighted that facilitators' attitudes toward interventions are crucial for implementing innovative psychosocial interventions. However, in the emerging implementation science field, little research has examined how organizational and individual factors may influence teachers' positive attitudes and readiness toward evidence-based interventions. The current study investigated the association between teachers' quality of work life and their attitudes toward an innovative psychosocial intervention for children affected by parental HIV/AIDS; the study also probed the potential indirect roles of self-efficacy and burnout. A total of 157 teachers with different levels of involvement in the intervention study were recruited from 47 schools. Our results revealed that teachers' quality of work life was positively associated with their attitudes toward the intervention directly and indirectly through enhanced self-efficacy and reduced burnout. The findings highlight the importance of organizational and individual factors in successfully implementing innovative psychosocial interventions for vulnerable children in organizations such as schools. Researchers should work with organizations to provide the necessary quality of work life and sufficient training to semi-professionals in order to boost their self-efficacy, reduce their burnout, and improve their attitudes toward innovative intervention programs to achieve the expected effectiveness of the interventions, particularly in resource-limited regions.

Bahamas National Implementation Project: Proposal for Sustainability of an Evidence-based HIV Prevention Intervention in a School Setting

BACKGROUND

Sustained implementation of school-based prevention programs is low. Effective strategies are needed to enhance both high-level implementation fidelity and sustainability of prevention programs.

OBJECTIVE

This proposed study aims to determine if the provision of either biweekly monitoring and feedback and site-based assistance and mentorship or both to at-risk and moderate-performing teachers with monitoring through an enhanced decision-making platform by the Ministry of Education (MOE) and Ministry of Health (MOH) based on the real-time implementation data will increase national implementation fidelity and result in sustained implementation over time.

METHODS

This study will target government schools including 200 grade 6 teachers in 80 primary schools and 100 junior/middle high school teachers (and their classes) on 12 Bahamian islands. Teacher and school coordinator training will be conducted by the MOE in year 1, followed by an optimization trial among teachers in the capital island. Informed by these results, an implementation intervention will be conducted to train using different levels of educational intensity all at-risk and moderate-performing teachers. Subsequently selected training and implementation strategies will be evaluated for the national implementation of Focus on Youth in the Caribbean and Caribbean Informed Parents and Children Together in years 2 to 5.

RESULTS

It is hypothesized that a more intensive training and supervision program for at-risk and moderate-performing teachers will enhance their implementation fidelity to the average level of the high-performing group (85%), an HIV prevention program delivered at the national level can be implemented with fidelity in grade 6 and sustained over time (monitored annually), and student outcomes will continue to be highly correlated with implementation fidelity and be sustained over time (assessed annually through grade 9). The proposed study is funded by the National Institute of Child Health and Human Development from August 1, 2018, through May 31, 2023.

CONCLUSIONS

The study will explore several theory-driven implementation strategies to increase sustained teacher implementation fidelity and thereby increase the general public health impact of evidence-based interventions. The proposed project has potential to make significant contributions to advancing school-based HIV prevention research and implementation science and serve as a global model for the Fast Track strategy.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/14816.

Minimizing Health-Compromising Behaviors via School-Based Programs: An Optimization Approach

School health programs are united by their desire to promote health and health-related outcomes among youth. They are also united by the fact that their expected effects are contingent on successful program implementation, which is often impeded by a multitude of real-world barriers. Techniques used in management science may help optimize school-based programs by accounting for implementation barriers. In this exploratory study, we present a detailed example of the first known application of linear programming (LP), which is an optimization technique, to Positive Action (PA). PA is a social emotional and character development program that includes a six-unit, teacher-delivered, classroom curriculum. We specify how we used LP to calculate the optimal levels of program implementation needed to minimize substance use, subject to known levels of implementation barriers (e.g., disruptive behavior, teacher education, teacher attitudes towards character development, school resources, and school safety). We found that LP is a technique that can be applied to data from a school health program. Specifically, we were able to develop a model that calculated the number of lessons that should be taught to minimize a specific health-compromising behavior, given expected levels of predetermined implementation barriers. Our findings from this exploratory study support the utility of applying LP during the program planning and implementation processes of school health programs.