Evaluating the Implementation Quality of a Social and Emotional Learning Program: A Mixed Methods Approach

A scoping review of the implementation and cultural adaptation of school-based mental health promotion and prevention interventions in low-and middle-income countries

Effective school-based mental health promotion and prevention interventions in low-and middle-income countries (LMICs) can positively impact the mental health and well-being of large numbers of young people. This scoping review aimed to investigate the implementation of effective mental health promotion and prevention interventions in LMIC schools. A scoping review of the international literature was conducted and followed the Preferred Reporting Items for Systematic reviews and Meta-Analysis extension for Scoping Reviews guidelines. Medline, PsycInfo, Scopus, Embase, CINAHL and Cochrane were searched for peer-reviewed literature published from 2014 to 2022. PsycExtra, Google Scholar and the websites of key organisations were searched for relevant grey literature. Study selection focussed on mental health promotion interventions, including the development of social and emotional skills and mental health literacy, and prevention interventions, including anti-bullying and skill-based interventions for "at-risk" students. Twenty-seven studies evaluating 25 school-based interventions in 17 LMICs were included in the review. Fifteen interventions were developed in the implementing country and 10 were adapted from high-income countries (HICs) or other settings. Findings from the studies reviewed were generally positive, especially when interventions were implemented to a high quality. Universal life-skills interventions were found to increase social and emotional skills, decrease problem behaviours and positively impact students' mental health and well-being. Mental health literacy interventions increased mental health knowledge and decreased stigma among students and school staff. Outcomes for externally facilitated anti-bullying interventions were less positive. All 19 effective studies reported on some aspects of programme implementation, and 15 monitored implementation fidelity. Eleven studies outlined the programme's underpinning theoretical model. Only four studies reported on the cultural adaptation of programmes in detail. Including young people in the adaptation process was reported to facilitate natural cultural adaptation of programmes, while input from programme developers was considered key to ensuring that the core components of interventions were retained. The review findings indicate increasing evidence of effective mental health interventions in LMIC schools. To facilitate the sustainability, replication and scaling-up of these interventions, greater attention is needed to reporting on intervention core components, and the processes of implementation and cultural adaptation in the local setting.

Effectiveness of Health Promotion Interventions in Primary Schools-A Mixed Methods Literature Review

School-based health promotion interventions (HPIs) are commonly used in schools, but scientific evidence about the structures of effective interventions is lacking. Therefore, we conducted a mixed methods systematic literature review to recognize the HPI structures related to their effectiveness. Based on the inclusion criteria, 49 articles were selected for the literature review. The articles, published in 2011-2022, described 46 different school-based interventions conducted in 20 different countries. The average duration of the interventions was 12 months, and they were implemented mostly with an RCT study design (61.2%) and by targeting children (69.4%). Three main groups of interventions were identified and explained: (1) extensive and long-term interventions; (2) school policy-changing interventions; and (3) highly effective interventions. Effective school-based HPIs included multiple target groups, multiple providers with external experts, and an efficient duration and timing of follow-ups. The implications for educational research and school practice are presented. Evidence on the effectiveness of health-related interventions is still lacking and needs to be addressed in further studies.

Positive Attitude Upper Middle School social and emotional learning program: influences of implementation quality on program outcome

Introduction

There is an increased call for studies analyzing how implementation quality influences Social and Emotional Learning (SEL) program effectiveness.

Methods

The current dissemination study analyzed the effectiveness of the Positive Attitude Upper Middle School SEL program on a Portuguese nationwide sample composed of 813 middle school students (7th and 8th grade; 51.7% boys; M_age = 12.41, S.D. = 1.06), from 36 classrooms (M_classroom = 22.58; S.D. = 2.86), distributed between the control group (179 students), and three intervention groups (643 students) that reflected low, middle, and high implementer experience (respectively, Gulbenkian Academies of Knowledge, Positive Attitude Cadaval and Positive Attitude Torres Vedras). Dosage and fidelity (as implementation quality dimensions), gender, and classroom size (as individual and classroom-level variables) were also analyzed. Self-report questionnaires were administered pre- and post-intervention and at a 6-month follow-up.

Results

Multilevel models were employed, and results showed that participating in the PAUMS SEL program led to more positive trajectories in self-control, social awareness, relationship skills, and responsible decision-making when compared with control groups. Regarding implementation quality, only the implementer's experience impacted the effectiveness of the PAUMS SEL program; students in the Gulbenkian Academies of Knowledge intervention group displayed a less positive trajectory in self-control than students in the Positive Attitude Torres Vedras intervention group.

Discussion

Altogether, results showed that the PAUMS SEL program is ready for dissemination in Portugal, although a higher level of implementer experience is needed to achieve the best effectiveness, and they support the importance of analyzing implementer experience in SEL programs' effectiveness studies.

Monitoring system of implementation of the Promoting Mental Health at Schools (PROMEHS) program

Effective school-based mental health programs are a research field with growing interest and great social value. At the stage of development and initial testing of the program, as well as during dissemination, and adaptation in other cultures, it is important that the implementation is carried out in the way that was originally intended. Fidelity or adherence is the most often used concept relating to the extent to which the implemented intervention corresponds to the originally intended program. Therefore, monitoring of the implementation is an essential element necessary to integrate into contemporary evidence-based program. The current paper describes the monitoring system developed for the Promoting Mental Health at Schools (PROMEHS) project. The monitoring was done on both the structural and procedural aspects of the program's implementation, involving the evaluation of five core aspects: fidelity, dosage, quality, responsiveness, and adaptation. This methods article aims to describe the development of the monitoring system and to analyze the strengths of the qualitative-quantitative multi-informant approach in the monitoring of the intervention's implementation. In the future, this would support further research on effectiveness of the PROMEHS program.

Children’s Perspectives on Using Serious Games as a Complement to Promoting Their Social–Emotional Skills

The use of serious games may be an appealing and complementary way to motivate curriculum-based social and emotional learning (SEL); still, investigation into this potential usefulness is scarce. This study aims to address the usefulness of serious games within the program ‘Me and Us of Emotions’. Specifically, we analyzed the differences in children’s satisfaction in sessions that did or did not use serious games as a complement to the intervention, explored the contribution of using serious games to the global satisfaction with the program, and explored children’s qualitative feedback regarding the sessions. The participants were 232 children (122 boys and 110 girls) aged between 8 and 12 years old (M = 9.09, SD = 0.80). The measures were based on the subjective appraisals of the sessions made by the participating children, including quantitative and qualitative assessments of the degree of satisfaction of the participants. The results showed that there were similar levels of satisfaction with the sessions that did or did not use serious games as a complement to the program. However, only satisfaction with the sessions that used serious games (and not satisfaction with the sessions that did not use them) contributed significantly to explaining both the enjoyment of the activities and the interest in the subjects. Satisfaction with serious games was significantly and positively associated with fun, easiness, ability to understand the session, and ability to cope with emotions. Qualitative analysis showed three main themes, namely: positive aspects, negative aspects, and opportunities for improvement of the program. Overall, these results indicate that children’s satisfaction with the ‘Me and Us of Emotions’ program is related to serious games, suggesting the relevance of using this complementary tool more often when intervening with younger generations.

Innovations in Social and Emotional Learning Research and Practice: Building from Evidence and Applying Behavioral Insights to the Design of a Social and Emotional Learning Intervention in Northeast Nigeria

Recent studies suggest that social and emotional learning (SEL) programming has the potential to be effective in conflict-affected regions, yet evidence is limited, and findings to date are mixed. One hypothesis about why SEL interventions in education in emergencies (EiE) settings have not generated the anticipated results is that the SEL content and materials have not been sufficiently localized to the context, leading to poor cultural relevance or fit. A second hypothesis is that SEL program demands tend to be high and capacity for implementation is low, undermining sustainability and impact. The current study addresses these challenges by investing in locally driven SEL content and design as a way to ensure that SEL materials are grounded in local values and needs, culturally appropriate, relevant to the specific context, and feasible to implement. The study draws on the developmental and prevention sciences as well as the field of behavioral insights to test evidence-based interventions intended to encourage desired behaviors around uptake and implementation. This paper documents the activities conducted during the project’s design phase, including landscape research, creation of initial prototypes, design workshops and rapid prototyping, and field testing. Findings suggest that using local values, practices, and framing in SEL programming increases relevance and appropriateness for the Northeast Nigeria setting. Furthermore, targeted behavioral insights interventions increased the uptake, habitual and regular use, as well as correct implementation of SEL activities. The findings contribute to the emerging literature on the cultural variability of SEL and the need to consider the context when designing and implementing programs in any setting.

Social inequalities in availability of health-promoting interventions in Québec elementary schools

Availability of health-promoting interventions (HPIs) may vary across schools serving students with different socioeconomic backgrounds. Our objectives were to describe social inequalities across elementary schools in: (i) level of importance that school principals attribute to 13 common health-related issues among students in their school; (ii) availability of HPIs within their school addressing eight health topics and (iii) (mis)alignment between perceived importance and HPI availability. Data were collected in telephone interviews with school principals in a convenience sample of 171 elementary schools (59% of 291 schools contacted). Schools were categorized as serving very advantaged, moderately advantaged or disadvantaged students. Principals reported how important 13 health issues were among students in their schools and provided data on HPI availability for 8 health issues, 4 of which required school action under government mandates. Higher proportions of principals in schools serving disadvantaged students (36% of all 171 schools) perceived most health issues as important. The mean number of HPIs in the past year was 12.0, 12.1 and 11.7 in schools serving very advantaged, moderately advantaged and disadvantaged students, respectively. Only availability of mental health HPIs differed by school deprivation (60, 43 and 30% in very, moderately and disadvantaged, respectively). Although most schools offered oral health HPIs, dental problems were not perceived as important. Smoking was perceived as not important and smoking-related HPIs were relatively rare (9%). Given rapid evolution in public health priorities, (mis)alignment between perceived importance of specific health issues and HPI availability in elementary schools warrants ongoing reflection.

The Effects of Implementation Quality of a School-Based Social and Emotional Well-Being Program on Students' Outcomes

School-based social and emotional learning (SEL) programs can be effective in producing positive outcomes for students. However, when the implementation quality is poor, these programs often lose their effectiveness and fail to produce the expected positive outcomes. The current study evaluates a school-based SEL program for 15-18-year-olds in Ireland by determining the impact of implementation quality on program outcomes. The study also examines the effects on outcomes of different implementation dimensions including Dosage, Adherence, Quality of Delivery, and Participant Responsiveness. Employing a cluster randomized controlled trial design, this study collected student outcome data (n = 675) from 32 disadvantaged schools across three time points (pre-, post-, 12-month follow-up) and compared these data across three treatment groups (high-implementation, low-implementation, and control). Linear mixed models (LMM) were used to determine the relationships between the implementation data and student outcome data longitudinally. The findings revealed that the positive effects of the program were only observed with the high-, but not the low-implementation group (reduced suppression of emotions (p = 0.049); reduced avoidance coping (p = 0.006); increased social support coping (p = 0.009); reduced levels of stress (p = 0.035) and depressive symptoms (p = 0.025). The comparison of implementation dimensions revealed that only Quality of Delivery had a significant effect on all of the tested outcomes. This study highlights the importance of high-quality implementation in producing positive outcomes and supports the need to evaluate implementation using multiple dimensions.