Program theory-driven evaluation science in a youth development context

Examining holistically the experiences of mentors in school-based programs: A logic analysis

The article presents a Logic Analysis of the Scottish MCR ("Motivation, Commitment and Resilience") Pathways school-based mentoring scheme. MCR Pathways provides vulnerable secondary school students with one-to-one support, helping them realize their full potential through education. The perceptions of 12 mentors were explored through interviews, thematically analyzed and mapped to derive the program's Theory of Change as regards the volunteers themselves. This model was then assessed against the evidence base yielded from prior studies. The evaluation highlighted a mismatch between mentors' outcome expectations and what they actually gained from the experience. Furthermore, some themes (e.g., being driven by community concerns) turned out to be more prominent in the context of this specific scheme than in the wider literature, as opposed to other ones (e.g., developing friendships). The study generates insights into the ways to attract and retain growing numbers of volunteers, as well as to advance scientific knowledge.

Positive Adolescent Development: Effects of a Psychosocial Intervention Program in a Rural Setting

The Positive Youth Development (PYD) approach identifies adolescents as resources to be empowered rather than problems to be solved. All adolescents have strengths and will fully develop when these strengths are integrated with healthy resources in the diverse environments where they live and interact. The objective of this study was twofold: (1) to present the Positive Development Program for Adolescents living in rural areas (DPAR Program) and (2) to pilot test the intervention program. The DPAR program was evaluated using a repeated-measures design before and after the intervention, with an intervention group and a control group. The sample consisted of 176 adolescents between 11 and 15 years old (M = 12.89, SD = 0.90) who belonged to two high schools with similar characteristics located in rural settings. A mixed-design analysis of variance was performed for each dependent variable. Results showed a significant increase in most of the study variables (self-esteem, self-efficacy, group identity, empathy, relational skills, assertiveness, and conflict resolution) and a significant decrease in alexithymia, as well as better academic performance. All this evidence indicates that the DPAR program is effective in promoting positive adolescent development and addresses the lack of programs based on the PYD approach in rural areas.

Entwicklung einer Programmtheorie als Grundlage für die Evaluation einer Demenzstation

Zusammenfassung. Hintergrund: Ein bestehendes Konzept einer Demenzstation soll evaluiert werden. Damit die Evaluation gut aufgesetzt werden kann, bedarf es eines theoretischen Verständnisses für die implementierte Intervention, welches bisher nicht explizit formuliert ist. Eine Möglichkeit dafür ist die Entwicklung einer Programmtheorie. Ziel: Ziel ist es, eine Methode vorzustellen, wie eine Programmtheorie für das bestehende Wohn- und Betreuungskonzept einer Demenzstation entwickelt wird, die im Rahmen eines grösseren Projekts dazu dienen soll, dieses Konzept theoriegeleitet zu evaluieren. Methode: Die Evaluation wurde in den Strukturrahmen von van Belle et al. (2010) eingebettet und ein Action Model und ein Change Model nach Chen (2015) erstellt. Die Contribution Analysis ( Mayne, 2011 ) dient der Konkretisierung des Change Models. Es wurden Workshops mit den Entwicklern / Entwicklerinnen und den Pflegenden der Demenzstation sowie eine Literaturrecherche zu den Interventionen und Outcomes durchgeführt. Die Ergebnisse wurden in einem Konsensworkshop zusammengeführt. Ergebnisse: Das Action Model beschreibt die Interventionen der Demenzstation, die Implementierer / Implementiererinnen, die Organisation und den Kontext. Im Change Model wird dargelegt, mit welchen Mechanismen die Intervention Outcomes verändert. Schlussfolgerung: Die Ergebnisse der Programmtheorie können für die Wahl der Datenerhebungsmethoden und Instrumente der Evaluation genutzt werden. Anhand der Ergebnisse der Evaluation kann in weiterer Folge die Programmtheorie verfeinert und adaptiert werden.

Towards program theory validation: Crowdsourcing the qualitative analysis of participant experiences

This exploratory study examines a novel tool for validating program theory through crowdsourced qualitative analysis. It combines a quantitative pattern matching framework traditionally used in theory-driven evaluation with crowdsourcing to analyze qualitative interview data. A sample of crowdsourced participants are asked to read an interview transcript and identify whether program theory components (Activities and Outcomes) are discussed and to highlight the most relevant passage about that component. The findings indicate that using crowdsourcing to analyze qualitative data can differentiate between program theory components that are supported by a participant's experience and those that are not. This approach expands the range of tools available to validate program theory using qualitative data, thus strengthening the theory-driven approach.

Using Theories of Change to inform implementation of health systems research and innovation: experiences of Future Health Systems consortium partners in Bangladesh, India and Uganda

Background

The Theory of Change (ToC) is a management and evaluation tool supporting critical thinking in the design, implementation and evaluation of development programmes. We document the experience of Future Health Systems (FHS) Consortium research teams in Bangladesh, India and Uganda with using ToC. We seek to understand how and why ToCs were applied and to clarify how they facilitate the implementation of iterative intervention designs and stakeholder engagement in health systems research and strengthening.

Methods

This paper combines literature on ToC, with a summary of reflections by FHS research members on the motivation, development, revision and use of the ToC, as well as on the benefits and challenges of the process. We describe three FHS teams’ experiences along four potential uses of ToCs, namely planning, communication, learning and accountability.

Results

The three teams developed ToCs for planning and evaluation purposes as required for their initial plans for FHS in 2011 and revised them half-way through the project, based on assumptions informed by and adjusted through the teams’ experiences during the previous 2 years of implementation. All teams found that the revised ToCs and their accompanying narratives recognised greater feedback among intervention components and among key stakeholders. The ToC development and revision fostered channels for both internal and external communication, among research team members and with key stakeholders, respectively. The process of revising the ToCs challenged the teams’ initial assumptions based on new evidence and experience. In contrast, the ToCs were only minimally used for accountability purposes.

Conclusions

The ToC development and revision process helped FHS research teams, and occasionally key local stakeholders, to reflect on and make their assumptions and mental models about their respective interventions explicit. Other projects using the ToC should allow time for revising and reflecting upon the ToCs, to recognise and document the adaptive nature of health systems, and to foster the time, space and flexibility that health systems strengthening programmes must have to learn from implementation and stakeholder engagement.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-017-0272-y) contains supplementary material, which is available to authorized users.

The Impact of the Project K Youth Development Program on Self-Efficacy: A Randomized Controlled Trial

A key issue for youth development programs is whether the learning they provide is transferred to participants' daily lives. It is also important that they are effective for the diverse range of participants they attract. This study used a randomized controlled trial design to measure the impact of Project K, a New Zealand-based youth development program, on academic and social self-efficacy. Project K combines a 3-week wilderness adventure, a 10 day community service component, and 1 year of mentoring to promote positive growth in 14-15 year olds with low self-efficacy. At baseline, the evaluation included 600 Project K (46 % female) and 577 Control participants (48 % female) and revealed that Project K was effective in improving both social and academic self-efficacy from pre- to post-program with effects being sustained 1 year later. Parents' perceptions of changes in the participants' interpersonal skills supported these findings. Differential program effects were found across participant subgroups, particularly 1 year after program completion. The implications of these differences are discussed.