A Theory-Informed Approach to Locally Managed Learning School Systems: Integrating Treatment Integrity and Youth Mental Health Outcome Data to Promote Youth Mental Health

A Longitudinal Analysis of Trajectories and Predictors of Fidelity Using the SafeCare Parenting Model

Evidence-based practice (EBP) fidelity, understood as the extent to which a program is implemented as the developers intended, is a key implementation variable which likely relates to consumer outcomes. However, studies that track fidelity longitudinally and at large scale are uncommon, and finding reliable predictors of fidelity has proven to be a complex challenge. Further, attitudes toward EBP are a potentially important predictor of fidelity, but results across the literature have been mixed. The purpose of the present study is to use data from the ongoing implementation and dissemination of the SafeCare model to better understand (1) the characteristics of SafeCare implementation fidelity trajectories, and (2) whether individual level factors predict differences in fidelity and fidelity trajectories, especially provider attitudes toward EBP. The analyses reported here include 14,778 observed fidelity sessions by 868 providers in 172 agencies. We use multilevel modeling to examine fidelity, fidelity trajectories over time, and several potential individual-level predictors of fidelity, including demographics, work history, and attitudes toward EBP. We found: (1) that SafeCare fidelity begins high at baseline (93.85% on average); (2) that SafeCare fidelity displays a statistically significant trend of positive linear growth, even among those with less positive attitudes; and (3) that positive attitudes are associated with slightly higher fidelity on average and at baseline, while negative attitudes are associated with slightly lower fidelity on average and at baseline. To our knowledge, this is the largest longitudinal analysis of EBP fidelity in a child welfare program to date, and our findings support the notion that intensive coaching supports which are titrated over time can be sufficient to ensure sustained high fidelity, at least in some cases. Further, these findings indicate that robust training and coaching processes can ensure high fidelity and fidelity growth even among providers with less positive attitudes toward EBP.

A case study of pragmatic measure development of the Treatment Integrity for Elementary Settings self-report measure for teachers

Introduction

Due to usability, feasibility, and acceptability concerns, observational treatment fidelity measures are often challenging to deploy in schools. Teacher self-report fidelity measures with specific design features might address some of these barriers. This case study outlines a community-engaged, iterative process to adapt the observational Treatment Integrity for Elementary Settings (TIES-O) to a teacher self-report version designed to assess the use of practices to support children's social-emotional competencies in elementary classrooms.

Method

Cognitive walkthrough interviews were conducted with teachers to improve the usability of the teacher self-report measure, called the Treatment Integrity for Elementary Schools-Teacher Report (TIES-T). Qualitative content analysis was used to extract themes from the interviews and inform changes to the measure.

Results

Increasing clarity and interactive elements in the measure training were the dominant themes, but suggestions for the measure format and jargon were also suggested.

Conclusion

The suggested changes resulted in a brief measure, training, and feedback system designed to support the teacher's use of practices to support children's social-emotional competencies in elementary classrooms. Future research with the TIES-T will examine the score reliability and validity of the measure.