Examining the Correspondence Between Teacher- and Observer-Report Treatment Integrity Measures

A case study of pragmatic measure development of the Treatment Integrity for Elementary Settings self-report measure for teachers

Introduction

Due to usability, feasibility, and acceptability concerns, observational treatment fidelity measures are often challenging to deploy in schools. Teacher self-report fidelity measures with specific design features might address some of these barriers. This case study outlines a community-engaged, iterative process to adapt the observational Treatment Integrity for Elementary Settings (TIES-O) to a teacher self-report version designed to assess the use of practices to support children's social-emotional competencies in elementary classrooms.

Method

Cognitive walkthrough interviews were conducted with teachers to improve the usability of the teacher self-report measure, called the Treatment Integrity for Elementary Schools-Teacher Report (TIES-T). Qualitative content analysis was used to extract themes from the interviews and inform changes to the measure.

Results

Increasing clarity and interactive elements in the measure training were the dominant themes, but suggestions for the measure format and jargon were also suggested.

Conclusion

The suggested changes resulted in a brief measure, training, and feedback system designed to support the teacher's use of practices to support children's social-emotional competencies in elementary classrooms. Future research with the TIES-T will examine the score reliability and validity of the measure.

Therapeutic techniques and session impact: A practice-research network study in private practice

Objective: This study investigated the relationship between therapeutic techniques and session impact, by examining the replicability of findings observed in a university-based training clinic (Boswell et al., 2010) in another practice-oriented setting: private practice.

Method: N = 8 therapists completed session-level assessments of their technique use for N = 38 clients. The same client sample completed session-level assessments of session outcome. Technique-outcome associations were examined with multilevel models.

Results: As in Boswell et al., common factors were associated with positive session impact. For clients who received higher average common factor techniques (relative to their own therapist's caseload), session impact was the poorest in sessions with higher behavioral change techniques use (relative to the client's own average). Moreover, clients with the lowest average common factor techniques (relative to their therapist's caseload) reported better session impact in sessions that involved a higher degree of session-level behavioral change techniques (relative to their own average).

Conclusion: In line with Boswell et al., therapists should be mindful of the consistency of their routine technique use between- and within-clients, and this can be aided through collection of their own practice-oriented data.

Exploring Intervention Sustainment and Intervention Spread Following a Randomized Clinical Trial of the MOSAIC Program

The randomized trial of the Making Socially Accepting Inclusive Classrooms (MOSAIC) program included intensive coaching from research staff to support teachers' implementation of MOSAIC strategies and resulted in positive student outcomes (Mikami et al., J. Clin. Child Adolesc. Psychol. 51(6):1039-1052, 2022). However, these intensive procedures are costly (in time, money, and resources) and serve as barriers to intervention adoption under typical school conditions. In this study, we explored the extent to which MOSAIC-trained teachers could sustain practices under typical practice conditions (sustainment), the extent to which teachers who did not participate in the trial could adopt the practices under typical practice conditions (spread), and the extent to which strategy use in the follow-up year was associated with participation in MOSAIC-focused professional learning communities (PLCs). Participants were 30 elementary school teachers, including (a) 13 teachers who received intensive coaching on MOSAIC practices during the previous year (MOSAIC group), and (b) seven teachers who participated in the trial in the control condition, plus 10 new teachers interested in MOSAIC (new-to-MOSAIC group). We assessed MOSAIC strategy use over the school year via monthly observations and biweekly teacher self-report surveys. Observation data revealed high sustainment in the MOSAIC group, with teachers showing less than 20% decline in the use of most strategies between the two years of participation. New-to-MOSAIC teachers implemented some core MOSAIC strategies, although not to the extent as those in the MOSAIC group. Higher strategy use was modestly associated with PLC attendance. We discuss implications for encouraging sustainment and intervention spread after initial, intensive supports are withdrawn.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12310-022-09555-w.

Pragmatic fidelity measurement in youth service settings

Background

Fidelity measurement is critical for developing, evaluating, and implementing evidence-based treatments (EBTs). However, traditional fidelity measurement tools are often not feasible for community-based settings. We developed a short fidelity rating form for the Collaborative Problem Solving (CPS) approach from an existing manualized coding system that requires extensive training. We examined the reliability and accuracy of this short form when completed by trained observers, untrained observers, and self-reporting providers to evaluate multiple options for reducing barriers to fidelity measurement in community-based settings.

Methods

Community-based treatment providers submitted recordings of youth service sessions in which they did, or did not, use CPS. For 60 recordings, we compared short-form fidelity ratings assigned by trained observers and untrained observers to those provided by trained observers on the manualized coding system. For 141 recordings, we compared providers' self-reported fidelity on the short form to ratings provided by trained observers on the manualized coding system and examined providers' accuracy as a function of their global fidelity.

Results & Conclusions

The short form was reliable and accurate for trained observers. An assigned global integrity score and a calculated average of component scores on the short form, but not component scores themselves, were reliable and accurate for observers who had CPS expertise but no specific training on rating CPS fidelity. When providers self-reported fidelity on the short form, their global integrity score was a reliable estimate of their CPS integrity; however, providers with better CPS fidelity were most accurate in their self-reports. We discuss the costs and benefits of these more pragmatic fidelity measurement options in community-based settings.

What is the Status of Multi-Informant Treatment Fidelity Research?

OBJECTIVE

The precise measurement of treatment fidelity (quantity and quality in the delivery of treatment strategies in an intervention) is essential for intervention development, evaluation, and implementation. Various informants are used in fidelity assessment (e.g., observers, practitioners [clinicians, teachers], clients), but these informants often do not agree on ratings. This scoping review aims to ascertain the state of science around multi-informant assessment of treatment fidelity.

METHOD

A literature search of articles published through December 2021 identified 673 articles. Screening reduced the number of articles to 44, and the final study set included 35 articles.

RESULTS

There was substantial variability across studies regarding study design, how fidelity was operationalized, and how reliability was defined and assessed. Most studies evaluated the agreement between independent observers and practitioner-report, though several other informant pairs were assessed. Overall, findings suggest that concordance across fidelity informants was low to moderate, with a few key exceptions.

CONCLUSIONS

It is difficult to draw clear conclusions about the degree to which single versus multiple informant assessment is needed to produce an accurate and complete picture of treatment fidelity. The field needs to take steps to determine how to leverage multi-informant assessment to accurately assess treatment fidelity.

Treatment Fidelity and Outcome in CBT for Youth with Autism: The MEYA Fidelity Scale

OBJECTIVE

Assessing treatment fidelity in effectiveness research is critical to interpreting study findings. This paper details the development and initial psychometric evaluation of the Modular Evidence-Based Practices for Youth with Autism Fidelity Scale (MEYA-FS) designed to support the assessment of cognitive-behavioral treatments for youth with autism in effectiveness research.

METHOD

Recorded treatment sessions (N = 338) were randomly selected from 77 youth (M age = 9.65 years, SD = 1.87; 50.67% White; 85.33% male) who received the Schema, Emotion, and Behavior-Focused Therapy for Children (SEBASTIEN) (n = 51) or Coping Cat (n = 24) program.

RESULTS

The MEYA-FS Adherence items displayed acceptable interrater reliability, but more than half of the MEYA-FS Competence items did not. The magnitude and pattern of correlations supported the score validity of the MEYA-FS Adherence and Competence items and subscales. However, some corresponding Adherence and Competence items displayed significant overlap. Scores on each Adherence subscale distinguished between the SEBASTIEN and Coping Cat programs, providing support for discriminant validity. Finally, higher Adherence and Competence subscales predicted significant improvements in youth clinical outcomes (adjustment problems in the school setting, social-communication difficulties, restrictive/repetitive behaviors, and externalizing problems), providing initial evidence for predictive validity.

CONCLUSIONS

The psychometric properties of the MEYA-FS make it appropriate for supporting efforts to evaluate cognitive-behavioral interventions for youth with autism in effectiveness and implementation research.