Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention: A Quasi-Experimental Study

The Case for Assessing and Reporting on Facilitator Fidelity: Introducing the Fidelity of Implementation in Parenting Programs Guideline

The sizeable body of evidence indicating that parenting programs have a positive impact on children and families highlights the potential public health benefits of their implementation on a large scale. Despite evidence and global attention, beyond the highly controlled delivery of parenting programs via randomized trials, little is known about program effectiveness or how to explain the poorer results commonly observed when implemented in community settings. Researchers, practitioners, and policymakers must work together to identify what is needed to spur adoption and sustainment of evidence-based parenting programs in real-world service systems and how to enhance program effectiveness when delivered via these systems. Collecting, analyzing, and using facilitator fidelity data is an important frontier through which researchers and practitioners can contribute. In this commentary, we outline the value of assessing facilitator fidelity and utilizing the data generated from these assessments; describe gaps in research, knowledge, and practice; and recommend directions for research and practice. In making recommendations, we describe a collaborative process to develop a preliminary guideline-the Fidelity of Implementation in Parenting Programs Guideline or FIPP-to use when reporting on facilitator fidelity. Readers are invited to complete an online survey to provide comments and feedback on the first draft of the guideline.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43477-023-00092-5.

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care

Background

Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner.

Methods

In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes.

Results

Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83-1.02 to 0.62-0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81-27.3 to 0.62-19.50, resulting in an approximate average improvement of 18%.

Conclusions

These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented.

Trial registration

NCT03013309 ClinicalTrials.gov.

Improving the feasibility of fidelity measurement for community-based quality assurance: Partial- versus full-session observations of supervisor adherence and competence

Background

Clinical supervision is a common quality assurance method for supporting the implementation and sustainment of evidence-based interventions (EBIs) in community mental health settings. However, assessing and supporting supervisor fidelity requires efficient and effective measurement methods. This study evaluated two observational coding approaches that are potentially more efficient than coding full sessions: a randomly selected 15-min segment and the first case discussion of the session.

Method

Data were leveraged from a randomized trial of an Audit and Feedback (A&F) intervention for supervisor Adherence and Competence. Supervisors ( N = 57) recorded and uploaded weekly group supervision sessions for 7 months, with one session observationally coded each month ( N = 374). Of the coded sessions, one was randomly selected for each supervisor, and a random 15-min segment was coded. Additionally, the first case discussion was coded for the full sample of sessions.

Results

Across all models (and controlling for the proportion of the session covered by the partial observation), Adherence and Competence scores from partial observations were positively and significantly associated with scores from full sessions. In all cases, partial observations were most accurate when the level of Adherence and Competence was moderate. At lower levels, partial observations were underestimates, and at higher levels, they were overestimates.

Conclusions

The results suggest that efficient observational measurement can be achieved while retaining a general level of measurement effectiveness. Practically, first-case discussions are easier to implement, whereas 15-min segments have fewer potential threats to validity. Evaluation of resource requirements is needed, along with determining whether A&F effects are retained if feedback is based on partial observations. Nevertheless, more efficient observational coding could increase the feasibility of routine fidelity monitoring and quality assurance strategies, including A&F, which ultimately could support the implementation and sustainment of effective supervision practices and EBIs in community practice settings. Plain Language Summary: When delivering evidence-based mental health interventions in community-based practice settings, a common quality assurance method is clinical supervision. To support supervisors, assessment methods are needed, and those methods need to be both efficient and effective. Ideally, supervision sessions would be recorded, and trained coders would rate the supervisor’s use of specific strategies. In most settings, though, this requires too many resources. The present study evaluated a more efficient approach. The data came from an existing randomized trial of an Audit and Feedback intervention for enhancing supervisor Adherence and Competence. This included 57 supervisors and 374 sessions across seven months of monitoring. Instead of rating full supervision sessions, a more efficient approach was to have coders rate partial sessions. Two types of partial observations were considered: a randomly selected 15-minute segment of the session and the first case discussion of the session. The aim was to see if partial observations and full observations led to similar conclusions about Adherence and Competence. In all cases, they did. The scores were most similar for sessions with moderate levels of Adherence and Competence. If Adherence and Competence were low, partial observations were underestimates, but if they were high, partial observations were overestimates. Observing partial sessions is more efficient, but in terms of accuracy, the benefits and limitations should be evaluated in light of how the scores will be used. Additionally, future research should consider whether Audit and Feedback interventions have the same effect if feedback is based on observations of partial sessions.

Health behaviour outcomes of a family based intervention for paediatric obesity in primary care: A randomized type II hybrid effectiveness-implementation trial

BACKGROUND

Paediatric obesity is a multifaceted public health problem. Family based behavioural interventions are the recommended approach for the prevention of excess weight gain in children and adolescents, yet few have been tested under "real-world" conditions.

OBJECTIVES

To evaluate the effectiveness of a family based intervention, delivered in coordination with paediatric primary care, on child and family health outcomes.

METHODS

A sample of 240 families with racially and ethnically diverse (86% non-White) and predominantly low-income children (49% female) ages 6 to 12 years (M = 9.5 years) with body mass index (BMI) ≥85th percentile for age and gender were identified in paediatric primary care. Participants were randomized to either the Family Check-Up 4 Health (FCU4Health) program (N = 141) or usual care plus information (N = 99). FCU4Health, an assessment-driven individually tailored intervention designed to preempt excess weight gain by improving parenting skills was delivered for 6 months in clinic, at home and in the community. Child BMI and body fat were assessed using a bioelectrical impedance scale and caregiver-reported health behaviours (eg, diet, physical activity and family health routines) were obtained at baseline, 3, 6 and 12 months.

RESULTS

Change in child BMI and percent body fat did not differ by group assignment. Path analysis indicated significant group differences in child health behaviours at 12 months, mediated by improved family health routines at 6 months.

CONCLUSION

The FCU4Health, delivered in coordination with paediatric primary care, significantly impacted child and family health behaviours that are associated with the development and maintenance of paediatric obesity. BMI did not significantly differ.

Motivational Interviewing and Caregiver Engagement in the Family Check-Up 4 Health

Motivational interviewing (MI) is a therapeutic style in which a provider elicits client motivation and helps strengthen commitment to change (Miller and Rollnick 2002). The original Family Check-Up (FCU; Dishion and Stormshak 2007)-and the adapted version for improving health behaviors in primary care, the Family Check-Up 4 Health (FCU4Health; Smith et al. 2018a)-are brief, assessment-driven, and family-centered preventive interventions that use MI to improve parent engagement in services to improve parenting and prevent negative child outcomes. This study examines the role of MI in the Raising Healthy Children project, a randomized trial to test the effectiveness of the FCU4Health for the prevention of obesity in pediatric primary care, with data from the 141 families assigned to receive the FCU4Health. Families were eligible for the study if the child was between 5.5 and 12 years of age at the time of identification and had a BMI ≥ 85th percentile for age and gender at the most recent visit to their primary care provider. MI skills at the first session predicted caregiver in-session active engagement, attendance at follow-up parenting sessions, and improvements in motivation to address child health and behavior goals. Baseline characteristics of the family (i.e., child health diagnosis, caregiver baseline depression, motivation, and Spanish language preference) had differential associations with responsiveness and MI skills. This study has implications for program development, provider training, and fidelity monitoring.

Using Motivational Interviewing to Improve Parenting Skills and Prevent Problem Behavior During the Transition to Kindergarten

In this study, we examined the efficacy of a version of the Family Check-Up (FCU) adapted for kindergarten school entry with regard to parenting skills during the transition to school. We also examined whether improvements in parenting skills would mediate improvements in parent- and teacher-rated child behavior problems from kindergarten to second grade. The FCU is a motivational interviewing (MI) intervention designed to engage parents in treatment to improve parenting skills. Participants were parents of 365 children enrolled in one of five elementary schools in the Pacific Northwestern United States. Main and indirect effects were tested with structural equation path modeling using an intent-to-treat approach. The FCU was associated with improved change in parenting skills, and changes in parenting skills, in turn, predicted reductions in child behavior problems. Implications for embedding MI in family-centered interventions at kindergarten school entry are discussed. Trial registration: NCT02289092.

Effects of the Family Check-Up 4 Health on Parenting and Child Behavioral Health: A Randomized Clinical Trial in Primary Care

The Family Check-Up 4 Health (FCU4Health) is an adaptation of the Family Check-Up (FCU) for delivery in primary care settings. While maintaining the original FCU's focus on parenting and child behavioral health, we added content targeting health behaviors. This study evaluated whether the adapted FCU maintained positive effects on parenting (positive behavior support, limit setting, parental warmth) and child behavioral health (self-regulation, conduct problems, emotional problems). Pediatric (6-12 years) primary care patients with a BMI ≥ 85^th%ile (n = 240) were recruited from primary care clinics in Phoenix. Children were 75% Latino, 49% female, and 73% Medicaid recipients. This type 2 effectiveness-implementation hybrid trial compared families randomized to FCU4Health (n = 141) or usual care (n = 99). FCU4Health was delivered over a period of 6 months. This study focuses on a priori secondary outcomes included parenting and child behavioral health targets of the original FCU, assessed at baseline and 3, 6, and 12 months. Significant improvements were found for the FCU4Health condition, compared to usual care, in parenting from baseline to the 3-month assessment [β = .17 (.01; .32)]. Parenting predicted improvements in child self-regulation at 6-months [β = .17 (.03; .30)], which in turn predicted reductions in conduct problems [β = - .38 (- .51; - .23)] and emotional problems [β = - .24 (- .38; - .09)] at 12 months. Ethnicity and language of delivery (English or Spanish) did not moderate these effects. The FCU4Health can improve parenting and child behavioral health outcomes when delivered in primary care.Trial Registration Trial registration number: NCT03013309 ClinicalTrials.gov.

The Family Check-Up 4 Health: Study protocol of a randomized type II hybrid effectiveness-implementation trial in integrated primary care (the healthy communities 4 healthy students study)

BACKGROUND

Parenting interventions like the Family Check-Up have demonstrated effects on child physical and behavioral health outcomes. However, access to these programs is limited, particularly for populations experiencing health disparities. Primary care settings have become recognized as a potential delivery system in which these programs may be implemented at scale. The purpose of this trial is to test the effectiveness of the Family Check-Up 4 Health (FCU4Health) program, an adaptation of the FCU for primary care, and assess program implementation in an integrated primary care setting.

METHODS

We will conduct a hybrid type 2 effectiveness-implementation trial in partnership with a primary care clinic in a low-income, majority Latino community. Families with 2- to 5-year-old children will be eligible to participate. Families will be randomized to receive the intervention (n = 130) or services as usual (n = 70) and will be assessed annually over three years. Outcomes are informed by the RE-AIM framework (i.e., reach, effectiveness, adoption, implementation, and maintenance). Effectiveness outcomes include child health behaviors (e.g., Dietary Screener Questionnaire), behavioral health (e.g., Strengths and Difficulties Questionnaire), and parenting (e.g., Proactive Parenting). Early stage implementation outcomes are also included (e.g., cost, acceptability, appropriateness, and feasibility). Effectiveness outcomes will be assessed via intent-to-treat (ITT) analyses. Implementation outcomes will be primarily descriptive with comparisons to prior trials of FCU4Health and the original FCU.

PROJECTED OUTCOMES

This trial will provide evidence related to the potential of integrated primary care settings to deliver evidence-based preventive interventions with a dual focus on behavioral and physical health.

Provider Readiness and Adaptations of Competency Drivers During Scale-Up of the Family Check-Up

We used provider (n = 112) data that staff at the agency disseminating the Family Check-Up (FCU; REACH Institute) collected to profile provider diversity in community settings and to examine whether provider profiles are related to implementation fidelity. Prior to FCU training, REACH Institute staff administered the FCU Provider Readiness Assessment (PRA), a provider self-report measure that assesses provider characteristics previously linked with provider uptake of evidence-based interventions. We conducted a latent class analysis using PRA subscale scores as latent class indicators. Results supported four profiles: experienced high readiness (ExHR), experienced low readiness (ExLR), moderate experience (ME), and novice. The ExHR class was higher than all other classes on: (1) personality variables (i.e., agreeableness, conscientiousness, openness, extraversion); (2) evidence-based practice attitudes; (3) work-related enthusiasm and engagement; and (4) their own well-being. The ExHR class was also higher than ExLR and ME classes on clinical flexibility. The ME class was lowest of all classes on conscientiousness, supervision, clinical flexibility, work-related enthusiasm and engagement, and well-being. During the FCU certification process, FCU Consultants rated providers' fidelity to the model. Twenty-three of the 112 providers that completed the PRA also participated in certification. We conducted follow-up regression analyses using fidelity data for these 23 providers to explore associations between probability of class membership and fidelity. The likelihood of being in the ExHR class was related to higher FCU fidelity, whereas the likelihood of being in the ExLR class was related to lower fidelity. We discuss how provider readiness assessment data can be used to guide the adaptation of provider selection, training, and consultation in community settings.