The Cascading Effects of Multiple Dimensions of Implementation on Program Outcomes: a Test of a Theoretical Model

Engagement Measures in Maltreatment Prevention Studies: A Scoping Review

Prevention services can promote public health by building protective factors and reducing maltreatment risk. Yet, engaging caregivers in prevention services presents a unique set of challenges. Measurement studies are important first steps to increase the knowledge of caregiver engagement in prevention services. The purpose of this scoping review was to investigate how family engagement has been measured and operationalized in the studies of maltreatment prevention/positive parenting programs. The review examined quantitative and mixed methods studies conducted in the U.S., which measured multiple dimensions of client engagement, including behavioral, attitudinal, and relational domains. A total of 88 studies selected from PubMed, CINAHL, ERIC, PsycINFO, Social Work Abstracts, Academic Search Premier, and Web of Science were included in this review. Results indicated that studies examine engagement constructs in all three domains of engagement with a primary focus on behavioral engagement. The attitudinal and relational engagement was mostly assessed through general satisfaction surveys, and a limited number of studies utilized validated measures to assess those constructs. While most studies reported acceptable internal reliabilities, only two studies reported other dimensions of psychometric qualities. Only one validated measure was found, which assessed client perceptions of provider cultural competence. More measurement studies are needed to further incorporate multiple dimensions of engagement into the studies of maltreatment prevention programs, which can inform the effort to develop tailored implementation strategies to fully engage various groups of parents in maltreatment prevention programs.

Adjunctive interventions: change methods directed at recipients that support uptake and use of health innovations

BACKGROUND

Implementation science groups change methods into two categories: (1) clinical, behavioral, or biomedical intervention targeting recipient's health outcomes and (2) implementation strategies targeting the delivery system. Differentiating interventions from strategies based on their intended functions is critical to accurately attributing their effects to health or implementation outcomes. However, in coordinating 200+ HIV implementation research projects and conducting systematic reviews, we identified change methods that had characteristics of both interventions and strategies that were inconsistently categorized. To alleviate confusion and improve change method specification, we propose that implementation science should adopt an extant but rarely used term-adjunctive interventions-to classify change methods that are distinct from the common intervention/strategy taxonomy.

MAIN TEXT

Adjunctive interventions as change methods that target recipients (e.g., patients, participants) of a health intervention but are designed to increase recipients' motivation, self-efficacy, or capacity for initiating, adhering to, complying with, or engaging with the health intervention over time. In two of our published reviews on implementation of HIV interventions, 25 out of 45 coded change methods fell into this gray area between strategy and intervention. We also noted instances in which the same change method was labelled as the intervention ("the thing"), as an adjunctive intervention, or an implementation strategy in different studies-further muddying the waters. Adjunctive interventions are distinguished from other change methods by their intended targets, desired outcomes, and theory of action and causal processes. Whereas health interventions target recipients and have a direct, causal effect on the health outcome, adjunctive interventions enhance recipients' attitudes and behaviors to engage with the intervention and have an indirect causal link to the health outcome via increasing the probability of recipients' utilization and adherence to the intervention. Adjunctive interventions are incapable of directly producing the health outcome and will themselves require implementation strategies to effectively impact sustained uptake, utilization, and adherence. Case examples, logic modeling, and considerations (e.g., relationship to consumer engagement strategies) for adjunctive intervention research are provided.

CONCLUSION

Conceptualizing adjunctive interventions as a separate type of change method will advance implementation research by improving tests of effectiveness, and the specification of mechanisms and outcomes.

Moving beyond dosage and adherence: A protocol for capturing dimensions of active child engagement as a measure of fidelity for social-emotional learning interventions

Social-emotional competencies are important for school-readiness and can be supported through social-emotional learning (SEL) interventions in the preschool years. However, past research has demonstrated mixed efficacy of early SEL interventions across varied samples, highlighting a need to unpack the black box of which early interventions work, under what conditions, and for whom. In the present article we discuss the critical implementation component of active child engagement in an intervention as a potential point of disconnect between the intervention as designed and as implemented. Children who are physically present but unengaged during an intervention may lead to decreased average impacts of an intervention. Furthermore, measuring young children's active engagement with an intervention may help to guide iterative intervention development. We propose a four-step protocol for capturing the multi-dimensional and varied construct of active child engagement in a SEL intervention. To illustrate the utility of the protocol, we apply it to data from a pilot study of a researcher-implemented, semi-structured block play intervention focused on supporting the development of SEL and math skills in preschoolers. We then present future directions for the integration of active participant engagement into the measurement of implementation of SEL interventions for young children.

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care

Background

Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner.

Methods

In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes.

Results

Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83-1.02 to 0.62-0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81-27.3 to 0.62-19.50, resulting in an approximate average improvement of 18%.

Conclusions

These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented.

Trial registration

NCT03013309 ClinicalTrials.gov.

Effects of the A+ intervention on elementary-school teachers' social and emotional competence and occupational health

Teaching is, to date, one of the most prone jobs to experiencing occupational stress and burnout. Owing to burnout's negative personal, social, organizational and economic impacts, researchers, practitioners and education policy leaders are interested in developing practices and interventions aimed at preventing/reducing its prevalence. With teachers' main professional demands to be of a social and emotional nature, interventions designed with a view to promote teachers' social and emotional competence appears to be particularly promising, positively impacting teachers' well-being and personal accomplishment and contributing to a decrease in their psychological distress, namely emotional exhaustion. However, theoretical and empirically grounded interventions with ecological validity and specifically targeting teachers are still scarce. Thus, to bridge the previously identified gaps, the present study aimed to evaluate the efficacy and the quality of the intervention's implementation of the A+, an online social and emotional learning intervention for elementary-school teachers. A quasi-experimental study was conducted with a total of 81 participants (96.3% female, MAge = 46.21, SDAge  = 4.82, n = 42 assigned to the experimental group) from three different school contexts. School clusters were similar in size, organizational structure and socioeconomic level, and as regards previous attendance at social and emotional learning interventions; however, they differed with regards to perceived organizational climate. Data on the efficacy of the A+ was collected across four waves using a set of self-report questionnaires that assessed proximal variables (i.e., social and emotional skills) and distal variables (e.g., well-being, burnout symptoms), and analyzed through Robust Linear Mixed-Effects Models. Coefficient omegas suggested adequate reliability of the measures. Additionally, two trained observers completed an observation grid to evaluate the quality of the A+ implementation (e.g., participant responsiveness, fidelity), with excellent inter-rater reliability. Results suggested that, over time, the A+ had positive impacts across proximal (e.g., increased self-regulation, positive relationship, conflict management skills) and distal variables (e.g., increased emotional well-being, decreased occupational stress and emotional exhaustion symptoms) favoring the experimental group. However, results differed across the school contexts. These findings were accompanied by good implementation quality indicators, namely high fidelity in the delivery of the A+ contents and high participants' responsiveness. Despite its limitations, this study contributes to a growing body of research which reinforces the importance of investing in social and emotional learning interventions to prevent teachers' burnout and improve their occupational health. Furthermore, it highlights the importance of implementation quality research as a component of program planning with a view to enhancing programs' efficacy, as well as the need to adapt and consider context variables in research and practice.

Responsiveness to mHealth Intervention for Cannabis Use in Young Adults Predicts Improved Outcomes

Mobile health (mHealth) interventions have proliferated rapidly in part because of their advantages in reducing consumer and provider burden, but less attention has been paid to participant responsiveness to mHealth programs and how this may affect outcomes. This study adds to that literature by examining whether participant responsiveness to a text messaging-delivered intervention was predictive of treatment outcomes over baseline levels of the outcome. We analyzed data from a pilot-randomized controlled trial of a text messaging-intervention to treat young adults with cannabis use disorder (treatment arm, N = 47), examining three indicators of responsiveness (two behavioral: treatment completion and booster message participation; and one subjective: perceived helpfulness of treatment) on abstinence from cannabis use and use-related problems measured at 3-month follow-up. With the exception of completion, the indicators were positively correlated with each other. Each of the indicators was predictive of better treatment outcomes above and beyond baseline risk. Treatment completion and booster participation-measured via technical data captured during intervention administration-appeared to be stronger predictors of improved outcomes than self-reported perceived helpfulness. Results suggest that behavioral and subjective responsiveness measures appear to be valid indicators of treatment response to mHealth interventions for substance use. Responsiveness measured via technical data captured during intervention administration may be a stronger and more efficient strategy for monitoring continued engagement. We discuss implications of these findings for deploying mHealth interventions at scale and monitoring responsiveness.

Pilot randomized controlled trial of an attachment- and trauma-focused intervention for kinship caregivers

BACKGROUND

Kinship care placements are increasing in many Western countries, however families in kinship care are underserved partly due to the lack of evidence-based interventions addressing their unique needs.

OBJECTIVE

We conducted a pilot randomized controlled trial (RCT) to examine research feasibility and the acceptability, fidelity, and preliminary outcomes of an attachment- and trauma-focused intervention for kinship caregivers in Australia.

PARTICIPANTS AND SETTING

Participants included 26 kinship caregivers (M_age = 55.00 years; 54% grandparents) with youth (M_age = 10.58 years; 38% female), and 19 practitioners from child protection, out-of-home care and other services.

METHOD

Kinship caregivers were randomized to Connect for Kinship Parents (Connect-KP) or care-as-usual (CAU), and completed assessments at baseline, post-intervention, and 6-month follow-up. Placement changes were evaluated at 6-month follow-up. A subset of practitioners received both training and supervision in implementing Connect-KP, and their videorecorded sessions were coded for fidelity.

RESULTS

Practitioners rated the training highly and demonstrated excellent fidelity to program content and process. Results supported the research feasibility and all kinship caregivers reported high levels of program satisfaction and had very high attendance and completion rates. Direction of effects favored Connect-KP vs. CAU for all nine caregiver and youth outcomes, with the largest effects observed for significant reductions in caregiver strain, caregiver psychological aggression, and youth affect suppression following intervention. At 6-month follow-up, more youth from CAU (15%) vs. Connect-KP (0%) experienced an unplanned placement change.

CONCLUSIONS

Our findings show promise of Connect-KP as a potentially effective intervention that can be successfully implemented in child welfare services, and provide impetus for a larger-scale RCT.

The Development, Evaluation, and Implementation of Parenting-focused Prevention Programs in Collaboration with Family Court

Promoting the well-being and best interests of children in separated and divorcing families is a shared value among family court professionals and prevention scientists who develop and evaluate intervention programs. This article chronicles the development, evaluation, and implementation of two programs - the New Beginnings Program (NBP), a parenting intervention for separated/divorcing parents and the Family Transitions Guide (FTG), an intervention designed to motivate high conflict separated/divorcing parents to attend the NBP. The development and evaluation of these programs was facilitated by a long-standing collaboration with Maricopa Family Court. We discuss the process of developing these programs, their underlying small theories, and the evaluation of their effects in randomized trials. We also describe our collaboration with the family court and ways that the court promoted the development and evaluation of these programs. Finally, we summarize lessons learned and discuss future directions to bolster the public health impact of evidence-based programs for separated/divorcing families.

Motivational Interviewing and Caregiver Engagement in the Family Check-Up 4 Health

Motivational interviewing (MI) is a therapeutic style in which a provider elicits client motivation and helps strengthen commitment to change (Miller and Rollnick 2002). The original Family Check-Up (FCU; Dishion and Stormshak 2007)-and the adapted version for improving health behaviors in primary care, the Family Check-Up 4 Health (FCU4Health; Smith et al. 2018a)-are brief, assessment-driven, and family-centered preventive interventions that use MI to improve parent engagement in services to improve parenting and prevent negative child outcomes. This study examines the role of MI in the Raising Healthy Children project, a randomized trial to test the effectiveness of the FCU4Health for the prevention of obesity in pediatric primary care, with data from the 141 families assigned to receive the FCU4Health. Families were eligible for the study if the child was between 5.5 and 12 years of age at the time of identification and had a BMI ≥ 85th percentile for age and gender at the most recent visit to their primary care provider. MI skills at the first session predicted caregiver in-session active engagement, attendance at follow-up parenting sessions, and improvements in motivation to address child health and behavior goals. Baseline characteristics of the family (i.e., child health diagnosis, caregiver baseline depression, motivation, and Spanish language preference) had differential associations with responsiveness and MI skills. This study has implications for program development, provider training, and fidelity monitoring.

Family as a health promotion setting: A scoping review of conceptual models of the health-promoting family

Background

The family is a key setting for health promotion. Contemporary health promoting family models can establish scaffolds for shaping health behaviors and can be useful tools for education and health promotion.

Objectives

The objective of this scoping review is to provide details as to how conceptual and theoretical models of the health promoting potential of the family are being used in health promotion contexts.

Design

Guided by PRISMA ScR guidelines, we used a three-step search strategy to find relevant papers. This included key-word searching electronic databases (Medline, PSycINFO, Embase, and CINAHL), searching the reference lists of included studies, and intentionally searching for grey literature (in textbooks, dissertations, thesis manuscripts and reports.)

Results

After applying inclusion and exclusion criteria, the overall search generated 113 included manuscripts/chapters with 118 unique models. Through our analysis of these models, three main themes were apparent: 1) ecological factors are central components to most models or conceptual frameworks; 2) models were attentive to cultural and other diversities, allowing room for a wide range of differences across family types, and for different and ever-expanding social norms and roles; and 3) the role of the child as a passive recipient of their health journey rather than as an active agent in promoting their own family health was highlighted as an important gap in many of the identified models.

Conclusions

This review contributes a synthesis of contemporary literature in this area and supports the priority of ecological frameworks and diversity of family contexts. It encourages researchers, practitioners and family stakeholders to recognize the value of the child as an active agent in shaping the health promoting potential of their family context.

Implementation methodology from a social systems informatics and engineering perspective applied to a parenting training program

OBJECTIVE

For implementation of an evidence-based program to be effective, efficient, and equitable across diverse populations, we propose that researchers adopt a systems approach that is often absent in efficacy studies. To this end, we describe how a computer-based monitoring system can support the delivery of the New Beginnings Program (NBP), a parent-focused evidence-based prevention program for divorcing parents.

METHOD

We present NBP from a novel systems approach that incorporates social system informatics and engineering, both necessary when utilizing feedback loops, ubiquitous in implementation research and practice. Examples of two methodological challenges are presented: how to monitor implementation, and how to provide feedback by evaluating system-level changes due to implementation.

RESULTS

We introduce and relate systems concepts to these two methodologic issues that are at the center of implementation methods. We explore how these system-level feedback loops address effectiveness, efficiency, and equity principles. These key principles are provided for designing an automated, low-burden, low-intrusive measurement system to aid fidelity monitoring and feedback that can be used in practice.

DISCUSSION

As the COVID-19 pandemic now demands fewer face-to-face delivery systems, their replacement with more virtual systems for parent training interventions requires constructing new implementation measurement systems based on social system informatics approaches. These approaches include the automatic monitoring of quality and fidelity in parent training interventions. Finally, we present parallels of producing generalizable and local knowledge bridging systems science and engineering method. This comparison improves our understanding of system-level changes, facilitates a program's implementation, and produces knowledge for the field. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

What got in the way? Caregiver-reported challenges to home practice of assigned intervention skills

Background

Evidence-based intervention (EBI) effectiveness is hindered by low rates of caregiver home practice, or caregiver rehearsal of intervention skills at home. Although home practice is essential to intervention success, we know little about what makes home practice difficult to complete. We sought to identify the challenges caregivers face when attempting home practice assignments within the context of community implementation of a family-based preventative EBI. We also aimed to identify barriers, perceived need, and implementation-related factors that may contribute to frequency of home practice challenges.

Methods

The family-based preventative intervention Bridges was delivered to caregivers and adolescents in three public middle schools in North America within a randomized efficacy-effectiveness hybrid trial. Caregivers (N = 233) completed worksheets in which they reported challenges to home practice of intervention skills. Caregiver-endorsed challenges were coded for emergent themes and categorized into higher-order home practice challenge domains. Additional data was gathered through multiple methods. In addition to descriptive analyses, multilevel linear regressions were conducted to identify factors associated with frequency of home practice challenge domains.

Results

Over 85% of caregivers reported one or more home practice challenges. Challenges fell into six domains: daily stresses, cognitions/beliefs about the intervention/practice, caregiver wellness/regulation, adolescent responsiveness, relevance/acceptability of intervention or assignment, and external barriers to practice. Adolescent externalizing symptoms, academic performance, and caregiver participation in the English-language intervention group (compared to Spanish-language delivery) were associated with greater frequency of reported challenges.

Conclusions

Among caregivers already attending intervention sessions, specific support to overcome challenges in daily stresses and beliefs about the intervention/practice can support greater home practice compliance, which in turn may impact overall intervention effectiveness. Findings suggest that providing services in caregivers' preferred language may avert potential challenges. Further research into home practice challenges is necessary to identify appropriate home practice implementation supports.

Plain Language Summary

Low rates and poor quality of caregiver home practice of intervention skills hampers intervention effectiveness, but no prior studies have identified the difficulties caregivers face when attempting home practice of intervention skills and their rates of occurrence. Within a community implementation of a family-based preventative intervention, we found that difficulties were common and largely fell in the domains theorized to impact engagement, such as daily stresses and beliefs about the intervention/practice. Findings also suggest that providing services and materials in caregivers' preferred language may avert potential home practice challenges. By identifying common home practice challenges and correlates of different challenges, our findings point to a need to develop appropriate and effective home practice implementation support.

Measures of Facilitator Competent Adherence Used in Parenting Programs and Their Psychometric Properties: A Systematic Review

Implementation fidelity is a critical component of intervention science, which aims to understand how interventions unfold in practice to improve outcomes. A key element of fidelity is facilitator competent adherence-the extent to which a program is delivered as prescribed with the specified level of quality. We conducted a two-part systematic review examining these aspects in parenting programs aiming to reduce child behavior problems and maltreatment. Part One reviews measures of facilitator competent adherence and Part Two examines the psychometric properties of the observational measures found. Searches identified 9153 articles from electronic databases, citation tracking, and expert input. After screening using pre-specified criteria, 156 (Part One) and 41 (Part Two) articles remained. In Part One, measure, facilitator, and intervention characteristics were extracted and synthesized from 65 measures. Most measures were observational, used by facilitators and researchers, and employed Likert-scale ratings. In Part Two, evidence on the reliability (internal consistency, inter-rater, intra-rater, test-retest) and validity (content, construct, convergent/divergent, criterion) of 30 observational measures identified from Part One was synthesized and evaluated. An adapted COSMIN checklist was used to assess study and measure quality. We found most studies to be of reasonably high quality. This is the first review to summarize and critically appraise measures of facilitator competent adherence used in the parenting program literature and establish their psychometric properties. The findings underscore the need to advance research on measures of facilitator competent adherence; reliable, valid, and high-quality implementation measures allow for evidence-based decisions regarding the delivery and scale-up of parenting programs. PROSPERO Registration Number: CRD42020167872.

Redesigning Implementation Measurement for Monitoring and Quality Improvement in Community Delivery Settings

The field of prevention has established the potential to promote child adjustment across a wide array of outcomes. However, when evidence-based prevention programs have been delivered at scale in community settings, declines in implementation and outcomes have resulted. Maintaining high quality implementation is a critical challenge for the field. We describe steps towards the development of a practical system to monitor and support the high-quality implementation of evidence-based prevention programs in community settings. Research on the implementation of an evidence-based parenting program for divorcing families called the "New Beginnings Program" serves as an illustration of the promise of such a system. As a first step, we describe a multidimensional theoretical model of implementation that links aspects of program delivery with improvements in participant outcomes. We then describe research on the measurement of each of these implementation dimensions and test their relations to intended program outcomes. As a third step, we develop approaches to the assessment of these implementation constructs that are feasible to use in community settings and to establish their reliability and validity. We focus on the application of machine learning algorithms and web-based data collection systems to assess implementation and provide support for high quality delivery and positive outcomes. Examples are presented to demonstrate that valid and reliable measures can be collected using these methods. Finally, we envision how these measures can be used to develop an unobtrusive system to monitor implementation and provide feedback and support in real time to maintain high quality implementation and program outcomes.

Annual Research Review: The state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field

BACKGROUND

Scientists have developed evidence-based interventions that improve the symptoms and functioning of youth with psychiatric disorders; however, these interventions are rarely used in community settings. Eliminating this research-to-practice gap is the purview of implementation science, the discipline devoted to the study of methods to promote the use of evidence-based practices in routine care.

METHODS

We review studies that have tested factors associated with implementation in child psychology and psychiatry, explore applications of social science theories to implementation, and conclude with recommendations to advance implementation science through the development and testing of novel, multilevel, causal theories.

RESULTS

During its brief history, implementation science in child psychology and psychiatry has documented the implementation gap in routine care, tested training approaches and found them to be insufficient for behavior change, explored the relationships between variables and implementation outcomes, and initiated randomized controlled trials to test implementation strategies. This research has identified targets related to implementation (e.g., clinician motivation, organizational culture) and demonstrated the feasibility of activating these targets through implementation strategies. However, the dominant methodological approach has been atheoretical and predictive, relying heavily on a set of variables from heuristic frameworks.

CONCLUSIONS

Optimizing the implementation of effective treatments in community care for youth with psychiatric disorders is a defining challenge of our time. This review proposes a new direction focused on developing and testing integrated causal theories. We recommend implementation scientists: (a) move from observational studies of implementation barriers and facilitators to trials that include causal theory; (b) identify a core set of implementation determinants; (c) conduct trials of implementation strategies with clear targets, mechanisms, and outcomes; (d) ensure that behaviors that are core to EBPs are clearly defined; and (e) agree upon standard measures. This agenda will help fulfill the promise of evidence-based practice for improving youth behavioral health.

How Program Users Enhance Fidelity: Implementing the WITS Programs in Rural Canadian Elementary Schools

Research on the processes that enhance implementation fidelity is needed to increase understanding of ways to advance the uptake and sustainability of evidence-based programs (Berkel et al. in Prevention Science, 12, 23-33, 2011; Berkel et al. 2017). We propose and test a theoretical model of interrelations among implementation fidelity (i.e., adherence to program components), and rarely investigated activities of program users that may enhance fidelity; namely integration of program strategies into daily activities and children's use of program strategies (CUoPS). These were assessed across the initial 2 years of the implementation of the WITS peer victimization prevention programs in 16 Canadian rural schools. WITS stands for Walk away, Ignore, Talk it out, and Seek help. All schools were implementing the program. We examined the interrelations among these implementation indicators and their effects on child outcomes targeted by the WITS Programs (i.e., social responsibility, prosocial leadership, peer victimization, emotional symptoms, and aggression). Four implementation assessments were collected from children (n = 1326), their parents, and teachers in the fall and spring of two academic years. The within-time correlations among fidelity, integration, and CUoPS were significant at each assessment. Cross-lagged models showed fidelity and integration, and CUoPS and integration were reciprocally related during each academic year. CUoPS predicted higher subsequent levels of fidelity during and across academic years. Suggestions are given for training and coaching to enhance teachers' integration of program strategies into daily life and for creating opportunities for school staff to observe children using program strategies in order to enhance implementation fidelity.