Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention

Measuring the active elements of cognitive-behavioral therapies

Understanding how and for whom cognitive-behavioral therapies work is central to the development and improvement of mental health interventions. Suboptimal quantification of the active elements of cognitive-behavioral therapies has hampered progress in elucidating mechanisms of change. To advance process research on cognitive-behavioral therapies, we describe a theoretical measurement framework that focuses on the delivery, receipt, and application of the active elements of these interventions. We then provide recommendations for measuring the active elements of cognitive-behavioral therapies aligned with this framework. Finally, to support measurement harmonization and improve study comparability, we propose the development of a publicly available repository of assessment tools: the Active Elements of Cognitive-Behavioral Therapies Measurement Kit.

Common Elements Approaches to Implementation Research and Practice: Methods and Integration with Intervention Science

We propose that common elements approaches can advance implementation research and practice and facilitate pragmatic use of intervention and implementation evidence. Common elements are practices or processes frequently shared by interventions or implementations. Traditional common elements methodologies use synthesis, distillation, and statistics to describe and evaluate the merit of common ingredients in effective interventions. Recent developments include identifying and testing common configurations of elements, processes, and context variables across the literature of effective interventions and implementations. While common elements thinking has grown popular in intervention science, it has rarely been utilized in implementation science, and specifically, combined with the intervention literature. The goals of this conceptual methodology paper are to (1) provide an overview of the common elements concept and how it may advance implementation research and usability for practice, (2) give a step-by-step guide to systematic common elements reviews that synthesizes and distills the intervention and implementation literature together, and (3) offer recommendations for advancing element-level evidence in implementation science. A narrative review of the common elements literature was conducted with attention to applications to implementation research. A six-step guide to using an advanced common elements methodology was provided. Examples of potential results are presented, along with a review of the implications for implementation research and practice. Finally, we reviewed methodological limitations in current common elements approaches, and identified steps towards realizing their potential. Common elements methodologies can (a) synthesize and distill the implementation science literature into practical applications, (b) generate evidence-informed hypotheses about key elements and determinants in implementation and intervention processes and mechanisms, and (c) promote evidence-informed precision tailoring of intervention and implementation to context. To realize this potential, common elements approaches need improved reporting of details from both successful and unsuccessful intervention and implementation research, more data availability, and more testing and investigation of causal processes and mechanisms of change from diverse theories.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43477-023-00077-4.

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care

Background

Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner.

Methods

In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes.

Results

Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83-1.02 to 0.62-0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81-27.3 to 0.62-19.50, resulting in an approximate average improvement of 18%.

Conclusions

These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented.

Trial registration

NCT03013309 ClinicalTrials.gov.

Three Flavorings for a Soup to Cure what Ails Mental Health Services

With new tools from artificial intelligence and new perspectives on personalizing interventions, we could revolutionize the way mental health services are delivered and achieve major gains in improving the public's mental health. We examine Dr. Bickman's vision around these technological and paradigm changes that would usher in major scientific, workforce training, and societal cultural changes. We argue that additional efforts in research evaluations in implementation have the potential to scale up and adapt existing interventions and scale them out to diverse populations and service systems. The next stage of this work involves testing the effectiveness of personalized interventions that are preferred by the public and integrating these choices into sustainable service systems. We note cautions on the delivery of these programs as automated algorithmic recommendations are heretofore foreign to humans.

Implementation methodology from a social systems informatics and engineering perspective applied to a parenting training program

OBJECTIVE

For implementation of an evidence-based program to be effective, efficient, and equitable across diverse populations, we propose that researchers adopt a systems approach that is often absent in efficacy studies. To this end, we describe how a computer-based monitoring system can support the delivery of the New Beginnings Program (NBP), a parent-focused evidence-based prevention program for divorcing parents.

METHOD

We present NBP from a novel systems approach that incorporates social system informatics and engineering, both necessary when utilizing feedback loops, ubiquitous in implementation research and practice. Examples of two methodological challenges are presented: how to monitor implementation, and how to provide feedback by evaluating system-level changes due to implementation.

RESULTS

We introduce and relate systems concepts to these two methodologic issues that are at the center of implementation methods. We explore how these system-level feedback loops address effectiveness, efficiency, and equity principles. These key principles are provided for designing an automated, low-burden, low-intrusive measurement system to aid fidelity monitoring and feedback that can be used in practice.

DISCUSSION

As the COVID-19 pandemic now demands fewer face-to-face delivery systems, their replacement with more virtual systems for parent training interventions requires constructing new implementation measurement systems based on social system informatics approaches. These approaches include the automatic monitoring of quality and fidelity in parent training interventions. Finally, we present parallels of producing generalizable and local knowledge bridging systems science and engineering method. This comparison improves our understanding of system-level changes, facilitates a program's implementation, and produces knowledge for the field. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Sustainability planning in the US response to the opioid crisis: An examination using expert and text mining approaches

Between January 2016 and June 2020, the Substance Abuse and Mental Health Services Administration rapidly distributed $7.5 billion in response to the U.S. opioid crisis. These funds are designed to increase access to medications for addiction treatment, reduce unmet treatment need, reduce overdose death rates, and provide and sustain effective prevention, treatment and recovery activities. It is unclear whether or not the services developed using these funds will be sustained beyond the start-up period. Based on 34 (64%) State Opioid Response (SOR) applications, we assessed the states' sustainability plans focusing on potential funding sources, policies, and quality monitoring. We found variable commitment to sustainability across response plans with less than half the states adequately describing sustainability plans. States with higher proportions of opioid prescribing, opioid misuse, and poverty had somewhat higher scores on sustainment. A text mining/machine learning approach automatically rated sustainability in SOR applications with an 82% accuracy compared to human ratings. Because life saving evidence-based programs and services may be lost, intentional commitment to sustainment beyond the bolus of start-up funding is essential.

Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale

Background

Enhancing the sustainability of evidence-based prevention programs for mental and behavioral health requires tools for measuring both sustainability determinants and sustainment outcomes. The aim of this study was to develop the Sustainment Measurement System Scale (SMSS) and to assess its reliability and construct validity for measuring both determinants and outcomes of efforts to sustain prevention programs and initiatives.

Methods

A 42-item scale comprised of items identified from qualitative data collected from 45 representatives of 10 programs and 8 SAMHSA program officers was administered to 186 representatives of 145 programs funded by 7 SAMHSA prevention grant initiatives. Cronbach’s alphas were used to determine inter-item reliability. Convergent validity was assessed by comparisons of a global measure of sustainment with current SAMHSA-funding status and continued operation in the same form. Discriminant validity was assessed by comparisons of sustainability determinants with whether or not the program had undergone adaptations.

Results

Confirmatory factor analysis provided support for a 35-item model fit to the data. Cronbach’s alpha was .84 for the sustainment outcome construct and ranged from .70 to .93 for the sustainability determinant constructs. All of the determinant constructs were significantly associated with sustainment outcome individual and global measures for the entire sample (p < 0.01 to 0.001) and for community-based programs and programs with a substance abuse focus (p < 0.05 to 0.001). Convergent validity was supported by significant associations between the global sustainment measure and current SAMHSA funding status and continued operation in the same form (p < 0.001). Four of the sustainability determinant constructs (responsive to community needs; coalitions, partnerships, and networks; organizational staff capability; and evaluation, feedback, and program outcomes) were also significantly associated with current SAMHSA funding status (p < 0.5 to 0.01). With the exception of organizational staff capability, all sustainability determinants were unrelated to program adaptation as predicted.

Conclusions

The SMSS demonstrated good reliability and convergent and discriminant validity in assessing likelihood of sustainment of SAMHSA funded prevention programs and initiatives. The measure demonstrates potential in identifying predictors of program sustainment and as a tool for enhancing the likelihood of successful sustainment through ongoing evaluation and feedback.

Redesigning Implementation Measurement for Monitoring and Quality Improvement in Community Delivery Settings

The field of prevention has established the potential to promote child adjustment across a wide array of outcomes. However, when evidence-based prevention programs have been delivered at scale in community settings, declines in implementation and outcomes have resulted. Maintaining high quality implementation is a critical challenge for the field. We describe steps towards the development of a practical system to monitor and support the high-quality implementation of evidence-based prevention programs in community settings. Research on the implementation of an evidence-based parenting program for divorcing families called the "New Beginnings Program" serves as an illustration of the promise of such a system. As a first step, we describe a multidimensional theoretical model of implementation that links aspects of program delivery with improvements in participant outcomes. We then describe research on the measurement of each of these implementation dimensions and test their relations to intended program outcomes. As a third step, we develop approaches to the assessment of these implementation constructs that are feasible to use in community settings and to establish their reliability and validity. We focus on the application of machine learning algorithms and web-based data collection systems to assess implementation and provide support for high quality delivery and positive outcomes. Examples are presented to demonstrate that valid and reliable measures can be collected using these methods. Finally, we envision how these measures can be used to develop an unobtrusive system to monitor implementation and provide feedback and support in real time to maintain high quality implementation and program outcomes.

Towards text mining therapeutic change: A systematic review of text-based methods for Therapeutic Change Process Research

Therapeutic Change Process Research (TCPR) connects within-therapeutic change processes to outcomes. The labour intensity of qualitative methods limit their use to small scale studies. Automated text-analyses (e.g. text mining) provide means for analysing large scale text patterns. We aimed to provide an overview of the frequently used qualitative text-based TCPR methods and assess the extent to which these methods are reliable and valid, and have potential for automation. We systematically reviewed PsycINFO, Scopus, and Web of Science to identify articles concerning change processes and text or language. We evaluated the reliability and validity based on replicability, the availability of code books, training data and inter-rater reliability, and evaluated the potential for automation based on the example- and rule-based approach. From 318 articles we identified four often used methods: Innovative Moments Coding Scheme, the Narrative Process Coding Scheme, Assimilation of Problematic Experiences Scale, and Conversation Analysis. The reliability and validity of the first three is sufficient to hold promise for automation. While some text features (content, grammar) lend themselves for automation through a rule-based approach, it should be possible to automate higher order constructs (e.g. schemas) when sufficient annotated data for an example-based approach are available.

Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention: A Quasi-Experimental Study

Assessment of fidelity that is effective, efficient, and differentiates from usual practices is critical for effectively implementing evidence-based programs for families. This quasi-experiemntal study sought to determine whether observational ratings of fidelity to the Family Check-Up (FCU) could differentiate between levels of clinician training in the model, and from services as usual, and whether rating segments of sessions could be equivalent to rating complete sessions. Coders rated 75 videotaped sessions-complete and 20-min segments-for fidelity, using a valid and reliable rating system across three groups: (a) highly trained in FCU with universal, routine monitoring; (b) minimally trained in FCU with optional, variable monitoring; and (c) services as usual with no training in the FCU. We hypothesized that certain dimensions of fidelity would differ by training, whereas others would not. The results indicated that, as expected, one dimension of fidelity to the FCU, Conceptually accurate to the FCU, was reliably different between the groups (χ² = 44.63, p < .001). The differences observed were in the expected direction, showing higher scores for therapists with more training. The rating magnitude of session segments largely did not differ from those of complete session ratings; however, interrater reliabilities were low for the segments. Although observational ratings were shown to be sensitive to the degree of training in the FCU on a unique and theoretically critical dimension, observational coding of complete sessions is resource intensive and limits scalability. Additional work is needed to reduce the burden of assessing fidelity to family-centered programs.

George SM, Prado G, Mauricio AM, Chiapa A, Bruening MM, Dishion TJ. The Family Check-Up 4 Health (FCU4Health): Applying Implementation Science Frameworks to the Process of Adapting an Evidence-Based Parenting Program for Prevention of Pediatric Obesity and Excess Weight Gain in Primary Care

Implementation experts have recently argued for a process of "scaling out" evidence-based interventions, programs, and practices (EBPs) to improve reach to new populations and new service delivery systems. A process of planned adaptation is typically required to integrate EBPs into new service delivery systems and address the needs of targeted populations while simultaneously maintaining fidelity to core components. This process-oriented paper describes the application of an implementation science framework and coding system to the adaptation of the Family Check-Up (FCU), for a new clinical target and service delivery system-prevention of obesity and excess weight game in primary care. The original FCU has demonstrated both short- and long-term effects on obesity with underserved families across a wide age range. The advantage of adapting such a program is the existing empirical evidence that the intervention improves the primary mediator of effects on the new target outcome. We offer a guide for determining the levels of evidence to undertake the adaptation of an existing EBP for a new clinical target. In this paper, adaptation included shifting the frame of the intervention from one of risk reduction to health promotion; adding health-specific assessments in the areas of nutrition, physical activity, sleep, and media parenting behaviors; family interaction tasks related to goals for health and health behaviors; and coordinating with community resources for physical health. We discuss the multi-year process of adaptation that began by engaging the FCU developer, community stakeholders, and families, which was then followed by a pilot feasibility study, and continues in an ongoing randomized effectiveness-implementation hybrid trial. The adapted program is called the Family Check-Up 4 Health (FCU4Health). We apply a comprehensive coding system for the adaptation of EBPs to our process and also provide a side-by-side comparison of behavior change techniques for obesity prevention and management used in the original FCU and in the FCU4Health. These provide a rigorous means of classification as well as a common language that can be used when adapting other EBPs for context, content, population, or clinical target. Limitations of such an approach to adaptation and future directions of this work are discussed.

An individually tailored family-centered intervention for pediatric obesity in primary care: study protocol of a randomized type II hybrid effectiveness-implementation trial (Raising Healthy Children study)

BACKGROUND

Pediatric obesity is a multi-faceted public health concern that can lead to cardiovascular diseases, cancers, and early mortality. Small changes in diet, physical activity, or BMI can significantly reduce the possibility of developing cardiometabolic risk factors. Family-based behavioral interventions are an underutilized, evidence-based approach that have been found to significantly prevent excess weight gain and obesity in children and adolescents. Poor program availability, low participation rates, and non-adherence are noted barriers to positive outcomes. Effective interventions for pediatric obesity in primary care are hampered by low family functioning, motivation, and adherence to recommendations.

METHODS

This (type II) hybrid effectiveness-implementation randomized trial tests the Family Check-Up 4 Health (FCU4Health) program, which was designed to target health behavior change in children by improving family management practices and parenting skills, with the goal of preventing obesity and excess weight gain. The FCU4Health is assessment driven to tailor services and increase parent motivation. A sample of 350 families with children aged 6 to 12 years who are identified as overweight or obese (BMI ≥ 85th percentile for age and gender) will be enrolled at three primary care clinics [two Federally Qualified Healthcare Centers (FQHCs) and a children's hospital]. All clinics serve predominantly Medicaid patients and a large ethnic minority population, including Latinos, African Americans, and American Indians who face disparities in obesity, cardiometabolic risk, and access to care. The FCU4Health will be coordinated with usual care, using two different delivery strategies: an embedded approach for the two FQHCs and a referral model for the hospital-based clinic. To assess program effectiveness (BMI, body composition, child health behaviors, parenting, and utilization of support services) and implementation outcomes (such outcomes as acceptability, adoption, feasibility, appropriateness, fidelity, and cost), we use a multi-method and multi-informant assessment strategy including electronic health record data, behavioral observation, questionnaires, interviews, and cost capture methods.

DISCUSSION

This study has the potential to prevent excess weight gain, obesity, and health disparities in children by establishing the effectiveness of the FCU4Health and collecting information critical for healthcare decision makers to support sustainable implementation of family-based programs in primary care.

TRIAL REGISTRATION

NCT03013309 ClinicalTrials.gov.

Accelerating Digital Mental Health Research From Early Design and Creation to Successful Implementation and Sustainment

Mental health problems are common and pose a tremendous societal burden in terms of cost, morbidity, quality of life, and mortality. The great majority of people experience barriers that prevent access to treatment, aggravated by a lack of mental health specialists. Digital mental health is potentially useful in meeting the treatment needs of large numbers of people. A growing number of efficacy trials have shown strong outcomes for digital mental health treatments. Yet despite their positive findings, there are very few examples of successful implementations and many failures. Although the research-to-practice gap is not unique to digital mental health, the inclusion of technology poses unique challenges. We outline some of the reasons for this gap and propose a collection of methods that can result in sustainable digital mental health interventions. These methods draw from human-computer interaction and implementation science and are integrated into an Accelerated Creation-to-Sustainment (ACTS) model. The ACTS model uses an iterative process that includes 2 basic functions (design and evaluate) across 3 general phases (Create, Trial, and Sustain). The ultimate goal in using the ACTS model is to produce a functioning technology-enabled service (TES) that is sustainable in a real-world treatment setting. We emphasize the importance of the service component because evidence from both research and practice has suggested that human touch is a critical ingredient in the most efficacious and used digital mental health treatments. The Create phase results in at least a minimally viable TES and an implementation blueprint. The Trial phase requires evaluation of both effectiveness and implementation while allowing optimization and continuous quality improvement of the TES and implementation plan. Finally, the Sustainment phase involves the withdrawal of research or donor support, while leaving a functioning, continuously improving TES in place. The ACTS model is a step toward bringing implementation and sustainment into the design and evaluation of TESs, public health into clinical research, research into clinics, and treatment into the lives of our patients.

Factors associated with the implementation of the Familias Unidas intervention in a type 3 translational trial

This study highlights how Familias Unidas, a Hispanic-specific, evidence-based, family centered preventive intervention, progressed from intervention development (type 1 translation; T1) through rigorous evaluation (T2) and examines the role of intervention fidelity-adherence and competence-in a T3 trial. Effects of participant, provider, and organizational variables on direct (observational) and indirect (self-reported) fidelity were examined as were effects of fidelity. Two structural equation models were estimated using data from 367 Hispanic parent-adolescent dyads randomized to Familias Unidas. Facilitator perceptions of parental involvement in schools, school performance grade, and school socioeconomic status predicted indirect adherence ratings, which were positively related to adolescent substance use. Facilitator openness to evidence-based practices was associated with indirect competence ratings, school performance grade and size were associated with direct competence ratings, and direct competence ratings were negatively associated with substance use. Findings highlight unique contributions of direct and indirect fidelity ratings in the implementation of Familias Unidas.

Increasing the Delivery of Preventive Health Services in Public Education

The delivery of prevention services to children and adolescents through traditional healthcare settings is challenging for a variety of reasons. Parent- and community-focused services are typically not reimbursable in traditional medical settings, and personal healthcare services are often designed for acute and chronic medical treatment rather than prevention. To provide preventive services in a setting that reaches the widest population, those interested in public health and prevention often turn to school settings. This paper proposes that an equitable, efficient manner in which to promote health across the life course is to integrate efforts from public health, primary care, and public education through the delivery of preventive healthcare services, in particular, in the education system. Such an integration of systems will require a concerted effort on the part of various stakeholders, as well as a shared vision to promote child health via community and institutional stakeholder partnerships. This paper includes (1) examination of some key system features necessary for delivery of preventive services that improve child outcomes; (2) a review of the features of some common models of school health services for their relevance to prevention services; and (3) policy and implementation strategy recommendations to further the delivery of preventive services in schools. These recommendations include the development of common metrics for health outcomes reporting, facilitated data sharing of these metrics, shared organization incentives for integration, and improved reimbursement and funding opportunities.

Automatic classification of communication logs into implementation stages via text analysis

BACKGROUND

To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes.

RESULTS

We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage.

CONCLUSIONS

This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.

Human Subjects Protection and Technology in Prevention Science: Selected Opportunities and Challenges

Internet-connected devices are changing the way people live, work, and relate to one another. For prevention scientists, technological advances create opportunities to promote the welfare of human subjects and society. The challenge is to obtain the benefits while minimizing risks. In this article, we use the guiding principles for ethical human subjects research and proposed changes to the Common Rule regulations, as a basis for discussing selected opportunities and challenges that new technologies present for prevention science. The benefits of conducting research with new populations, and at new levels of integration into participants' daily lives, are presented along with five challenges along with technological and other solutions to strengthen the protections that we provide: (1) achieving adequate informed consent with procedures that are acceptable to participants in a digital age; (2) balancing opportunities for rapid development and broad reach, with gaining adequate understanding of population needs; (3) integrating data collection and intervention into participants' lives while minimizing intrusiveness and fatigue; (4) setting appropriate expectations for responding to safety and suicide concerns; and (5) safeguarding newly available streams of sensitive data. Our goal is to promote collaboration between prevention scientists, institutional review boards, and community members to safely and ethically harness advancing technologies to strengthen impact of prevention science.

"It sounds like...": A natural language processing approach to detecting counselor reflections in motivational interviewing

The dissemination and evaluation of evidence-based behavioral treatments for substance abuse problems rely on the evaluation of counselor interventions. In Motivational Interviewing (MI), a treatment that directs the therapist to utilize a particular linguistic style, proficiency is assessed via behavioral coding-a time consuming, nontechnological approach. Natural language processing techniques have the potential to scale up the evaluation of behavioral treatments such as MI. We present a novel computational approach to assessing components of MI, focusing on 1 specific counselor behavior-reflections, which are believed to be a critical MI ingredient. Using 57 sessions from 3 MI clinical trials, we automatically detected counselor reflections in a maximum entropy Markov modeling framework using the raw linguistic data derived from session transcripts. We achieved 93% recall, 90% specificity, and 73% precision. Results provide insight into the linguistic information used by coders to make ratings and demonstrate the feasibility of new computational approaches to scaling up the evaluation of behavioral treatments.