Study protocol for a group randomized controlled trial of a classroom-based intervention aimed at preventing early risk factors for drug abuse: integrating effectiveness and implementation research

Testing the Impact of the Whole-Day Good Behavior Game on Aggressive Behavior: Results of a Classroom-Based Randomized Effectiveness Trial

Three generations of developmental epidemiologically based randomized field trials of the Good Behavior Game (GBG) have been delivered to Baltimore elementary schools. With the collaboration of family and community partners, all three trials were directed at decreasing proximal targets of aggressive behavior and improving learning in first-grade classrooms with distal mental health and substance abuse outcomes. GBG is a group-contingent classroom behavior management strategy that promotes classmate/peer concern for each child's positive behavior by rewarding teams with below-criterion levels of aggressive, disruptive behavior. GBG targets early risk factors for the above distal outcomes: aggressive, disruptive behavior, family/school relationships, and school failure. Here, we report on the third-generation randomized prevention trial of the GBG (whole-day first grade program (WD)), including 12 elementary schools. WD enhanced the standard curriculum in the areas of classroom behavior management; academic instruction, particularly reading; and family-classroom partnerships. Using a within-school classroom randomized trial design, we: 1) evaluate the effectiveness of the WD program by sex and cohort and 2) measure variation in WD impact by the quality of teachers' behavior management practices. Data from 961 first graders were used in general growth mixture modeling that accounts for classroom randomization to identify distinct developmental trajectories of aggressive, disruptive behavior and GBG impact on these trajectories. In the chronic high aggression trajectory of males, ratings of aggression after WD implementation and to the end of third grade were significantly lower in the WD condition than in controls in classrooms with a higher WD dosage (Cohort 2) and especially in classrooms with higher quality of WD implementation. For females, we found a modest but significant benefit of GBG in the low trajectory class when cohorts were combined. Regarding policy implications, embedding GBG into the curricula in teacher's colleges could better support student learning and behavior. Clinical Trials Registration number: NCT00257088.

An Overview of Research and Evaluation Designs for Dissemination and Implementation

The wide variety of dissemination and implementation designs now being used to evaluate and improve health systems and outcomes warrants review of the scope, features, and limitations of these designs. This article is one product of a design workgroup that was formed in 2013 by the National Institutes of Health to address dissemination and implementation research, and whose members represented diverse methodologic backgrounds, content focus areas, and health sectors. These experts integrated their collective knowledge on dissemination and implementation designs with searches of published evaluations strategies. This article emphasizes randomized and nonrandomized designs for the traditional translational research continuum or pipeline, which builds on existing efficacy and effectiveness trials to examine how one or more evidence-based clinical/prevention interventions are adopted, scaled up, and sustained in community or service delivery systems. We also mention other designs, including hybrid designs that combine effectiveness and implementation research, quality improvement designs for local knowledge, and designs that use simulation modeling.

Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention

Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146-156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on "joining," which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

Measuring sustainment of prevention programs and initiatives: a study protocol

BACKGROUND

Sustaining prevention efforts directed at substance use and mental health problems is one of the greatest, yet least understood, challenges in the field of implementation science. A large knowledge gap exists regarding the meaning of the term "sustainment" and what factors predict or even measure sustainability of effective prevention programs and support systems.

METHODS/DESIGN

The U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) supports a diverse portfolio of prevention and treatment grant programs that aim to improve population and individual level behavioral health. This study focuses on four SAMHSA prevention grant programs, two of which target substance abuse prevention at the state or single community level, one targets suicide prevention, and one targets prevention of aggressive/disruptive behavior in elementary schools. An examination of all four grant programs simultaneously provides an opportunity to determine what is meant by the term sustainment and identify and support both the unique requirements for improving sustainability for each program as well as for developing a generalizable framework comprised of core components of sustainment across diverse prevention approaches. Based on an analysis of qualitative and quantitative data of 10 grantees supported by these four programs, we will develop a flexible measurement system, with both general and specific components, that can bring precision to monitoring sustainment of infrastructure, activities, and outcomes for each prevention approach. We will then transform this system for use in evaluating and improving the likelihood of achieving prevention effort sustainment. To achieve these goals, we will (1) identify core components of sustainment of prevention programs and their support infrastructures; (2) design a measurement system for monitoring and providing feedback regarding sustainment within the four SAMHSA's prevention-related grant programs; and (3) pilot test the predictability of this multilevel measurement system across these programs and the feasibility and acceptability of a measurement system to evaluate and improve the likelihood of sustainment.

DISCUSSION

This project is intended to improve sustainment of the supporting prevention infrastructure, activities, and outcomes that are funded by federal, state, community, and foundation sources.

Characterizing implementation strategies using a systems engineering survey and interview tool: a comparison across 10 prevention programs for drug abuse and HIV sexual risk behavior

Background

Although many behavioral interventions have proven to be efficacious, new methodologies are required beyond efficacy trials to understand how to adopt, implement with fidelity, and sustain behavioral interventions in community settings. In this paper, we present a new approach, based on systems engineering concepts and methods, for characterizing implementation strategies that are used to deliver evidence-based behavioral interventions in health and social service settings. We demonstrate the use of this approach with implementation strategies, used or being used for broader dissemination of 10 evidence-based prevention program projects focused on the prevention of drug or HIV sex risk behaviors.

Results

The results indicate that there are wide variations in intervention approaches and that there are challenges in program implementation including maintaining program fidelity, serving community needs, and adequate resources. The results also indicate that implementation requires a committed partnership between the program developers, implementation researchers, and community partners. In addition, there is a need for adaptability within programs to meet community needs, resources, and priorities while maintaining program fidelity.

Conclusions

Our methodological approach enabled us to highlight challenges associated with the community implementation of health risk prevention interventions. We also demonstrate how comprehensive descriptions of interventions facilitate understanding of the requirements of program implementation and decisions about the feasibility of implementing a program in community settings.

Universal school-based prevention for illicit drug use

BACKGROUND

Drug addiction is a chronic, relapsing disease. Primary interventions should aim to reduce first use or to prevent the transition from experimental use to addiction. School is the appropriate setting for preventive interventions.

OBJECTIVES

To evaluate the effectiveness of universal school-based interventions in reducing drug use compared to usual curricular activities or no intervention.

SEARCH METHODS

We searched the Cochrane Drugs and Alcohol Group's Trials Register (September 2013), the Cochrane Central Register of Controlled Trials (2013, Issue 9), PubMed (1966 to September 2013), EMBASE (1988 to September 2013) and other databases. We also contacted researchers in the field and checked reference lists of articles.

SELECTION CRITERIA

Randomised controlled trials (RCT) evaluating school-based interventions designed to prevent illicit drugs use.

DATA COLLECTION AND ANALYSIS

We used the standard methodological procedures expected by The Cochrane Collaboration.

MAIN RESULTS

We included 51 studies, with 127,146 participants. Programmes were mainly delivered in sixth and seventh grade pupils. Most of the trials were conducted in the USA. Social competence approach versus usual curricula or no intervention Marijuana use at < 12 months follow-up: the results favoured the social competence intervention (risk ratio (RR) 0.90; 95% confidence interval (CI) 0.81 to 1.01, four studies, 9456 participants, moderate quality evidence). Seven studies assessed this outcome (no data for meta-analysis): two showed a positive significant effect of intervention, three showed a non-significant effect, one found a significant effect in favour of the control group and one found a trend in favour of the control group.Marijuana use at 12+ months: the results favoured the social competence intervention (RR 0.86; 95% CI 0.74 to 1.00, one study, 2678 participants, high quality evidence). Seven studies assessed this outcome (no data for meta-analysis): two showed a significant positive effect of intervention, three showed a non-significant effect, one found a significant effect in favour of the control group and one a trend in favour of the control group.Hard drug use at < 12 months: we found no difference (RR 0.69; 95% CI 0.40 to 1.18, one study, 2090 participants, moderate quality evidence). Two studies assessed this outcome (no data for meta-analysis): one showed comparable results for the intervention and control group; one found a statistically non-significant trend in favour of the social competence approach.Hard drug use at 12+ months: we found no difference (mean difference (MD) -0.01; 95% CI -0.06 to 0.04), one study, 1075 participants, high quality evidence). One study with no data for meta-analysis showed comparable results for the intervention and control group.Any drug use at < 12 months: the results favoured social competence interventions (RR 0.27; 95% CI 0.14 to 0.51, two studies, 2512 participants, moderate quality evidence). One study with 1566 participants provided continuous data showing no difference (MD 0.02; 95% CI -0.05 to 0.09, moderate quality evidence). Social influence approach versus usual curricula or no intervention Marijuana use at < 12 months: we found a nearly statistically significant effect in favour of the social influence approach (RR 0.88; 95% CI 0.72 to 1.07, three studies, 10,716 participants, moderate quality evidence). One study with 764 participants provided continuous data showing results that favoured the social influence intervention (MD -0.26; 95% CI -0.48 to -0.04).Marijuana use at 12+ months: we found no difference (RR 0.95; 95% CI 0.81 to 1.13, one study, 5862 participants, moderate quality evidence). One study with 764 participants provided continuous data and showed nearly statistically significant results in favour of the social influence intervention (MD -0.22; 95% CI -0.46 to 0.02). Of the four studies not providing data for meta-analysis a statistically significant protective effect was only found by one study.Hard drug use at 12+ months: one study not providing data for meta-analysis found a significant protective effect of the social influence approach.Any drug use: no studies assessed this outcome. Combined approach versus usual curricula or no intervention Marijuana use at < 12 months: there was a trend in favour of intervention (RR 0.79; 95% CI 0.59 to 1.05, three studies, 8701 participants, moderate quality evidence). One study with 693 participants provided continuous data and showed no difference (MD -1.90; 95% CI -5.83 to 2.03).Marijuana use at 12+ months: the results favoured combined intervention (RR 0.83; 95% CI 0.69 to 0.99, six studies, 26,910 participants, moderate quality evidence). One study with 690 participants provided continuous data and showed no difference (MD -0.80; 95% CI -4.39 to 2.79). Two studies not providing data for meta-analysis did not find a significant effect.Hard drug use at < 12 months: one study with 693 participants provided both dichotomous and continuous data and showed conflicting results: no difference for dichotomous outcomes (RR 0.85; 95% CI 0.63 to 1.14), but results in favour of the combined intervention for the continuous outcome (MD -3.10; 95% CI -5.90 to -0.30). The quality of evidence was high.Hard drug use at 12+ months: we found no difference (RR 0.86; 95% CI 0.39 to 1.90, two studies, 1066 participants, high quality evidence). One study with 690 participants provided continuous data and showed no difference (MD 0.30; 95% CI -1.36 to 1.96). Two studies not providing data for meta-analysis showed a significant effect of treatment.Any drug use at < 12 months: the results favoured combined intervention (RR 0.76; 95% CI 0.64 to 0.89, one study, 6362 participants).Only one study assessed the effect of a knowledge-focused intervention on drug use and found no effect. The types of comparisons and the programmes assessed in the other two groups of studies were very heterogeneous and difficult to synthesise.

AUTHORS' CONCLUSIONS

School programmes based on a combination of social competence and social influence approaches showed, on average, small but consistent protective effects in preventing drug use, even if some outcomes did not show statistical significance. Some programmes based on the social competence approach also showed protective effects for some outcomes.Since the effects of school-based programmes are small, they should form part of more comprehensive strategies for drug use prevention in order to achieve a population-level impact.

Guided by Theory, Informed by Practice: Training and Support for the Good Behavior Game, a Classroom-based Behavior Management Strategy

Moving evidence-based practices for classroom behavior management into real-world settings is a high priority for education and public health. This paper describes the development and use of a model of training and support for the Good Behavior Game (GBG), one of the few preventive interventions shown to have positive outcomes for elementary school children lasting through to young adulthood, ages 19-21, including reductions in the use of drugs and alcohol, school-based mental health services, and suicide ideation and attempts. We first describe the conceptual framework guiding the development of the model of training and support. Data on implementation of the model, from an ongoing trial of GBG being conducted in partnership with the Houston Independent School District, are then presented. We end with a discussion of the lessons learned and the implications for the next stage of research and practice.

Addressing core challenges for the next generation of type 2 translation research and systems: the translation science to population impact (TSci Impact) framework

Evidence-based preventive interventions developed over the past two decades represent great potential for enhancing public health and well-being. Research confirming the limited extent to which these interventions have been broadly and effectively implemented, however, indicates much progress is needed to achieve population-level impact. In part, progress requires Type 2 translation research that investigates the complex processes and systems through which evidence-based interventions are adopted, implemented, and sustained on a large scale, with a strong orientation toward devising empirically-driven strategies for increasing their population impact. In this article, we address two core challenges to the advancement of T2 translation research: (1) building infrastructure and capacity to support systems-oriented scaling up of evidence-based interventions, with well-integrated practice-oriented T2 research, and (2) developing an agenda and improving research methods for advancing T2 translation science. We also summarize a heuristic "Translation Science to Population Impact (TSci Impact) Framework." It articulates key considerations in addressing the core challenges, with three components that represent: (1) four phases of translation functions to be investigated (pre-adoption, adoption, implementation, and sustainability); (2) the multiple contexts in which translation occurs, ranging from community to national levels; and (3) necessary practice and research infrastructure supports. Discussion of the framework addresses the critical roles of practitioner-scientist partnerships and networks, governmental agencies and policies at all levels, plus financing partnerships and structures, all required for both infrastructure development and advances in the science. The article concludes with two sets of recommended action steps that could provide impetus for advancing the next generation of T2 translation science and, in turn, potentially enhance the health and well-being of subsequent generations of youth and families.

A computational future for preventing HIV in minority communities: how advanced technology can improve implementation of effective programs

African Americans and Hispanics in the United States have much higher rates of HIV than non-minorities. There is now strong evidence that a range of behavioral interventions are efficacious in reducing sexual risk behavior in these populations. Although a handful of these programs are just beginning to be disseminated widely, we still have not implemented effective programs to a level that would reduce the population incidence of HIV for minorities. We proposed that innovative approaches involving computational technologies be explored for their use in both developing new interventions and in supporting wide-scale implementation of effective behavioral interventions. Mobile technologies have a place in both of these activities. First, mobile technologies can be used in sensing contexts and interacting to the unique preferences and needs of individuals at times where intervention to reduce risk would be most impactful. Second, mobile technologies can be used to improve the delivery of interventions by facilitators and their agencies. Systems science methods including social network analysis, agent-based models, computational linguistics, intelligent data analysis, and systems and software engineering all have strategic roles that can bring about advances in HIV prevention in minority communities. Using an existing mobile technology for depression and 3 effective HIV prevention programs, we illustrated how 8 areas in the intervention/implementation process can use innovative computational approaches to advance intervention adoption, fidelity, and sustainability.

Developing a collaboration with the Houston independent school district: testing the generalizability of a partnership model

Moving evidence-based practices into real-world settings is a high priority for education and public health. This paper describes the development of a partnership among the Houston Independent School District, the American Institutes of Research, and the Houston Federation of Teachers to support research on and program sustainability for the Good Behavior Game, a team-based classroom behavior management strategy that has shown positive impact in randomized field trials. The conceptual framework guiding partnership development is presented, followed by an application of the framework in Houston. Lessons learned and implications for the next stage of research and practice are then discussed.

Partnerships for the design, conduct, and analysis of effectiveness, and implementation research: experiences of the prevention science and methodology group

What progress prevention research has made comes through strategic partnerships with communities and institutions that host this research, as well as professional and practice networks that facilitate the diffusion of knowledge about prevention. We discuss partnership issues related to the design, analysis, and implementation of prevention research and especially how rigorous designs, including random assignment, get resolved through a partnership between community stakeholders, institutions, and researchers. These partnerships shape not only study design, but they determine the data that can be collected and how results and new methods are disseminated. We also examine a second type of partnership to improve the implementation of effective prevention programs into practice. We draw on social networks to studying partnership formation and function. The experience of the Prevention Science and Methodology Group, which itself is a networked partnership between scientists and methodologists, is highlighted.

Design elements in implementation research: a structured review of child welfare and child mental health studies

Implementation science is an emerging field of research with considerable penetration in physical medicine and less in the fields of mental health and social services. There remains a lack of consensus on methodological approaches to the study of implementation processes and tests of implementation strategies. This paper addresses the need for methods development through a structured review that describes design elements in nine studies testing implementation strategies for evidence-based interventions addressing mental health problems of children in child welfare and child mental health settings. Randomized trial designs were dominant with considerable use of mixed method designs in the nine studies published since 2005. The findings are discussed in reference to the limitations of randomized designs in implementation science and the potential for use of alternative designs.

Examining implementer fidelity: Conceptualizing and measuring adherence and competence

The large gap between intervention outcomes demonstrated in efficacy trials and the apparent ineffectiveness of these same programs in community settings has prompted investigators and practitioners to look closely at implementation fidelity. Critically important, but often overlooked, are the implementers who deliver evidence-based programs -- the effectiveness of programs cannot surpass skill levels of the people implementing them. This article distinguishes fidelity at the programmatic level from implementer fidelity. Two components of implementer fidelity are defined. Implementer adherence and competence are proposed to be related but unique constructs that can be reliably measured for training¸ monitoring, and outcomes research. Observational measures from a school-based preventive intervention are provided and the unique contributions of implementer adherence and competence are illustrated. Distinguishing implementer adherence to the manual and competence in program delivery is a critical next step in child mental health program implementation research.