Measuring Quality of Delivery in a Substance Use Prevention Program

A national effectiveness trial of an eHealth program to prevent alcohol and cannabis misuse: responding to the replication crisis

BACKGROUND

The burden of disease attributable to alcohol and other drug (AOD) use in young people is considerable. Prevention can be effective, yet few programs have demonstrated replicable effects. This study aimed to replicate research behind Climate Schools: Alcohol and Cannabis course among a large cohort of adolescents.

METHODS

Seventy-one secondary schools across three States participated in a cluster-randomised controlled trial. Year 8 students received either the web-based Climate Schools: Alcohol and Cannabis course (Climate, n = 3236), or health education as usual (Control, n = 3150). Outcomes were measured via self-report and reported here for baseline, 6- and 12-months for alcohol and cannabis knowledge, alcohol, cannabis use and alcohol-related harms.

RESULTS

Compared to Controls, students in the Climate group showed greater increases in alcohol- [standardised mean difference (SMD) 0.51, p < 0.001] and cannabis-related knowledge (SMD 0.49, p < 0.001), less increases in the odds of drinking a full standard drink[(odds ratio (OR) 0.62, p = 0.014], and heavy episodic drinking (OR 0.49, p = 0.022). There was no evidence for differences in change over time in the odds of cannabis use (OR 0.57, p = 0.22) or alcohol harms (OR 0.73, p = 0.17).

CONCLUSIONS

The current study provides support for the effectiveness of the web-based Climate Schools: Alcohol and Cannabis course in increasing knowledge and reducing the uptake of alcohol. It represents one of the first trials of a web-based AOD prevention program to replicate alcohol effects in a large and diverse sample of students. Future research and/or adaptation of the program may be warranted with respect to prevention of cannabis use and alcohol harms.

Formative Evaluation to Build an Online Parenting Skills and Youth Drug Prevention Program: Mixed Methods Study

Background

Family-based drug prevention programs that use group-based formats with trained facilitators, such as the Strengthening Families Program (SFP), are effective in preventing underage drinking and youth drug use. However, these programs are resource-intensive and have high costs and logistical demands. Tailoring them for Web-based delivery is more cost-effective and makes it easier to scale these programs for widespread dissemination. This requires the active involvement of all key stakeholders to determine content and delivery format.

Objective

The aim was to obtain consumer, agency stakeholder, and expert input into the design of a Web-based parenting skills training and youth drug prevention program.

Methods

We conducted 10 focus groups with 85 adults (range 4-10, average 8 per group), 20 stakeholder interviews with family services agency staff, and discussed critical design considerations with 10 prevention scientists and e-learning experts to determine the optimal program content and technology features for SFP Online. Focus group participants also answered survey questions on perceived barriers to use, desired navigational features, preferred course format, desired content, preferred reward structures, course length, interactive components, computer efficacy, and technology use. Descriptive statistics were used to examine consumer characteristics; linear regression was used to examine relations between SFP exposure and four continuous outcome measures, including desired program content, interactive technology, and concerns that may inhibit future use of SFP Online. Logistic regression was used as a binary measure of whether consumers desired fun games in the SFP Online program.

Results

Three broad thematic categories emerged from the qualitative interviews enumerating the importance of (1) lesson content, (2) logistics for program delivery, and (3) multimedia interactivity. Among the many significant relations, parents who viewed more SFP lessons reported more reasons to use an online program (beta=1.48, P=.03) and also wanted more interactivity (6 lessons: beta=3.72, P=.01; >6 lessons: beta=2.39, P=.01), parents with less interest in a mixed delivery format (class and online) reported fewer reasons to use the online program (beta=−3.93, P=.01), comfort using computers was negatively associated with concerns about the program (beta=−1.83, P=.01), having mobile phone access was related to fewer concerns about online programs (beta=−1.63, P=.02), willingness to view an online program using a mobile phone was positively associated with wanting more online components (beta=1.95, P=.02), and parents who wanted fun games wanted more interactivity (beta=2.28, P=.01).

Conclusions

Formative evaluation based on user-centered approaches can provide rich information that fuels development of an online program. The user-centered strategies in this study lay the foundation for improving SFP Online and provide a means to accommodate user interests and ensure the product serves as an effective prevention tool that is attractive to consumers, engaging, and can overcome some of the barriers to recruitment and retention that have previously affected program outcomes in family-based prevention.

Self-Reported Engagement in a Drug Prevention Program: Individual and Classroom Effects on Proximal and Behavioral Outcomes

Numerous studies emphasize the role of student engagement in academic learning and performance. Less known is whether engagement plays a role in drug prevention program outcomes. We examined a self-report measure of engagement as part of the All Stars Core drug prevention program evaluation, assessing its impact on target risk mechanisms and behavioral outcomes. Students completed pretests just prior to and posttests just after completing the intervention. Surveys assessed demographics, proximal intervening measures (i.e., commitments to avoid substance use and antisocial behavior, perceived lifestyle incongruence with substance use and antisocial behavior, normative beliefs about substance use and antisocial behavior, and parental attentiveness), and distal outcome measures of alcohol, cigarette use, and antisocial behaviors. A brief 6-item posttest measure including items tapping the students' perspective on the quality of teaching the program material and their level of engagement with the program was internally consistent (α = .79). Multi-level analyses positing engagement effects at both the classroom- and individual-level indicated that classroom average engagement was significantly associated with all the targeted risk mechanisms, and outcomes of antisocial behavior and alcohol use, controlling for pretest measures and classroom size. Individual student engagement relative to classroom peers was significantly associated with all posttest target risk mechanisms and behavioral outcomes. The current findings suggest that students should routinely provide assessments of engagement and perceived quality of teaching, which would improve our understanding of how prevention programs work. Teachers can improve engagement by paying attention to students when they speak in class, making the program enjoyable to participants, encouraging students to share opinions, stimulating attentiveness, being well prepared to deliver the intervention, and helping students think broadly about implications of drug prevention as it affects their lives. This type of support will ultimately engage students in ways that will enhance the likelihood that these programs will have their desired effects.

Typology of delivery quality: latent profile analysis of teacher engagement and delivery techniques in a school-based prevention intervention, keepin' it REAL curriculum

Enhancing the delivery quality of school-based, evidence-based prevention programs is one key to ensuring uniform program effects on student outcomes. Program evaluations often focus on content dosage when implementing prevention curricula, however, less is known about implementation quality of prevention content, especially among teachers who may or may not have a prevention background. The goal of the current study is to add to the scholarly literature on implementation quality for a school-based substance use prevention intervention. Twenty-five schools in Ohio and Pennsylvania implemented the original keepin' REAL (kiR) substance use prevention curriculum. Each of the 10, 40-45 min lessons of the kiR curriculum was video recorded. Coders observed and rated a random sample of 276 videos reflecting 78 classes taught by 31 teachers. Codes included teachers' delivery techniques (e.g., lecture, discussion, demonstration and role play) and engagement with students (e.g. attentiveness, enthusiasm and positivity). Based on the video ratings, a latent profile analysis was run to identify typology of delivery quality. Five profiles were identified: holistic approach, attentive teacher-orientated approach, enthusiastic lecture approach, engaged interactive learning approach and skill practice-only approach. This study provides a descriptive typology of delivery quality while implementing a school-based substance use prevention intervention.

School-based programmes for preventing smoking

BACKGROUND

Helping young people to avoid starting smoking is a widely endorsed public health goal, and schools provide a route to communicate with nearly all young people. School-based interventions have been delivered for close to 40 years.

OBJECTIVES

The primary aim of this review was to determine whether school smoking interventions prevent youth from starting smoking. Our secondary objective was to determine which interventions were most effective. This included evaluating the effects of theoretical approaches; additional booster sessions; programme deliverers; gender effects; and multifocal interventions versus those focused solely on smoking.

SEARCH METHODS

We searched the Cochrane Central Register of Controlled Trials (CENTRAL), the Cochrane Tobacco Addiction Group's Specialised Register, MEDLINE, EMBASE, PsycINFO, ERIC, CINAHL, Health Star, and Dissertation Abstracts for terms relating to school-based smoking cessation programmes. In addition, we screened the bibliographies of articles and ran individual MEDLINE searches for 133 authors who had undertaken randomised controlled trials in this area. The most recent searches were conducted in October 2012.

SELECTION CRITERIA

We selected randomised controlled trials (RCTs) where students, classes, schools, or school districts were randomised to intervention arm(s) versus a control group, and followed for at least six months. Participants had to be youth (aged 5 to 18). Interventions could be any curricula used in a school setting to deter tobacco use, and outcome measures could be never smoking, frequency of smoking, number of cigarettes smoked, or smoking indices.

DATA COLLECTION AND ANALYSIS

Two reviewers independently assessed studies for inclusion, extracted data and assessed risk of bias. Based on the type of outcome, we placed studies into three groups for analysis: Pure Prevention cohorts (Group 1), Change in Smoking Behaviour over time (Group 2) and Point Prevalence of Smoking (Group 3).

MAIN RESULTS

One hundred and thirty-four studies involving 428,293 participants met the inclusion criteria. Some studies provided data for more than one group.Pure Prevention cohorts (Group 1) included 49 studies (N = 142,447). Pooled results at follow-up at one year or less found no overall effect of intervention curricula versus control (odds ratio (OR) 0.94, 95% confidence interval (CI) 0.85 to 1.05). In a subgroup analysis, the combined social competence and social influences curricula (six RCTs) showed a statistically significant effect in preventing the onset of smoking (OR 0.49, 95% CI 0.28 to 0.87; seven arms); whereas significant effects were not detected in programmes involving information only (OR 0.12, 95% CI 0.00 to 14.87; one study), social influences only (OR 1.00, 95% CI 0.88 to 1.13; 25 studies), or multimodal interventions (OR 0.89, 95% CI 0.73 to 1.08; five studies). In contrast, pooled results at longest follow-up showed an overall significant effect favouring the intervention (OR 0.88, 95% CI 0.82 to 0.96). Subgroup analyses detected significant effects in programmes with social competence curricula (OR 0.52, 95% CI 0.30 to 0.88), and the combined social competence and social influences curricula (OR 0.50, 95% CI 0.28 to 0.87), but not in those programmes with information only, social influence only, and multimodal programmes.Change in Smoking Behaviour over time (Group 2) included 15 studies (N = 45,555). At one year or less there was a small but statistically significant effect favouring controls (standardised mean difference (SMD) 0.04, 95% CI 0.02 to 0.06). For follow-up longer than one year there was a statistically nonsignificant effect (SMD 0.02, 95% CI -0.00 to 0.02).Twenty-five studies reported data on the Point Prevalence of Smoking (Group 3), though heterogeneity in this group was too high for data to be pooled.We were unable to analyse data for 49 studies (N = 152,544).Subgroup analyses (Pure Prevention cohorts only) demonstrated that at longest follow-up for all curricula combined, there was a significant effect favouring adult presenters (OR 0.88, 95% CI 0.81 to 0.96). There were no differences between tobacco-only and multifocal interventions. For curricula with booster sessions there was a significant effect only for combined social competence and social influences interventions with follow-up of one year or less (OR 0.50, 95% CI 0.26 to 0.96) and at longest follow-up (OR 0.51, 95% CI 0.27 to 0.96). Limited data on gender differences suggested no overall effect, although one study found an effect of multimodal intervention at one year for male students. Sensitivity analyses for Pure Prevention cohorts and Change in Smoking Behaviour over time outcomes suggested that neither selection nor attrition bias affected the results.

AUTHORS' CONCLUSIONS

Pure Prevention cohorts showed a significant effect at longest follow-up, with an average 12% reduction in starting smoking compared to the control groups. However, no overall effect was detected at one year or less. The combined social competence and social influences interventions showed a significant effect at one year and at longest follow-up. Studies that deployed a social influences programme showed no overall effect at any time point; multimodal interventions and those with an information-only approach were similarly ineffective.Studies reporting Change in Smoking Behaviour over time did not show an overall effect, but at an intervention level there were positive findings for social competence and combined social competence and social influences interventions.

Engagement matters: lessons from assessing classroom implementation of steps to respect: a bullying prevention program over a one-year period

Steps to Respect: A Bullying Prevention Program (STR) relies on a social-ecological model of prevention to increase school staff awareness and responsiveness, foster socially responsible beliefs among students, and teach social-emotional skills to students to reduce bullying behavior. As part of a school-randomized controlled trial of STR, we examined predictors and outcomes associated with classroom curriculum implementation in intervention schools. Data on classroom implementation (adherence and engagement) were collected from a sample of teachers using a weekly on-line Teacher Implementation Checklist system. Pre-post data related to school bullying-related outcomes were collected from 1,424 students and archival school demographic data were obtained from the National Center for Education Statistics. Results of multilevel analyses indicated that higher levels of program engagement were influenced by school-level percentage of students receiving free/reduced lunch, as well as classroom-level climate indicators. Results also suggest that higher levels of program engagement were related to lower levels of school bullying problems, enhanced school climate and attitudes less supportive of bullying. Predictors and outcomes related to program fidelity (i.e., adherence) were largely nonsignificant. Results suggest that student engagement is a key element of program impact, though implementation is influenced by both school-level demographics and classroom contexts.

Describing teacher-student interactions: a qualitative assessment of teacher implementation of the 7th grade keepin' it REAL substance use intervention

Variations in the delivery of school-based substance use prevention curricula affect students' acquisition of the lesson content and program outcomes. Although adaptation is sometimes viewed as a lack of fidelity, it is unclear what types of variations actually occur in the classroom. This observational study investigated teacher and student behaviors during implementation of a middle school-based drug prevention curriculum in 25 schools across two Midwestern states. Trained observers coded videos of 276 lessons, reflecting a total of 31 predominantly Caucasian teachers (10 males and 21 females) in 73 different classes. Employing qualitative coding procedures, the study provides a working typology of implementation patterns based on varying levels of teacher control and student participation. These patterns are fairly consistent across lessons and across classes of students, suggesting a teacher-driven delivery model where teachers create a set of constraints within which students vary their engagement. Findings provide a descriptive basis grounded in observation of classroom implementation that can be used to test models of implementation fidelity and quality as well as impact training and other dissemination research.

Measuring Fidelity and Adaptation: Reliability of a Instrument for School-Based Prevention Programs

There is a need to standardize methods for assessing fidelity and adaptation. Such standardization would allow program implementation to be examined in a manner that will be useful for understanding the moderating role of fidelity in dissemination research. This article describes a method for collecting data about fidelity of implementation for school-based prevention programs, including measures of adherence, quality of delivery, dosage, participant engagement, and adaptation. We report about the reliability of these methods when applied by four observers who coded video recordings of teachers delivering All Stars, a middle school drug prevention program. Interrater agreement for scaled items was assessed for an instrument designed to evaluate program fidelity. Results indicated sound interrater reliability for items assessing adherence, dosage, quality of teaching, teacher understanding of concepts, and program adaptations. The interrater reliability for items assessing potential program effectiveness, classroom management, achievement of activity objectives, and adaptation valences was improved by dichotomizing the response options for these items. The item that assessed student engagement demonstrated only modest interrater reliability and was not improved through dichotomization. Several coder pairs were discordant on items that overall demonstrated good interrater reliability. Proposed modifications to the coding manual and protocol are discussed.

Translational Research in South Africa: Evaluating Implementation Quality Using a Factorial Design

BACKGROUND: HealthWise South Africa: Life Skills for Adolescents (HW) is an evidence-based substance use and sexual risk prevention program that emphasizes the positive use of leisure time. Since 2000, this program has evolved from pilot testing through an efficacy trial involving over 7,000 youth in the Cape Town area. Beginning in 2011, through 2015, we are undertaking a new study that expands HW to all schools in the Metro South Education District. OBJECTIVE: This paper describes a research study designed in partnership with our South African collaborators that examines three factors hypothesized to affect the quality and fidelity of HW implementation: enhanced teacher training; teacher support, structure and supervision; and enhanced school environment. METHODS: Teachers and students from 56 schools in the Cape Town area will participate in this study. Teacher observations are the primary means of collecting data on factors affecting implementation quality. These factors address the practical concerns of teachers and schools related to likelihood of use and cost-effectiveness, and are hypothesized to be "active ingredients" related to high-quality program implementation in real-world settings. An innovative factorial experimental design was chosen to enable estimation of the individual effect of each of the three factors. RESULTS: Because this paper describes the conceptualization of our study, results are not yet available. CONCLUSIONS: The results of this study may have both substantive and methodological implications for advancing Type 2 translational research.

The role of teacher communicator style in the delivery of a middle school substance use prevention program

We examine whether teachers' communicator style relates to student engagement, teacher-student relationships, student perceptions of teacher immediacy, as well as observer ratings of delivery skills during the implementation of All Stars, a middle school-based substance use prevention program. Data from 48 teachers who taught All Stars up to 3 consecutive years and their respective seventh-grade students (n = 2,240) indicate that having an authoritative communication style is negatively related to student engagement with the curriculum and the quality of the student-teacher relationship, while having an expressive communicator style improves teachers' immediacy to student needs. Adaptations made by a subsample of teachers (n = 27) reveal that those who were more expressive asked students more questions, used more motivational techniques, and introduced more new concepts than authoritarian teachers.

Putting the pieces together: an integrated model of program implementation

Considerable evidence indicates that variability in implementation of prevention programs is related to the outcomes achieved by these programs. However, while implementation has been conceptualized as a multidimensional construct, few studies examine more than a single dimension, and no theoretical framework exists to guide research on the effects of implementation. We seek to address this need by proposing a theoretical model of the relations between the dimensions of implementation and outcomes of prevention programs that can serve to guide future implementation research. In this article, we focus on four dimensions of implementation, which we conceptualize as behaviors of program facilitators (fidelity, quality of delivery, and adaptation) and behaviors of participants (responsiveness) and present the evidence supporting these as predictors of program outcomes. We then propose a theoretical model by which facilitator and participant dimensions of implementation influence participant outcomes. Finally, we provide recommendations and directions for future implementation research.

Implementation of the Project P.A.T.H.S. in Hong Kong: observations based on the Co-Walker Scheme

The implementation quality of the Tier 1 Program (Secondary 1 Program) of the Project P.A.T.H.S. in the third year of the Full Implementation Phase was examined in this study, with classroom observations of 233 units in 157 schools. Results demonstrated that the overall level of program adherence was generally high with an average of 72.96%. A total of 13 aspects regarding program delivery were significantly correlated. Multiple regression analyses indicated that overall implementation quality was significantly predicted by student participation and involvement, classroom control, use of positive and supportive feedback, degree of achievement of the objectives, and time management. Student participation and involvement, classroom control, degree of achievement of the objectives, and time management were significant predictors of success of implementation.

Preliminary research in the application of integrated learning and teacher-centredness in undergraduate education in China

BACKGROUND

The General Practice Department of Fudan University recognised that a traditional didactic educational approach will not achieve expected learning outcomes. Therefore, the Department adopts an integrated learning and teacher-centred approach.

AIM

To evaluate the effect of introducing integrated learning and a teacher-centred approach to undergraduate medical education in China.

METHODS

The concept of integrated learning and the use of a teacher-centred approach was introduced to the General Practice Department of Fudan University's 'Doctor-Patient Communication Skills' undergraduate course. A self-designed questionnaire and a questionnaire used by Fudan University to evaluate the students' satisfaction with their tutors were used to survey 58 medical students.

RESULTS

The self-designed questionnaire gave good reliability and validity results. Based on the survey, 88% of the students both enjoyed the course and rated it highly. The students also showed a high degree of satisfaction with their tutors.

CONCLUSION

Although the student numbers were low, their comments have indicated that the newly introduced, but optional course has achieved a highly desirable effect amongst the students. We believe that this effect has been created through a change from a didactic approach to teaching to a more student-centred approach and by giving meaning and purpose to the students learning; in effect making teaching and learning an enjoyable and satisfying experience for all.

Putting the pieces together: an integrated model of program implementation

Considerable evidence indicates that variability in implementation of prevention programs is related to the outcomes achieved by these programs. However, while implementation has been conceptualized as a multidimensional construct, few studies examine more than a single dimension, and no theoretical framework exists to guide research on the effects of implementation. We seek to address this need by proposing a theoretical model of the relations between the dimensions of implementation and outcomes of prevention programs that can serve to guide future implementation research. In this article, we focus on four dimensions of implementation, which we conceptualize as behaviors of program facilitators (fidelity, quality of delivery, and adaptation) and behaviors of participants (responsiveness) and present the evidence supporting these as predictors of program outcomes. We then propose a theoretical model by which facilitator and participant dimensions of implementation influence participant outcomes. Finally, we provide recommendations and directions for future implementation research.

Studying implementation quality of a school-based prevention curriculum in frontier Alaska: application of video-recorded observations and expert panel judgment

This study assesses the implementation quality of Think Smart, a school-based drug prevention curriculum that was designed to reduce use of harmful legal products (HLPs; e.g., inhalants and over-the-counter drugs), alcohol, tobacco, and other drugs among 5th- and 6th-grade students in frontier Alaska. Participating in the study were eight communities that took part in a larger randomized control trial to assess the short-term effects of the Think Smart curriculum. Video-recorded observations of the 12 core and 3 booster lessons were conducted in 20 classrooms. Ninety-five sessions were randomly selected from 228 usable videodiscs, and two pairs of researchers observed each video recording to code level of dosage, adherence to curriculum design, and teachers' delivery skills. Inter-rater reliability for all implementation quality measures was very high. An expert panel consisting of 16 scientists reviewed the results of the implementation study and concluded that the level of dosage and adherence to the curriculum design was at least as high as those yielded by similar studies. However, the panel assessed the delivery quality to be only marginal in comparison to results of other studies. The experts concluded that the implementation quality of the Think Smart curriculum was adequate even though the teachers' delivery skills were only marginal. A bootstrapping analysis, in which 1,000 samples were drawn for each implementation quality result, found the expert judgments to be reliable. The authors conclude that despite some limitations, video-recorded observations, as well as expert judgment, provide strong methodologies that should be considered for future implementation quality studies.

Teachers' delivery skills and substance use prevention program outcomes: the moderating role of students' need for cognition and impulse decision making

The purpose of this study was to extend the literature in both substance use implementation and persuasive health communication by examining the extent to which students' need for cognition and impulsive decision-making moderated the relationship between teachers' classroom communication behavior and program outcomes in an evidence-based middle school substance use prevention curriculum. Participants included 48 teachers and their respective 7th grade students who participated in a randomized trial testing the effectiveness of personal coaching as a means to improve the quality with which teachers implemented the All Stars curriculum. Need for cognition and impulse decision-making were both associated with positive changes in lifestyle incongruence and commitments to not use substances for students whose teachers displayed greater interactive teaching. Further, need for cognition was associated with lower alcohol use rates while impulse decision making related to lower rates of marijuana use in classes with interactive teaching.