Zero effects in substance abuse programs. Avoiding false positives and false negatives in the evaluation of community-based programs

Artikel

Information-based school programs aimed at reducing alcohol and drug use tend to have limited effects. It is claimed that this is because the aims of such programs are to influence the students' behavior and social life outside the school system.

The aim of this article is to describe different views on how to evaluate substance prevention school programs. The analysis is based on publications in the philosophy of science, sociology, psychology, social-psychology and prevention science.

There is an ongoing debate about how prevention programs ought to be evaluated. One group of researchers argues that experiments involving control groups is the only reliable way to assess prevention programs. Others maintain that this kind of experimental data to determine the outcomes has to be complemented by process data. This is because we need to know how programs interact with the local environment to optimize their effects. Knowledge about the processes will enable us to develop more effective programs in the future.

Based on the analysis presented we conclude that both outcome and process studies are necessary to form a basis for the development of more efficient programs. Such programs must bring about positive interactions among the students and between the students and the instructors, which in turn will affect the outcomes of the prevention programs.

Application of a Mixed Model Approach for Assessment of Interventions and Evaluation of Programs

Many social programs and programs for prevention of drug use are designed to affect a wide variety of targets, including individuals, families and neighborhoods, and organizations such as schools, companies, or hospitals. The nature of the intervention and the design of the particular study determine the choice of the appropriate unit of analysis in assessments of outcome. When the units of assignment and units of observation differ from one another, that is, when clusters of persons rather than persons are assigned at random to treatments, analyses performed at lower levels in the study hierarchy provide inefficient estimates of parameters and often lead to inappropriate significance tests. The present goal was to illustrate the applications of linear mixed models for evaluating statistically the effectiveness of programs.

Community-Based Prevention Programs in the War on Drugs: Findings from the “Fighting Back” Demonstration

Illegal drug use remains one of the United States' most serious health problems, and the “War on Drugs” continues without an end in sight. Antidrug programs, which offer the potential to reduce substance abuse problems, are a component of efforts to deal with the problem, but they operate absent adequate scientific analysis. Although policy has shifted from a focus on supply control to one that includes prevention and treatment, research and theory lag behind program implementation. Thus, for example, community-based programs designed to change norms and systems of substance use have been widely promoted despite the lack of data to support their use. The present paper summarizes findings from an evaluation of a large national demonstration program, “Fighting Back.” Results of the evaluation of broad-based community initiatives in a dozen communities show that the programs failed to reduce rates of substance use and associated harms. These findings, along with other evidence, place reliance on community-based programs at odds with public rhetoric. To deal more effectively with substance abuse, there is a need to move from “grading” programs to understanding why and how interventions function.

An evaluation of the 4-H Health Rocks program: implications for program improvement

The National 4-H Council developed the Health Rocks substance abuse educational program to prevent youth from engaging in risky behaviors. The program was presented in 2010 to more than 8,000 middle school youth in Georgia. A post-then-pre evaluation was conducted with youth who completed 10 hours of instruction to determine if changes in youth knowledge, beliefs/attitudes, skills, and behavioral intentions occurred during the course of the program. This study sought to measure the impact of the program and critically evaluate the questionnaire used. The data revealed statistically significant increases in knowledge, beliefs/attitudes, skills, and behavioral intentions of participating youth. Suggestions for improvement of the questionnaire included utilizing questions that are more specific to the curriculum and adding questions to measure the influence of peer pressure.

Effects of the Communities That Care model in pennsylvania on youth risk and problem behaviors

We undertook the first broad-scale quasi-experimental evaluation of youth outcomes in communities using the Communities That Care program (Hawkins, J. D., & Catalano, R. F., Jr. San Francisco, CA, USA: Jossey-Bass Inc, Publishers, 1992a), which targets adolescent problem behaviors. We evaluated 15 risk factors and 6 outcomes (substance use and delinquent behaviors) for 38,107 youth in 2001 and 98,436 youth in 2003 in Pennsylvania schools. Multilevel analyses compared student reports in communities with CTC programs to comparable communities without CTC, while controlling for level of poverty in the community. Results favored the CTC communities at greater than chance levels in terms of lower rates of some risk factors and outcomes. In a follow-up analysis, CTC community grade cohorts were included only if the grade cohort was expected to benefit from a CTC sponsored program (based on timing of program implementation and target age of the program). Evidence of CTC effects for grade cohorts that received evidence-based programs was even stronger. These findings suggest that community coalitions can affect adolescent public health problems at a population level, especially when evidence-based programs are utilized.

How cultural factors influence school-based substance use prevention programs

This article analyzes school-based substance use prevention programs, comparing programs in the United States and Nordic countries, explores how cultural factors influence the ways in which prevention programs are designed and implemented, and how evaluation is affected by design and implementation.

An example of international drug politics--the development and distribution of substance prevention programs directed at adolescents

Many substance use prevention programs directed at adolescents exist that have been developed by researchers in the United States and are intended to be used in school settings. Some of the problems associated with such programs are reviewed, including their accessibility, ease of use, copyright status, evaluation options, program scales, and ratings, together with an overall consideration of the factors and processes posited to be associated with substance use and non-use (posited "at-risk" and "protective" mechanisms). The authors contend that there is a great need to: (a) develop substance use prevention programs which are commercially available but are not protected by copyright, (b) assess empirically each component in a program separately, and (c) encourage funding bodies to be more active in supporting the production of manuals and evaluation instruments for substance use prevention programs directed at adolescents. We need more and better process evaluations that are also sensitive to both endogenous and exogenous forces in order to know the processes by which a successful prevention program achieves its effects, is prevented from doing so and which processes are irrelevant. A social competence framework might be used as both a goal and as a theoretical base to achieve a better understanding of the processes by which substance use prevention programs reach their effects.

Toward a comprehensive strategy for effective practitioner-scientist partnerships and larger-scale community health and well-being

This article articulates joint priorities for the fields of prevention science and community psychology. These priorities are intended to address issues raised by the frequent observation of natural tensions between community practitioners and scientists. The first priority is to expand the knowledge base on practitioner-scientist partnerships, particularly on factors associated with positive outcomes within communities. To further articulate this priority, the paper first discusses the rapid growth in community-based partnerships and the emergent research on them. Next described is an illustrative research project on a partnership model that links state university extension and public school delivery systems. The article then turns to the second, related priority of future capacity-building for diffusion of effective partnership-based interventions to achieve larger-scale health and well-being across communities. It outlines two salient tasks: clarification of a conceptual framework and the formulation of a comprehensive capacity-building strategy for diffusion. The comprehensive strategy would require careful attention to the expansion of networks of effective partnerships, partnership-based research agendas, and requisite policy-making.

Readiness, functioning, and perceived effectiveness in community prevention coalitions: a study of communities that care

This paper examined whether community readiness, prevention knowledge, coalition functioning, and barriers are linked to perceived effectiveness of community prevention coalitions. Interviews were conducted with 203 key leaders in Communities That Care (CTC) prevention boards in 21 Pennsylvania communities. Community-level means for the reliable self-report measures were utilized separately and in combination with research staff ratings, state technical assistant staff ratings, and other data. The results indicated that the strong link between readiness and perceived effectiveness was mediated by internal coalition functioning. The extent of CTC linkage with outside community entities was not linked to perceived effectiveness. The study concludes that community readiness is an important condition for success of a prevention coalition, and exerts effects mainly through the quality of the coalition's internal functioning. Member turnover and infighting appear to be important factors related to internal functioning. Linkage with outside entities may be more important for coalition models where the coalition is more dependent on local institutions for resources.

Reconsidering community-based health promotion: promise, performance, and potential

Contemporary public health emphasizes a community-based approach to health promotion and disease prevention. The evidence from the past 20 years indicates, however, that many community-based programs have had only modest impact, with the notable exception of a number of HIV prevention programs. To better understand the reasons for these outcomes, we conducted a systematic literature review of 32 community-based prevention programs. Reasons for poor performance include methodological challenges to study design and evaluation, concurrent secular trends, smaller-than-expected effect sizes, limitations of the interventions, and limitations of theories used. The effectiveness of HIV programs appears to be related in part to extensive formative research and an emphasis on changing social norms.

Curtailing tobacco use among youth: evaluation of project 4-health

This article reports the development and evaluation of Project 4-Health, a theory-driven, research-based program to prevent tobacco use among youth enrolled in 4-H clubs throughout California. Additional goals were to involve youth in discouraging others' tobacco use and to develop youth leadership for tobacco control. To assess program effectiveness, 72 clubs were matched and randomly assigned to the program or control condition. Of 1,853 eligible club members, 88.6% completed the pretest. Of these, 79.5% completed a posttest 4 months after conclusion of program delivery, and 77.6% completed a second posttest 2 years later.Short-term effects were found on 7 of 24 outcome measures, indicating changes in knowledge, attitudes, and behavioral intention, but not in social influence variables or behaviors. No long-term effects were observed. Discussion considers how the challenges of designing, delivering, and evaluating the intervention influenced results and implications for preventing tobacco use through community-based youth groups.

Survey estimates of drug-use trends in urban communities: general principles and cautionary examples

Surveys to depict substance abuse rates and monitor trends in specific areas have become increasingly important policy tools. Yet, as illustrated by two national multiwave surveys, using small sample survey data and making longitudinal comparisons is fraught with interpretative problems. In the case of the metropolitan area "oversample" of the National Household Survey on Drug Abuse, for example, interpreting apparent declines in drug use has to take account of the devastating effects of Hurricane Andrew in the Miami Metropolitan area. In the case of a 41-community survey sponsored by the Robert Wood Johnson Foundation to evaluate substance abuse prevention, the difficulty is how to interpret small differences in drug use, which seem to follow no reasonable pattern with respect to treatment or comparison sites. Inferences from such surveys are confounded with statistical anomalies and unforeseen events. They are limited by the sample size. In part, the solution to these problems is to use other survey and nonsurvey data to validate their conclusions and to note their limitations.

The substance use system: social and neighborhood environments associated with substance use and misuse

Substance use theory and practice have traditionally focused on individuals who misuse substances or who are at risk for substance misuse, but this emphasis is shifting. The present study views both substance use and misuse systematically, assessing the relationships between the physical and social environments and substance use and misuse in dynamic interplay. This substance use system was examined through a survey of approximately 10,000 persons, aged 22-44, from primarily inner-city neighborhoods in the United States. Individual indicators such as race, sex, age, socioeconomic status, education, and religious service attendance relate to both the physical and interpersonal environments, even when each is controlled for the others. Qualities of both environments are strongly associated with substance dependency, even after individual indicators are controlled. These findings suggest the difficulty of bringing about change in drug and alcohol use without fundamental change in the environments where use takes place.