More appropriate evaluation methods for community-level health interventions. Introduction to the special issue

Avaliação da implantação de serviços de saúde reprodutiva no Município de Maringá, Paraná, Brasil

O objetivo deste estudo foi desenvolver um instrumento para avaliar a implantação da assistência em contracepção em serviços de saúde, bem como aplicá-lo nas 23 unidades básicas de saúde no Município de Maringá, Paraná, Brasil. Elaborou-se o modelo teórico-lógico, correspondente a uma "imagem-objetivo" do programa de planejamento familiar. Por meio da técnica Delphi e conferência de consenso, seis especialistas validaram a imagem-objetivo do programa, que contemplou três dimensões e 60 critérios de avaliação. Elaborou-se um instrumento para coleta de dados e uma planilha para avaliar o grau de implantação do programa de planejamento familiar, que constituíram o Questionário de Avaliação de Serviços de Saúde Reprodutiva (QASAR). A grande maioria das unidades básicas de saúde (91,3%) recebeu a classificação "intermediária" na implantação do programa de planejamento familiar, 8,7% foram categorizadas "incipientes" e nenhuma obteve escore para ser considerada "avançada". O grau de implantação "avançado" na dimensão estrutural contrastou com as dimensões organizacional e assistencial. O instrumento constitui ferramenta para avaliar programas de saúde reprodutiva, aplicável aos processos de planejamento e gestão dos serviços de saúde.

Small-group hypertension health education programme: a process and outcome evaluation

AIM

This paper reports a study to explore the expectations, attainments and opinions of the participants, the objectives and experiences of nurse facilitators, and the changes in nurse-client relationship after a small-group hypertension health education programme.

BACKGROUND

Effective control of blood pressure among hypertensive patients is essential in the reduction of cerebrovascular-related morbidity and mortality. A 'small-group hypertension education programme' has been implemented in all general outpatient clinics in Hong Kong since 2000. In order to identify what aspects of the programme were beneficial and how it benefited clients, a process evaluation study was conducted.

METHODS

Following implementation of a small-group hypertension health education programme, two focus groups were run to collect data from eight nurse facilitators, and individual semi-structured interviews were conducted with 16 programme participants. The data were collected in 2001.

FINDINGS

The findings indicate that both clients and nurses gained a great deal from and enjoyed the process of this health education programme, and their relationships were improved after the programme. About half of the clients reported having adopted health-enhancing behaviour. The study also revealed discrepancies between the nurses and clients in their perceptions of the objectives of the programme. This indicates that there is a need to clarify the objectives with clients to minimize disappointments from unrealistic expectations.

CONCLUSION

There is a need for the nurse facilitators to clarify the goal of the programme as an education group rather than as a support group. Also, needs assessment is suggested before group member recruitment in order to make the group more homogeneous.

Concepts and approaches in the evaluation of health promotion

The demands and tensions surrounding evidence-based policy (EBP) as part of results-based management have frequently indicated a gap between these concepts and the complex nature of health promotion interventions. This article discusses the challenges associated with the conceptual field of Health Promotion and the requirements for "proof" of effectiveness and efficiency faced by managers, evaluators, and local agents in the development of inter-sector health programs. The authors identify the limitations of epidemiological trials for the evaluation of social policies and use arguments related to "theories of change" in order to discuss the relationship of the "constructs" in the social policy intervention model and provide the basis for the "analysis of the contribution" of its effects. Systematic reviews of the "realist synthesis" type are discussed, due to their capacity for highlighting the theoretical framework of a specific program and explaining the underlying action mechanisms common to different programs and/or contexts. The authors argue that the expression and maintenance of expected social changes require the construction of collaborative processes, considering the set of (bottom-up) stakeholders involved in all stages of the process of developing and evaluating interventions.

Avaliacão da implantação do programa de desenvolvimento integrado em Manguinhos: impasses na formulação de uma agenda local

O artigo discute a implantação do programa de Desenvolvimento Integrado e Sustentável em Manguinhos, iniciado através da constituição de parcerias entre poder público, iniciativa privada e organizações sociais, visando às políticas e ações intersetoriais de promoção da saúde nas comunidades da região. A avaliação tomou como referência a discussão sobre a implantação do programa e a importância do contexto, considerando as dimensões da mobilização dos atores, da constituição de um espaço de interlocução e de ação coletiva entre as lideranças locais, objetivando uma agenda comum. Foi feita uma análise da vida associativa e da dinâmica de constituição do chamado Fórum Regional, que representou um esforço de agregação destes atores e lideranças da região. A análise revelou uma fragilidade nas relações e ações coletivas vis-à-vis a ausência da implementação de ações públicas intersetoriais. Mobilização, organização social e mudanças institucionais foram vistos como fatores cruciais para equacionar o problema da violência e do padrão alterado de sociabilidade local, desencadeando medidas mais efetivas de promoção da saúde e de melhoria na qualidade de vida em Manguinhos.

Heart failure: how can we prevent the epidemic?

Heart failure prevalence is increasing because of the ageing of the population and the longer survival of people experiencing myocardial infarction and heart failure. The lifetime risk of developing heart failure in Western countries is about 20%. The increasing prevalence of overweight, obesity and diabetes is likely to accelerate heart failure incidence. While there have been major advances in treating heart failure, a preventive approach promises greater benefit to a larger proportion of the community. The medical strategy for heart failure prevention, based on calculation of individual risk, is focused on the minority of individuals who exceed an arbitrary risk threshold. A public health strategy targeting the whole population offers a greater prospect of reducing the incidence of heart failure and other cardiovascular disease. A multitiered approach, encompassing environmental determinants of lifestyle, legislation, and education about healthy lifestyles throughout life, in addition to aggressive control of risk factors in high-risk individuals, is likely to have the greatest impact.

Evaluating partnerships for community health improvement: tracking the footprints

Private-public partnerships are increasingly seen as an important mechanism for improving community health. Despite their popularity, traditional evaluations of these efforts have produced negative or mixed results. This is often attributed to weak interventions or an insufficient period of time to observe an impact. This study examines two additional possibilities--the need for a well-articulated shared vision and the governance and management capabilities of the partnership itself. We conducted a midstream process evaluation of twenty-five community partnerships associated with the Community Care Network (CCN) Demonstration Program. We examined how the roles of a common shared vision, strong governance, and effective management influence a partnership's ability to achieve its objectives. The findings, based on both qualitative and quantitative analyses, underscore the importance of membership organizations' perceived benefits and costs of participation and management capabilities to the partnership's progress toward a vision. Based on the qualitative data, six key governance and management characteristics are identified that separate the top performing partnerships from the lowest performing ones. We explore the implications of this research for future evaluations of public-private community health partnerships.

Atenção básica na agenda da saúde

O artigo discute a política de reorganização da atenção básica a partir do processo recente de descentralização do SUS no Brasil. Enfatiza-se o papel indutor do governo central, que, através de um conjunto de medidas e programas específicos (PAB e PACS/PSF, principalmente), transfere para os municípios a responsabilidade com a atenção básica. Assim, é no nível municipal que ocorre o processo de implementação dessa política, gerando efeitos de difícil avaliação, dada a diversidade de contextos locais. A argumentação central enfatiza a importância de se avaliarem processos e resultados intermediários voltados para o desempenho institucional, que podem ser traduzidos em vontade política e compromisso público, capacidade de gestão e maior controle e participação social, mais do que exatamente efeitos ou impactos mais diretos sobre a oferta de serviços. A conclusão é de que apesar de todas as dificuldades e obstáculos o processo tem implicado o fortalecimento da capacidade de gestão municipal no que diz respeito à organização da atenção básica em saúde.

Using theory to understand the multiple determinants of low participation in worksite health promotion programs

Low participation at the employee or worksite level limits the potential public health impact of worksite-based interventions. Ecological models suggest that multiple levels of influence operate to determine participation patterns in worksite health promotion programs. Most investigations into the determinants of low participation study the intrapersonal, interpersonal, and institutional influences on employee participation. Community- and policy-level influences have not received attention, nor has consideration been given to worksite-level participation issues. The purpose of this article is to discuss one macrosocial theoretical perspective--political economy of health--that may guide practitioners and researchers interested in addressing the community- and policy-level determinants of participation in worksite health promotion programs. The authors argue that using theory to investigate the full spectrum of determinants offers a more complete range of intervention and research options for maximizing employee and worksite levels of participation.

Effectiveness of comprehensive tobacco control programmes in reducing teenage smoking in the USA

OBJECTIVE

To describe the extent to which comprehensive statewide tobacco control programmes in the USA have made progress toward reducing teenage smoking.

DATA SOURCES

Literature search of Medline for reviews of effectiveness of programme and policy elements, plus journal articles and personal request for copies of publicly released reports and working papers from evaluation staff in each of the state programmes of California, Massachusetts, Arizona, Oregon, and Florida.

STUDY SELECTION

All studies, reports, and commentaries that provided information on aspects of programme implementation and evaluation.

DATA SYNTHESIS

Statewide comprehensive programmes show high levels of advertising recall and generally positive improvement in smoking related beliefs and attitudes among teenagers. More fully funded programmes lead to increased mass media campaign advertising and community initiatives; a greater capacity to implement school based smoking prevention programmes; and an increase in the passage of local ordinances that create smoke free indoor environments and reduce cigarette sales to youth. The combination of programme activity and increased tobacco tax reduce cigarette consumption more than expected as a result of price increases alone, and these effects seem to apply to adolescents as well as adults. Programmes are associated with a decline in adult smoking prevalence, with these effects observed to date in California, Massachusetts, and Oregon. Arizona and Florida have yet to examine change in adult prevalence associated with programme exposure. California and Massachusetts have demonstrated relative beneficial effects in teenage smoking prevalence, and Florida has reported promising indications of reduced prevalence. Arizona has yet to report follow up data, and Oregon has found no change in teenage smoking, but has only two years of follow up available. One of the most critical factors in programme success is the extent of programme funding, and consequent level of programme implementation, and the degree to which this is undermined by the tobacco industry and other competitors for funding.

CONCLUSIONS

Despite the different strengths and combinations of programme messages and strategies used in these comprehensive programmes, there is evidence that they lead to change in factors that influence teenage smoking, and to reductions in teenage smoking.

The American Stop Smoking Intervention Study. Conceptual framework and evaluation design

Reducing tobacco use, especially cigarette smoking, is a public health priority. The American Stop Smoking Intervention Study (ASSIST) was initiated in 1991 to prevent and reduce tobacco use primarily through policy-based approaches to alter the social-political environment. This article describes the conceptual design, research framework, evaluation components, and analytic strategies that are guiding the evaluation of this demonstration research endeavor. The ASSIST evaluation is a unique analysis of the complex relationships between the social context, public health activity at the state level, tobacco use, and individual behavior. The measures of tobacco control activity developed for this evaluation may be useful in ongoing national cancer control surveillance efforts, and the lessons learned will enhance the development of tobacco control programs.

Evaluation of service provision patterns during a public-funded dental program

OBJECTIVE

The Commonwealth Dental Health Program was introduced in 1994 to reduce geographic and financial barriers which prevented adult health card holders receiving timely and appropriate dental care. This paper compares the pattern of service provision over the first three years of the program.

METHOD

Cross-sectional surveys of adult public-funded dental patients in Australia in 1994 (17,653 visits), 1995 (n = 80,098 visits) and 1996 (n = 69,159 visits).

RESULTS

There was some reduction in emergency visits, changing from 38.9% in 1994, to 33.6% in 1995 and 35.6% in 1996 (p < 0.05; chi-square). Services were associated with age of patients, visit type, geographic location and year of the program. Poisson regression analyses controlling for patient age, visit type and geographic location showed that between 1994 and 1996 there were significant differences (p < 0.05) in rates (rate ratio: 95% CI) of: restorative (1.27; 1.23-1.31), prosthodontic (0.90: 0.85-0.96), periodontic (0.72; 0.68-0.75), preventive (0.43; 0.40-0.45) and endodontic (1.88; 1.67-2.11) services. The total number of services per visit declined over the period of the program (0.87; 0.86-0.89).

CONCLUSIONS

Over the three years, the service pattern changed to include higher rates of restorative and endodontic services, and lower rates of prosthodontic services. Rates of preventive and periodontic services declined, and there was no reduction in extraction rates.

IMPLICATIONS

Continued improvement in service patterns may require longer programs, incorporating structural changes, to shift the type of care further away from emergency visits, and towards prevention and maintenance.

Evaluation of welfare reform. A framework for addressing the urgent and the important

Assessing the effects of changes still emerging under welfare reform and its overarching policy of devolution presents a challenge to evaluators. Given such features as time limits and benefits caps that vary widely across states and communities, it is necessary not only to attend to urgent issues of immediate relevance for individuals on public assistance but also to focus on important long-term analyses of this complex intergovernmental set of policies. The authors present a conceptual framework based on evaluation utilization and illustrate it with research questions under the rubric of welfare reform. The approach crosses three types of utilization--conceptual, instrumental, and political--with two utilization settings--policy adoption and program implementation. Evaluation strategies are linked to the utilization framework and illustrated with examples from studies of welfare reform. In the aggregate, evaluation studies represent a reasonable range of urgent and important issues across most utilization types and settings.

Analysis approaches to community evaluation

Analysis approaches to the evaluation of community interventions must be sensitive to a wide variety of analytic contaminants that may bias the statistical assessment of changes in outcome measures. These contaminants include model misspecifications related to failures to control for community-specific time trends, temporal autocorrelated errors in equations, spatial autocorrelated errors among geographic units, and other failures of unit independence otherwise indexed by estimated intraclass correlations. Although an enormous amount of progress has been made toward the solution of many of these analytic problems over the past years, the contemporary evaluator of community interventions is left with a number of unenviable design and analysis choices; choices that inevitably force an assessment of the relative threats of different sources of error to the internal and external validity of the evaluation. This article describes the choices made for the evaluation of the Community Trial Project outcome data.