The bottom-up approach to integrative validity: a new perspective for program evaluation

A review of implementation and evaluation frameworks for public health interventions to inform co-creation: a Health CASCADE study

BACKGROUND

By including the needs and perspectives of relevant stakeholders, co-creation is seen as a promising approach for tackling complex public health problems. However, recommendations and guidance on how to plan and implement co-creation are lacking. By identifying and analysing existing implementation and evaluation frameworks for public health, this study aims to offer key recommendations for professional stakeholders and researchers wanting to adopt a co-creation approach to public health interventions.

METHODS

Firstly, PubMed and CINAHL databases were screened for articles introducing original implementation and evaluation frameworks for public health interventions. Backwards snowballing techniques were applied to the included papers. Secondly, identified frameworks were classified and relevant data extracted, including steps and constructs present in the frameworks. Lastly, recommendations were derived by conducting thematic analysis on the included frameworks.

RESULTS

Thirty frameworks were identified and data related to their nature and scope extracted. The frameworks' prominent steps and constructs were also retrieved. Recommendations related to implementation and evaluation in the context of co-creation were included.

CONCLUSION

When engaging in co-creation, we recommend including implementation considerations from an early stage and suggest adopting a systems thinking as a way to explore multiple levels of influence, contextual settings and systems from an early planning stage. We highlight the importance of partnering with stakeholders and suggest applying an evaluation design that is iterative and cyclical, which pays particular attention to the experience of the engaged co-creators.

WASH interventions and child diarrhea at the interface of climate and socioeconomic position in Bangladesh

Many diarrhea-causing pathogens are climate-sensitive, and populations with the lowest socioeconomic position (SEP) are often most vulnerable to climate-related transmission. Household Water, Sanitation, and Handwashing (WASH) interventions constitute one potential effective strategy to reduce child diarrhea, especially among low-income households. Capitalizing on a cluster randomized trial population (360 clusters, 4941 children with 8440 measurements) in rural Bangladesh, one of the world's most climate-sensitive regions, we show that improved WASH substantially reduces diarrhea risk with largest benefits among children with lowest SEP and during the monsoon season. We extrapolated trial results to rural Bangladesh regions using high-resolution geospatial layers to identify areas most likely to benefit. Scaling up a similar intervention could prevent an estimated 734 (95% CI 385, 1085) cases per 1000 children per month during the seasonal monsoon, with marked regional heterogeneities. Here, we show how to extend large-scale trials to inform WASH strategies among climate-sensitive and low-income populations.

The Acti-Pair program helps men with prostate cancer increase physical activity with peer support: a mixed method pilot study

Background

Although the health benefits of physical activity (PA) are recognized, prostate cancer patients do not follow PA recommendations. Barriers to PA, whether physical, environmental or organizational, are known. Furthermore, even when these barriers are overcome, this achievement is not systematically accompanied by lifestyle change. Many strategies have shown to be effective in increasing patient adherence to PA. This study aims to assess the feasibility and the viability of the Acti-Pair program which combines three strategies: peer support, a personalized and realistic PA project, and support from health and adapted physical activity professionals in a local context.

Methods and analysis

We conducted a pilot study utilizing a mixed qualitative and quantitative methodology, employing feasibility and viability assessments. Quantitative assessments included recruitment, retention adherence rates, process and potential effectiveness (PA and motivation) indicators; while qualitative methods were used to evaluate the program's practicality, suitability and usefulness. Indicators of potential effectiveness were assessed before and after the intervention using a Wilcoxon test for matched data. Qualitative data were collected through semistructured interviews conducted by two researchers with various program stakeholders. The study lasted for 3 years.

Results

Twenty-four patients were recruited over a 25-month period. Forty-two percent of patients completed the program 3 months after the beginning. We recruited 14 peers and trained nine peers over a 10-month period. The program was coordinated extensively by adapted PA professionals, while health professionals were involved in recruiting patients and peers. Self-reporting of moderate to vigorous PA was increased after the Acti-Pair program initiation [42.86 (30.76) at baseline to 53.29 (50.73)]. Intrinsic motivation significantly increased after participation in the Acti-Pair program [1.76 (1.32) before the intervention vs. 2.91 (1.13) after the intervention]. The key player to support the Acti-Pair program in the field has been the PA support system. The main challenge has been the difficulty of health professionals in promoting PA.

Discussion

This pilot study has shown that the Acti-Pair program is feasible and viable. It will allow us to extend the peer support intervention to other contexts and assess the effectiveness of this intervention and its generalization.

How to co-design a health literacy-informed intervention based on a needs assessment study in chronic obstructive pulmonary disease

OBJECTIVE

To develop a co-designed health literacy (HL)-informed intervention for people with chronic obstructive pulmonary disease (COPD) that enables them to find, understand, remember, use and communicate the health information needed to promote and maintain good health.

DESIGN

This study used a co-design approach informed by the programme logic of the Ophelia (Optimising Health Literacy and Access) process. The co-design included workshops where possible solutions for an HL-informed intervention were discussed based on an HL needs assessment study.

SETTINGS

Five workshops were performed in a local community setting in the specialist and municipality healthcare services in Oslo, Norway.

PARTICIPANTS

People with COPD, multidisciplinary healthcare professionals (HCPs) from the municipality and specialist healthcare services, and researchers (n=19) participated in the workshops. The co-designed HL-informed intervention was based on seven focus groups with people with COPD (n=14) and HCPs (n=21), and a cross-sectional study of people with COPD using the Health Literacy Questionnaire (n=69).

RESULTS

The workshop co-design process identified 45 action points and 51 description points for possible intervention solutions to meet the HL needs of people with COPD. The final recommendation for an HL-informed intervention focused on tailored follow-up after hospitalisation, which uses motivational interviewing techniques, is based on the individual's HL, self-management and quality of life needs and is implemented in cooperation with HCPs in both the specialist and municipality healthcare services.

CONCLUSION

During the codesign process, the workshop group generated several ideas for how to help patients find, understand, remember, use and communicate health information in order to promote and maintain good health. People with COPD need tailored follow-up based on their individual HL needs by HCPs that have knowledge of COPD and are able to motivate them for self-management tasks and help them to improve their quality of life (QOL) and decrease hospitalisation.

Development of an intervention (PICASO) to optimise the palliative care capacity of social workers in Flanders: a study protocol based on phase I of the Medical Research Council framework

INTRODUCTION

An important challenge for future palliative care delivery is the growing number of people with palliative care needs compared with the limited qualified professional workforce. Existing but underused professional potential can further be optimised. This is certainly the case for social work, a profession that fits well in multidisciplinary palliative care practice but whose capacities remain underused. This study aims to optimise the palliative care capacity of social workers in Flanders (Belgium) by the development of a Palliative Care Program for Social Work (PICASO).

METHODS AND ANALYSIS

This protocol paper covers the steps of the development of PICASO, which are based on phase I of the Medical Research Council framework. However, additional steps were added to the original framework to include more opportunities for stakeholder involvement. The development of PICASO follows an iterative approach. First, we will identify existing evidence by reviewing the international literature and describe the problem by conducting quantitative and qualitative research among Flemish social workers. Second, we will further examine practice and identify an appropriate intervention theory by means of expert panels. Third, the process and outcomes will be depicted in a logic model.

ETHICS AND DISSEMINATION

Ethical approval for this study was given by the KU Leuven Social and Societal Ethics Committee (SMEC) on 14 April 2021 (reference number: G-2020-2247-R2(MIN)). Findings will be disseminated through professional networks, conference presentations and publications in scientific journals.

Realist evaluation of three programmes aimed at reducing harm and risks associated with alcohol consumption in the Nouvelle Aquitaine region of France: the ECIAE study protocol

INTRODUCTION

In Europe, alcohol consumption is responsible for many diseases, disabilities, injuries and premature deaths. In France, alcohol consumption represents an important health burden, due to its frequency, scale and the serious damage it causes. One of the keys to addressing the problem would appear to be the adoption of harm and risk reduction approaches. In order to operationalise this strategy, the Nouvelle Aquitaine Regional Health Agency is funding three different programmes to reduce the harm and risks associated with alcohol consumption: Alcochoix, Iaca and ETP (Education Thérapeutique du Patient) Conso-repère. We are interested in understanding how, under what circumstances, through which mechanisms and for which population the different programmes work.

METHODS AND ANALYSIS

The ECIAE study (a cross-evaluation of the 3 programs Iaca/Alcochoix/ ETP (Patient Therapeutic Education) is a theory-based evaluation where the realist evaluation method is used to explore effects, intervention mechanisms and the influence of context on outcomes. This realist evaluation is based on multiple case studies in two nested levels. At the first level, each centre implementing the programme will represent a case. At the second level, each programme will represent a case in which a set of activities is conducted to achieve risk reduction objectives.

ETHICS AND DISSEMINATION

The project will be carried out in full compliance with existing legislation and international conventions. It was subject to analysis, including a privacy impact assessment conducted by the Data Protection Officer of the University of Bordeaux. The University of Bordeaux has ensured that all the regulatory procedures related to the ECIAE study have been carried out. The dissemination plan includes scientific papers, seminars, a report and recommendation and a public restitution. The study will provide evidence-based results to help health authorities roll out strategies to reduce risks and harm associated with alcohol use.

Realist evaluation of the impact, viability and transferability of an alcohol harm reduction support programme based on mental health recovery: the Vitae study protocol

INTRODUCTION

Addiction is considered a chronic disease associated with a high rate of relapse as a consequence of the addictive condition. Most of the current therapeutic work focuses on the notion of relapse prevention or avoidance and the control of its determinants. Since only a small portion of patients can access alcohol addiction treatment, it is crucial to find a way to offer new support towards safe consumptions, reductions or cessations. The harm reduction (HR) approach and mental health recovery perspective offers another way to support the patient with alcohol addiction. Vitae is a realist evaluation of the impact, viability and transferability of the IACA! programme, an HR programme based on the principle of psychosocial recovery for people with alcohol use disorders.

METHODS AND ANALYSIS

The Vitae study adheres to the theory-driven evaluation framework where the realist evaluation method and contribution analysis are used to explore the effects, mechanisms and influence of context on the outcomes and to develop and adjust an intervention theory. This study is a 12-month, multi-case, longitudinal descriptive pilot study using mixed methods. It is multi-centred, and carried out in 10 addiction treatment or prevention centres. In this study, outcomes are related to the evolution of alcohol use and the beneficiaries trajectory in terms of psychosocial recovery during these 12 months after the start of IACA!. The target number of participants are 100 beneficiaries and 23 professionals.

ETHICS AND DISSEMINATION

This research was approved by the Committee for the Protection of Persons Ouest V n°: 21/008-3HPS and was reported to the French National Agency for the Safety of Health Products. All participants will provide consent prior to participation. The results will be reported in international peer-reviewed journals and presented at scientific and public conferences.

TRIAL REGISTRATION NUMBERS

NCT04927455; ID-RCB2020-A03371-38.

What does it mean for an evaluation to be 'valid'? A critical synthesis of evaluation literature

Validity is both a critical foundation for, and widely contested concept in the field of program evaluation. Given the wide and divergent treatment of the term, this article sought to enhance the field's understanding and application of validity by systematically identifying, collating and integrating existing conceptions of validity. A critical interpretive synthesis approach was taken to combine the best elements of systematic literature searching with integrative review. We found that conceptions of validity could be roughly collated into three categories pertaining to the evaluation research, evaluative judgement and reporting, and beyond the evaluation proper. The result was a wide-ranging map of validity concepts against the various stages of the evaluation process. This made clear several gaps in the validity thinking, resulting in the development of comprehensive but not yet complete framework that weaves together related aspects identified into a cohesive whole. It is our hope that this initial iteration of a cohesive validity framework will be the starting point for a cooperative effort to improve professional evaluation practice.

Biologically grounded scientific methods: The challenges ahead for combating epidemics

The protracted COVID 19 pandemic may indicate failures of scientific methodologies. Hoping to facilitate the evaluation and/or update of methods relevant in Biomedicine, several aspects of scientific processes are here explored. First, the background is reviewed. In particular, eight topics are analyzed: (i) the history of Higher Education models in reference to the pursuit of science and the type of student cognition pursued, (ii) whether explanatory or actionable knowledge is emphasized depending on the well- or ill-defined nature of problems, (iii) the role of complexity and dynamics, (iv) how differences between Biology and other fields influence methodologies, (v) whether theory, hypotheses or data drive scientific research, (vi) whether Biology is reducible to one or a few factors, (vii) the fact that data, to become actionable knowledge, require structuring, and (viii) the need of inter-/trans-disciplinary knowledge integration. To illustrate how these topics interact, a second section describes four temporal stages of scientific methods: conceptualization, operationalization, validation and evaluation. They refer to the transition from abstract (non-measurable) concepts (such as 'health') to the selection of concrete (measurable) operations (such as 'quantification of ́anti-virus specific antibody titers'). Conceptualization is the process that selects concepts worth investigating, which continues as operationalization when data-producing variables viewed to reflect critical features of the concepts are chosen. Because the operations selected are not necessarily valid, informative, and may fail to solve problems, validations and evaluations are critical stages, which require inter/trans-disciplinary knowledge integration. It is suggested that data structuring can substantially improve scientific methodologies applicable in Biology, provided that other aspects here mentioned are also considered. The creation of independent bodies meant to evaluate biologically oriented scientific methods is recommended.

Understanding the complexity of population health interventions: assessing intervention system theory (ISyT)

Given their inherent complexity, we need a better understanding of what is happening inside the “black box” of population health interventions. The theory-driven intervention/evaluation paradigm is one approach to addressing this question. However, barriers related to semantic or practical issues stand in the way of its complete integration into evaluation designs. In this paper, we attempt to clarify how various theories, models and frameworks can contribute to developing a context-dependent theory, helping us to understand the black box of population health interventions and to acknowledge their complexity. To achieve this goal, we clarify what could be referred to as “theory” in the theory-driven evaluation of the interventional system, distinguishing it from other models, frameworks and classical theories. In order to evaluate the interventional system with a theory-driven paradigm, we put forward the concept of interventional system theory (ISyT), which combines a causal theory and an action model. We suggest that an ISyT could guide evaluation processes, whatever evaluation design is applied, and illustrate this alternative method through different examples of studies. We believe that such a clarification can help to promote the use of theories in complex intervention evaluations, and to identify ways of considering the transferability and scalability of interventions.

[Innovative transformation of the health system through a preventive transition]

Given today's evolution of the healthcare system, organizational transformations, technological developments and major challenges, innovation has taken on primordial importance. In this context and with considerable support, many experimentations have taken place. Unfortunately, few have managed to scale up. What results is a congeries of innovations without a future, possibly avoidable squandering of resources, a number of missed opportunities, and the grim prospect of inventor burnout. As regards prevention, innovation is at the heart of an anticipated "preventive transition" of the health system that has yet to achieve operational status. In this article we attempt to redesign the contours of innovation in health, considering it first and foremost in regard to its social utility. We will go on to explore the limitations of innovative practices that delay the arrival of advances in health. Four types of obstacles appear: faulty evaluation; insufficient dialogue between researchers, stakeholders and decision-makers; lack of visibility and, finally, conceptions and perceptions of innovation characterized by tunnel vision. In the concluding section of this paper, we will present several tracks through which the innovation process could be impelled to drive health system transformation. They consist in: (i) incorporating an evaluative and comprehensive research into innovation processes, (ii) elaborating "bottom-up" approaches giving special consideration to innovations instigated by stakeholders and brought to fruition under real-life conditions, (iii) breaking from standardization by thinking from the outset of the adaptability of innovations and, finally, (iv) tying in the experimental approach with a decision-making process.

Towards a standardized approach of assessing social context of persons receiving home care in Flanders, Belgium: the development and test of a social supplement to the interRAI instruments

BACKGROUND

Apart from a person's physical functioning, the early identification of social context indicators which affect patient outcomes - such as environmental and psychosocial issues - is key for high quality and comprehensive care at home. During a home care assessment, a person's biomedical and functional problems are typically considered. Harder to define concepts, such as psychosocial well-being or living arrangements, are not routinely documented, even though research shows they also affect functioning and health outcomes. The purpose of this study is to develop and test a concise, integrated assessment (BelRAI Social Supplement) that evaluates these social context indicators for persons receiving home care to complement existing interRAI- instruments.

METHODS

The development of the BelRAI Social Supplement is a multi-stage process, based upon the revised MRC-framework, involving both qualitative and quantitative research with stakeholders such as; clients, informal caregivers, care professionals and policy makers. The developmental process encompasses four stages: (I) item generation based on multiple methods and content validation by a panel of stakeholders (II) assessing feasibility and piloting methods, (III) early evaluation, and (IV) final evaluation. Stage II and III are covered in this paper.

RESULTS

During Stages I and II, a testable version of the BelRAI Social Supplement was developed in an iterative process. In Stage III, 100 care professionals assessed 743 individuals receiving home care in Flanders between December 2018 and December 2019. Using inter-item correlation matrixes, frequency distributions and regular feedback from the participants, the BelRAI Social Supplement was improved and prepared for Stage IV. The updated version of the instrument consists of four main sections: (1) environmental assessment; (2) civic engagement; (3) psychosocial well-being; and (4) informal care and support. In total, the BelRAI Social Supplement contains a maximum of 76 items.

CONCLUSIONS

The BelRAI Social Supplement was reviewed and shortened in close collaboration with care professionals and other experts in Flanders. This study resulted in an instrument that documents need-to-know social context determinants of home dwelling adults.

TABADO 2 : une stratégie d'accompagnement au sevrage tabagique des adolescents en milieu scolaire

La consommation de tabac est une des premières causes de mortalité évitable au niveau mondial. La dépendance au tabac s'observe dès l'adolescence. En complément des actions visant à prévenir l'entrée des jeunes dans le tabagisme, il est donc nécessaire de développer des stratégies spécifiques pour les adolescents déjà fumeurs. Un programme d'accompagnement au sevrage tabagique pour adolescents, TABADO, a montré son efficacité lors d'un essai contrôlé mené dans des centres de formation des apprentis en 2007/2009. En 2018, l'Institut national du cancer a souhaité généraliser TABADO au niveau national et l'étendre aux lycées professionnels. Pour accompagner ce processus de mise à l'échelle, il était essentiel d'élaborer la théorie d'intervention et d'analyser la transférabilité de TABADO en conditions de vie réelle et dans de nouveaux contextes, et de proposer les adaptations correspondantes. Une recherche spécifique a été conduite et détaillée par ailleurs. L'objectif de cette publication pragmatique est de présenter la nouvelle stratégie TABADO 2 aux acteurs et décideurs de santé publique, et le guide afférent développé pour les accompagner dans la mise en œuvre.Une étude de cas multiples (n = 10) a été menée à partir de la mise en œuvre de TABADO dans trois régions françaises, qui reposait sur des observations, entretiens et séminaires de retour d'expérience. Elle a mis en évidence l'adaptation de l'intervention aux contextes locaux, ainsi que de nouveaux leviers interventionnels mis en œuvre. De plus, les investigations ont montré que pour instaurer un climat favorable à la démarche d'arrêt de la consommation de tabac et soutenir son maintien sur la durée, il était nécessaire d'inscrire TABADO dans une stratégie globale de l'établissement scolaire et de son environnement. Cette démarche a permis de proposer une transformation de l'intervention TABADO en une nouvelle stratégie - TABADO 2 - et de proposer un guide pour accompagner son déploiement national.

Development of a fostering purpose intervention

Evolutionary Evaluation is a framework for understanding the evolution of programs. Just as programs grow and develop, so can standalone interventions. This paper focused on building and evaluating a purpose intervention to scaffold adolescents and young adults' search for purpose. Four studies were conducted at each phase of the intervention's lifespan to assess if modifications were needed and to test the viable validity of using online activities to increase both the search for and the discovery of a purpose in life. These studies can be viewed through the lens of the Evolutionary Evaluation framework. Study 1 evaluated the intervention at its initiation phase, which consisted of developing and testing the intervention activities. Study 2 tested the intervention during the development phase to determine which activities most effectively increased rates of purpose. Study 3 evaluated the intervention in its stability phase by experimentally testing all activities together in a full curriculum via Amazon's Mechanical Turk. Finally, study 4 assessed the dissemination of the intervention by testing the full curriculum with over 500 high school and college students. These studies suggest that the fostering purpose intervention cultivated purpose among adolescents and young adults. Additionally, the success of the intervention demonstrates that the Evolutionary Evaluation framework may be an effective model with which to develop an intervention, from initiation through dissemination phases.

Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study

BACKGROUND

In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement.

OBJECTIVE

We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels.

METHODS

Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations.

RESULTS

Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested.

CONCLUSIONS

There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.

Communities for Healthy Living (CHL) A Community-based Intervention to Prevent Obesity in Low-Income Preschool Children: Process Evaluation Protocol

BACKGROUND

Process evaluation can illuminate barriers and facilitators to intervention implementation as well as the drivers of intervention outcomes. However, few obesity intervention studies have documented process evaluation methods and results. Community-based participatory research (CBPR) requires that process evaluation methods be developed to (a) prioritize community members' power to adapt the program to local needs over strict adherence to intervention protocols, (b) share process evaluation data with implementers to maximize benefit to participants, and (c) ensure partner organizations are not overburdened. Co-designed with low-income parents using CBPR, Communities for Healthy Living (CHL) is a family-centered intervention implemented within Head Start to prevent childhood obesity and promote family well-being. We are currently undertaking a randomized controlled trial to test the effectiveness of CHL in 23 Head Start centers in the greater Boston area. In this protocol paper, we outline an embedded process evaluation designed to monitor intervention adherence and adaptation, support ongoing quality improvement, and examine contextual factors that may moderate intervention implementation and/or effectiveness.

METHODS

This mixed methods process evaluation was developed using the Pérez et al. framework for evaluating adaptive interventions and is reported following guidelines outlined by Grant et al. Trained research assistants will conduct structured observations of intervention sessions. Intervention facilitators and recipients, along with Head Start staff, will complete surveys and semi-structured interviews. De-identified data for all eligible children and families will be extracted from Head Start administrative records. Qualitative data will be analyzed thematically. Quantitative and qualitative data will be integrated using triangulation methods to assess intervention adherence, monitor adaptations, and identify moderators of intervention implementation and effectiveness.

DISCUSSION

A diverse set of quantitative and qualitative data sources are employed to fully characterize CHL implementation. Simultaneously, CHL's process evaluation will provide a case study on strategies to address the challenges of process evaluation for CBPR interventions. Results from this process evaluation will help to explain variation in intervention implementation and outcomes across Head Start programs, support CHL sustainability and future scale-up, and provide guidance for future complex interventions developed using CBPR.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT03334669 . Registered on October 10, 2017.

Realist evaluation of a theory-based life skills programme aiming to prevent addictive behaviours in adolescents: the ERIEAS study protocol

INTRODUCTION

Adolescence is a sensitive life stage during which tobacco, alcohol and cannabis are used as ways to learn and adopt roles. There is a great deal of interest in substance use (SU) prevention programmes for young people that work to change representations of these products and help with mobilisation of life skills. Unfortunately, few existing programmes are evidence-based.In France, a programme called Expériences Animées (EA, Animated Experiences) has been developed, inspired by life skills development programmes that have been proven to be successful. The EA programme uses animated short movies and talks with high school and secondary school pupils about the use of psychoactive substances and addictions. By allowing life skills mobilisation and modifying representations and beliefs about SU, it is aimed at delaying initiation of use of psychoactive substances, preventing adolescents from becoming regular consumers, reducing the risks and harms related to the use of these substances and opening the way for adapted support measures.We are interested in understanding how, under what circumstances, through which mechanisms and among which adolescents the EA programme works. Therefore, we have developed the ERIEAS study ('Evaluation Réaliste de l'Intervention Expériences Animées en milieu Scolaire'; Realist Evaluation of the EA Intervention in Schools).

METHODS AND ANALYSIS

EA will be conducted in 10 schools. A multi-case approach will be adopted with the aim of developing and adjusting an intervention theory. The study comes under the theory-driven evaluation framework. The investigation methodology will include four stages: (i) elaboration of a middle-range theory; (ii) data collection for validating/adjusting the theory; (iii) data analysis; and (iv) refinement and adjustment of the middle-range theory and definition of the programme's key functions.

ETHICS AND DISSEMINATION

The study will provide evidence-based results to health authorities to help in the rollout of health promotion strategies in schools. It will provide knowledge about the strategic configurations most suitable for leading to life skills mobilisation and change young people's representations about SU. The project will be carried out with full respect of current relevant legislation (eg, the Charter of Fundamental Rights of the European Union) and international conventions (eg, Helsinki Declaration). It follows the relevant French legislation of the research category on interventional research protocol involving the human person. The protocol was approved by the Comité et Protection des Personnes (CPP), that is, Committee for the Protection of Persons CPP SUD-EST VI n°: AU 1525 and was reported to the Agence Française de Sécurité Sanitaire des Produits de Santé (ANSM) that is, the French National Agency for the Safety of Health Products. It is in conformity with reference methodology MR003 of Bordeaux University Hospital (CNIL n° 2 026 779 v0).Trial registration detailsThis research has been registered on ClinicalTrials.gov (No. NCT04110626).The research project is registered in the European database ID-RCB (No. 2019-A01003-54).

Current challenges in population health intervention research

Population health interventions (PHIs) are generally complex; their results depend on their interaction with the context of their implementation. Moreover, the distinction between intervention and context is arbitrary: we need rather to consider an ‘interventional system’, including both interventional and contextual components. Evaluation must go beyond effectiveness and must include two key research questions: a viability analysis, to verify that the intervention can be routinised in a real-life context; and a theory-based evaluation, to analyse mechanisms and to understand what produces effects among components and their interactions with each other and with the context. PHI research is a question not only of the object but also of perspectives. This means doing research differently, making use of interdisciplinarity and involving stakeholders. Such an approach may contribute to the development, transfer, implementation and scaling-up of innovative interventions.

Population health intervention research: what is the place for pilot studies?

Background

An international workshop on population health intervention research (PHIR) was organized to foster exchanges between experts from different disciplines and different fields.

Aims

This paper aims to summarize the discussions around one of the issues addressed: the place or role of pilot studies in PHIR. Pilot studies are well-established in biomedical research, but the situation is more ambiguous for PHIR, in which a pilot study could refer to different purposes.

Methods

The workshop included formal presentations of participants and moderated discussions. An oral synthesis was carried out by a rapporteur to validate by expert consensus the key points of the discussion and the recommendations. All discussions have been recorded and fully transcribed.

Discussion

PHIR generally addresses complex interventions. Thus, numerous tasks may be required to inform the intervention and test different aspects of its design and implementation. While in clinical research the pilot study mainly concerns the preparation of the trial, in PHIR the pilot study focuses on the preparation of both the intervention and the trial. In particular, pilot studies in PHIR could be used for viability evaluation and theory development.

Recommendations from the workshop participants

The following recommendations were generated by consensus from the workshop discussions: i) terms need to be clarified for PHIR; ii) reporting and publication should be standardized and transparency should be promoted; iii) the objectives and research questions should drive the methods used and be clearly stated; iv) a pilot study is generally needed for complex intervention evaluation and for research-designed programs; and v) for field-designed programs, it is important to integrate evaluability assessments as pilot studies .

Conclusion

Pilot studies play an important role in intervention development and evaluation. In particular, they contribute to a better understanding of the mechanisms of intervention and the conditions of its applicability and transferability. Pilot studies could therefore facilitate evidence-based decisions about design and conduct of main studies aimed to generate evidence to inform public health policy.

Feasibility evaluation of a mindfulness-based stress reduction program for primary care professionals in Brazilian national health system

BACKGROUND AND PURPOSE

In the last decades, an increasing body of scientific studies has shown mindfulness-based interventions as efficacious for reducing stress, including among primary care professionals. Despite the strength of the evidence, mindfulness-based interventions still are not widely adopted as a clinical practice in national health systems. The aim of the present study was to conduct a feasibility evaluation of a mindfulness-based stress reduction program for primary care professionals in Brazilian national health system.

MATERIALS AND METHODS

A pilot mindfulness program was conducted through the course of four weekly encounters in the municipality of Biguaçu with the participation of 26 primary care professionals. Data was collected through direct observations and four self-report questionnaires. The information was used to complete an evaluation matrix and reach a value judgment about the feasibility level of the components of the mindfulness-based program.

RESULTS

The subdimensions Integration, Demand, and Acceptability were judged as "Highly Feasible", the subdimension Practicality was judged as "Feasible" and the subdimension Adaptation was judged as "Lowly Feasible". The results indicated that there is a high demand for stress-reduction interventions within the context of primary care and the acceptability by the stakeholders (participants and management) was excellent. However, the two main barriers found were the need for a brief version of the program to accommodate the restrictive timetable of the primary care professionals and the low retention rates.

CONCLUSION

The implementation of a mindfulness-based program for primary care professionals in Brazilian national health system was judged as "Feasible". Further studies need to conduct the feasibility evaluation in other municipalities and with larger sample sizes to ensure the generalizability of these results.

Implementing Web-Based Interventions in HIV Primary Care Clinics: Pilot Implementation Evaluation of Positive Health Check

Background

Web-based interventions can help people living with HIV achieve better clinical outcomes and behaviors, but integrating them into clinical practice remains challenging. There is a gap in understanding the feasibility of implementing these interventions in HIV clinic settings from the clinicians’ perspective.

Objective

The goal of the research was to determine whether Positive Health Check (PHC)—a Web-based, tailored video counseling tool focused on increasing patient adherence and retention in care and reducing HIV risk among HIV-positive patients—was acceptable, appropriate, and feasible for HIV primary care clinic staff to implement in clinic workflows.

Methods

A multiple-case study design was used to evaluate the pilot implementation. Four primary care clinics located in the southeastern United States implemented PHC over a 1-month period. Nine clinic staff across the clinics participated in structured interviews before, during, and after the implementation. In total, 54 interviews were conducted. We used a framework analysis approach to code the data and identify themes related to implementation outcomes, including acceptability, appropriateness, and feasibility. We also analyzed patient intervention use metrics (n=104) to quantify patient intervention completion rates (n=68).

Results

Overall, clinicians viewed PHC as acceptable and appropriate. Themes that emerged related to these implementation outcomes include the ability for PHC to increase provider-patient communication and its ability to engage patients due to the tailored and interactive design. While generally feasible to implement, challenges to the clinic workflow and physical environment were areas that clinics needed to manage to make PHC work in their clinics.

Conclusions

Findings from this pilot implementation suggest that clinical staff viewed PHC as acceptable and appropriate, especially as more patients used the intervention over the pilot period. Feasibility of implementation was challenging in some cases, and lessons learned from this pilot implementation can provide information for larger scale tests of the intervention that include assessment of both implementation outcomes and clinical outcomes.

Context Considerations for Developing the In Defense of Food Nutrition Education Curriculum

OBJECTIVE

To examine perceptions of teachers and afterschool program staff related to the positive aspects and challenges of afterschool education for youth to guide the development of the "In Defense of Food" nutrition education curriculum.

METHODS

Semi-structured in-depth interviews with a convenience sample of teachers and afterschool program staff experienced serving at-risk youth were conducted in New York, NY. They were audio-recorded, transcribed verbatim, and analyzed using inductive summative content analysis to identify, count, and compare themes.

RESULTS

Interviews (n = 12) generated 2 meta-themes (Pedagogical Elements for Successful Learning and Pragmatic Considerations for hosting health promotion programs in the afterschool context). Learner-centered and tailored approaches and building social connection were considered important for fostering learning in the afterschool context.

CONCLUSIONS AND IMPLICATIONS

This study emphasizes the importance of considering pedagogical elements related to delivery that minimizes didactic instruction and focuses on learner-centered approaches. Further research is needed to compare outcomes and process measures common approaches used in designing nutrition education curricula with those that have been described in this study.

Evaluating program effects: Conceptualizing and demonstrating a typology

Outcome evaluation is very important for program evaluation and has been becoming increasingly so in the age of accountability. Typically, outcome evaluation is conducted for a single program from a single perspective. However, in a real-life situation, many programs exist in a system, and the effects could be viewed from various perspectives. The authors illustrate a typology of program effects in a system. It moves from the paradigm of a single program's single effect to that of a set of programs' multiple effects. Methodological implications are discussed.

Factors influencing workplace health promotion intervention: a qualitative systematic review

Although workplace health promotion (WHP) has evolved over the last 40 years, systematically collected knowledge on factors influencing the functioning of WHP is scarce. Therefore, a qualitative systematic literature review was carried out to systematically identify and synthesize factors influencing the phases of WHP interventions: needs assessment, planning, implementation and evaluation. Research evidence was identified by searching electronic databases (Scopus, PubMed, Social Sciences Citation Index, ASSIA, ERIC, IBBS and PsycINFO) from 1998 to 2013, as well as by cross-checking reference lists of included peer-reviewed articles. The inclusion criteria were: original empirical research, description of WHP, description of barriers to and/or facilitators of the planning, implementation and/or evaluation of WHP. Finally, 54 full texts were included. From these, influencing factors were extracted and summarized using thematic analysis. The majority of influencing factors referred to the implementation phase, few dealt with planning and/or evaluation and none with needs assessment. The influencing factors were condensed into topics with respect to factors at contextual level (e.g. economic crisis); factors at organizational level (e.g. management support); factors at intervention level (e.g. quality of intervention concept); factors at implementer level (e.g. resources); factors at participant level (e.g. commitment to intervention) and factors referring to methodological and data aspects (e.g. data-collection issues). Factors regarding contextual issues and organizational aspects were identified across three phases. Therefore, future research and practice should consider not only the influencing factors at different levels, but also at different phases of WHP interventions.

Predictors of School Garden Integration: Factors Critical to Gardening Success in New York City

The purpose of this study was to determine the level of integration of school gardens and identify factors that predict integration. 211 New York City schools completed a survey that collected demographic information and utilized the School Garden Integration Scale. A mean garden integration score was calculated, and multiple regression analysis was conducted to determine independent predictors of integration and assess relationships between individual integration characteristics and budget. The average integration score was 34.1 (of 57 points) and ranged from 8 to 53. Operating budget had significant influence on integration score, controlling for all other factors ( p < .0001). Partner organizations, evaluation/feedback, planning the physical space, and characteristics of the physical space were positively and significantly related to budget. The results of this study indicate that any garden can become well integrated, as budget is a modifiable factor. When adequate funding is secured, a well-integrated garden may be established with proper planning and sound implementation.

Feasibility evaluation of a mindfulness-based intervention for primary care professionals: Proposal of an evaluative model

A critical issue in the contemporary field of public health is the organizational stress experienced by healthcare professionals. An integrative and complementary therapy that research has shown as efficacious in helping healthcare professionals to cope with stress is mindfulness meditation. An intervention, however, can't merely be efficacious, it also needs to be feasible to be implemented in a specific political and organizational context. This paper proposes a theoretical logical model and evaluation matrix of the feasibility of a mindfulness-based stress reduction program for primary care professionals. The literature review about the topic was the source for the construction of the theoretical logical model and evaluation matrix, and the validation of those was given by consensus methods, gathering data from experts in the field.

A Mixed-Methods Comparison of Classroom Context During Food, Health & Choices, a Childhood Obesity Prevention Intervention

BACKGROUND

Schools are frequent settings for childhood obesity prevention; however, intervention results are mixed. Classroom context may hold important clues to improving these interventions.

METHODS

We used mixed methods to examine classroom context during a curriculum intervention taught by trained instructors in fifth grade classrooms. We identified classrooms with high and low buy-in using cluster analysis and compared intervention delivery and reception, student energy balance-related behavior, and student perceptions about supports and barriers to energy balance.

RESULTS

Delivery and reception did not differ between the groups. Between-group differences in unhealthy behaviors were smaller at posttest, due to improvement in low buy-in classes. Although student perceptions of supports and barriers to energy balance were similar across groups, students in high buy-in classes indicated food preferences as a support while students in low buy-in classes indicated food preferences as a barrier. Neighborhood environment emerged as a universal barrier.

CONCLUSIONS

Trained instructors may improve intervention delivery and reception regardless of classroom context. Social norms could explain high levels of unhealthy behavior in low buy-in classes at baseline; improvement at posttest suggests that the curriculum may have motivated change. All children need more strategies and supportive policies to overcome a challenging food environment.

"Layers of translation" - evidence literacy in public health practice: a qualitative secondary analysis

BACKGROUND

Strengthening public health systems has been a concern in Canada in the wake of public health emergencies. In one Canadian province, British Columbia, a high priority has been placed on the role of evidence to guide decision making; however, there are numerous challenges to using evidence in practice. The National Collaborating Centre for Methods and Tools therefore developed the Evidence Informed Public Health Framework (EIPH), a seven step guide to assist public health practitioners to use evidence in practice. We used this framework to examine the evidence literacy of public health practitioners in BC.

METHODS

We conducted a secondary analysis of two separate qualitative studies on the public health renewal process in which the use and understanding of evidence were key interview questions. Using constant comparative analysis, we analyzed the evidence-related data, mapping it to the categories of the EIPH framework.

RESULTS

Participants require both data and evidence for multiple purposes in their daily work; data may be more important to them than research evidence. They are keen to provide evidence-based programs in which research evidence is balanced with community knowledge and local data. Practitioners recognise appraisal as an important step in using evidence, but the type of evidence most often used in daily practice does not easily lend itself to established methods for appraising research evidence. In the synthesis stage of the EIPH process, synthesized evidence in the form of systematic reviews and practice guidelines is emphasized. Participants, however, need to synthesize across the multiple forms of evidence they use and see the need for more skill and resources to help them develop skill in this type of synthesis.

CONCLUSIONS

Public health practitioners demonstrated a good level of evidence literacy, particularly at the collective level in the organization. The EIPH framework provides helpful guidance in how to use research evidence in practice, but it lacks support on appraising and synthesizing across the various types of evidence that practitioners consider essential in their practice. We can better support practitioners by appreciating the range of evidence they use and value and by creating tools that help them to do this.

Interaction of Theory and Practice to Assess External Validity

BACKGROUND

Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread.

OBJECTIVE

To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice.

METHODS

We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically.

RESULTS

Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges.

CONCLUSIONS

In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

External Validity: The Next Step for Systematic Reviews?

BACKGROUND

Systematic reviews-which identify, assess, and summarize existing research-are usually designed to determine whether research shows that an intervention has evidence of effectiveness, rather than whether an intervention will work under different circumstances. The reviews typically focus on the internal validity of the research and do not consistently incorporate information on external validity into their conclusions.

OBJECTIVES

In this article, we focus on how systematic reviews address external validity.

METHODS

We conducted a brief scan of 19 systematic reviews and a more in-depth examination of information presented in a systematic review of home visiting research.

RESULTS

We found that many reviews do not provide information on generalizability, such as statistical representativeness, but focus on factors likely to increase heterogeneity (e.g., numbers of studies or settings) and report on context. The latter may help users decide whether the research characteristics-such as sample demographics or settings-are similar to their own. However, we found that differences in reporting, such as which variables are included and how they are measured, make it difficult to summarize across studies or make basic determinations of sample characteristics, such as whether the majority of a sample was unemployed or married.

CONCLUSION

Evaluation research and systematic reviews would benefit from reporting guidelines for external validity to ensure that key information is reported across studies.

Hierarchical Logic Models as a Tool to Evaluate Programmatic Initiatives: Practical Solutions to Identified Problems

INTRODUCTION

Large programmatic grants advance the missions of funding agencies or organizations. This article describes the programmatic impact of using "hierarchical" logic models in two Centers funded by the National Institute of Occupational Safety and Health (NIOSH) that were designed to achieve NIOSH goals. Such models are supportive of priority setting, policy implementation, and effective evaluation.

METHODS

Two NIOSH Centers, an Agricultural Center and an Occupational Safety and Health Education and Research Center, used the same hierarchical logic model process to support the NIOSH programmatic goal of improving worker health and safety in their respective occupational categories. The logic model development processes were led by the same evaluator.

RESULTS

Case studies describe the utilization of "hierarchical" logic models: in each case, NIOSH was the "grandparent", the Center was its descendant (parent) and the cores were the children. This lineage was articulated through the Center-wide logic model and through the logic model of each of its core programmatic areas (core). The Center-wide logic model ensured that the Center's goals, and the intended outcomes and impact of its work were linked to the mission and goals of NIOSH. Each core's logic model articulated how its goals, activities, and outcomes were specifically linked to the Center-wide model.

DISCUSSION

A hierarchical logic model process ensures that the objectives of the funding agency or organization are addressed, and enables stakeholders to articulate the linkages between each layer. This facilitates the process of developing, implementing and evaluating programmatic elements within the framework of strategic planning.

Exploration of the functions of health impact assessment in real-world policymaking in the field of social health inequality: towards a conception of conceptual learning

With the implementation of health impact assessment (HIA)'s conceptual model into real-world policymaking, a number of fundamental issues arise concerning its decision-support function. Rooted in a rational vision of the decision-making process, focus regarding both conceptualisation and evaluation has been mainly on the function of instrumental policy-learning. However, in the field of social health inequalities, this function is strongly limited by the intrinsic 'wickedness' of the policy issue. Focusing almost exclusively on this instrumental function, the real influence HIA can have on policymaking in the longer term is underestimated and remains largely unexploited. Drawing insights from theoretical models developed in the field of political science and sociology, we explore the different decision-support functions HIA can fulfill and identify conceptual learning as potentially the most important. Accordingly, dominant focus on the technical engineering function, where knowledge is provided in order to 'rationalise' the policy process and to tackle 'tame' problems, should be complemented with an analysis of the conditions for conceptual learning, where knowledge introduces new information and perspectives and, as such, contributes in the longer term to a paradigm change.

Evidence and Evidence-Based Practices: Are We There Yet?

The purpose of this article is to move the field of intellectual and closely related developmental disabilities (IDD) towards a better understanding of evidence and evidence-based practices. To that end, we discuss (a) different perspectives on and levels of evidence, (b) commonly used evidence-gathering strategies, (c) standards to evaluate evidence, (d) the distinction between internal and external validity, and (e) guidelines for establishing evidence-based practices. We also describe how the conceptualization and use of evidence and evidence-based practices are changing to accommodate recent trends in the field.

Assessing local capacity to expand rural breast cancer screening and patient navigation: An iterative mixed-method tool

BACKGROUND

Despite federal funding for breast cancer screening, fragmented infrastructure and limited organizational capacity hinder access to the full continuum of breast cancer screening and clinical follow-up procedures among rural-residing women. We proposed a regional hub-and-spoke model, partnering with local providers to expand access across North Texas. We describe development and application of an iterative, mixed-method tool to assess county capacity to conduct community outreach and/or patient navigation in a partnership model.

METHODS

Our tool combined publicly-available quantitative data with qualitative assessments during site visits and semi-structured interviews.

RESULTS

Application of our tool resulted in shifts in capacity designation in 10 of 17 county partners: 8 implemented local outreach with hub navigation; 9 relied on the hub for both outreach and navigation. Key factors influencing capacity: (1) formal linkages between partner organizations; (2) inter-organizational relationships; (3) existing clinical service protocols; (4) underserved populations. Qualitative data elucidate how our tool captured these capacity changes.

CONCLUSIONS

Our capacity assessment tool enabled the hub to establish partnerships with county organizations by tailoring support to local capacity and needs. Absent a vertically integrated provider network for preventive services in these rural counties, our tool facilitated a virtually integrated regional network to extend access to breast cancer screening to underserved women.

Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity

Public health researchers and practitioners are calling for greater focus on external validity, the ability to generalize findings of evidence-based interventions (EBIs) beyond the limited number of studies testing effectiveness. For public health, the goal is applicability: to translate, disseminate, and implement EBIs for an impact on population health. This article is a review of methods and how they might be combined to better assess external validity. The methods include (a) better description of EBIs and their contexts; (b) combining of statistical tools and logic to draw inferences about study samples; (c) sharper definition of the theory behind the intervention and core intervention components; and (d) more systematic consultation of practitioners. For population impact, studies should focus on context features that are likely to be both important (based on program theory) and frequently encountered by practitioners. Mixed-method programs of research will allow public health to expand causal generalizations.

Testing an Integrated Model of Program Implementation: the Food, Health & Choices School-Based Childhood Obesity Prevention Intervention Process Evaluation

Childhood obesity is a complex, worldwide problem. Significant resources are invested in its prevention, and high-quality evaluations of these efforts are important. Conducting trials in school settings is complicated, making process evaluations useful for explaining results. Intervention fidelity has been demonstrated to influence outcomes, but others have suggested that other aspects of implementation, including participant responsiveness, should be examined more systematically. During Food, Health & Choices (FHC), a school-based childhood obesity prevention trial designed to test a curriculum and wellness policy taught by trained FHC instructors to fifth grade students in 20 schools during 2012-2013, we assessed relationships among facilitator behaviors (i.e., fidelity and teacher interest); participant behaviors (i.e., student satisfaction and recall); and program outcomes (i.e., energy balance-related behaviors) using hierarchical linear models, controlling for student, class, and school characteristics. We found positive relationships between student satisfaction and recall and program outcomes, but not fidelity and program outcomes. We also found relationships between teacher interest and fidelity when teachers participated in implementation. Finally, we found a significant interaction between fidelity and satisfaction on behavioral outcomes. These findings suggest that individual students in the same class responded differently to the same intervention. They also suggest the importance of teacher buy-in for successful intervention implementation. Future studies should examine how facilitator and participant behaviors together are related to both outcomes and implementation. Assessing multiple aspects of implementation using models that account for contextual influences on behavioral outcomes is an important step forward for prevention intervention process evaluations.

Research design issues for evaluating complex multicomponent interventions in neighborhoods and communities

Major advances in population health will not occur unless we translate existing knowledge into effective multicomponent interventions, implement and maintain these in communities, and develop rigorous translational research and evaluation methods to ensure continual improvement and sustainability. We discuss challenges and offer approaches to evaluation that are key for translational research stages 3 to 5 to advance optimized adoption, implementation, and maintenance of effective and replicable multicomponent strategies. The major challenges we discuss concern (a) multiple contexts of evaluation/research, (b) complexity of packages of interventions, and (c) phases of evaluation/research questions. We suggest multiple alternative research designs that maintain rigor but accommodate these challenges and highlight the need for measurement systems. Longitudinal data collection and a standardized continuous measurement system are fundamental to the evaluation and refinement of complex multicomponent interventions. To be useful to T3-T5 translational research efforts in neighborhoods and communities, such a system would include assessments of the reach, implementation, effects on immediate outcomes, and effects of the comprehensive intervention package on more distal health outcomes.

Interfacing theories of program with theories of evaluation for advancing evaluation practice: Reductionism, systems thinking, and pragmatic synthesis

Theories of program and theories of evaluation form the foundation of program evaluation theories. Theories of program reflect assumptions on how to conceptualize an intervention program for evaluation purposes, while theories of evaluation reflect assumptions on how to design useful evaluation. These two types of theories are related, but often discussed separately. This paper attempts to use three theoretical perspectives (reductionism, systems thinking, and pragmatic synthesis) to interface them and discuss the implications for evaluation practice. Reductionism proposes that an intervention program can be broken into crucial components for rigorous analyses; systems thinking view an intervention program as dynamic and complex, requiring a holistic examination. In spite of their contributions, reductionism and systems thinking represent the extreme ends of a theoretical spectrum; many real-world programs, however, may fall in the middle. Pragmatic synthesis is being developed to serve these moderate- complexity programs. These three theoretical perspectives have their own strengths and challenges. Knowledge on these three perspectives and their evaluation implications can provide a better guide for designing fruitful evaluations, improving the quality of evaluation practice, informing potential areas for developing cutting-edge evaluation approaches, and contributing to advancing program evaluation toward a mature applied science.

Collaborative mental health services in primary care systems in Latin America: contextualized evaluation needs and opportunities

AIM

This study examined Latin American evaluation needs regarding the development of a collaborative mental health care (CMHC) evaluation framework as seen by local key health-care leaders and professionals. Potential implementation challenges and opportunities were also identified.

METHODS

This multisite research study used an embedded mixed methods approach in three public health networks in Mexico, Nicaragua and Chile. Local stakeholders participated: decision-makers in key informant interviews, front-line clinicians in focus groups and other stakeholders through a survey. The analysis was conducted within site and then across sites.

RESULTS

A total of 22 semi-structured interviews, three focus groups and 27 questionnaires (52% response rate) were conducted. Participants recognized a strong need to evaluate different areas of CMHC in Latin America, including access, types and quality of services, human resources and outcomes related to mental disorders, including addiction. A priority was to evaluate collaboration within the health system, including the referral system. Issues of feasibility, including the weaknesses of information systems, were also identified.

CONCLUSION

Local stakeholders strongly supported the development of a comprehensive evaluation framework for CMHC in Latin America and cited several dimensions and contextual factors critical for inclusion. Implementation must allow flexibility and adaptation to the local context.

A Pilot Study of the Creation and Implementation of a Teaching Development Assessment Tool

BACKGROUND

The importance of effective clinical teaching skills is well established in the literature. However, reliable tools with validity evidence that are able to measure the development of these skills and can effectively be used by nonphysician raters do not exist.

OBJECTIVE

Our initiative had 2 aims: (1) to develop a teaching development assessment tool (TDAT) that allows skill assessment along a continuum, and (2) to determine if trained nonphysicians can assess clinical teachers with this tool.

METHODS

We describe the development of the TDAT, including identification of 6 global teaching domains and observable teaching behaviors along a 3-level continuum (novice/beginner, competent/proficient, expert) and an iterative revision process involving local and national content experts. The TDAT was studied with attending physicians during inpatient rounds with trained physician and nonphysician observers over 6 months.

RESULTS

The TDAT showed emerging evidence of content, construct, and viable validity (the degree to which an assessment tool is practical, affordable, suitable, evaluable, and helpful in the real world) for the evaluation of attending physicians on inpatient rounds. Moderate to near perfect interrater reliability was seen between physician and nonphysician raters for the domains of promotion of clinical reasoning, control of the learning environment, ability to teach to multiple levels of learners, and provision of feedback.

CONCLUSIONS

The TDAT holds potential as a valid and reliable assessment tool for clinical teachers to track the development of each individual's teaching skills along the continuum from early development to mastery.

"They just know": the epistemological politics of "evidence-based" non-formal education

Community education and outreach programs should be evidence-based. This dictum seems at once warranted, welcome, and slightly platitudinous. However, the "evidence-based" movement's more narrow definition of evidence--privileging randomized controlled trials as the "gold standard"--has fomented much debate. Such debate, though insightful, often lacks grounding in actual practice. To address that lack, the purpose of the study presented in this paper was to examine what actually happens, in practice, when people support the implementation of evidence-based programs (EBPs) or engage in related efforts to make non-formal education more "evidence-based." Focusing on three cases--two adolescent sexual health projects (one in the United States and one in Kenya) and one more general youth development organization--I used qualitative methods to address the questions: (1) How is evidence-based program and evidence-based practice work actually practiced? (2) What perspectives and assumptions about what non-formal education is are manifested through that work? and (3) What conflicts and tensions emerge through that work related to those perspectives and assumptions? Informed by theoretical perspectives on the intersection of science, expertise, and democracy, I conclude that the current dominant approach to making non-formal education more evidence-based by way of EBPs is seriously flawed.

Using the exhibited generalization approach to evaluate a carbon monoxide alarm ordinance

Current interests in enhancing the focus of external validity or transferability call for developing practical evaluation approaches and illustrating their applications in this area for meeting the need. This study takes the challenge by introducing an innovative evaluation approach, named the exhibited generalization approach, and applying it in evaluating the carbon monoxide (CO) alarm ordinance passed by Mecklenburg County, North Carolina. The stakeholders specifically asked evaluators to determine the answers to the following two questions: (1) Does the alarm ordinance work? (2) What generalizable information can the Mecklenburg experience provide to other jurisdictions trying to decide if the alarm ordinance's planning, implementation, adoption, and outcomes are transferable to their communities? This study illustrates how to apply the exhibited generalization approach to provide the stakeholders with answers to these questions. Our results indicate that the alarm ordinance was effective in increasing CO alarm ownerships and reducing CO poisoning cases. The evaluation provides potential users and other interested parties with the necessary information on contextual factors and the causal mechanism underlying the CO alarm ordinance, so that these parties and users could decide whether the Mecklenburg alarm ordinance would be transferable to their own communities. Discussions include implications of this study for contributing in further advancing evaluation theory in addressing transferability or external validity issues.

Evolutionary Evaluation: implications for evaluators, researchers, practitioners, funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of "evidence-based programs". From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.

What counts? For whom?: Cultural beacons and unexpected areas of programmatic impact

The present article was motivated by our observations that (1) current methods for gathering data do not wholly capture program-related transformations, and (2) grassroots ways of knowing yield legitimate data and can enrich programmatic efforts and evaluations. Accordingly, our work seeks to leverage grassroots knowledge in order to both recognize and respect cultural beacons (CBs) - culturally embedded, user-defined aids for understanding program-related change. Simply, these inductively gathered, locally identified CBs illuminate what to measure and/or how to measure it. Our analysis of participatory evaluations from four international, social change interventions offers four sites for detecting CBs: material possessions, community landscape features, social behaviors, and community-inspired art. We examine the methodological and relational contexts that facilitate CB discovery, discuss unexpected areas of programmatic impact, and utilize lessons learned across projects to make suggestions for continued exploration of CBs in monitoring and evaluation design and practice.

Pricing health behavior interventions to promote adoption: lessons from the marketing and business literature

The relatively high cost of delivering many public health interventions limits their potential for broad public impact by reducing their likelihood of adoption and maintenance over time. Practitioners identify cost as the primary factor for which interventions they select to implement, but researchers rarely disseminate cost information or consider its importance when developing new interventions. A new approach is proposed whereby intervention developers assess what individuals and agencies adopting their interventions are willing to pay and then design interventions that are responsive to this price range. The ultimate goal is to develop effective and affordable interventions, called lean interventions, which are widely adopted and have greater public health impact.

A systems-based partnership learning model for strengthening primary healthcare

Background

Strengthening primary healthcare systems is vital to improving health outcomes and reducing inequity. However, there are few tools and models available in published literature showing how primary care system strengthening can be achieved on a large scale. Challenges to strengthening primary healthcare (PHC) systems include the dispersion, diversity and relative independence of primary care providers; the scope and complexity of PHC; limited infrastructure available to support population health approaches; and the generally poor and fragmented state of PHC information systems.

Drawing on concepts of comprehensive PHC, integrated quality improvement (IQI) methods, system-based research networks, and system-based participatory action research, we describe a learning model for strengthening PHC that addresses these challenges. We describe the evolution of this model within the Australian Aboriginal and Torres Strait Islander primary healthcare context, successes and challenges in its application, and key issues for further research.

Discussion

IQI approaches combined with system-based participatory action research and system-based research networks offer potential to support program implementation and ongoing learning across a wide scope of primary healthcare practice and on a large scale. The Partnership Learning Model (PLM) can be seen as an integrated model for large-scale knowledge translation across the scope of priority aspects of PHC. With appropriate engagement of relevant stakeholders, the model may be applicable to a wide range of settings. In IQI, and in the PLM specifically, there is a clear role for research in contributing to refining and evaluating existing tools and processes, and in developing and trialling innovations. Achieving an appropriate balance between funding IQI activity as part of routine service delivery and funding IQI related research will be vital to developing and sustaining this type of PLM.

Summary

This paper draws together several different previously described concepts and extends the understanding of how PHC systems can be strengthened through systematic and partnership-based approaches. We describe a model developed from these concepts and its application in the Australian Indigenous primary healthcare context, and raise questions about sustainability and wider relevance of the model.

Seeking best practices: a conceptual framework for planning and improving evidence-based practices

How can we encourage ongoing development, refinement, and evaluation of practices to identify and build an evidence base for best practices? On the basis of a review of the literature and expert input, we worked iteratively to create a framework with 2 interrelated components. The first — public health impact — consists of 5 elements: effectiveness, reach, feasibility, sustainability, and transferability. The second — quality of evidence — consists of 4 levels, ranging from weak to rigorous. At the intersection of public health impact and quality of evidence, a continuum of evidence-based practice emerges, representing the ongoing development of knowledge across 4 stages: emerging, promising, leading, and best. This conceptual framework brings together important aspects of impact and quality to provide a common lexicon and criteria for assessing and strengthening public health practice. We hope this work will invite and advance dialogue among public health practitioners and decision makers to build and strengthen a diverse evidence base for public health programs and strategies.