Barriers and facilitators to evaluation of health policies and programs: Policymaker and researcher perspectives

Predictors of body image perceptions among healthcare providers in Terengganu, Malaysia

Objectives

The study's aim was to find predictors of body image perception scores among health care providers (HCPs) in Terengganu, Malaysia.

Methods

A cross-sectional study among 201 health care providers (HCPs) was conducted from January to March 2023. This study participants were drawn from a pool of HCPs with a BMI of 25 kg/m2 or higher. Body image perceptions were scored using the Malay version of the Body Self-Image Questionnaire-Short Form. Multiple linear regression was used to determine the predictors of body image perception scores.

Results

Age and longer work hours correlate with higher body satisfaction among HCPs, while higher BMI lowers satisfaction. Nurses typically experience more height dissatisfaction. Higher income is associated with better self-ratings in looks, health, and fitness, but higher education correlates with poorer self-ratings. Married HCPs report lower attractiveness, health, fitness, and self-esteem but are more satisfied with their height.

Conclusion

This situational analysis identifies predictors, such as sociodemographic factors and occupational aspects, that can aid in customising interventions to address specific body image concerns among HCPs, with the goal of reducing obesity prevalence among them.

Barriers and Enablers to Evaluating Outcomes From Public Involvement in Health Service Design: An Interpretive Description

While health services are expected to have public involvement in service (re)design, there is a dearth of evaluation of outcomes to inform policy and practice. There are major gaps in understanding why outcome evaluation is under-utilised. The aims of this interpretive descriptive study were to explore researcher participants' experiences of and/or attitudes towards evaluating health service outcomes from public involvement in health service design in high-income countries. Additionally, the aims were to explore barriers and enablers of evaluation, and reasons for the use of evaluation tools or frameworks. Semi-structured interviews (n = 13) were conducted with researchers of published studies where the public was involved in designing health services. Using framework analysis, four themes were developed that captured participants' experiences: Public involvement is hard - evaluation is harder; power, a diversity of agendas, and the invisible public; practical and methodological challenges; and genuineness and authenticity matter. Evaluation is driven by stakeholder requirements, including decision-makers, funding bodies, researchers, and academics, and evaluation tools are rarely used. The public is largely absent from the outcome evaluation agenda. There is a lack of commitment and clarity of purpose of public involvement and its evaluation. Outcome evaluation must be multi-layered and localised and reflect the purpose of public involvement, what constitutes success (and to whom), and use the most appropriate methods. Multi-level supports should include increased resources, such as funding, time, and expertise. Without improved evaluation, outcomes of investment in public involvement in health service design/redesign remain unknown.

How practitioner, organisational and system-level factors act to influence health promotion evaluation capacity: Validation of a conceptual framework

The need to improve the practice and quality of evaluation in the health promotion and disease prevention field is widely recognised. In order to plan, implement and evaluate health promotion evaluation capacity building efforts, there is a need to better understand the practitioner, organisational and system-level determinants of evaluation capacity and practice. This study aimed to assess the validity Evaluation Practice Analysis Survey (EPAS) constructs using confirmatory factor analysis and validate a conceptual framework of health promotion evaluation capacity using path analysis. Experienced Australian health promotion practitioners completed the survey (n = 219). Twenty-one of the original 23 EPAS scales were assessed as reliable and valid. The final model was found to have good fit (χ²₁₄ = 18.72, p = 0.18, root mean square error of approximation = 0.04, 90% CI 0.00-0.82, Comparative Fit Index = 1.00, standardised root mean square residual = 0.04). This model supports the role of the organisation in facilitating evaluation practice through leadership, culture, systems, support and resources. It builds on existing frameworks from other fields to incorporate political, funding and administrative factors. This study provides an evidence-based model of evaluation capacity that organisations, funders and policy makers can use to plan and implement more effective evaluation capacity building strategies within organisations and the wider prevention field.

Benefits of not smoking during pregnancy for non-Aboriginal women and their babies in New South Wales, Australia: a record linkage study

BACKGROUND

Smoking rates among pregnant women in New South Wales (NSW) have plateaued at 8-9%. To inform relevant smoking reduction efforts, we aimed to quantify the benefits of not smoking during pregnancy for non-Aboriginal NSW mothers and their babies. The benefits of not smoking during pregnancy for NSW Aboriginal mothers have previously been described. These data are important inputs in modelling health and economic impacts of smoking cessation interventions.

METHODS

This population-based cohort study used linked-data from routinely collected data sets. Not smoking during pregnancy was the exposure of interest among all NSW non-Aboriginal women who became mothers of singleton babies in 2012-2016. Unadjusted and adjusted relative risks (aRR) were used to examine associations between not smoking during pregnancy and adverse outcomes including severe morbidity, inter-hospital transfer, perinatal death, preterm birth and small-for-gestational age. Population attributable fractions (PAFs) were calculated to quantify adverse perinatal outcomes avoided in the population if all mothers were non-smokers.

RESULTS

Compared with babies born to mothers who smoked during pregnancy, babies born to non-smoking mothers had a lower risk of all adverse perinatal outcomes including perinatal death (aRR = 0.68, 95%CI 0.61-0.76), preterm birth (aRR = 0.58, 95%CI 0.56-0.61) and small-for-gestational age (aRR = 0.48, 95%CI 0.47-0.50). PAFs(%) were 3.9% for perinatal death, 5.6% for preterm birth and 7.3% for small-for-gestational-age. Compared with women who smoked during pregnancy (n = 36,518), those who did not smoke (n = 413,072) had a lower risk of suffering severe maternal morbidity (aRR = 0.87, 95%CI 0.81-0.93) and being transferred to another hospital (aRR = 0.92, 95%CI 0.86-0.99).

CONCLUSIONS

Mothers who reported not smoking during pregnancy had a small reduction in their risk of morbidity and of being transferred to another hospital whilst their babies had substantially reduced risks of all adverse perinatal outcomes. Results have implications for clinician training, clinical care standards, and performance management.

A systems approach to the exploration of research activity and relationships within a local authority

Background

Organizations with responsibilities for public health are increasingly required to use evidence-based practice to inform programme delivery, requiring research to generate relevant evidence, and dissemination and use of evidence to inform decisions and practices. Understanding how relationships between organizational structures, systems and processes influence evidence-based practices is critical to improving practice at both an institutional and system level, yet how these relationships should best operate is not well understood. Understanding how to better support research within local authorities, the elected administrative bodies responsible for services including public health at a regional level in the United Kingdom, is a priority for the National Institute for Health Research (NIHR) Public Health Research. This study is based on Norfolk County Council, a local authority in the east of England. We aimed to apply a systems perspective to develop a better understanding of the structures, systems and processes that support a local authority in becoming research-active, identifying gaps in understanding and recommendations for action to address them.

Methods

Taking a participatory action research approach, we applied qualitative methods to explore research activity and relationships in Norfolk County Council. We surveyed employees and used network analysis to map individuals, departments and external partners involved in research activities and the connections between them. We then applied participatory approaches to conduct a series of focus groups and semi-structured interviews to explore stakeholders’ experiences and perceptions of being involved in research at, or with, the authority, and their ideas for recommendations for future actions.

Results

A range of research activity is undertaken at the local authority, with an emphasis on applied work to improve service delivery. We identified several examples of effective practice and models of research collaboration in some departments. Challenges such as limitations in resources, capacity and knowledge exchange were evident, yet there was a readiness amongst key stakeholders to develop and implement actions that may better support the authority in becoming more research-active.

Conclusion

In large complex organizations, a key challenge is how to share learning across teams and implement good practice at an organizational and system level. Our findings highlight the potential for developing improved collaborative partnership models and systems to support sustainable processes and practices for research and knowledge exchange at an institutional and interorganizational level. The insights gained and shared will support other local authorities and similar large, multilevel organizations with responsibilities for evidence-based public health to explore their own setting and implement change where needed, and provide stimulus for further research into system-level change.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-021-00792-0.

Opportunities and disconnects in the use of primary research on schistosomiasis and soil-transmitted helminths for policy and practice: results from a survey of researchers

Even with efforts to facilitate use of evidence in health policy and practice, limited attention has been paid to researchers' perspectives on use of their research in informing public health policy and practice at local, national, and global levels. We conducted a systematic literature search to identify published primary research related to schistosomiasis or soil-transmitted helminths, or both. We then surveyed corresponding authors. Results indicate differences by locations of authors and in conduct of research, especially for research conducted in low- and middle-income countries. Our findings exemplify disparities in research leadership discussed elsewhere. Researchers' perspectives on the use of their work suggest limited opportunities and 'disconnects' that hinder their engagement with policy and other decision-making processes. These findings highlight a need for additional efforts to address structural barriers and enable engagement between researchers and decision-makers.

Exploring influences on evaluation practice: a case study of a national physical activity programme

BACKGROUND

Interventions to improve physical activity behaviour are a core part of public health policy and practice. It is essential that we evaluate these interventions and use the evidence to inform decisions to improve population health. Evaluation of 'real-world' interventions provide an opportunity to generate practice-relevant evidence, however these interventions are difficult to evaluate. Various guidelines have been developed to facilitate evaluation, but evidence about their effectiveness in practice is limited. To explore influences on evaluation practice in an applied context, we conducted a case study of Sport England's 'Get Healthy Get Active' (GHGA) programme. This was a national programme that funded 33 projects that were delivered and evaluated across England. The programme was chosen as it was designed to generate evidence on the role of sport in increasing physical activity and improving health. The study aimed to explore and appraise whether strategies intended to facilitate project evaluation, including funder requirements to use a standardised evaluation framework and specific data collection methods, were effective in generating evidence that enabled the programme to meet its aims.

METHODS

We applied a collective case study design involving 35 semi-structured interviews, and documentary analysis of multiple sources of evidence from 23 physical activity projects funded by GHGA. We applied thematic and framework analysis. We developed a logic model and mapped actual outcomes against intended outcomes. A narrative synthesis is provided. We discuss implications for the effective commissioning and evaluation of public health interventions.

RESULTS

We identified five main themes of influences on evaluation practices that can act as barriers and facilitators to good practice: programme and project design; evaluation design; partnerships; resources; and organisational structures and systems. These influences are context-specific and operate through a complex set of interactions.

CONCLUSION

Developing a better understanding of how influences on evaluation practice can act as facilitators or barriers is vital to help close current gaps in the evidence-based practice cycle. Critically, organisational structures and systems are needed to facilitate collaborative decision making; integration of projects and evaluation across partners organisations; transfer of knowldege and insights between stakeholders; and more rapid feedback and dissemination.

Planning, implementation and evaluation of antenatal weight management programmes: What are the key components? A mixed methods study

OBJECTIVE

To provide an understanding of the influential components required for the planning, delivery and evaluation of antenatal weight management programmes for women who are overweight or obese.

DESIGN

Two phase sequential explanatory mixed methods design comprising of an online survey and one-to-one telephone interviews.

SETTING AND PARTICIPANTS

All UK maternity services (n = 168) and local government councils (n = 417).

FINDINGS

From the 378 responses, 49 maternity services and 28 local government councils reported having an antenatal weight management programme. Of the 62 responses that specified BMI as an inclusion criterion only two-fifths (40.3% n = 25) adhered to the recommended threshold to include women with a BMI ≥30 kg/m². Although having a multi-disciplinary team was identified as fundamental when planning a programme, only 20.6% (n = 14/68) of the programmes involved service users during the planning phase. How the programme was communicated to a woman was a key factor which determined whether she partook in the programme or not. Having administrative support and staff with expertise in data collection were essential when evaluating a programme. Staff having protected time was identified as important when planning, delivering and evaluating a programme. Three overarching themes were identified from the individual interviews (n = 14) 'choices and decisions', 'demands and resources' and 'engagement and disengagement'.

KEY CONCLUSIONS

National guidelines recommendations regarding service user involvement when planning programmes and the BMI threshold used for inclusion are not being met. In addition to having adequate time, personnel and finances, successful programmes are dependent on the confidence and communication skills of midwives to raise the issue of obesity with these women at the booking appointment. Without staff having the time and necessary knowledge and skills, evaluation, and hence demonstrating programme impact, will likely remain difficult. Organisational support is needed to release resources to plan, deliver and evaluate these programmes. Strategic communication strategies are needed to promote the programme to both women and staff within organisations. Future programmes need to ensure there is engagement with service users from the planning of the programme through to evaluation.

An assessment of program evaluation methods and quality in Australian prevention agencies

ISSUE ADDRESSED

This study aimed to examine evaluation methods and quality in Australian health promotion agencies and the factors associated with this. The evidence base for prevention strategies is limited, with the evidence generated through program evaluation by health promotion and disease prevention agencies lacking rigour. Despite the need to improve the quality of evaluation, there is limited evidence of what influences evaluation quality in the prevention field.

METHODS

Data were collected using the Evaluation Practice Analysis Survey and an audit and appraisal of evaluation reports. Descriptive analysis was used to examine evaluation characteristics and multivariable regression was used to explore the association between evaluation and organisational attributes and evaluation quality.

RESULTS

In total, 392 evaluation reports were reviewed from 78 government and non-government agencies. Process evaluation was conducted most frequently, followed by impact evaluation. Overall evaluation quality was low (median 24.5%). In multivariable regression analysis, only two factors were associated with evaluation quality: health promotion budget (ratio of geometric means 1.53 [95% CI 1.02-2.29]); and, conducting statewide or national prevention programs (1.38 [95% CI 1.05-1.82]).

CONCLUSIONS

The findings show that the potential to improve evaluation quality is greatest in smaller organisations that deliver health promotion at a local or regional scale. SO WHAT?: By improving the rigour of existing evaluation, there is opportunity to build the evidence base for prevention strategies, which highlights the importance of embedding the enablers of program learning and evidence generation within health promotion and prevention organisations.

Understanding the factors that influence health promotion evaluation: The development and validation of the evaluation practice analysis survey

The demand for improved quality of health promotion evaluation and greater capacity to undertake evaluation is growing, yet evidence of the challenges and facilitators to evaluation practice within the health promotion field is lacking. A limited number of evaluation capacity measurement instruments have been validated in government or non-government organisations (NGO), however there is no instrument designed for health promotion organisations. This study aimed to develop and validate an Evaluation Practice Analysis Survey (EPAS) to examine evaluation practices in health promotion organisations. Qualitative interviews, existing frameworks and instruments informed the survey development. Health promotion practitioners from government agencies and NGOs completed the survey (n = 169). Principal components analysis was used to determine scale structure and Cronbach's α used to estimate internal reliability. Logistic regression was conducted to assess predictive validity of selected EPAS scale. The final survey instrument included 25 scales (125 items). The EPAS demonstrated good internal reliability (α > 0.7) for 23 scales. Dedicated resources and time for evaluation, leadership, organisational culture and internal support for evaluation showed promising predictive validity. The EPAS can be used to describe elements of evaluation capacity at the individual, organisational and system levels and to guide initiatives to improve evaluation practice in health promotion organisations.

Barriers to the evaluation of evidence-based public health policy

Public health policy has the potential to produce great benefits for individuals and communities. There is growing demand that such efforts be rigorously evaluated to ensure that the expected benefits are, in fact, realised. Commonly, public health policy is evaluated by consumer acceptability, reach, or changes in knowledge and attitudes. Non-robust research designs are often used. But these approaches to evaluation do not answer three critical questions: Has a change in the desired outcome occurred? Was it a consequence of the policy and not some extraneous factor? Was the size of the change considered significant and cost-effective? We, a team of government and academic scholars working in research and evaluation, have examined some of the more common impediments to robust evaluation: political impediments, a lack of investment in evaluation capacity within bureaucracy, and the failure of academic researchers to understand the need for the evaluation of public health policy.

Organizational determinants of evaluation practice in Australian prevention agencies

Program evaluation is essential to inform decision making, contribute to the evidence base for strategies, and facilitate learning in health promotion and disease prevention organizations. Theoretical frameworks of organizational learning, and studies of evaluation capacity building describe the organization as central to evaluation capacity. Australian prevention organizations recognize limitations to current evaluation effectiveness and are seeking guidance to build evaluation capacity. This qualitative study identifies organizational facilitators and barriers to evaluation practice, and explores their interactions in Australian prevention organizations. We conducted semi-structured interviews with 40 experienced practitioners from government and non-government organizations. Using thematic analysis, we identified seven key themes that influence evaluation practice: leadership, organizational culture, organizational systems and structures, partnerships, resources, workforce development and training and recruitment and skills mix. We found organizational determinants of evaluation to have multi-level interactions. Leadership and organizational culture influenced organizational systems, resource allocation and support of staff. Partnerships were important to overcome resource deficits, and systems were critical to embed evaluation within the organization. Organizational factors also influenced the opportunities for staff to develop skills and confidence. We argue that investment to improve these factors would allow organizations to address evaluation capacity at multiple levels, and ultimately facilitate effective evaluation practice.

Disability Policy Evaluation: Combining Logic Models and Systems Thinking

Policy evaluation focuses on the assessment of policy-related personal, family, and societal changes or benefits that follow as a result of the interventions, services, and supports provided to those persons to whom the policy is directed. This article describes a systematic approach to policy evaluation based on an evaluation framework and an evaluation process that combine the use of logic models and systems thinking. The article also includes an example of how the framework and process have recently been used in policy development and evaluation in Flanders (Belgium), as well as four policy evaluation guidelines based on relevant published literature.