Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia

Do community-based digital health inclusion programmes contribute to tackling health inequalities in disadvantaged population groups?: a qualitative study of experiences of a city-wide programme in the North of England

BACKGROUND

Advances in digital healthcare and health information provide benefits to the public. However, lack of digital skills together with access, confidence, trust and motivation issues present seemingly insurmountable barriers for many. Such digital health exclusion exacerbates existing health inequalities experienced by older people, people with less income, less education or who don't have English as a first language. This study examines the role of a city-wide digital inclusion programme in the North of England, which works with Voluntary Sector Community Organisations (VCSOs) to provide digital support to disadvantaged communities (Digital Health Hubs). The aim was to explore if and how Digital Health Hubs contribute to tackling health inequalities, with a specific focus on impacts on service-users and how these impacts are produced.

METHODS

We used qualitative semi-structured interviews to explore the experiences of service-users receiving digital support, and perspectives of staff working for organisations coordinating or providing digital support (n = 30).

RESULTS

Inductive thematic analysis was used to identify four major themes and mapped to 'Consolidated Framework for Implementation Research' constructs. These were: programme components, recipient-centred support, impacts on service-users and system-level barriers to digital health services. Findings suggest moderators of implementation are provision of community spaces, social activities and devices and Wi-fi, and recipient-centred support through community understanding, personalised regular support and trusting relationships. Impacts on service-users include improved social and emotional resilience and basic digital skills. Health system-level barriers to digital health services, such as inconsistency in service provision, were also identified.

CONCLUSIONS

Themes highlight the importance of recipient-centred support by Voluntary and Community Sector Organisations (VCSOs) which centres the unique needs of specific communities. Our data provide policymakers with a model for implementation that could be replicated and scaled-up. Our data add to the public health, primary care and digital health research by proposing an initial programme theory for how Digital Health Hubs may reduce health inequalities, and recommendations for evaluation which consider short-, medium- and long-term outcomes, and contextual factors which are likely to be crucial to Digital Health Hubs' effectiveness.

A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance

The UK Medical Research Council's widely used guidance for developing and evaluating complex interventions has been replaced by a new framework, commissioned jointly by the Medical Research Council and the National Institute for Health Research, which takes account of recent developments in theory and methods and the need to maximise the efficiency, use, and impact of research.

Workplace-Based Interventions for Mental Health in Africa: A Scoping Review

Mental health problems are one of the leading contributors to the global burden of disease. Workplaces provide a valuable and accessible setting for interventions to improve worker health. However, little is known about mental health interventions on the African continent, particularly those based in the workplace. This review aimed to identify and report on the literature about workplace-based interventions for mental health in Africa. This review followed the JBI and PRISMA ScR guidelines for scoping reviews. We searched 11 databases for qualitative, quantitative and mixed-methods studies. The grey literature was included, and no language or date limits were applied. Title and abstract screening and full text review, were completed independently by two reviewers. A total of 15 514 titles were identified, of which, 26 titles were included. The most common study designs were qualitative studies (n = 7) and pre-experimental, single-group, pre-test, post-test studies (n = 6). Workers with depression, bipolar mood disorder, schizophrenia, intellectual disability, alcohol and substance abuse, stress and burnout were included in the studies. The participants were mostly skilled and professional workers. A wide variety of interventions were offered, of which, most were multi-modal. There is a need to develop multi-modal interventions in partnership with stakeholders, particularly for semi-skilled and unskilled workers.

Where Is Equity in HiAP? Comment on "A Realist Explanatory Case Study Investigating How Common Goals, Leadership, and Committed Staff Facilitate Health in All Policies Implementation in the Municipality of Kuopio, Finland"

Health equity is no longer a central feature of Health in All Policies (HiAP) approaches despite its presence in select definitions of HiAP. In other words, HiAP is not just about considering health, but also health equity. But as HiAP has become more mainstream, its success around health equity has been muted and largely non-existent. Given the normative underpinning and centrality of equity in HiAP, equity should be better considered in HiAP and particularly when considering what 'successful' implementation may look like. Raising health on the radar of policy-makers is not mutually exclusive from considering equity. Taking an incremental approach to considering equity in HiAP can yield positive results. This article discusses these ideas and presents potential actions to restore HiAP's once central equity objectives, which include: seeking synergies focused on health equity with those who hold different convictions, both in terms of goals and measures of success; considering the conditions that allow HiAP to be fostered, such as good governance; and drawing on research on HiAP and other multisectoral approaches.

How to Evaluate Health in All Policies at the Local Level: Methodological Insights Within Municipalities From the WHO French Healthy Cities Network

BACKGROUND

This article proposes a method for analysing the degree of maturity of Health in All Policies (HiAP) among World Health Organization-French Healthy Cities Network (WHO-FHCN) as part of the GoveRnance for Equity, EnviroNment and Health in the City (GREENH-City) project. We focused on the creation or enhancement of health-promoting environments, and more specifically, public green spaces.

METHODS

We conducted a cross-sectional quantitative study guided by the evaluative framework of the HiAP maturity level developed by Storm et al mixed with a qualitative interpretation. A self-administered questionnaire was sent to elected officials and health department officers in the 85 member cities of the WHO-FHCN in 2017. Subsequently 58 cities were included in the analysis, which was based on a multiple correspondence analysis (MCA) and a hierarchical ascending classification (HAC).

RESULTS

Thirty-two criteria among a total of 100 were identified and were used to organize the cities into 8 groups which was then reduced to three profiles among the cities: a less advanced HiAP profile, an established HiAP profile and an advanced HiAP profile. This process allows us to identify 4 dimensions that make it possible to evaluate the level of maturity of cities in the HiAP process, namely: (1) the consideration of social inequalities in health and/or health issues in the policies/actions of the sector studied, (2) occasional intersectoral collaboration, ie, one-off initiatives between the health department and others sectors, (3) the existence of joint projects, ie, common projects between two or more sectors, (4) the existence of intersectoral bodies, in this case on the theme of urban green spaces including an intersectoral committee and/or working groups.

CONCLUSION

Four dimensions which allow to the measurement of the degree of progress in implementing health-all-policies are proposed. With a view to integrating knowledge into public action, this study carried out under real conditions offers a realistic method to evaluate HiAP.

Evaluating ‘Health in All Policies’ in Norwegian Municipalities

The Health in All Policies (HiAP) approach has emerged as a central strategy for promoting health at local, regional, and state levels in different countries. Representing a complex and complicated strategy, evaluations of HIAP at the local level are scarce, and scholars call for more knowledge and critical discussions regarding how to evaluate at this level. In this conceptual paper, I discuss how summative and formative evaluation approaches might supplement each other when evaluating HiAP at the local level. First, I discuss the possibilities of using summative and formative evaluation of HiAP. Further, I discuss how formative-dialogue research might represent possibilities for evaluation by combining the two approaches. Finally, I ask if there has been a shift in the Norwegian evaluation discourse, from the promotion of summative evaluation alone to a combination of both summative and formative methods.

Australian State and Territory Eclectic Approaches to Obesity Prevention in the Early Years: Policy Mapping and Perspectives of Senior Health Officials

Background

The international increase in the prevalence of childhood obesity has hastened in recent decades. This rise has coincided with the emergence of comorbidities in childhood-such as type II diabetes, non-alcoholic fatty liver disease, metabolic syndrome, sleep apnoea and hypertension-formerly only described in adulthood. This phenomenon suggests global social and economic trends are impacting on health supportive environments. Obesity prevention is complex and necessitates both long-term and systems approaches. Such an approach considers the determinants of health and how they interrelate to one another. Investment in the early years (from conception to about 5 years of age) is a key life stage to prevent obesity and establish lifelong healthy habits relating to nutrition, physical activity, sedentary behavior and sleep. In Australia, obesity prevention efforts are spread across national and state/territory health departments. It is not known from the literature how, with limited national oversight, state and territory health departments approach obesity prevention in the early years.

Methods

We conducted a qualitative study including policy mapping and interviews with senior officials from each Australian state/territory health department. A series of questions were developed from the literature to guide the policy mapping, drawing on the World Health Organisation Ending Childhood Obesity Report, and adapted to the state/territory context. The policy mapping was iterative. Prior to the interviews initial policy mapping was undertaken. During the interviews, these policies were discussed, and participants were asked to supply any additional policies of relevance to obesity prevention. The semi-structured interviews explored the approaches to obesity prevention taken in each jurisdiction and the barriers and enablers faced for policy implementation. Thematic analysis was used to analyse the data, using NVivo software.

Results

State and territory approaches to obesity prevention are eclectic and while there are numerous similarities between jurisdictions, no two states are the same. The diversity of approaches between jurisdictions is influenced by the policy culture and unique social, geographic, and funding contexts in each jurisdiction. No Australian state/territory had policies against all the guiding questions. However, there are opportunities for sharing and collaborating within and between Australian jurisdictions to establish what works, where, and for whom, across Australia's complex policy landscape.

Conclusions

Even within a single country, obesity prevention policy needs to be adaptable to local contexts. Opportunities for jurisdictions within and between countries to share, learn, and adapt their experiences should be supported and sustained funding provided.

Community-Based Interventions for the Treatment and Management of Conflict-Related Trauma in Low-Middle Income, Conflict-Affected Countries: a Realist Review

Where low- and middle-income countries have limited economic resources to provide individualized mental health services to people exposed to conflict, community-based interventions may be more appropriate. We aimed to evaluate community level interventions for improving mental health outcomes in Low- and Middle-income countries (LMIC). A realist review of community-based interventions (CBIs) to improve mental health for people in LMIC following conflict. Five databases (Cochrane, PubMed, PsychINFO, Medline, and CINAHL) and a manual search of individual papers. We found 1318 articles, of which 29 were selected. Out of the 29 primary articles, 19 showed successful results, 4 showed mixed results, 1 showed inconclusive results, and 1 showed unsuccessful results. After analyzing the results, we found 3 mechanisms that may influence the effectiveness of these CBIs: the use of lay community members as intervention deliverers, the application of transdiagnostic approaches, and customized outcome assessment tools. Community-based approaches to improve mental health in LMICs are rare and evidence for their effectiveness is limited. Interventions that have a wide scope, train lay mental health workers, and use contextually adapted outcome assessment tools show promise.

Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

BACKGROUND

The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods.

METHODS

There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments.

RESULTS

Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six 'core elements' that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout.

LIMITATIONS

The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development.

CONCLUSIONS

This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as 'events in systems' that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers.

FUTURE WORK

We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources.

FUNDING

This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).

Developing a tool to monitor knowledge translation in the health system: results from an international Delphi study

BACKGROUND

It is generally accepted that evidence-informed decision making contributes to better health system performance and health outcomes, yet we are lacking benchmarks to monitor the impact of national health information systems (HIS) in policy and practice. Hence in this study, we have aimed to identify criteria for monitoring Knowledge Translation (KT) capacity within countries.

METHODS

We conducted a web-based Delphi with over 120 public health professionals from 45 countries to reach agreement on criteria to monitor KT at the level of national HIS. Public health professionals participated in three survey rounds, in which they ranked 85 preselected criteria and could suggest additional criteria.

RESULTS

Experts working in national (public) health agencies and statistical offices, as well as in health policy and care agreed on 29 criteria which constitute the Health Information (HI)-Impact Index. The criteria cover four essential domains of evaluation: the production of high-quality evidence, broad access and dissemination, stakeholder engagement and knowledge integration across sectors and in civil society. The HI-Impact Index was pretested by officials working in ministries of health and public health agencies in eight countries; they found the tool acceptable and user-friendly.

CONCLUSIONS

The HI-Impact Index provides benchmarks to monitor KT so that countries can assess whether high-quality evidence can be easily accessed and used by the relevant stakeholders in health policy and practice, by civil society and across sectors. Next steps include further refining the procedure for conducting the assessment in routine, and sharing experiences from HIS evaluations using the HI-Impact Index.

Opportunity for HiAP through a Healthy Cities initiative in Taiwan: a multiple streams analysis

Health in All Policies (HiAP) is an effective approach to promote population health through addressing comprehensive social determinants of health. In 1997, the World Health Organization designed a 20-step protocol for developing a Healthy Cities (HC) project to build healthy public policies. Taiwan adopted the concept of HC in 2002 and established the first demonstration project in Tainan City in 2003. This study explores the impact of the HC initiative on the development of HiAP, as well as how a window of opportunities for HiAP was opened through the HC movement, using Tainan City as an example. Tainan was selected as the case for this study because of its relative maturity in the development of an HC initiative. A theory-driven thematic analysis was conducted in the study with archived documents between 2003 and 2010. We first adopted a Multiple Streams Approach to examine how the city government opened a window for HiAP through the HC initiative and then drew on the Maturity Model to evaluate the stage of HiAP in the Tainan HC project. After analyzing all related documents, we concluded that HiAP in the Tainan HC project had reached Stage V Institutionalization after 7 years. Key contexts, factors, challenges and strategies were identified. This study suggests that the HC initiative can indeed contribute to the development of HiAP and that the window was opened for HiAP through a concurrence of the three streams of problem, policy and politics along with the HC movement process.

The future of public health policymaking after COVID-19: a qualitative systematic review of lessons from Health in All Policies

Background: 'Health in All Policies' (HiAP) describes the pursuit of health equity. It has five main elements: treat health as a human right; identify evidence of the 'social determinants' of health inequalities, recognise that most powers to affect health are not held by health departments, promote intersectoral policymaking and collaboration inside and outside of government, and generate political will. Studies describe its potential but bemoan a major implementation gap. Some HiAP scholars learn from policymaking research how to understand this gap, but the use of policy theories is patchy. In that context, our guiding research question is: How does HiAP research use policy theory to understand policymaking? It allows us to zoom-out to survey the field and zoom-in to identify: the assumed and actual causes of policy change, and transferable lessons to HiAP scholars and advocates. Methods: Our qualitative systematic review (two phases, 2018 and 2020) identified 4972 HiAP articles. Of these, 113 journal articles (research and commentary) provide a non-trivial reference to policymaking (at least one reference to a policymaking concept). We use the 113 articles to produce a general HiAP narrative and explore how the relatively theory-informed articles enhance it. Results: Most articles focus on policy analysis (identifying policy problems and solutions) rather than policy theory (explaining policymaking dynamics). They report a disappointing gap between HiAP expectations and policy outcomes. Theory-informed articles contribute to a HiAP playbook to close that gap or a programme theory to design and evaluate HiAP in new ways.   Conclusions: Few HiAP articles use policy theories for their intended purpose. Policy theories provide lessons to aid critical reflection on power, political dilemmas, and policymaking context. HiAP scholars seek more instrumental lessons, potentially at the cost of effective advocacy and research.

The future of public health policymaking after COVID-19: a qualitative systematic review of lessons from Health in All Policies

Background: 'Health in All Policies' (HiAP) describes the pursuit of health equity. It has five main elements: treat health as a human right; identify evidence of the 'social determinants' of health inequalities, recognise that most powers to affect health are not held by health departments, promote intersectoral policymaking and collaboration inside and outside of government, and generate political will. Studies describe its potential but bemoan a major implementation gap. Some HiAP scholars learn from policymaking research how to understand this gap, but the use of policy theories is patchy. In that context, our guiding research question is: How does HiAP research use policy theory to understand policymaking? It allows us to zoom-out to survey the field and zoom-in to identify: the assumed and actual causes of policy change, and transferable lessons to HiAP scholars and advocates. Methods: Our qualitative systematic review (two phases, 2018 and 2020) identified 4972 HiAP articles. Of these, 113 journal articles (research and commentary) provide a non-trivial reference to policymaking (at least one reference to a policymaking concept). We use the 113 articles to produce a general HiAP narrative and explore how the relatively theory-informed articles enhance it. Results: Most articles focus on policy analysis (identifying policy problems and solutions) rather than policy theory (explaining policymaking dynamics). They report a disappointing gap between HiAP expectations and policy outcomes. Theory-informed articles contribute to a HiAP playbook to close that gap or a programme theory to design and evaluate HiAP in new ways.   Conclusions: Few HiAP articles use policy theories for their intended purpose. Policy theories provide lessons to aid critical reflection on power, political dilemmas, and policymaking context. HiAP scholars seek more instrumental lessons, potentially at the cost of effective advocacy and research.

WITHDRAWN: The role of medical and health policies in the health risk management system

Ahead of Print article withdrawn by publisher.

Promoting Intersectoral Collaboration Through the Evaluations of Public Health Interventions: Insights From Key Informants in 6 European Countries

BACKGROUND

Intersectoral collaboration is critical to the successful implementation of many public health interventions (PHIs). Little attention has been paid to whether and how processes at the stage of evaluation can promote intersectoral collaboration. The objective of this study was to examine European experiences and views on whether and how the evaluation of PHIs promote intersectoral collaboration.

METHODS

A qualitative study design was used. We conducted semi-structured interviews with 15 individuals centrally involved in the evaluation of PHIs in 6 European countries (Austria, Denmark, England, Germany, Norway, and Switzerland). Questions pertained to current processes for evaluating PHIs in the country and current and potential strategies for promoting intersectoral collaboration. Transcripts were analyzed using thematic analysis to identify key themes responding to our primary objective.

RESULTS

Experiences with promoting intersectoral collaboration through the evaluation of PHIs could be summarized in 4 themes: (1) Early involvement of non-health sectors in the evaluative process and inclusion of non-health benefits can promote intersectoral collaboration, but should be combined with greater influence of these sectors in shaping PHIs; (2) Harmonization of methodological approaches may enable comparison of results and facilitate intersectoral collaboration, but should not be an overriding goal; (3) Involvement in health impact assessments (HIAs) can promote intersectoral collaboration, but needs to be incentivized and be conducted without putting overwhelming demands on non-health sectors; (4) A designated body for evaluating PHIs may promote intersectoral collaboration, but its design needs to take account of realities of policy-making.

CONCLUSION

The full potential for promoting intersectoral collaboration through the evaluation of PHIs appears currently unrealized in the settings we studied. To further promote intersectoral collaboration, evaluators and decision-makers may consider the full range of strategies characterized in this study. This may be most effective if the strategies are deployed so that they reinforce each other, value outcomes beyond health, and are tailored to maximize political priority for PHIs across sectors.

Towards a Program Theory for Family-Focused Practice in Adult Mental Health Care Settings: An International Interview Study With Program Leaders

Objectives: In several high-income countries, family-focused practice programs have been introduced in adult mental health care settings to identify and support children whose parents live with mental health problems. Whilst their common goal is to reduce the impact of parental mental illness on children, the mechanisms by which they improve outcomes in different systems and settings are less well known. This kind of knowledge can importantly contribute to ensuring that practice programs achieve pre-defined impacts. Methods: The aim of this study was to develop knowledge about relationships between contextual factors, mechanisms and impact that could inform a program theory for developing, implementing, and evaluating family-focused practice. Principles of a realist evaluation approach and complex system thinking were used to conceptualize the design of semi-structured in-depth interviews with individuals who led the implementation of programs. Seventeen individuals from eight countries participated in the study. Results: Interviewees provided rich accounts of the components that programs should include, contextual factors in which they operated, as well as the behavior changes in practitioners that programs needed to achieve. Together with information from the literature, we developed an initial program theory, which illustrates the interconnectedness between changes that need to co-occur in practitioners, parents, and children, many of which related to a more open communication about parental mental health problems. Stigma, risk-focused and fragmented health systems, and a lack of management commitment were the root causes explaining, for example, why conversations about parents' mental illness did not take place, or not in a way that they could help children. Enabling practitioners to focus on parents' strengths was assumed to trigger changes in knowledge, emotions and behaviors in parents that would subsequently benefit children, by reducing feelings of guilt and improving self-esteem. Conclusion: To our knowledge, this is the first research, which synthesizes knowledge about how family-focused practice programs works in a way that it can inform the design, implementation, and evaluation of programs. Stakeholder, who fund, design, implement or evaluate programs should start co-developing and using program theories like the one presented in this paper to strengthen the design and delivery of family-focused practice.

New frontiers in alcohol and gender: The role of health promotion policy and practice in Australia

Scholarship indicates that gender norms influence drinking behaviours, yet the consequences of this for health professionals and health promotion remains neglected. To address this gap, we discuss the implications of gender and alcohol consumption for Australian health promotion and practice. We convey how a more integrated public health approach, aimed at promoting healthy gender expectations and enhancing gender relations, is warranted. We also discuss how changing gender norms pose new challenges for health professionals. By confronting these contentious issues, this commentary helps the health sector consider innovative measures to combat alcohol-related harms.

Creating a whole-of-government approach to promoting healthy weight: What can Health in All Policies contribute?

OBJECTIVES

This paper examines the opportunities and barriers that the South Australian Health in all Policies (SA HiAP) approach encountered when seeking to establish a whole-of-government response to promoting healthy weight.

METHODS

The paper draws on data collected during 31 semi-structured interviews, analysis of 113 documents, and a program logic model developed via workshops to show the causal links between strategies and anticipated outcomes.

RESULTS

A South Australian Government target to increase healthy weight was supported by SA HiAP to develop a cross-government response. Our analysis shows what supported and hindered implementation. A combination of economic and systemic framing, in conjunction with a co-benefits approach, facilitated intersectoral engagement. The program logic shows how implementation can be expected to contribute to a population with healthy weight.

CONCLUSIONS

The HiAP approach achieved some success in encouraging a range of government departments to contribute to a healthy weight target. However, a comprehensive approach requires national regulation to address the commercial determinants of health and underlying causes of population obesity in addition to cross-government action to promote population healthy weight through regional government action.

Sharing a vision. Do participants in integrated care programmes have the same goals and objectives?

Integrating health and social care services remains one of the most difficult undertakings in the field of care delivery. One of the key requirements for success in integration programmes is a shared vision amongst care providers. Shared visions may contain views as to what the new services should look like, how it should operate and what it should be able to achieve. The paper reports findings of an evaluation of a service integration programme in the North of England. It confirms that a programme consensus on issues such as aims and objectives and programme logics is seen by participants as a key to success. Yet, the study also found that there is a specific window of opportunity in integration programmes when participating organisations start on relatively high levels of commitment and enthusiasm which tend to tail off relatively quickly. The paper closes with a discussion about the implications of the findings for programme designers and service planners.

To what extent can the activities of the South Australian Health in All Policies initiative be linked to population health outcomes using a program theory-based evaluation?

Background

This paper reports on a five-year study using a theory-based program logic evaluation, and supporting survey and interview data to examine the extent to which the activites of the South Australian Health in All Policies initiative can be linked to population health outcomes.

Methods

Mixed-methods data were collected between 2012 and 2016 in South Australia (144 semi-structured key informant interviews; two electronic surveys of public servants in 2013 (n = 435) and 2015 (n = 483); analysis of state government policy documents; and construction of a program logic model to shape assessment of the feasibility of attribution to population health outcomes).

Results

Multiple actions on social determinants of health in a range of state government sectors were reported and most could be linked through a program logic model to making some contribution to future population health outcomes. Context strongly influences implementation; not all initiatives will be successful and experimentation is vital. Successful initiatives included HiAP influencing the urban planning department to be more concerned with the health impacts of planning decisions, and encouraging the environment department to be concerned with the health impacts of its work.

Conclusions

The theory-based program logic suggests that SA HiAP facilitated improved population health through working with multiple government departments. Public servants came to appreciate how their sectors impact on health. Program logic is a mechanism to evaluate complex public health interventions in a way that takes account of political and economic contexts. SA HiAP was mainly successful in avoiding lifestyle drift in strategy. The initiative encouraged a range of state government departments to tackle conditions of daily living. The broader underpinning factors dictating the distribution of power, money and resources were not addressed by HiAP. This reflects HiAP’s use of a consensus model which was driven by (rather than drove) state priorities and sought ‘win-win’ strategies.

Reducing Health Inequities Through Intersectoral Action: Balancing Equity in Health With Equity for Other Social Goods

Significant attention has been devoted to developing intersectoral strategies to reduce health inequities; however, these strategies have largely neglected to consider how equity in health ought to be weighted and balanced with the pursuit of equity for other social goods (eg, education equity). Research in this domain is crucial, as the health sector’s pursuit of health equity may be at odds with policies in other sectors, which may consider the reduction of health inequities to be peripheral to, if not incompatible with, their own equity-related aims. It is therefore critical that intersectoral strategies to reduce health inequities be guided by a more general account of social justice that is capable of carefully balancing equity in health against the pursuit of equity in other sectors.

Researching Healthy Public Policy: Navigating the 'Black Box' Means Thinking More About Power Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

Lawless et al provide a valuable narrative of using program logic to develop an evaluation of Health in All Policies (HiAP) in South Australia. In this commentary I argue that the paper and analysis is an extremely useful example of navigating the supposed black box of policy-making. However the original makes the reader work too hard and is distracting from the main narrative of explaining the logic behind the HiAP approach in South Australia. My response covers avoiding epistemological traps and weighing up the pragmatics of collaborative policy research with more complex institutional policy issues like power.

Policy, Theory, and Evaluation: Stop Mixing the Fruit Salad Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

The study of Health in All Policies (HiAP) is gaining momentum. Authors are increasingly turning to wide swathes of political and social theory to frame (Program) Theory Based (or Informed) Evaluation (TBE) approaches. TBE for HiAP is not only prudent, it adds a level of elegance and insight to the research toolbox. However, it is still necessary to organize theoretical thinking appropriately. A commentary on a recent Int J Health Policy Manag paper argued that the framing of context and causality were hard to establish. This paper argues that this is not the most pressing issue. Rather, it claims we need to go back to basics to establish an appropriate HiAP evaluation paradigm. Such a basic paradigm would hinge on an understanding of power.

Evaluating Health in All Policies Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

Health in All Policies (HiAP) has gained attention as a potential tool to address complex health and societal challenges at global, regional, national and subnational levels. In a recent article, Lawless et al propose an evaluation framework developed in the context of the South Australia HiAP initiative. Strategies, mediators, activities and impacts identified in the framework could potentially be useful for evaluating HiAP in other settings. Creating and sustaining political will, managing conflicts of interest and achieving financially, politically and conceptually sustainable HiAP initiatives are challenges that could be further strengthened in the current framework.

Ideas for Extending the Approach to Evaluating Health in All Policies in South Australia Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

Since 2008, the government of South Australia has been using a Health in All Policies (HiAP) approach to achieve their strategic plan (South Australia Strategic Plan of 2004). In this commentary, we summarize some of the strengths and contributions of the innovative evaluation framework that was developed by an embedded team of academic researchers. To inform how the use of HiAP is evaluated more generally, we also describe several ideas for extending their approach, including: deeper integration of interdisciplinary theory (eg, public health sciences, policy and political sciences) to make use of existing knowledge and ideas about how and why HiAP works; including a focus on implementation outcomes and using developmental evaluation (DE) partnerships to strengthen the use of HiAP over time; use of systems theory to help understand the complexity of social systems and changing contexts involved in using HiAP; integrating economic considerations into HiAP evaluations to better understand the health, social and economic benefits and trade-offs of using HiAP.

How Do We Evaluate Health in All Policies? Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

It is well-established that population health is influenced by a multitude of factors, many of which lie outside the scope of the health sector. In the public health literature it is often assumed that intersectoral engagement with nonhealth sectors will be instrumental in addressing these social determinants of health. Due to the expected desirable outcomes in population health, several countries have introduced Health in All Policies (HiAP). However, whether this systematic, top-down approach to whole-of-government action (which HiAP entails) is efficient in changing government policies remains unclear. A systematic evaluation of HiAP is therefore much needed. Lawless and colleagues present an evaluation framework for HiAP in their article: "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia." This work is an important endeavor in addressing this problem (of uncertainty as to whether HiAP is effective) and represents an essential contribution to the HiAP literature. Nonetheless, in the spirit of encouraging ongoing reflection on this topic, we wish to highlight some challenges in the presented framework, which may pose difficulties in operationalization. We find that the evaluation framework faces two main limitations: its unclear causal logic and its level of complexity. We argue that in order to function as a tool for evaluation, the framework should be explicit about the mechanisms of change and enable us to trace whether the assumed causal relations resulted in changes in practice. Developing manageable evaluation frameworks, albeit simplified, may then be an important part of cumulating the theoretical insights aspired in theory-based evaluation. On this basis, we highlight how HiAP processes and healthy public policies respectively involve different mechanisms, and thus argue that different program theories are needed.

From Mid-Level Policy Analysis to Macro-Level Political Economy Comment on "Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia"

This latest contribution by the evaluation research team at Flinders University/Southgate Institute on their multiyear study of South Australia’s Health in All Policies (HiAP) initiative is simultaneously frustrating, exemplary, and partial. It is frustrating because it does not yet reveal the extent to which the initiative achieved its stated outcomes; that awaits further papers. It is exemplary in describing an evaluation research design in which the research team has excelled over the years, and in adding to it an element of theory testing and re-testing. It is partial, in that the political and economic context considered important in examining both process and outcome of the HiAP initiative stops at the Australian state’s borders as if the macro-level national and global political economy (and its power relations) have little or no bearing on the sustainability of the policy learning that the initiative may have engendered. To ask that of an otherwise elegant study design that effectively engages policy actors in its implementation may be demanding too much; but it may now be time that more critical political economy theories join with those that elaborate well the more routine praxis of public policy-making.

Health in All Policies in South Australia-Did It Promote and Enact an Equity Perspective?

Mobilising cross-sectoral action is helpful in addressing the range of social determinants that contribute to health inequities. The South Australian Health in All Policies (SA HiAP) approach was implemented from 2007 to stimulate cross-sector policy activity to address the social determinants of health to improve population wellbeing and reduce health inequities. This paper presents selected findings from a five year multi-methods research study of the SA HiAP approach and draws on data collected during interviews, observation, case studies, and document analysis. The analysis shows that SA HiAP had dual goals of facilitating joined-up government for co-benefits (process focus); and addressing social determinants of health and inequities through cross-sectoral policy activity (outcomes focus). Government agencies readily understood HiAP as providing tools for improving the process of intersectoral policy development, while the more distal outcome-focused intent of improving equity was not well understood and gained less traction. While some early rhetorical support existed for progressing an equity agenda through SA HiAP, subsequent economic pressures resulted in the government narrowing its priorities to economic goals. The paper concludes that SA HiAP’s initial intentions to address equity were only partially enacted and little was done to reduce inequities. Emerging opportunities in SA, and internationally, including the UN Sustainable Development Goals, may revive interest in addressing equity.