Barriers and Enablers to Evaluating Outcomes From Public Involvement in Health Service Design: An Interpretive Description

While health services are expected to have public involvement in service (re)design, there is a dearth of evaluation of outcomes to inform policy and practice. There are major gaps in understanding why outcome evaluation is under-utilised. The aims of this interpretive descriptive study were to explore researcher participants' experiences of and/or attitudes towards evaluating health service outcomes from public involvement in health service design in high-income countries. Additionally, the aims were to explore barriers and enablers of evaluation, and reasons for the use of evaluation tools or frameworks. Semi-structured interviews (n = 13) were conducted with researchers of published studies where the public was involved in designing health services. Using framework analysis, four themes were developed that captured participants' experiences: Public involvement is hard - evaluation is harder; power, a diversity of agendas, and the invisible public; practical and methodological challenges; and genuineness and authenticity matter. Evaluation is driven by stakeholder requirements, including decision-makers, funding bodies, researchers, and academics, and evaluation tools are rarely used. The public is largely absent from the outcome evaluation agenda. There is a lack of commitment and clarity of purpose of public involvement and its evaluation. Outcome evaluation must be multi-layered and localised and reflect the purpose of public involvement, what constitutes success (and to whom), and use the most appropriate methods. Multi-level supports should include increased resources, such as funding, time, and expertise. Without improved evaluation, outcomes of investment in public involvement in health service design/redesign remain unknown.

Voices of society: the emergence of civil-society practices aiming to engage in the management of the COVID-19 pandemic in the Netherlands

During the COVID-19 pandemic, public groups mobilised themselves in civil-society engagement practices (CSEPs) aiming to improve or suggest alternative epidemic management. This study explores the motivation to establish CSEPs and their perceived contributions to epidemic management, to gain insight whether integrating views of CSEPs could add value. A systematic online search was executed to identify CSEPs focused on COVID-19 management between January 2020 and January 2022 in the Netherlands. In order to create a comprehensible overview of the identified CSEPs, relevant characteristics were gathered and mapped, for example, local or national scope, subject of action and goals. A selection of CSEPs was interviewed between April and June 2022 to study their motivators to start the CSEPs and perceived contributions to management. The search resulted in the identification of 22 CSEPs, of which members of 14 CSEPs were interviewed. These members indicated several issues that motivated the start of their CSEP, namely; shortage of equipment, sense of solidarity, and a perceived lack of governmental action, lack of democratic values and lack in diversity of perspectives in epidemic management. All respondents believed to have contributed to policy or society, by influencing opinions, and occasionally by altering policy. However, respondents encountered obstacles in their attempts to contribute such as inability to establish contact with authorities, feeling unheard or undermined, and complications due to the interplay of political interests. In conclusion, CSEPs have fulfilled various roles such as providing alternative management policies, producing equipment, representing the needs of vulnerable populations, and supporting citizens and providing citizens with other viewpoints and information. The identified motivators to establish CSEPs in this study uncover room for improvements in policy. These insights, together with the identified perceived barriers of CSEPs, can be used to improve the connection between (future) epidemic management and public priorities and interests.

Consumers' and health providers' views and perceptions of partnering to improve health services design, delivery and evaluation: a co-produced qualitative evidence synthesis

BACKGROUND

Partnering with consumers in the planning, delivery and evaluation of health services is an essential component of person-centred care. There are many ways to partner with consumers to improve health services, including formal group partnerships (such as committees, boards or steering groups). However, consumers' and health providers' views and experiences of formal group partnerships remain unclear. In this qualitative evidence synthesis (QES), we focus specifically on formal group partnerships where health providers and consumers share decision-making about planning, delivering and/or evaluating health services. Formal group partnerships were selected because they are widely used throughout the world to improve person-centred care. For the purposes of this QES, the term 'consumer' refers to a person who is a patient, carer or community member who brings their perspective to health service partnerships. 'Health provider' refers to a person with a health policy, management, administrative or clinical role who participates in formal partnerships in an advisory or representative capacity. This QES was co-produced with a Stakeholder Panel of consumers and health providers. The QES was undertaken concurrently with a Cochrane intervention review entitled Effects of consumers and health providers working in partnership on health services planning, delivery and evaluation.

OBJECTIVES

1. To synthesise the views and experiences of consumers and health providers of formal partnership approaches that aimed to improve planning, delivery or evaluation of health services. 2. To identify best practice principles for formal partnership approaches in health services by understanding consumers' and health providers' views and experiences.

SEARCH METHODS

We searched MEDLINE, Embase, PsycINFO and CINAHL for studies published between January 2000 and October 2018. We also searched grey literature sources including websites of relevant research and policy organisations involved in promoting person-centred care.

SELECTION CRITERIA

We included qualitative studies that explored consumers' and health providers' perceptions and experiences of partnering in formal group formats to improve the planning, delivery or evaluation of health services.

DATA COLLECTION AND ANALYSIS

Following completion of abstract and full-text screening, we used purposive sampling to select a sample of eligible studies that covered a range of pre-defined criteria, including rich data, range of countries and country income level, settings, participants, and types of partnership activities. A Framework Synthesis approach was used to synthesise the findings of the sample. We appraised the quality of each study using the CASP (Critical Appraisal Skill Program) tool. We assessed our confidence in the findings using the GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research) approach. The Stakeholder Panel was involved in each stage of the review from development of the protocol to development of the best practice principles.

MAIN RESULTS

We found 182 studies that were eligible for inclusion. From this group, we selected 33 studies to include in the final synthesis. These studies came from a wide range of countries including 28 from high-income countries and five from low- or middle-income countries (LMICs). Each of the studies included the experiences and views of consumers and/or health providers of partnering in formal group formats. The results were divided into the following categories. Contextual factors influencing partnerships: government policy, policy implementation processes and funding, as well as the organisational context of the health service, could facilitate or impede partnering (moderate level of confidence). Consumer recruitment: consumer recruitment occurred in different ways and consumers managed the recruitment process in a minority of studies only (high level of confidence). Recruiting a range of consumers who were reflective of the clinic's demographic population was considered desirable, particularly by health providers (high level of confidence). Some health providers perceived that individual consumers' experiences were not generalisable to the broader population whereas consumers perceived it could be problematic to aim to represent a broad range of community views (high level of confidence). Partnership dynamics and processes: positive interpersonal dynamics between health providers and consumers facilitated partnerships (high level of confidence). However, formal meeting formats and lack of clarity about the consumer role could constrain consumers' involvement (high level of confidence). Health providers' professional status, technical knowledge and use of jargon were intimidating for some consumers (high level of confidence) and consumers could feel their experiential knowledge was not valued (moderate level of confidence). Consumers could also become frustrated when health providers dominated the meeting agenda (moderate level of confidence) and when they experienced token involvement, such as a lack of decision-making power (high level of confidence) Perceived impacts on partnership participants: partnering could affect health provider and consumer participants in both positive and negative ways (high level of confidence). Perceived impacts on health service planning, delivery and evaluation: partnering was perceived to improve the person-centredness of health service culture (high level of confidence), improve the built environment of the health service (high level of confidence), improve health service design and delivery e.g. facilitate 'out of hours' services or treatment closer to home (high level of confidence), enhance community ownership of health services, particularly in LMICs (moderate level of confidence), and improve consumer involvement in strategic decision-making, under certain conditions (moderate level of confidence). There was limited evidence suggesting partnering may improve health service evaluation (very low level of confidence). Best practice principles for formal partnering to promote person-centred care were developed from these findings. The principles were developed collaboratively with the Stakeholder Panel and included leadership and health service culture; diversity; equity; mutual respect; shared vision and regular communication; shared agendas and decision-making; influence and sustainability.

AUTHORS' CONCLUSIONS

Successful formal group partnerships with consumers require health providers to continually reflect and address power imbalances that may constrain consumers' participation. Such imbalances may be particularly acute in recruitment procedures, meeting structure and content and decision-making processes. Formal group partnerships were perceived to improve the physical environment of health services, the person-centredness of health service culture and health service design and delivery. Implementing the best practice principles may help to address power imbalances, strengthen formal partnering, improve the experiences of consumers and health providers and positively affect partnership outcomes.

Community input in a genomic health implementation program: Perspectives of a community advisory group

Consumer and community engagement (CCE) in the implementation of genomics into health services and associated research is needed to ensure that changes benefit the affected patients. Queensland Genomics was a program to implement genomics into a public health service. We describe its Community Advisory Group’s (CAG) structure and function and provide recommendations based on the CAG members’ perspectives. The CAG provided advice to the Queensland Genomics program and its projects in an advisory capacity. The CAG was also resourced to develop and lead community-focused activities. Key enablers for CAG included; diversity of CAG members’ skills and experience, adequate resourcing, and the CAG’s ability to self-determine their direction. The CAG experienced limitations due to a lack of mechanisms to implement CCE in the Program’s projects. Here, we provide insights and commentary on this CAG, which will be useful for other initiatives seeking to undertake CCE in genomic research and health care.

Medical Students’ Perception Regarding Health Policy Teaching and Their Participation in Health Policy Roles: A Survey at a Public University in Malaysia

Background: Health policy is a set of comprehensive principles and legislations that guide how healthcare should be effectively delivered in the community. Medical schools should prepare students to undertake managerial responsibilities by incorporating health policy into the curriculum to deal with the intricacies of healthcare systems and their clinical roles in their future professional careers. Objective: To examine medical students’ perception at a Public University in Malaysia regarding teaching health policy and their participation in health policy roles. Material and Methods: A cross-sectional study using universal sampling was carried out among the medical students using a paper-based questionnaire to collect the data. Results: Most respondents opined their willingness to learn health policy (80.9%) and that teaching health policy (83.6%) should be compulsory for medical students. The respondents thought health policy should be introduced earlier in Year 1 or 2. The student scores on their knowledge regarding health policy and year of study were significantly associated with their involvement in the health policy roles in both the simple and multiple logistic regression. Both statistical tests reported higher participation in health policy roles with the higher year of study, though only Year 4 and 5 were significant in the simple logistic regression and only Year 5 in the multiple logistic regression compared to Year 1. On the other hand, age and type of admission show significant results only in the simple logistic regression, while the race was only significant at the multivariate level. Conclusions: This study demonstrated that most respondents showed their willingness to learn health policy, participate in the health policy programs, and recommend that health policy be considered an essential topic in the medical curriculum, which should be taught right from the first year of medical school. We recommend encouraging students’ participation in health policy activities.

Engaging citizens in local health policymaking. A realist explorative case-study

Background

Municipalities have been trying to involve citizens as citizen participation is thought to improve municipalities’ accountability, the quality of services, and to align policies and services to communities’ needs. This study examined citizens’ participation preferences in policymaking by investigating their health policy priorities, expectations of involvement, and required support.

Methods

For this case-study the realist evaluation approach was applied to focus groups with citizens and to a workshop with a local panel consisting of professionals, citizens and citizen representatives.

Results

This study showed that citizens want to be involved in (health) policymaking with the aim of improving their communities’ quality of life and living environment and prioritised local services and amenities (e.g. suitable housing, public transport, health and care services). Instead, professionals’ priorities were focussed on singular public health issues related to prevention and lifestyle factors. The results also show that citizens felt responsible for driving citizen participation and representing community needs to the municipality, but needed the municipality to improve their communication and accessibility in order to do so successfully. Furthermore, the professionals on the panel indicated that they needed training on how to reach out to citizens. Such training should highlight how to better align their language to citizens’ lived experiences. They also wanted their organisations to provide more space, flexibility and resources to build relationships with citizens in order to provide improved communication and accessibility to citizens.

Conclusion

The difference in priorities between citizens and professionals highlights the importance of involving citizens in policymaking. Moreover, citizens’ involvement can act as a lever for change to bring a wider range of services and policy sectors together and has the potential to better align policies to citizens’ lived experiences and hopefully increase the democratic legitimacy of policymaking. However, to fulfil such potential municipalities will need to invest in improving their accessibility and communication with communities.

Can We Afford to Exclude Patients Throughout Health Technology Assessment?

Health technology assessment (HTA) is intended to determine the value of health technologies and, once a technology is recommended for funding, bridge clinical research and practice. Understanding the values and beliefs expressed by patients and health professionals can help guide this knowledge transfer and work toward managing the expectations of end users. We gathered patient and patient group leader experiences to gain insights into the roles that patients and patient advocacy groups are playing. We argue that through partnerships and co-creation between HTA professionals, researchers and patient advocates we can strengthen the HTA process and better align with service delivery where person-centered care and shared decision making are key elements. Patient experiences and knowledge are important to the democratization of evidence and the legitimacy of HTAs. Patient preference studies are used to balance benefits with potential harms of technologies, and patient-reported outcomes (PROs) can measure what matters to patients over time. A change in culture in HTA bodies is occurring and with further transformative thinking patients can be involved in every step of the HTA process. Patients have a right to be involved in HTAs, with patients' values central to HTA deliberations on a technology and where patients can provide valuable insights to inform HTA decision-making; and in ensuring that HTA methodologies evolve. By evaluating the implementation of HTA recommendations we can determine how HTA benefits patients and their communities. Our shared commitment can positively effect the common good and provide benefits to individual patients and their communities.

Making community voices heard in a research-health service alliance, the evolving role of the Community Advisory Group: a case study from the members' perspective

BACKGROUND

The Melbourne Genomics Health Alliance (the Alliance) is a collaboration of leading hospitals, research and academic organisations, supported by its member organisations and the Victorian Government. The Alliance was set up by its members in 2013 to steer the translation of genomics, making it an integral part of health care in Victoria, Australia. The Community Advisory Group (CAG) was formed soon after, to give input and advice across the program. This was to ensure consideration of community values, perspectives and priorities, and knowledge translation for patient care. The CAG was charged with providing a strong community voice for the duration of the program. Appointed members were experienced consumer advocates with developed connections to the community.

MAIN BODY

The Alliance progressed from an initial Demonstration Project (2013-2015) to a multifaceted program (2016-2020). The CAG worked strategically to help address complex issues, for example, communication, privacy, informed consent, ethics, patient experience, measurement and evaluation standards and policies, data storage and re-use of genomic data. Many aspects of translating genomics into routine care have been tackled, such as communicating with patients invited to have genomic testing, or their caregivers, and obtaining informed consent, clinical questions across 16 areas of health care, training and education of health and laboratory professionals, genomic data management and data-sharing. Evidence generated around clinical utility and cost-effectiveness led to government funding of testing for complex genetic conditions in children.

CONCLUSION

The CAG activities, recorded in a CAG-inspired Activity register, span the full spectrum of information sharing and consultation to co-design and partnership. The CAG were involved at multiple levels of participation and in all tiers of activity including governance, development of policies and procedures, program planning and evaluation. Working relationships were built up and a level of trust instilled to advance the Alliance work program in ensuring an effective patient-care model of delivery of genomics. CAG input into project deliverables has been tangible. Less tangible contributions included presentations at external meetings and conferences, direct interactions at meetings with Alliance members, interactions with visitors and external experts, taking part in consultations with experts, state and federal government.

Framing the nanny (state): an analysis of public submissions to a parliamentary inquiry on personal choice and community safety

OBJECTIVE

To examine public submissions to a parliamentary inquiry on personal choice and community safety, exploring framing used to support or oppose current public health regulatory approaches.

METHODS

Descriptive content analysis summarised the characteristics of electronic submissions. Framing analysis examined submissions according to the devices: problem and causes; principles and values; recommendations; data and evidence; and salience.

RESULTS

We categorised one hundred and five (n=105) submissions by source as Individual, Industry, Public Health and Other. Individuals made more than half the submissions. Overarching frames were choice and rights (Individuals); progress and freedom (Industry); protection and responsibility (Public Health). Most submissions opposed current regulations. Cycling, including mandatory helmet legislation, was most cited, with three-quarters of submissions opposing current legislation.

CONCLUSIONS

Framing analysis provided insights into policy actor agendas concerning government regulation. We found a high degree of resistance to public health regulation that curtails individual autonomy across various health issues. Investigating the influence of different frames on community perception of public health regulation is warranted. Implications for public health: Action is required to counteract 'nanny state' framing by industry and to problematise community understanding of the 'nanny state' in the context of balancing the public's liberties and the public's health.

Themes for evaluating the quality of initiatives to engage patients and family caregivers in decision-making in healthcare systems: a scoping review

OBJECTIVE

To identify the key themes for evaluating the quality of initiatives to engage patients and family caregivers in decision-making across the organisation and system domains of healthcare systems.

METHODS

We conducted a scoping review. Seven databases of journal articles were searched from their inception to June 2019. Eligible articles were literature reviews published in English and provided useful information for determining aspects of engaging patients and family caregivers in decision-making to evaluate. We extracted text under three predetermined categories: structure, process and outcomes that were adapted from the Donabedian conceptual framework. These excerpts were then independently open-coded among four researchers. The subsequent themes and their corresponding excerpts were summarised to provide a rich description of each theme.

RESULTS

Of 7747 unique articles identified, 366 were potentially relevant, from which we selected the 42 literature reviews. 18 unique themes were identified across the three predetermined categories. There were six structure themes: engagement plan, level of engagement, time and timing of engagement, format and composition, commitment to support and environment. There were four process themes: objectives, engagement approach, communication and engagement activities. There were eight outcome themes: decision-making process, stakeholder relationship, capacity development, stakeholder experience, shape policy/service/programme, health status, healthcare quality, and cost-effectiveness.

CONCLUSIONS

The 18 themes and their descriptions provide a foundation for identifying constructs and selecting measures to evaluate the quality of initiatives for engaging patients and family caregivers in healthcare system decision-making within the organisation and system domains. The themes can be used to investigate the mechanisms through which relevant initiatives are effective and investigate their effectiveness.

Public and patient involvement (PPI) in health policy decisionmaking on the health system-level: protocol for a systematic scoping review

INTRODUCTION

Public and patient involvement (PPI) in healthcare decisions at the health system-level (macro-level) has become increasingly important during recent years. Existing evidence indicates that PPI increase patient centredness and the democracy of healthcare decisions as well as patients' trust and acceptance of these decisions. However, different methods for PPI exist, and an overview of the outcomes and influential contextual factors has not yet been conducted. Therefore, this scoping review aims to provide an overview of the different methods used for PPI in health system decisionmaking and the reported outcomes and contextual factors for these methods.

METHODS AND ANALYSIS

The structure of this protocol is guided by the advanced scoping studies framework of Arksey and O' Malley, developed by Levac, Colquhoun and O'Brien, and the PRISMA-ScR Statement. We will systematically search electronic databases (MEDLINE, Cochrane Library, Scopus, CINAHL, PDQ-Evidence, Web of Science and PsycINFO) for peer-reviewed literature and screen the reference lists of included studies. Additionally, we will search for relevant grey literature and consult experts from the field to identify further information. Studies focusing on PPI in the context of health policy decision-making at the macro-level will be eligible for full-text screening. Studies focusing on decisions at the individual treatment-level (micro-level) and the organisational-level (mesolevel) as well as those dealing with PPI in health research will be excluded. A qualitative analysis will dissect how the included studies define PPI and its desirable outcomes, the achieved outcomes and reported contextual factors.

ETHICS AND DISSEMINATION

We will present the results at relevant conferences and in an open-access journal. Additionally, we will share them with the experts involved in the research process and consider ways in which to transfer the findings into practice. As only secondary and previously published information will be used, ethical approval is not necessary.

Environmental impact assessment of cement industries using mathematical matrix method: case of Ghayen cement, South Khorasan, Iran

Cement industries produce millions of tons of by-products each year, including dust, toxic gases, and heavy metals, which pose respiratory health hazards and environmental pollution. Although several literatures have investigated environmental impact in cement factories, the extent to which these assessments affect the population has received limited responsiveness. This study investigates this gap by examining the environmental impacts of the Ghayen Cement Industry on environmental components and public health. Using a mathematical matrix, it found that the effect of the cement factory on the environment produced four categories of significant effects: L, low; M, medium; H, high; and VH, very high. Results in terms of basic criteria revealed 0.70 of the most destructive effect was related to water pollution. Regarding the interaction effect of project activities (basic and complimentary basic criteria) on environmental components, the highest destructive effect was excavation and land deformation while the lowest effect was on public health. The study also found out that the use of compensation factors reduced the negative effect of the project on the environment. The use of common methods of environmental improvement such as erosion control, proper location of facilities, and the creation of green spaces is essential to mitigate the impact of projects on the environment.

Creating a patient and community advisory committee at the Canadian Agency for Drugs and Technologies in Health

In recognition of patients' roles using, and contributing to, a publicly funded health system, the Canadian Agency for Drugs and Technologies in Health (CADTH) created a Patient and Community Advisory Committee. Twelve members bring lived experiences of chronic illness, progressive illness, mental illness, trauma, traveling long distances for treatment, and caregiving to an ill child, parent, or spouse. Members contribute their own insights and ideas but do not represent specific organizations or viewpoints. This paper explores how CADTH determined the committee's role, whether to have individuals or organizations as members, and how to recruit for diversity. The creation of this committee is changing how CADTH engages with patients.

Evaluation of a digital platform that engages stakeholders in the co-creation of healthcare innovations: A mixed-methods study

Introduction

Engaging public and patients in the decision-making processes is on the agenda of many healthcare systems towards sustainable healthcare delivery. While many engagement initiatives are performed face-to-face, an increasing number is conducted online. An example of the latter is the Dutch digital platform Gezonde Mening that engages patients, healthcare professionals and other stakeholders in the co-creation of healthcare innovations. This study aimed to evaluate the effectiveness of stakeholder engagement performed on Gezonde Mening by focusing on the process of planning, execution and transition of engagement activities.

Methods

A mixed-methods study was performed by conducting eight semi-structured interviews with developers and funders of Gezonde Mening and an assessment of the psychometric properties of two questionnaires administrated via Gezonde Mening to seek stakeholders’ inputs. While the interviews were analysed deductively and inductively, data from the assessment of psychometric properties were analysed in a descriptive quantitative manner. Data were interpreted through triangulation.

Results

Assessment of the planning of stakeholder engagement identified needs for having more stakeholders on the platform to enable subgroup analysis and robust insights. Moreover, questionnaires administered by Gezonde Mening showed low validity and reliability. Assessment of the execution of stakeholder engagement indicated that stakeholders are sufficiently informed about engagement. Assessment of the transition of engagement activities showed needs to provide direct results to stakeholders and allow their evaluation of the platform.

Conclusion

Gezonde Mening digital platform facilitates communication between innovators and stakeholders during engagement activities. However, the study identified room for improvement regarding the planning and transition activities.