Monitoring and Evaluation of Patient Engagement in Health Product Research and Development: Co-Creating a Framework for Community Advisory Boards

PURPOSE

While patient engagement is becoming more customary in developing health products, its monitoring and evaluation to understand processes and enhance impact are challenging. This article describes a patient engagement monitoring and evaluation (PEME) framework, co-created and tailored to the context of community advisory boards (CABs) for rare diseases in Europe. It can be used to stimulate learning and evaluate impacts of engagement activities.

METHODS

A participatory approach was used in which data collection and analysis were iterative. The process was based on the principles of interactive learning and action and guided by the PEME framework. Data were collected via document analysis, reflection sessions, a questionnaire, and a workshop.

RESULTS

The tailored framework consists of a theory of change model with metrics explaining how CABs can reach their objectives. Of 61 identified metrics, 17 metrics for monitoring the patient engagement process and short-term outcomes were selected, and a "menu" for evaluating long-term impacts was created. Example metrics include "Industry representatives' understanding of patients' unmet needs;" "Feeling of trust between stakeholders;" and "Feeling of preparedness." "Alignment of research programs with patients' needs" was the highest-ranked metric for long-term impact.

CONCLUSIONS

Findings suggest that process and short-term outcome metrics could be standardized across CABs, whereas long-term impact metrics may need to be tailored to the collaboration from a proposed menu. Accordingly, we recommend that others adapt and refine the PEME framework as appropriate. The next steps include implementing and testing the evaluation framework to stimulate learning and share impacts.

Evaluation of patient engagement in medicine development: A multi-stakeholder framework with metrics

BACKGROUND

Patient engagement is becoming more customary in medicine development. However, embedding it in organizational decision-making remains challenging, partly due to lack of agreement on its value and the means to evaluate it. The objective of this project was to develop a monitoring and evaluation framework, with metrics, to demonstrate impact and enhance learning.

METHODS

A consortium of five patient groups, 15 biopharmaceutical companies and two academic groups iteratively created a framework in a multi-phase participatory process, including analysis of its application in 24 cases.

RESULTS

The framework includes six components, with 87 metrics and 15 context factors distributed among (sub)components: (a) Input: expectations, preparations, resources, representativeness of stakeholders; (b) Activities/process: structure, management, interactions, satisfaction; (c) Learnings and changes; (d) Impacts: research relevance, study ethics and inclusiveness, study quality and efficiency, quality of evidence and uptake of products, empowerment, reputation and trust, embedding of patient engagement; (e) Context: policy, institutional, community, decision-making contextual factors. Case study findings show a wide variation in use of metrics. There is no 'one size fits all' set of metrics appropriate for every initiative or organization. Presented sample sets of metrics can be tailored to individual situations.

CONCLUSION

Introducing change into any process is best done when the value of that change is clear. This framework allows participants to select what metrics they value and assess to what extent patient engagement has contributed.

PATIENT CONTRIBUTION

Five patient groups were involved in all phases of the study (design, conduct, interpretation of data) and in writing the manuscript.

Giving patients a voice: a participatory evaluation of patient engagement in Newfoundland and Labrador Health Research

BACKGROUND

Efforts to engage patients as partners in health research have grown and thereby the need for feedback and evaluation. In this pilot evaluation study, we aimed to 1) evaluate patient engagement in health research projects in Newfoundland and Labrador, Canada, and 2) learn more about how to best monitor and evaluate patient engagement. This paper presents the results of our participatory evaluation study and the lessons learned. The evaluation of the projects was driven by questions patients wanted answered.

METHODS

We conducted a formative evaluation of patient engagement in health research projects. Projects spanned a variety of topics, target groups, research designs and methods of patient engagement. Participants included principal investigators (n = 6) and their patient partners (n = 14). Furthermore, graduate students (n = 13) working on their own research projects participated. Participants completed an online survey with closed and open-ended questions about their patient engagement efforts, experiences and preliminary outcomes. Patients were involved as co-investigators in the entire evaluation study. We used qualitative methods to evaluate our participatory process.

RESULTS

The evaluation study results show that most patients and researchers felt prepared and worked together in various phases of the research process. Both groups felt that the insights and comments of patients influenced research decisions. They believed that patient engagement improved the quality and uptake of research. Students felt less prepared and were less satisfied with their patient engagement experience compared to researchers and their patient partners. Involvement of patient co-investigators in this evaluation resulted in learnings, transparency, validation of findings and increased applicability. Challenges were to select evaluation questions relevant to all stakeholders and to adapt evaluation tools to local needs.

CONCLUSIONS

Our findings show that researchers, patient partners and students value patient engagement in health research. Capacity building at the supervisor level in academic institutions is needed to better support students. Sufficient time is also needed to permit observable outcomes. Participatory evaluation may increase the relevance and usefulness of information, but it also raises issues such as who defines and designs the content of evaluation tools. A co-creation process is required to develop appropriate monitoring and evaluation strategies.

Evaluating the "return on patient engagement initiatives" in medicines research and development: A literature review

Background

Showing how engagement adds value for all stakeholders can be an effective motivator for broader implementation of patient engagement. However, it is unclear what methods can best be used to evaluate patient engagement. This paper is focused on ways to evaluate patient engagement at three decision‐making points in the medicines research and development process: research priority setting, clinical trial design and early dialogues with regulators and health technology assessment bodies.

Objective

Our aim was to review the literature on monitoring and evaluation of patient engagement, with a focus on indicators and methods.

Search strategy and inclusion criteria

We undertook a scoping literature review using a systematic search, including academic and grey literature with a focus on evaluation approaches or outcomes associated with patient engagement. No date limits were applied other than a cut‐off of publications after July 2018.

Data extraction and synthesis

Data were extracted from 91 publications, coded and thematically analysed.

Main results

A total of 18 benefits and 5 costs of patient engagement were identified, mapped with 28 possible indicators for their evaluation. Several quantitative and qualitative methods were found for the evaluation of benefits and costs of patient engagement.

Discussion and conclusions

Currently available indicators and methods are of some use in measuring impact but are not sufficient to understand the pathway to impact, nor whether interaction between researchers and patients leads to change. We suggest that the impacts of patient engagement can best be determined not by applying single indicators, but a coherent set of measures.

Co-building a patient-oriented research curriculum in Canada

PLAIN ENGLISH SUMMARY

Foundations in Patient-Oriented Research is a course designed and piloted in Canada to help patients, researchers, health care professionals and health system decision-makers gain an introductory understanding of patient-oriented research, the research enterprise, and how to work in a team. The course curriculum was co-developed by a diverse group of people with different lived experiences and relevant expertise. The course is meant to be delivered in a 'co-learning format' with classes comprised of all the above stakeholder groups learning together. The purpose of this study was to explore the experiences of the project leaders, developers, facilitators and patient co-facilitators who were involved in the process of co-developing, piloting and revising the curriculum.Our findings suggest that co-developing a patient-oriented research curriculum increases its quality, uptake and credibility. The co-development process not only resulted in training that benefited the target learners, but it provided valuable learning experiences about patient-oriented research for the project leaders, developers, facilitators and patient co-facilitators. These findings and the resulting recommendations may provide guidance for other learning and development groups wishing to undertake a similar project.

ABSTRACT

Background Foundations in Patient-Oriented Research is a course designed and piloted in Canada to build mutually beneficial relationships for conducting patient-oriented research by ensuring that relevant stakeholders - patients, researchers, health care professionals and health system decision-makers - have a common foundational understanding of patient-oriented research, the research enterprise, and team dynamics. The curriculum was co-developed by a group of patients, researchers, patient engagement experts and curriculum development experts and involved consultations with broader groups of the relevant stakeholders mentioned above. It was designed to be delivered in a 'co-learning format' with classes comprised of all stakeholder groups learning together. The purpose of this study was to explore the experiences of individuals involved in the process of co-developing, piloting and revising Foundations in Patient-Oriented Research. Methods An embedded case study was conducted with individuals who were involved in the co-development, pilot and revision of Foundations in Patient-Oriented Research. These individuals took on different roles during the curriculum development process, including project co-lead, developer, facilitator, and patient co-facilitator. The constant comparison method was used to inductively develop themes from the two focus group sessions. Results Discussions from the focus groups revealed the value of co-building the content, co-facilitating the course sessions, and the importance of the co-learning format. The training itself was perceived as valuable and the systematic approach to co-development was perceived as a success. Several barriers were identified, including the amount of resources, time and commitment required to complete the project. There was a notable tension between maintaining the integrity of the content and having the freedom to adapt it to local contexts. Over the course of the project, the project co-leads, developers and facilitators found that their own understanding of patient-oriented research deepened. Conclusions The findings of this study suggest that co-developing a patient-oriented research curriculum increases its quality, uptake and credibility. The co-development process not only resulted in training that benefited the target learners, but also built capacity for patient-oriented research within the project co-leads, developers, facilitators and patient co-facilitators. Our findings and recommendations may provide guidance for other learning and development groups wishing to undertake a similar project.

Engaging patients in health research: identifying research priorities through community town halls

Background

The vision of Canada’s Strategy for Patient-Oriented Research is that patients be actively engaged as partners in health research. Support units have been created across Canada to build capacity in patient-oriented research and facilitate its conduct. This study aimed to explore patients’ health research priorities in the province of Newfoundland and Labrador (NL).

Methods

Eight town halls were held with members of the general public in rural and urban settings across the province. Sessions were a hybrid information-consultation event, with key questions about health research priorities and outcomes guiding the discussion.

Results

Sixty eight members of the public attended town hall sessions. A broad range of health experiences in the healthcare system were recounted. Key priorities for the public included access and availability of providers and services, disease prevention and health promotion, and follow-up support and community care. In discussing their health research priorities, participants spontaneously raised a broad range of suggestions for improving the healthcare system in our jurisdiction.

Conclusions

Public research priorities and suggestions for improving the provision of healthcare provide valuable information to guide Support Units’ planning and priority-setting processes. A range of research areas were raised as priorities for patients that are likely comparable to other healthcare systems. These create a number of health research questions that would be in line with public priorities. Findings also provide lessons learned for others and add to the evidence base on patient engagement methods.

Recruiting patients as partners in health research: a qualitative descriptive study

PLAIN ENGLISH SUMMARY

Increasingly, funders and researchers want to partner with patients in health research, but it can be challenging for researchers to find patient partners. More than taking part in research as participants, patient partners help design, carry out and manage research projects. The goal of this study was to describe ways that patient partners have been recruited by researchers and patient engagement leads (individuals within organizations responsible for promoting and supporting patients as research partners). We talked with researchers and patient engagement leads in Canada and the United Kingdom, as well as a patient representative. We found three ways that could help researchers and patients find each other. One way is a case-by-case basis, where patients are often sought with experience of a health condition that is the focus of the research. The other ways involved directories where projects were posted and could be found by patients and researchers, or a third party matched patients with research projects. We found four recruitment strategies:Social marketingCommunity outreachHealth systemPartnering with other organizations (e.g., advocacy groups) There are many influences on finding, selecting and retaining patient partners: patient characteristics, the local setting, the opportunity, work climate, education and support. We hope study results will provide a useful starting point for research teams in recruiting their patient partners.

ABSTRACT

Background Patient engagement in clinical trials and other health research continues to gain momentum. While the benefits of patient engagement in research are emerging, relatively little is known about recruiting patients as research partners. The purpose of this study was to describe recruitment strategies and models of recruiting patients as partners in health research. Methods Qualitative descriptive study. Thirteen patient engagement leads and health researchers from Canada and the United Kingdom, as well as one patient representative from a national patient organization (7 female) completed semi-structured interviews. Results Recruitment infrastructures available to respondents varied, but could be categorized into three models including the traditional, third-party and directory models. Four categories of recruitment strategies were identified, representing multiple ways of recruiting patient partners: social marketing recruitment, community outreach recruitment, health system recruitment, and partnering recruitment. Conclusions Multiple recruitment strategies were identified for engaging patient partners in research, and some common factors influenced recruitment. Study findings contribute to the evidence base in patient engagement and provide guidance for research teams to help identify potential recruitment methods for their patient partners.