Stakeholder engagement in research: a scoping review of current evaluation methods

Stakeholder Engagement in Type 1 Diabetes Research, Quality Improvement, and Clinical Care

The integration of stakeholder engagement (SE) in research, quality improvement (QI), and clinical care has gained significant traction. Type 1 diabetes is a chronic disease that requires complex daily management and care from a multidisciplinary team across the lifespan. Inclusion of key stakeholder voices, including patients, caregivers, health care providers and community advocates, in the research process and implementation of clinical care is critical to ensure representation of perspectives that match the values and goals of the patient population. This review describes the current framework for SE and its application to research, QI, and clinical care across the lifespan.

Reporting of equity in observational epidemiology: A methodological review

Background

Observational studies can inform how we understand and address persisting health inequities through the collection, reporting and analysis of health equity factors. However, the extent to which the analysis and reporting of equity-relevant aspects in observational research are generally unknown. Thus, we aimed to systematically evaluate how equity-relevant observational studies reported equity considerations in the study design and analyses.

Methods

We searched MEDLINE for health equity-relevant observational studies from January 2020 to March 2022, resulting in 16 828 articles. We randomly selected 320 studies, ensuring a balance in focus on populations experiencing inequities, country income settings, and coronavirus disease 2019 (COVID-19) topic. We extracted information on study design and analysis methods.

Results

The bulk of the studies were conducted in North America (n = 95, 30%), followed by Europe and Central Asia (n = 55, 17%). Half of the studies (n = 171, 53%) addressed general health and well-being, while 49 (15%) focused on mental health conditions. Two-thirds of the studies (n = 220, 69%) were cross-sectional. Eight (3%) engaged with populations experiencing inequities, while 22 (29%) adapted recruitment methods to reach these populations. Further, 67 studies (21%) examined interaction effects primarily related to race or ethnicity (48%). Two-thirds of the studies (72%) adjusted for characteristics associated with inequities, and 18 studies (6%) used flow diagrams to depict how populations experiencing inequities progressed throughout the studies.

Conclusions

Despite over 80% of the equity-focused observational studies providing a rationale for a focus on health equity, reporting of study design features relevant to health equity ranged from 0-95%, with over half of the items reported by less than one-quarter of studies. This methodological study is a baseline assessment to inform the development of an equity-focussed reporting guideline for observational studies as an extension of the well-known Strengthening Reporting of Observational Studies in Epidemiology (STROBE) guideline.

[Orientations de facilitation pour soutenir l'application de l'Algo dans les services de soutien à domicile des personnes aînées]

Comment soutenir le déploiement de connaissances coconstruites par des personnes cliniciennes, gestionnaires ou chercheures? Ce thème est abordé à partir de l'étude de l'application de l'Algo, un algorithme clinique décisionnel conçu pour la sélection des aides techniques visant à faciliter l'hygiène corporelle des personnes aînées vivant à domicile. L'objectif de cette note sur les politiques et les pratiques est de présenter les orientations de facilitation dégagées à la suite d'un devis mixte multiphases (2015-2019) mis en œuvre dans les services de soutien à domicile au Québec (Canada). Les orientations de facilitation centrée sur la tâche et holistique sont présentées en fonction des stades d'utilisation de l'Algo, afin de soutenir les personnes cliniciennes, gestionnaires et chercheures dans la poursuite de son application auprès des personnes aînées. De plus, cette note illustre l'apport des devis mixtes à la conduite et à la compréhension de l'application des connaissances coconstruites.

Evaluation of a stakeholder advisory board for an adolescent mental health randomized clinical trial

INTRODUCTION

Community engagement in research is widely accepted as best practice, despite gaps in existing frameworks to evaluate its process, context, and impact on research. The Screening in High Schools to Identify, Evaluate, and Lower Depression (SHIELD) study evaluated the use of a school-based major depressive disorder screening tool in the identification of symptoms and treatment initiation among adolescents, and was developed, implemented, and disseminated in partnership with a Stakeholder Advisory Board (SAB). We summarize outcomes of the evaluation strategy applied through our partnership with the SAB and explore gaps in the available engagement evaluation tools for mixed stakeholder populations including youth.

METHODS

SHIELD study SAB members (n = 13; adolescents, parents, mental health and primary care providers, and professionals from education and mental health organizations) advised on study design, implementation, and dissemination over a three-year period. Both SAB members and study team members (i.e., clinician researchers, project managers) were invited to quantitatively and qualitatively evaluate stakeholder engagement after each project year. At the conclusion of the study, SAB members and study team members were asked to evaluate the application of engagement principles in overall stakeholder engagement across the study period, using portions of the Research Engagement Survey Tool (REST).

RESULTS

SAB members and study team members responded similarly when evaluating engagement process (i.e., valued on team, voice represented); means ranged from 3.9 to 4.8 out of 5 points across all three project years. Reported engagement within study-specific engagement activities (i.e., meetings, study newsletter) varied from year to year, with some discrepancy between SAB member and study team evaluations. Using REST, SAB members reported the alignment of their experience with key engagement principles the same or higher than study team members. Qualitative feedback at the conclusion of the study generally matched quantitative measures; adolescent SAB members, however, reported disengagement from stakeholder activities that was not accurately or effectively captured in evaluation strategies employed across the study period.

CONCLUSIONS

Challenges exist in effectively engaging stakeholders and evaluating their engagement, particularly among heterogenous groups that include youth. Evaluation gaps should be addressed through the development of validated instruments that quantify the process, context, and impact of stakeholder engagement on study outcomes. Consideration should be given to collecting parallel feedback from stakeholders and study team members to fully understand the application and execution of engagement strategy.

Promoting Authentic Academic-Community Engagement to Advance Health Equity

Meaningful community engagement is critical to achieving the lofty goal of health equity. Nonetheless, implementing the principles of community engagement is not easy. Attempting to implement best practices for collaborating on transdisciplinary teams and working with community partners can be challenging, particularly in locales that have a long history of strained university-community relationships. The purpose of this paper is to provide additional context and consideration for researchers, community partners, and institutions interested in conducting community-engaged research. Here, we provide guidance and highlight exemplary programs that offer effective approaches to enhance the strength of community partnerships. These partnerships not only hold promise but are also essential in the development of the local, multi-factor solutions required to address racial/ethnic inequities in health.

Implementing an Electronic Patient-Reported Outcome and Decision Support Tool in Early Intervention

OBJECTIVE

The aim of the study is to identify and prioritize early intervention (EI) stakeholders' perspectives of supports and barriers to implementing the Young Children's Participation and Environment Measure (YC-PEM), an electronic patient-reported outcome (e-PRO) tool, for scaling its implementation across multiple local and state EI programs.

METHODS

An explanatory sequential (quan > QUAL) mixed-methods study was conducted with EI families (n = 6), service coordinators (n = 9), and program leadership (n = 7). Semi-structured interviews and focus groups were used to share select quantitative pragmatic trial results (e.g., percentages for perceived helpfulness of implementation strategies) and elicit stakeholder perspectives to contextualize these results. Three study staff deductively coded transcripts to constructs in the Consolidated Framework for Implementation Research (CFIR). Data within CFIR constructs were inductively analyzed to generate themes that were rated by national early childhood advisors for their relevance to longer term implementation.

RESULTS

All three stakeholder groups (i.e., families, service coordinators, program leadership) identified thematic supports and barriers across multiple constructs within each of four CFIR domains: (1) Six themes for "intervention characteristics," (2) Six themes for "process," (3) three themes for "inner setting," and (4) four themes for "outer setting." For example, all stakeholder groups described the value of the YC-PEM e-PRO in forging connections and eliciting meaningful information about family priorities for efficient service plan development ("intervention characteristics"). Stakeholders prioritized reaching families with diverse linguistic preferences and user navigation needs, further tailoring its interface with automated data capture and exchange processes ("process"); and fostering a positive implementation climate ("inner setting"). Service coordinators and program leadership further articulated the value of YC-PEM e-PRO results for improving EI access ("outer setting").

CONCLUSION

Results demonstrate the YC-PEM e-PRO is an evidence-based intervention that is viable for implementation. Optimizations to its interface are needed before undertaking hybrid type-2 and 3 multisite trials to test these implementation strategies across state and local EI programs with electronic data capture capabilities and diverse levels of organizational readiness and resources for implementation.

A scoping review of methods to measure and evaluate citizen engagement in health research

BACKGROUND

Citizen engagement, or partnering with interested members of the public in health research, is becoming more common. While ongoing assessment of citizen engagement practices is considered important to its success, there is little clarity around aspects of citizen engagement that are important to assess (i.e., what to look for) and methods to assess (i.e., how to measure and/ or evaluate) citizen engagement in health research.

METHODS

In this scoping review, we included peer-reviewed literature that focused primarily on method(s) to measure and/or evaluate citizen engagement in health research. Independently and in duplicate, we completed title and abstract screening and full-text screening and extracted data including document characteristics, citizen engagement definitions and goals, and methods to measure or evaluate citizen engagement (including characteristics of these methods).

RESULTS

Our search yielded 16,762 records of which 33 records (31 peer-reviewed articles, one government report, one conference proceeding) met our inclusion criteria. Studies discussed engaging citizens (i.e., patients [n = 16], members of the public [n = 7], service users/consumers [n = 4], individuals from specific disease groups [n = 3]) in research processes. Reported methods of citizen engagement measurement and evaluation included frameworks, discussion-based methods (i.e., focus groups, interviews), survey-based methods (e.g., audits, questionnaires), and other methods (e.g., observation, prioritization tasks). Methods to measure and evaluate citizen engagement commonly focused on collecting perceptions of citizens and researchers on aspects of citizen engagement including empowerment, impact, respect, support, and value.

DISCUSSION AND CONCLUSION

We found that methods to measure and/or evaluate citizen engagement in health research vary widely but share some similarities in aspect of citizen engagement considered important to measure or evaluate. These aspects could be used to devise a more standardized, modifiable, and widely applicable framework for measuring and evaluating citizen engagement in research.

PATIENT OR PUBLIC CONTRIBUTION

Two citizen team members were involved as equal partners in study design and interpretation of its findings.

SYSTEMATIC REVIEW REGISTRATION

Open Science Framework (10.17605/OSF.IO/HZCBR).

An evaluation of stakeholder engagement in comparative effectiveness research: lessons learned from SWOG S1415CD

Aim: Stakeholder engagement is central to comparative effectiveness research yet there are gaps in definitions of success. We used a framework developed by Lavallee et al. defining effective engagement criteria to evaluate stakeholder engagement during a pragmatic cluster-randomized trial. Methods: Semi-structured interviews were developed from the framework and completed to learn about members' experiences. Interviews were analyzed in a deductive approach for themes related to the effective engagement criteria. Results: Thirteen members participated and described: respect for ideas, time to achieve consensus, access to information and continuous feedback as areas of effective engagement. The primary criticism was lack of diversity. Discussion: Feedback was positive, particularly among themes of respect, trust and competence, and led to development of a list of best practices for engagement. The framework was successful for evaluating engagement. Conclusion: Standardized frameworks allow studies to formally evaluate their stakeholder engagement approach and develop best practices for future research.

Operationalizing Stakeholder Engagement Through the Stakeholder-Centric Engagement Charter (SCEC)

There is a need for clear strategies and procedures to operationalize stakeholder engagement in research studies. Clear guidelines that promote shared leadership among study investigators and research stakeholders are important for inclusive and sustainable partnerships. Such guidelines may take the form of a governance charter and can be a means for encouraging the participation and inclusion of stakeholders who may have little to no experience with research or are otherwise underrepresented in research. This perspective article presents the Stakeholder-Centric Engagement Charter (SCEC), one effort to operationalize a stakeholder engagement approach between researchers and an advisory committee as guided by the Patient-Centered Outcomes Research Institute's (PCORI) Research Engagement Principles (i.e., reciprocal relationships, partnerships, co-learning, transparency-honesty-trust). Building on the SCEC can help future investigators develop a study-specific, dynamic, governance document outlining advisory committee and research team preferences in areas such as role expectations, study governance, and decision-making procedures.

Evaluating the Impact of Stakeholder Engagement in a School-Based Type 1 Diabetes Study

OBJECTIVE

Although the importance of stakeholder engagement (SE) for patient-centered research is recognized, few studies document SE processes and influence on research outcomes in the diabetes field. We applied a research-informed framework to evaluate the impact of SE on a pediatric diabetes study exploring school nurse perspectives on modern diabetes devices.

METHODS

We recruited parents of children with type 1 diabetes, school nurses, and diabetes providers. Stakeholders convened virtually every 2 months for 12 months. Goals for SE included input on research materials, interpretation of findings, and future research directions. Processes were assessed using a validated survey. Immediate outcomes included changes to research materials and satisfaction. Secondary outcomes included research efficiency and value (acceptance by community partners).

RESULTS

Each role was represented at every meeting. The majority of stakeholders (>70%) completed the survey at study midpoint and end points. All surveyed indicated that they had received all desired information, shared feedback, and felt valued. Stakeholders were satisfied with the meeting frequency. Participants appreciated learning from each other and expressed enthusiasm for continued research participation. They described their role as one of consultant rather than research team members. SE resulted in five additional interview questions. Nearly 70 comments added to the interpretation of qualitative themes. Findings were published within 12 months and recognized by the state school nursing organization.

CONCLUSION

SE was well received and led to meaningful changes in content and dissemination of a diabetes study. A systematic approach to evaluating SE can increase scientific rigor and reproducibility and contribute to best practices for SE in diabetes research.

From research to international scale-up: stakeholder engagement essential in successful design, evaluation and implementation of paediatric HIV testing intervention

Stakeholder engagement between researchers, policymakers and practitioners is critical for the successful translation of research into policy and practice. The Counseling and Testing for Children at Home (CATCH) study evaluated a paediatric index case testing model, targeting the children of HIV-infected adults in care in Kenya. Researchers collaborated with stakeholders in the planning, execution and evaluation, and dissemination phases of CATCH. They included a community advisory board, the national HIV programme, County health departments, institutional ethics review bodies, a paediatric bioethics group, facility heads and frontline healthcare workers . Stakeholder analysis considered the power and interest of each stakeholder in the study. All stakeholders had some power to influence the success of the project in the different phases. However, support from institutions with higher hierarchical power increased acceptance of the study by stakeholders lower in the hierarchy. During the planning, execution and evaluation, and dissemination phases, the study benefitted from deliberate stakeholder engagement. Through engagement, changes were made in the approach to recruitment to ensure high external validity, placing recruitment optimally within existing clinic flow patterns. Choices in staffing home visits were made to include the appropriate cadre of staff. Adaptations were made to the consenting process that balanced the child's evolving autonomy and risks of HIV disclosure. Dissemination involved delivering site-specific results in each HIV clinic, local and international conferences and sharing of study tools, resulting in the study approach being scaled up nationally. The deliberate engagement of stakeholders early in intervention development optimized study validity and accelerated adoption of the CATCH approach in nationwide HIV testing campaigns by the Ministry of Health and inclusion of paediatric index-case testing in national HIV testing guidelines. Involving policymakers and frontline healthcare workers throughout the study cycle builds capacity in the implementing team for quick adoption and scale-up of the evidence-based practice.

Patient engagement in fertility research: bench research, ethics, and social justice

BACKGROUND

Patient and Public Involvement (PPI) in research is increasingly being utilized to better connect patients and researchers. The Patient Engagement Studio (PES) supports PPI in research by working directly with researchers throughout various stages of their projects. Recently, two researchers presented to the PES for assistance with their project, Embryo+™. The purpose of Embryo+™ is to decrease miscarriage rates using RNA sequencing technology that screens for the most viable embryos. To date, no examples of PPI directly in the planning or implementation of bench research concerning in vitro fertilization and embryo transfer have been identified.

MAIN BODY

Embryo+™ researchers met in-person with the PES two times (fall 2019; each meeting had 9 PES members in attendance) for initial feedback and protocol development. After these meetings, PES leadership and Embryo+™ researchers decided that the unique nature of the project merited a PPI evaluation. Subsequent evaluation of engagement efforts occurred by reviewing the PES reports for the Embryo+™ researchers, conducting two recorded web-based discussion meetings with the PES (summer 2020; meeting 1 n = 7; meeting 2 n = 6), and a brief survey (n = 13). The discussion meetings provided an opportunity for the PES members to define engagement themes through consensus via verbal agreement to the studio director's periodic summaries during the discussions. Combining survey results and PES themes allowed for a broad discussion for meaningful engagement. The Embryo+™ researchers established trust with the patients by changing some of their language in response to patient suggestions, allowing for unintended ethical conversations, and implementing the patient developed protocols. Overall, the patient experts thought this project was very meaningful and valuable, quantified by a mean loyalty score 89.43 (s.d. 10.29).

CONCLUSION

Bench science researchers may need additional PPI training prior to engaging with patient groups. PPI in this project was successful in large part due to this training, where the director emphasized the importance of gaining trust with the patients. The researchers applied what they learned and several examples of how to develop trust with patients are discussed. If trust is established, PPI in an ethically charged, basic science research study can be both valuable and successful.

Evaluating Stakeholder Engagement: Stakeholder-Centric Instrumentation Process (SCIP)

Evaluating engagement in a research partnership can capture the success and impact of the research team-stakeholder partnerships. This article describes the Stakeholder-Centric Instrumentation Process (SCIP), an iterative method to develop an evaluation that reflects research team-stakeholder collective values, language, and priorities. We describe our implementation of the SCIP and provide the Stakeholder-Centric Engagement Evaluation, an evaluation developed in collaboration with our advisory committee. Mean scores across three administrations of the tool remained constant. We monitored responses received from our advisory committee during each administration for changes in scores that guided refinements to our stakeholder engagement strategy. Face validity and acceptability questions showed high satisfaction for the tool's time required to complete, (M = 4.50, SD = 0.86), clarity (M = 4.56, SD = 0.78), and relevance (M = 4.67, SD = 0.49) (maximum score = 5). The SCIP methodology and the Stakeholder-Centric Engagement Tool can be used during study planning and data collection to capture research team-stakeholder collaborations that reflect stakeholder priorities.