1
|
Woodward EN, Castillo AIM, True G, Willging C, Kirchner JE. Challenges and promising solutions to engaging patients in healthcare implementation in the United States: an environmental scan. BMC Health Serv Res 2024; 24:29. [PMID: 38178131 PMCID: PMC10768202 DOI: 10.1186/s12913-023-10315-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2023] [Accepted: 11/13/2023] [Indexed: 01/06/2024] Open
Abstract
BACKGROUND One practice in healthcare implementation is patient engagement in quality improvement and systems redesign. Implementers in healthcare systems include clinical leadership, middle managers, quality improvement personnel, and others facilitating changes or adoption of new interventions. Patients provide input into different aspects of health research. However, there is little attention to involve patients in implementing interventions, especially in the United States (U.S.), and this might be essential to reduce inequities. Implementers need clear strategies to overcome challenges, and might be able to learn from countries outside the U.S. METHODS We wanted to understand existing work about how patients are being included in implementation activities in real world U.S. healthcare settings. We conducted an environmental scan of three data sources: webinars, published articles, and interviews with implementers who engaged patients in implementation activities in U.S. healthcare settings. We extracted, categorized, and triangulated from data sources the key activities, recurring challenges, and promising solutions using a coding template. RESULTS We found 27 examples of patient engagement in U.S. healthcare implementation across four webinars, 11 published articles, and seven interviews, mostly arranging patient engagement through groups and arranging processes for patients that changed how engaged they were able to be. Participants rarely specified if they were engaging a population experiencing healthcare inequities. Participants described eight recurring challenges; the two most frequently identified were: (1) recruiting patients representative of those served in the healthcare system; and (2) ensuring processes for equitable communication among all. We matched recurring challenges to promising solutions, such as logistic solutions on how to arrange meetings to enhance engagement or training in inclusivity and power-sharing. CONCLUSION We clarified how some U.S. implementers are engaging patients in healthcare implementation activities using less and more intensive engagement. It was unclear whether reducing inequities was a goal. Patient engagement in redesigning U.S. healthcare service delivery appears similar to or less intense than in countries with more robust infrastructure for this, such as Canada and the United Kingdom. Challenges were common across jurisdictions, including retaining patients in the design/delivery of implementation activities. Implementers in any region can learn from those in other places.
Collapse
Affiliation(s)
- Eva N Woodward
- VA Center for Mental Healthcare and Outcomes Research, 2200 Fort Roots Drive, Building 11, North Little Rock, AR, 72114, USA.
- Department of Psychiatry, University of Arkansas for Medical Sciences, 4301 W Markham St, Little Rock, AR, 72205, USA.
| | - Andrea Isabel Melgar Castillo
- VA Center for Mental Healthcare and Outcomes Research, 2200 Fort Roots Drive, Building 11, North Little Rock, AR, 72114, USA
- Graduate School, University of Arkansas for Medical Sciences, 4301 W Markham St, Little Rock, AR, 72205, USA
| | - Gala True
- South Central Mental Illness Research Education and Clinical Center, Southeast Louisiana Veterans Health Care System, 2400 Canal St, New Orleans, LA, 70119, USA
- Section on Community and Population Medicine, School of Medicine, Louisiana State University, 2400 Canal St (11F), New Orleans, LA, USA
| | - Cathleen Willging
- Pacific Institute for Research and Evaluation, 851 University Boulevard, Suite 101, Albuquerque, NM, 87106, USA
| | - JoAnn E Kirchner
- Department of Psychiatry, University of Arkansas for Medical Sciences, 4301 W Markham St, Little Rock, AR, 72205, USA
- Behavioral Health Quality Enhancement Research Initiative (QUERI), Central Arkansas Veterans Healthcare System, 2200 Fort Roots Drive, Building 11, North Little Rock, AR, 72114, USA
| |
Collapse
|
2
|
Woodward EN, Ball IA, Willging C, Singh RS, Scanlon C, Cluck D, Drummond KL, Landes SJ, Hausmann LRM, Kirchner JE. Increasing consumer engagement: tools to engage service users in quality improvement or implementation efforts. FRONTIERS IN HEALTH SERVICES 2023; 3:1124290. [PMID: 37560195 PMCID: PMC10407803 DOI: 10.3389/frhs.2023.1124290] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Accepted: 07/05/2023] [Indexed: 08/11/2023]
Abstract
Introduction Engaging service users or consumers in quality improvement or implementing a new service is important across settings and may reduce health inequities. Implementation strategies leveraging consumer engagement are neither commonly used nor robustly operationalized in implementation science. Implementers (e.g., middle managers, facilitators) want to involve consumers in implementation activities, but do not always feel confident in how to proceed. We developed a compendium of tools called Consumer Voice to guide others how to engage consumers in design/delivery of implementation strategies. Although generalizable to other settings, we developed Consumer Voice within the context of implementing suicide prevention treatments in healthcare to reach rural U.S. military veterans, as there are suicide inequities for people in rural areas. Methods We developed Consumer Voice using a multistep process and human-centered design methods. In between steps, a design team met to generate insights from data, and decide which prototypes to create/refine. In preliminary work, we conducted a scan of examples in healthcare of patient engagement in implementation activities and interviewed two implementation experts about preferred learning styles. In Step 1, we interviewed 26 participants with experience in community engagement, implementation, or lived experience as a rural U.S. veteran with suicidal thoughts/behavior. In Step 2, 11 implementers beta tested prototypes then share feedback in focus groups. In Step 3, we reconvened participants from prior steps to review tools and, using nominal group technique, prioritized remaining recommendations. Results Consumer Voice is online, modular, and nonlinear for self-guided learning tailored to beginner, intermediate, or advanced experience with consumer engagement. Tools consist of slides, audiovisual content with written text, and templates. Findings indicated there is not one "right" way to engage consumers in implementation activities, rather that implementers wanted tools showcasing core principles for consumer engagement and practical ideas. Discussion Consumer Voice can be used by implementers to reflect and decide on how to apply consumer engagement implementation strategies to improve equitable dissemination and uptake of evidence-based practices. Most insights generated by user data were explicitly to build trust between consumers and professionals representing institutions, which may be one component to reducing healthcare inequities.
Collapse
Affiliation(s)
- Eva N. Woodward
- Department of Psychiatry, University of Arkansas for Medical Sciences, Little Rock, AR, United States
- VA Center for Mental Healthcare & Outcomes Research, Central Arkansas Veterans Healthcare System, North Little Rock, AR, United States
| | - Irenia A. Ball
- VA Center for Mental Healthcare & Outcomes Research, Central Arkansas Veterans Healthcare System, North Little Rock, AR, United States
| | - Cathleen Willging
- Pacific Institute for Research and Evaluation, Southwest Center, Albuquerque, NM, United States
| | - Rajinder Sonia Singh
- Department of Psychiatry, University of Arkansas for Medical Sciences, Little Rock, AR, United States
- VA Center for Mental Healthcare & Outcomes Research, Central Arkansas Veterans Healthcare System, North Little Rock, AR, United States
| | - Celia Scanlon
- VA Center for Mental Healthcare & Outcomes Research, Central Arkansas Veterans Healthcare System, North Little Rock, AR, United States
| | - Damon Cluck
- Arkansas National Guard Foundation, North Little Rock, AR, United States
| | - Karen L. Drummond
- Department of Psychiatry, University of Arkansas for Medical Sciences, Little Rock, AR, United States
- VA Center for Mental Healthcare & Outcomes Research, Central Arkansas Veterans Healthcare System, North Little Rock, AR, United States
| | - Sara J. Landes
- Department of Psychiatry, University of Arkansas for Medical Sciences, Little Rock, AR, United States
- Behavioral Health Quality Enhancement Research Initiative (QUERI), Central Arkansas Veterans Healthcare System, North Little Rock, AR, United States
- South Central Mental Illness Research Education and Clinical Center (MIRECC), Central Arkansas Veterans Healthcare System, North Little Rock, AR, United States
| | - Leslie R. M. Hausmann
- VA Pittsburgh Healthcare System, Center for Health Equity Research and Promotion, Pittsburgh, PA, United States
- Division of General Internal Medicine, Department of Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA, United States
| | - JoAnn E. Kirchner
- Department of Psychiatry, University of Arkansas for Medical Sciences, Little Rock, AR, United States
- Behavioral Health Quality Enhancement Research Initiative (QUERI), Central Arkansas Veterans Healthcare System, North Little Rock, AR, United States
| |
Collapse
|
3
|
O'Hanlon CE, Giannitrapani KF, Lindvall C, Gamboa RC, Canning M, Asch SM, Garrido MM, Walling AM, Lorenz KA. Patient and Caregiver Prioritization of Palliative and End-of-Life Cancer Care Quality Measures. J Gen Intern Med 2022; 37:1429-1435. [PMID: 34405352 PMCID: PMC9086093 DOI: 10.1007/s11606-021-07041-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/04/2021] [Accepted: 07/14/2021] [Indexed: 10/20/2022]
Abstract
BACKGROUND Development and prioritization of quality measures typically relies on experts in clinical medicine, but patients and their caregivers may have different perspectives on quality measurement priorities. OBJECTIVE To inform priorities for health system implementation of palliative cancer and end-of-life care quality measures by eliciting perspectives of patients and caregivers. DESIGN Using modified RAND-UCLA Appropriateness Panel methods and materials tailored for knowledgeable lay participants, we convened a panel to rate cancer palliative care process quality measure concepts before and after a 1-day, in-person meeting. PARTICIPANTS Nine patients and caregivers with experience living with or caring for patients with cancer. MAIN MEASURES Panelists rated each concept on importance for providing patient- and family-centered care on a nine-point scale and each panelist nominated five highest priority measure concepts ("top 5"). KEY RESULTS Cancer patient and caregiver panelists rated all measure concepts presented as highly important to patient- and family- centered care (median rating ≥ 7) in pre-panel (mean rating range, 6.9-8.8) and post-panel ratings (mean rating range, 7.2-8.9). Forced choice nominations of the "top 5" helped distinguish similarly rated measure concepts. Measure concepts nominated into the "top 5" by three or more panelists included two measure concepts of communication (goals of care discussions and discussion of prognosis), one measure concept on providing comprehensive assessments of patients, and three on symptoms including pain management plans, improvement in pain, and depression management plans. Patients and caregivers nominated one additional measure concept (pain screening) back into consideration, bringing the total number of measure concepts under consideration to 21. CONCLUSIONS Input from cancer patients and caregivers helped identify quality measurement priorities for health system implementation. Forced choice nominations were useful to discriminate concepts with the highest perceived importance. Our approach serves as a model for incorporating patient and caregiver priorities in quality measure development and implementation.
Collapse
Affiliation(s)
- Claire E O'Hanlon
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Health Care System, Los Angeles, CA, USA. Claire.O'
| | - Karleen F Giannitrapani
- Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Palo Alto, CA, USA.,Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, CA, USA
| | - Charlotta Lindvall
- Department of Psychosocial Oncology and Palliative Care (POPC), Dana-Farber Cancer Institute, Boston, MA, USA.,Department of Medicine, Brigham and Women's Hospital, Boston, MA, USA
| | - Raziel C Gamboa
- Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Palo Alto, CA, USA
| | - Mark Canning
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Health Care System, Los Angeles, CA, USA
| | - Steven M Asch
- Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Palo Alto, CA, USA.,Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, CA, USA
| | - Melissa M Garrido
- Partnered Evidence-Based Policy Resource Center (PEPReC), VA Boston Healthcare System Research & Development, Boston, MA, USA.,Department of Health Law, Policy and Management, Boston University School of Public Health, Boston, USA
| | | | - Anne M Walling
- Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Health Care System, Los Angeles, CA, USA.,Division of General Internal Medicine and Health Services Research, University of California Los Angeles, Los Angeles, CA, USA
| | - Karl A Lorenz
- Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Palo Alto, CA, USA.,Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
4
|
A patient-centered framework for rehabilitation research in outpatient settings. Arch Phys Med Rehabil 2022; 103:1684-1692. [DOI: 10.1016/j.apmr.2022.02.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2022] [Accepted: 02/18/2022] [Indexed: 11/15/2022]
|
5
|
DePierro J, Lowe SM, Haugen PT, Cancelmo L, Schaffer J, Schechter CB, Dasaro CR, Todd AC, Crane M, Luft BJ, Moline JM, Harrison D, Udasin IG, Feder A, Southwick SM, Pietrzak RH. Mental health stigma and barriers to care in World Trade Center responders: Results from a large, population-based health monitoring cohort. Am J Ind Med 2021; 64:208-216. [PMID: 33241583 DOI: 10.1002/ajim.23204] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2020] [Revised: 09/26/2020] [Accepted: 10/31/2020] [Indexed: 12/20/2022]
Abstract
BACKGROUND Nearly 20 years after the terrorist attacks of September 11, 2001, multiple studies have documented the adverse mental consequences among World Trade Center (WTC) rescue, recovery, and clean-up workers. However, scarce research has examined mental health stigma and barriers to care in WTC-exposed individuals, and no known study has examined whether rates of endorsement may differ between police and "nontraditional" responders, the latter comprising a heterogeneous group of workers and volunteers. OBJECTIVE To identify the prevalence and correlates of mental health stigma and barriers to care in WTC responders. METHODS Mental health stigma and barriers to care and their correlates were examined in 6,777 police and 6,272 nontraditional WTC responders. RESULTS Nontraditional responders endorsed more stigma or barriers to care concerns than police responders. Within a subsample who screened positive for a psychiatric disorder, police were more likely than nontraditional responders to endorse "concerns that negative job consequences might result" (17.9% vs. 9.1%), while nontraditional responders were more likely to endorse "I don't know where to go to find counseling services" (18.4% vs.6.6%). Within this subsample, mental health service need and more severe WTC-related posttraumatic stress disorder symptoms were associated with increased likelihood of endorsing stigma or barriers; pre-9/11 psychiatric history and non-Hispanic Black race/ethnicity were associated with lower likelihood of endorsing stigma or barriers. CONCLUSIONS Results of this study underscore the burden of mental health stigma and barriers to care in WTC responders, and highlight the need for targeted interventions to address these concerns and promote mental healthcare utilization in this population.
Collapse
Affiliation(s)
- Jonathan DePierro
- Department of Psychiatry Icahn School of Medicine at Mount Sinai New York New York USA
| | - Sandra M. Lowe
- Department of Psychiatry Icahn School of Medicine at Mount Sinai New York New York USA
| | - Peter T. Haugen
- New York University Grossman School of Medicine New York New York USA
| | - Leo Cancelmo
- Department of Psychiatry Icahn School of Medicine at Mount Sinai New York New York USA
| | - Jamie Schaffer
- Department of Psychiatry Icahn School of Medicine at Mount Sinai New York New York USA
| | - Clyde B. Schechter
- Department of Family and Social Medicine Albert Einstein College of Medicine Bronx New York USA
| | - Christopher R. Dasaro
- Department of Environmental Medicine and Public Health Icahn School of Medicine New York NY 10029 USA
| | - Andrew C. Todd
- Department of Environmental Medicine and Public Health Icahn School of Medicine New York NY 10029 USA
| | - Michael Crane
- Department of Environmental Medicine and Public Health Icahn School of Medicine New York NY 10029 USA
| | - Benjamin J. Luft
- Stony Brook WTC Wellness Program Stony Brook University Stony Book New York USA
| | - Jacqueline M. Moline
- Department of Occupational Medicine, Epidemiology and Prevention Northwell Health Rego Park New York USA
| | - Denise Harrison
- New York University Grossman School of Medicine New York New York USA
| | - Iris G. Udasin
- Environmental and Occupational Health Sciences Institute (EOHSI) Clinical Center Rutgers University Piscataway New Jersey USA
| | - Adriana Feder
- Department of Psychiatry Icahn School of Medicine at Mount Sinai New York New York USA
| | - Steven M. Southwick
- Department of Psychiatry Yale School of Medicine New Haven Connecticut USA
- Department of Social and Behavioral Sciences Yale School of Public Health New Haven Connecticut USA
| | - Robert H. Pietrzak
- Department of Psychiatry Yale School of Medicine New Haven Connecticut USA
- Department of Social and Behavioral Sciences Yale School of Public Health New Haven Connecticut USA
- U.S. Department of Veterans Affairs National Center for Posttraumatic Stress Disorder VA Connecticut Healthcare System West Haven Connecticut USA
| |
Collapse
|
6
|
Oberhelman S, Boswell C, Jensen T, Swartz D, Bruhl E, O’Brien M, Angstman K. Student experiences and satisfaction with a novel clerkship patient scheduling. MEDICAL EDUCATION ONLINE 2020; 25:1742963. [PMID: 32174268 PMCID: PMC7144222 DOI: 10.1080/10872981.2020.1742963] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/20/2019] [Revised: 02/14/2020] [Accepted: 02/19/2020] [Indexed: 06/10/2023]
Abstract
Background: Outpatient primary care clerkships are an important part of medical students' education.Traditional clerkships usually partner a student with a single preceptor in that physician's clinic. However, it can be quite difficult for the preceptor to balance the educational needs of the students, the expectations of the patients and the organizational demands of the clinic practice.Objective: An innovative scheduling model (named "Patients as Teachers" [PAT] clinic) was developed as part of our third-year Family Medicine clerkship. The goal was to increase the students' opportunities for independence and improve their satisfaction without negatively impacting the flow of the clinic or patient satisfaction.Design: The third-year medical students spent part of their clerkship working in the PAT clinic and part of the time working with an individual preceptor in that preceptor's clinic in the traditional, usual fashion (PAU clinic-precepting as usual). The students completed patient-logs regarding the patients they saw and their level of participation. They also completed a voluntary survey regarding their experiences.Results: Students performed more independent interviews (90.3 vs 59.0%) and independent exams (96.2 vs 63.3%) in the PAT clinic than while working with their traditional preceptor (both p<0.01). Students were highly satisfied with the experience with 89.5% stating they would recommend it and 87.7% finding the PAT clinic to be an equal or superior experience to the PAU experience.Conclusions: Using a combination of time in the PAT clinic and time with a one on one preceptor in the usual fashion was successful in increasing opportunities for student autonomy and achieving a high level of student satisfaction in our third-year Family Medicine clerkship. Additional opportunities for innovative scheduling could be considered for meeting a variety of clerkship and clinic needs.
Collapse
Affiliation(s)
- Sara Oberhelman
- Department of Family Medicine, Mayo Clinic, Rochester, MN, USA
| | - Chris Boswell
- Department of Family Medicine, Mayo Clinic, Rochester, MN, USA
| | - Teresa Jensen
- Department of Family Medicine, Mayo Clinic, Rochester, MN, USA
| | - Daniel Swartz
- Department of Family Medicine, Liberty University College of Osteopathic Medicine, Lynchburg, VA, US
| | - Elliot Bruhl
- Southeast Alaska Regional Health Consortium, Juneau, AK, US
| | - Marcia O’Brien
- Department of Family Medicine, Mayo Clinic, Rochester, MN, USA
| | - Kurt Angstman
- Department of Family Medicine, Mayo Clinic, Rochester, MN, USA
| |
Collapse
|
7
|
Haesebaert J, Samson I, Lee-Gosselin H, Guay-Bélanger S, Proteau JF, Drouin G, Guimont C, Vigneault L, Poirier A, Sanon PN, Roch G, Poitras MÈ, LeBlanc A, Légaré F. "They heard our voice!" patient engagement councils in community-based primary care practices: a participatory action research pilot study. RESEARCH INVOLVEMENT AND ENGAGEMENT 2020; 6:54. [PMID: 32974050 PMCID: PMC7507740 DOI: 10.1186/s40900-020-00232-3] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/14/2020] [Accepted: 09/14/2020] [Indexed: 06/11/2023]
Abstract
BACKGROUND Patient engagement could improve the quality of primary care practices. However, we know little about effective patient engagement strategies. We aimed to assess the acceptability and feasibility of embedding advisory councils of clinicians, managers, patients and caregivers to conduct patient-oriented quality improvement projects in primary care practices. METHODS Using a participatory action research approach, we conducted our study in two non-academic primary care practices in Quebec City (Canada). Patient-experts (patients trained in research) were involved in study design, council recruitment and meeting facilitation. Advisory councils were each to include patients and/or caregivers, clinicians and managers. Over six meetings, councils would identify quality improvement priorities and plan projects accordingly. We assessed acceptability and feasibility of the councils using non-participant observations, audio-recordings and self-administered questionnaires. We used descriptive analyses, triangulated qualitative data and performed inductive thematic analysis. RESULTS Between December 2017 and June 2018, two advisory councils were formed, each with 11 patients (36% male, mean age 53.8 years), a nurse and a manager practising as a family physician (25% male, mean age 45 years). The six meetings per practice occurred within the study period with a mean of eight patients per meeting. Councils worked on two projects each: the first council on a new information leaflet about clinic organization and operation, and on communications about local public health programs; the second on methods to further engage patients in the practice, and on improving the appointment scheduling system. Median patient satisfaction was 8/10, and 66.7% perceived councils had an impact on practice operations. They considered involvement of a manager, facilitation by patient-experts, and the fostering of mutual respect as key to this impact. Clinicians and managers liked having patients as facilitators and the respect among members. Limiting factors were difficulty focusing on a single feasible project and time constraints. Managers in both practices were committed to pursuing the councils post-study. CONCLUSION Our results indicated that embedding advisory councils of clinicians, managers, patients and caregivers to conduct patient-oriented quality improvement projects in primary care practices is both acceptable and feasible. Future research should assess its transferability to other clinical contexts.
Collapse
Affiliation(s)
- Julie Haesebaert
- VITAM – Centre de recherche en santé durable, CIUSSS de la Capitale-Nationale, Pavillon Landry-Poulin, Room A-4574, 2525, chemin de la Canardière, Quebec City, Quebec G1J 0A4 Canada
- Tier 1 Canada Research Chair in Shared Decision Making and Knowledge Translation, Université Laval, Quebec City, Quebec Canada
| | - Isabelle Samson
- Department of Family Medicine and Emergency Medicine, Faculty of Medicine, Université Laval, Quebec City, Quebec Canada
| | - Hélène Lee-Gosselin
- Department of Management, Faculty of Business Administration, Université Laval, Quebec City, Quebec Canada
| | - Sabrina Guay-Bélanger
- VITAM – Centre de recherche en santé durable, CIUSSS de la Capitale-Nationale, Pavillon Landry-Poulin, Room A-4574, 2525, chemin de la Canardière, Quebec City, Quebec G1J 0A4 Canada
- Tier 1 Canada Research Chair in Shared Decision Making and Knowledge Translation, Université Laval, Quebec City, Quebec Canada
| | | | - Guy Drouin
- Clinic Manager, Quebec City, Quebec Canada
| | | | | | | | | | - Geneviève Roch
- Centre Hospitalier Universitaire de Québec – Université Laval Research Center, Hôpital Saint-François d’Assise, Quebec City, Quebec Canada
- Faculty of Nursing, Université Laval, Quebec City, Quebec Canada
| | - Marie-Ève Poitras
- Department of Family Medicine, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Chicoutimi, Quebec Canada
| | - Annie LeBlanc
- VITAM – Centre de recherche en santé durable, CIUSSS de la Capitale-Nationale, Pavillon Landry-Poulin, Room A-4574, 2525, chemin de la Canardière, Quebec City, Quebec G1J 0A4 Canada
- Department of Family Medicine and Emergency Medicine, Faculty of Medicine, Université Laval, Quebec City, Quebec Canada
| | - France Légaré
- VITAM – Centre de recherche en santé durable, CIUSSS de la Capitale-Nationale, Pavillon Landry-Poulin, Room A-4574, 2525, chemin de la Canardière, Quebec City, Quebec G1J 0A4 Canada
- Tier 1 Canada Research Chair in Shared Decision Making and Knowledge Translation, Université Laval, Quebec City, Quebec Canada
- Department of Family Medicine and Emergency Medicine, Faculty of Medicine, Université Laval, Quebec City, Quebec Canada
| |
Collapse
|
8
|
Tsang VWL, Chew SY, Junker AK. Facilitators and barriers to the training and maintenance of young persons' advisory groups (YPAGs). Int J Pediatr Adolesc Med 2019; 7:166-173. [PMID: 33319014 PMCID: PMC7729230 DOI: 10.1016/j.ijpam.2019.10.002] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2019] [Accepted: 10/14/2019] [Indexed: 12/20/2022]
Abstract
Introduction Increasing demands from public and private healthcare coupled with national initiatives in patient-oriented research has led to an increase in avenues to allow patients to be directly involved in research. In particular, the push towards participation of children and youth has resulted in the formation of pediatric patient advisory groups with broad partnerships and consultation requests across the globe. However, there is a lack of evidence to examine the challenges in formation and training of young persons’ advisory groups (YPAGs) and management processes required thereafter. Purpose and objectives This study’s purpose is to document YPAG formation and training protocols around the world, highlight common strengths, and evaluate pitfalls and challenges. The results from this study will subsequently inform the development of standardized training protocols for children and youth to be piloted globally. Methods In this study, 17 select YPAG team leaders from 7 countries were surveyed to determine current training techniques used within existing groups. 17 youth representatives and 16 team leaders were then interviewed to gather further qualitative data on facilitators and barriers that aid or prevent successful initiation and maintenance of these groups. Qualitative interview data was coded and analyzed using NVivo by two independent reviewers (SYC, VWLT). Any inconsistencies in thematic analysis was confirmed by a third reviewer (JB). Results The most common training topics include consent and assent (64.71%), clinical trials (64.71%), and patient safety (70.59%). There are significant discrepancies to the amount of training received by each team. Most YPAGs out of the 17 groups receive no formal training (58.82%) while training sessions in the remaining 7 groups vary in both duration and frequency. Collectively, meetings ranged from 15 minutes to 6 hours long, with the majority of team meetings being 2–3 hours long (58.82%). The most common training facilitators are a positive relationship with a local hospital (82.35%) and access to a dedicated team coordinator (64.71%). 70.59% of team leaders identified a lack of access to appropriate educational materials available as a drawback to the impact of their YPAG, making this the greatest common barrier. Conclusion Bringing children and youth to the forefront of paediatric trials and clinical research facilitates appropriate patient representation in subsequent research decision-making. There is an urgency to create and implement standardized protocols for the training of children and youth, especially in preparation for national and international research consultations. This low barrier framework may be of special interest to lower-middle-income countries who wish to encourage community participation in healthcare.
Collapse
Affiliation(s)
- V W L Tsang
- Faculty of Medicine, University of British Columbia, 317 - 2194 Health Sciences Mall, Vancouver, BC V6T 1Z3, Canada
| | - S Y Chew
- BC Children's Hospital, 4480 Oak St, Vancouver, BC V6H 3N1, Canada
| | - A K Junker
- Faculty of Medicine, University of British Columbia, 317 - 2194 Health Sciences Mall, Vancouver, BC V6T 1Z3, Canada.,BC Children's Hospital, 4480 Oak St, Vancouver, BC V6H 3N1, Canada
| |
Collapse
|
9
|
Haesebaert J, Samson I, Lee-Gosselin H, Guay-Bélanger S, Proteau JF, Drouin G, Guimont C, Vigneault L, Poirier A, Sanon PN, Roch G, Poitras MÈ, LeBlanc A, Légaré F. How to engage patients in research and quality improvement in community-based primary care settings: protocol for a participatory action research pilot study. RESEARCH INVOLVEMENT AND ENGAGEMENT 2018; 4:30. [PMID: 30288298 PMCID: PMC6166288 DOI: 10.1186/s40900-018-0113-3] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/06/2018] [Accepted: 08/15/2018] [Indexed: 06/08/2023]
Abstract
PLAIN ENGLISH SUMMARY Making primary care clinics more patient-centered is key to improving patients' experience of care. If patients themselves were engaged in helping define priorities and suggesting quality improvements in the clinic, care would respond better to their needs. However, patient engagement is a new phenomenon, particularly in community based primary care clinics. How to engage patients in quality improvement in these clinics, or what effect this might have, is not well known. The involvement of patients needs to be adapted to the way these clinics function. The aim of this study is to create and evaluate a new model of patient engagement for quality improvement in community based primary care clinics. Patients, primary care professionals and researchers will create advisory councils in two primary care clinics in Quebec City (Canada). In each clinic, the advisory council will include 12 patients or caregivers registered at the clinic, a clinician and a clinic manager. The advisory council will meet every 6 weeks for a total of six meetings. Two patient-experts will facilitate meetings. During meetings, members of the council will list their needs in order of importance. Then they will suggest improvements in line with these needs. We will study if our advisory council model is well adapted to community based primary care settings and meets participants' expectations. At the end of the study we will be able to offer guidance about engaging patients with health professionals in quality improvement in primary care clinics. ABSTRACT Background Involvement of end-users, including patients, managers and clinicians, in identifying quality improvement and research priorities might improve the relevance of projects and increase their impact. Few patient engagement initiatives have taken place in community based primary care practices (CBPCPs) and best practices for engaging patients in such settings are not well defined. The aim of this pilot study is to develop and assess the feasibility of a new collaborative model of advisory council involving clinicians, managers, patients and caregivers in CBPCP to strengthen their capacity to conduct quality improvement and patient-oriented research projects. Methods We will conduct a participatory action research project in two non-academic CBPCPs in Quebec City (Canada). In each CBPCP, the advisory council will include 12 patients or caregivers, a clinician and a clinic manager. Patients or their caregivers will be identified by clinicians and contacted by patient-experts. They will be eligible if they are registered at the practice, motivated, and available to attend meetings. The council will meet every 6 weeks for a total of six meetings. Two patient-experts will guide council members to identify quality improvement priorities and patient-oriented research questions based on their experience in the clinic. They will then be supported to plan actions to target these priorities. Analysis of meetings will be based on feasibility criteria, notes by non-participant observers in log books, audio-recording of the meetings and questionnaires to evaluate council members' perceptions and the likelihood they would engage in such councils. Discussion The results of this study will be a model of patient engagement and a discussion of factors to improve the model to fit the needs of primary care patients and professionals. This will lay the foundation for a sustainable structure for long-term patient engagement and contribute to the development of a patient-centered and quality-improvement culture in CBPCPs.
Collapse
Affiliation(s)
- Julie Haesebaert
- Centre de recherche sur les soins et services de première ligne de l’Université Laval, CIUSSS de la Capitale-Nationale, Pavillon Landry-Poulin, 2525, chemin de la Canardière, Quebec City, Quebec G1J 0A4 Canada
- Tier 1 Canada Research Chair in Shared Decision Making and Knowledge Translation, Université Laval, Quebec, QC Canada
| | - Isabelle Samson
- Department of Family Medicine and Emergency Medicine, Faculty of Medicine, Université Laval, Quebec City, Quebec Canada
| | - Hélène Lee-Gosselin
- Department of Management, Faculty of Business Administration, Université Laval, Quebec City, Quebec Canada
| | - Sabrina Guay-Bélanger
- Centre de recherche sur les soins et services de première ligne de l’Université Laval, CIUSSS de la Capitale-Nationale, Pavillon Landry-Poulin, 2525, chemin de la Canardière, Quebec City, Quebec G1J 0A4 Canada
| | | | | | | | | | | | | | - Geneviève Roch
- Centre Hospitalier Universitaire de Québec – Université Laval Research Centre, Hôpital Saint-François d’Assise, Quebec City, Quebec Canada
- Faculty of Nursing, Université Laval, Quebec City, Quebec Canada
| | - Marie-Ève Poitras
- Department of Health Sciences, Université du Québec à Chicoutimi, Chicoutimi, Quebec Canada
| | - Annie LeBlanc
- Centre de recherche sur les soins et services de première ligne de l’Université Laval, CIUSSS de la Capitale-Nationale, Pavillon Landry-Poulin, 2525, chemin de la Canardière, Quebec City, Quebec G1J 0A4 Canada
- Department of Family Medicine and Emergency Medicine, Faculty of Medicine, Université Laval, Quebec City, Quebec Canada
| | - France Légaré
- Centre de recherche sur les soins et services de première ligne de l’Université Laval, CIUSSS de la Capitale-Nationale, Pavillon Landry-Poulin, 2525, chemin de la Canardière, Quebec City, Quebec G1J 0A4 Canada
- Tier 1 Canada Research Chair in Shared Decision Making and Knowledge Translation, Université Laval, Quebec, QC Canada
- Department of Family Medicine and Emergency Medicine, Faculty of Medicine, Université Laval, Quebec City, Quebec Canada
| |
Collapse
|
10
|
van C, McInerney P, Cooke R. Patients' involvement in improvement initiatives: a qualitative systematic review. ACTA ACUST UNITED AC 2018; 13:232-90. [PMID: 26571293 DOI: 10.11124/jbisrir-2015-1452] [Citation(s) in RCA: 31] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/31/2022]
Abstract
BACKGROUND Over the last 20 years, quality improvement in health has become an important strategy in health services in many countries. With the emphasis on quality health care, there has been a shift in social paradigms towards including service users in their own health on different levels. There is growing evidence in literature on the positive impact on health outcomes where patients are active participants in their personal care. There is however less information available on the broader influence of users on improvement in systems. OBJECTIVES The objective of this review was to identify the barriers and enablers to patients being involved in quality improvement efforts directed towards their own health care. INCLUSION CRITERIA This review considered studies that included adults and children of any age experiencing any health problem.The review considered studies that explored patient or user participation in quality improvement and the factors enabling and hindering this processThe qualitative component of this review considered studies that focused on qualitative data, including, but not limited to, designs such as phenomenology, grounded theory, ethnography, action research and feminist research. Other texts such as opinion papers and reports were also considered. SEARCH STRATEGY The search strategy aimed to find both published and unpublished studies. A three-step search strategy was utilized in this review. The searches using all identified keywords and index terms included the databases PubMed, PsycINFO, Medline, Scopus, EBSCOhost and CINAHL.Qualitative, text and opinion papers were considered for inclusion in this review.Closely related concepts like community involvement, family involvement, patients' involvement in their own care (for example, in the case of shared decision making), and patient centeredness in the context of a consultation were excluded. METHODOLOGICAL QUALITY Qualitative and textual papers selected for retrieval were assessed by two independent reviewers for authenticity prior to inclusion in the review using the standardized critical appraisal instruments from the Joanna Briggs Institute. DATA EXTRACTION Qualitative and textual data were extracted from papers included in the review using the standardized data extraction tool from the Joanna Briggs Institute. DATA SYNTHESIS The above findings were pooled and through the identification of categories, a final meta-synthesis was formulated. RESULTS Two synthesized findings were created from the included papers. Firstly, there are barriers to patients' participation in quality improvement in health and in spite of policy support for user involvement in quality improvement, it is a difficult strategy to implement. The second synthesized finding was that there are enablers to patients' involvement in quality improvement: when patients are involved in quality improvement efforts in health care, there are innovative, often unexpected, outcomes at different levels of the process, and sustaining these efforts is possible with ongoing individual or group support.Five categories which supported the synthesized findings were created through the meta-aggregative process. CONCLUSIONS There are enablers and barriers to involving patients in quality improvement in health care that need to be considered when planning such interventions.Relationships and roles will need to be very clear from the outset. A developmental approach needs to be considered where support and training is part of the project. Where patients are truly engaged in service improvement, unexpected innovation occurs.There are many more reports and opinion papers published regarding this topic than there are rigorous research studies. This leaves the field open to the development of good methodological studies related to quality improvement and in particular to the participation of patients.
Collapse
Affiliation(s)
- Claire van
- 1Department of Family Medicine, University of the Witwatersrand, Johannesburg, South Africa2The Witwatersrand Center for Evidence Based Practice: an Affiliate Center of the Joanna Briggs Institute3Center for Health Science Education, Faculty of Health Science Education, University of the Witwatersrand.4Center for Rural Health, University of the Witwatersrand, Johannesburg, South Africa
| | | | | |
Collapse
|
11
|
Moffatt-Bruce SD, Hilligoss B, Gonsenhauser I. ERAS: Safety checklists, antibiotics, and VTE prophylaxis. J Surg Oncol 2017; 116:601-607. [PMID: 28846138 DOI: 10.1002/jso.24790] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2017] [Accepted: 07/10/2017] [Indexed: 01/25/2023]
Abstract
The concept rested on several components that many of us have now tried to adopt or improve on, inclusive of a multidisciplinary team, a multimodal approach to anesthesia and preoperative preparedness, evidence-based approach to care protocols; and a change in management using interactive and continuous audit prior to and post-procedure. This article describes the development of ERAS protocols relative to checklist implementation, antibiotic use, and venous thromboembolism (VTE) prevention, how these ideas are developed and operationalized as well as how they are evolving and spreading across the care continuum to achieve sustained outcome improvements.
Collapse
Affiliation(s)
| | - Brian Hilligoss
- College of Public Health, The Ohio State University, Columbus, Ohio
| | | |
Collapse
|
12
|
Abstract
Primary care practices are increasingly asked to engage patients in improving care delivery. We report early experiences with Patient and Family Advisory Councils (PFACs) from interviews of patients and practice staff in the Comprehensive Primary Care initiative, and identify ways to improve PFACs. Patients and practice staff report PFACs help practices elicit patient feedback and, in response, improve care delivery. Nonetheless, there are areas for refinement, including recruiting more diverse patients, providing an orientation to members, overcoming reticence of some patients to raise issues, and increasing transparency by sharing progress with PFAC members and patients in the practice more generally.
Collapse
|
13
|
Johnson KE, Mroz TM, Abraham M, Figueroa Gray M, Minniti M, Nickel W, Reid R, Sweeney J, Frosch DL, Ness DL, Hsu C. Promoting Patient and Family Partnerships in Ambulatory Care Improvement: A Narrative Review and Focus Group Findings. Adv Ther 2016; 33:1417-39. [PMID: 27352378 PMCID: PMC4969329 DOI: 10.1007/s12325-016-0364-z] [Citation(s) in RCA: 29] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2016] [Indexed: 01/17/2023]
Abstract
Introduction Ambulatory practices that actively partner with patients and families in quality improvement (QI) report benefits such as better patient/family interactions with physicians and staff, and patient empowerment. However, creating effective patient/family partnerships for ambulatory care improvement is not yet routine. The objective of this paper is to provide practices with concrete evidence about meaningfully involving patients and families in QI activities. Methods Review of literature published from 2000–2015 and a focus group conducted in 2014 with practice advisors. Results Thirty articles discussed 26 studies or examples of patient/family partnerships in ambulatory care QI. Patient and family partnership mechanisms included QI committees and advisory councils. Facilitators included process transparency, mechanisms for acting on patient/family input, and compensation. Challenges for practices included uncertainty about how best to involve patients and families in QI. Several studies found that patient/family partnership was a catalyst for improvement and reported that partnerships resulted in process improvements. Focus group results were concordant. Conclusion This paper describes emergent mechanisms and processes that ambulatory care practices use to partner with patients and families in QI including outcomes, facilitators, and challenges. Funding Gordon and Betty Moore Foundation. Electronic supplementary material The online version of this article (doi:10.1007/s12325-016-0364-z) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
| | | | - Marie Abraham
- Institute for Patient- and Family-Centered Care, Bethesda, MD, USA
| | | | - Mary Minniti
- Institute for Patient- and Family-Centered Care, Bethesda, MD, USA
| | - Wendy Nickel
- American College of Physicians, Philadelphia, PA, USA
| | - Robert Reid
- Trillium Health Partners, Mississauga, ON, Canada
| | - Jennifer Sweeney
- National Partnership for Women and Families, Washington, DC, USA
| | | | - Debra L Ness
- National Partnership for Women and Families, Washington, DC, USA
| | - Clarissa Hsu
- Group Health Research Institute, Seattle, WA, USA
| |
Collapse
|
14
|
Robbins M, Tufte J, Hsu C. Learning to "Swim" with the Experts: Experiences of Two Patient Co-Investigators for a Project Funded by the Patient-Centered Outcomes Research Institute. Perm J 2016; 20:85-8. [PMID: 27083011 DOI: 10.7812/tpp/15-162] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
The Patient-Centered Outcomes Research Institute (PCORI), established in 2010, launched a new model of incorporating stakeholder perspectives into health care research. To ensure that PCORI-funded studies address issues important to health care consumers, all projects must fully involve patients and other stakeholders in every step of the research process: from planning and design to implementation and dissemination of results.As members of the first cohort of PCORI-funded researchers, our team was on the forefront of developing new approaches to engaging patients in research. One innovation we pioneered was the creation of a "patient co-investigator" role for two nonscientists who were recruited to be active members of the research team throughout the project. This commentary, based on our experiences, aims to help other research teams to 1) understand how to effectively collaborate with stakeholder team members such as patients; 2) anticipate possible challenges; and 3) offer tools for the orientation, training, and integration of patients into a scientific team. Written from the perspective of two PCORI patient co-investigators, our commentary provides lessons learned and recommendations about incorporating nonscientists into research teams.Specifically, we suggest recruiting people with a record of relevant volunteer experience and commitment; establishing a formal application process that provides candidates with details about expectations and responsibilities; and providing comprehensive orientation with ongoing training, encouragement, and support. We hope the points in this commentary help research teams that are incorporating patient co-investigators move toward a positive and productive experience.
Collapse
Affiliation(s)
- Michele Robbins
- Patient Co-Investigator at the Group Health Research Institute in Seattle, WA.
| | - Janice Tufte
- Patient Co-Investigator at the Group Health Research Institute in Seattle, WA.
| | - Clarissa Hsu
- Assistant Investigator at the Center for Community Health and Evaluation at the Group Health Research Institute in Seattle, WA.
| |
Collapse
|
15
|
van Deventer C. Integration of non-communicable chronic diseases (NCDs) and HIV/AIDS and mental health care through the involvement of chronically ill patients using empowerment evaluation. S Afr Fam Pract (2004) 2015. [DOI: 10.1080/20786190.2015.1102499] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022] Open
|
16
|
Diabetes and obesity not associated with 6-month remission rates for primary care patients with depression. PSYCHOSOMATICS 2015; 56:354-61. [PMID: 26096322 DOI: 10.1016/j.psym.2014.05.012] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/13/2014] [Revised: 03/31/2014] [Accepted: 04/01/2014] [Indexed: 12/25/2022]
Abstract
BACKGROUND Complex interrelationships appear to exist among depression, diabetes, and obesity, and it has been proposed that both diabetes and obesity have an association with depression. OBJECTIVE The purpose of our study was to explore the effect of obesity and diabetes on response to the treatment of depression. Our hypothesis was that obesity and the diagnosis of diabetes in primary care patients with depression would have no effects on depression remission rates 6 months after diagnosis. METHODS A retrospective chart review analysis of 1894 adult (age ≥18y) primary care patients diagnosed with major depressive disorder or dysthymia and a Patient Health Questionnaire-9 score ≥10 from January 1, 2008, through September 30, 2012. Multiple logistic regression modeling retaining all independent variables was performed for the outcome of remission (Patient Health Questionnaire-9 < 5) 6 months after diagnosis. RESULTS The presence of obesity (odds ratio = 0.937, 95% CI: 0.770-1.140, p = 0.514) or the diagnosis of diabetes (odds ratio = 0.740, 95% CI: 0.535-1.022, p = 0.068) did not affect the likelihood of remission, while controlling for the other independent variables. CONCLUSIONS In primary care patients treated for depression, the presence of diabetes or obesity at the time of diagnosis of depression does not appear to significantly affect remission of depressive symptoms 6 months after diagnosis.
Collapse
|
17
|
Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, Lau J. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med 2014; 29:1692-701. [PMID: 24893581 PMCID: PMC4242886 DOI: 10.1007/s11606-014-2878-x] [Citation(s) in RCA: 305] [Impact Index Per Article: 30.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/13/2013] [Revised: 01/16/2014] [Accepted: 04/19/2014] [Indexed: 11/28/2022]
Abstract
OBJECTIVES We conducted a review of the peer-reviewed literature since 2003 to catalogue reported methods of stakeholder engagement in comparative effectiveness research and patient-centered outcomes research. METHODS AND RESULTS We worked with stakeholders before, during and after the review was conducted to: define the primary and key research questions; conduct the literature search; screen titles, abstracts and articles; abstract data from the articles; and analyze the data. The literature search yielded 2,062 abstracts. The review was conducted on 70 articles that reported on stakeholder engagement in individual research projects or programs. FINDINGS Reports of stakeholder engagement are highly variable in content and quality. We found frequent engagement with patients, modestly frequent engagement with clinicians, and infrequent engagement with stakeholders in other key decision-making groups across the healthcare system. Stakeholder engagement was more common in earlier (prioritization) than in later (implementation and dissemination) stages of research. The roles and activities of stakeholders were highly variable across research and program reports. RECOMMENDATIONS To improve on the quality and content of reporting, we developed a 7-Item Stakeholder Engagement Reporting Questionnaire. We recommend three directions for future research: 1) descriptive research on stakeholder-engagement in research; 2) evaluative research on the impact of stakeholder engagement on the relevance, transparency and adoption of research; and 3) development and validation of tools that can be used to support stakeholder engagement in future work.
Collapse
|