Engaging multilevel stakeholders in an implementation trial of evidence-based quality improvement in VA women's health primary care

Study protocol of a hybrid type 1 effectiveness-implementation multisite trial: Dialectical behavior therapy skills group for Veterans at high-risk for suicide attempt

BACKGROUND

Veterans of the United States Armed Forces are at disproportionately high risk for suicide death, requiring indicated strategies to mitigate that risk. Dialectical Behavior Therapy (DBT) is effective for reducing suicide attempts in individuals with emotional dysregulation and repeat suicidal behaviors or self-directed violence, but is a comprehensive, multi-component, resource-intensive treatment. A more resource-efficient component of DBT, the DBT Skills Group as an adjunctive treatment, with therapist consultation team (DBT-SG), has been shown to be as efficacious as comprehensive DBT in non-veteran samples, but its effectiveness and factors affecting its implementation have not been studied in the Veterans Health Administration (VHA). This research aims to assess the effectiveness of DBT-SG among high-risk veterans with recent and repeated suicide attempts and emotion dysregulation while systematically evaluating implementation barriers and facilitators.

METHODS

This hybrid type 1 effectiveness-implementation study will evaluate DBT-SG effectiveness among veterans at high-risk for suicide attempt with emotion dysregulation using a randomized controlled trial of 18 months duration. Study conditions are 24-session DBT-SG plus full-spectrum VHA mental health treatment-as-usual (TAU), or VHA TAU. Outcomes are assessed at 3-, 6-, 12-, and 18-months post-randomization. Before, during, and after the effectiveness trial, implementation determinants of DBT-SG as an adjunctive treatment in VHA will be assessed using a mixed methods evaluation grounded in the Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework.

CONCLUSIONS

This study will provide evidence for DBT-SG effectiveness for veterans at high risk for suicide and information about barriers and facilitators to support more widespread facilitation of implementing adjunctive DBT-SG in VHA if it is found effective.

CLINICAL TRIALS REGISTRATION

NCT05000749.

Enhancing Primary Care and Mental Health Integration for Women Veterans with Complex Healthcare Needs Using Evidence-Based Quality Improvement

BACKGROUND

Women Veterans with co-morbid medical and mental health conditions face persistent barriers accessing high-quality health care. Evidence-based quality improvement (EBQI) offers a systematic approach to implementing new care models that can address care gaps for women Veterans.

OBJECTIVE

This study examines factors associated with the successful deployment of EBQI within integrated health systems to improve primary care for women Veterans with complex mental health needs.

DESIGN

Following a 12-site (8 EBQI, 4 control) cluster randomized study to evaluate EBQI effectiveness, we conducted an in-depth case study analysis of one women's health clinic that used EBQI to improve integrated primary care-mental health services for women Veterans.

PARTICIPANTS

Our study sample included providers, program managers, and clinic staff at a women Veteran's health clinic that, at the time of the study, had one Primary Care and Mental Health Integration team and one women's health primary care provider serving 800 women. We analyzed interviews conducted 12 months, 24 months, and 4 years post-implementation and call summaries between the clinic and support team.

MAIN MEASURES

We conducted qualitative thematic analysis of interview and call summary data to identify EBQI elements, clinic characteristics, and reported challenges and successes within project development and execution.

KEY RESULTS

The clinic harnessed core EBQI elements (multi-level stakeholder engagement, data-driven progress-monitoring, PDSA cycles, sharing results) to accomplish pre-defined project goals, strengthen inter-disciplinary partnerships, and bolster team confidence. Clinic characteristics that facilitated implementation success included prior QI experience and an organizational culture responsive to innovation, while lack of pre-existing guidelines and limited access to centralized databases posed implementation challenges.

CONCLUSIONS

Successful practice transformation emerges through the interaction of evidence-based methods and site-specific characteristics. Examining how clinic characteristics support or impede EBQI adaptation can facilitate efforts to improve care within integrated health systems.

Implementing evidence-based practices to improve primary care for high-risk patients: study protocol for the VA high-RIsk VETerans (RIVET) type III effectiveness-implementation trial

BACKGROUND

Patients with significant multimorbidity and other factors that make healthcare challenging to access and coordinate are at high risk for poor health outcomes. Although most (93%) of Veterans' Health Administration (VHA) patients at high risk for hospitalization or death ("high-risk Veterans") are primarily managed by primary care teams, few of these teams have implemented evidence-based practices (EBPs) known to improve outcomes for the high-risk patient population's complex healthcare issues. Effective implementation strategies could increase adoption of these EBPs in primary care; however, the most effective implementation strategies to increase evidence-based care for high-risk patients are unknown. The high-RIsk VETerans (RIVET) Quality Enhancement Research Initiative (QUERI) will compare two variants of Evidence-Based Quality Improvement (EBQI) strategies to implement two distinct EBPs for high-risk Veterans: individual coaching (EBQI-IC; tailored training with individual implementation sites to meet site-specific needs) versus learning collaborative (EBQI-LC; implementation sites trained in groups to encourage collaboration among sites). One EBP, Comprehensive Assessment and Care Planning (CACP), guides teams in addressing patients' cognitive, functional, and social needs through a comprehensive care plan. The other EBP, Medication Adherence Assessment (MAA), addresses common challenges to medication adherence using a patient-centered approach.

METHODS

We will recruit and randomize 16 sites to either EBQI-IC or EBQI-LC to implement one of the EBPs, chosen by the site. Each site will have a site champion (front-line staff) who will participate in 18 months of EBQI facilitation.

ANALYSIS

We will use a mixed-methods type 3 hybrid Effectiveness-Implementation trial to test EBQI-IC versus EBQI-LC versus usual care using a Concurrent Stepped Wedge design. We will use the Practical, Robust Implementation and Sustainability Model (PRISM) framework to compare and evaluate Reach, Effectiveness, Adoption, Implementation, and costs. We will then assess the maintenance/sustainment and spread of both EBPs in primary care after the 18-month implementation period. Our primary outcome will be Reach, measured by the percentage of eligible high-risk patients who received the EBP.

DISCUSSION

Our study will identify which implementation strategy is most effective overall, and under various contexts, accounting for unique barriers, facilitators, EBP characteristics, and adaptations. Ultimately this study will identify ways for primary care clinics and teams to choose implementation strategies that can improve care and outcomes for patients with complex healthcare needs.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT05050643. Registered September 9th, 2021, https://clinicaltrials.gov/study/NCT05050643 PROTOCOL VERSION: This protocol is Version 1.0 which was created on 6/3/2020.

The Rapid Implementation Feedback (RIF) report: real-time synthesis of qualitative data for proactive implementation planning and tailoring

BACKGROUND

Qualitative methods are a critical tool for enhancing implementation planning and tailoring, yet rapid turn-around of qualitative insights can be challenging in large implementation trials. The Department of Veterans Affairs-funded EMPOWER 2.0 Quality Enhancement Research Initiative (QUERI) is conducting a hybrid type 3 effectiveness-implementation trial comparing the impact of Replicating Effective Programs (REP) and Evidence-Based Quality Improvement (EBQI) as strategies for implementing three evidence-based practices (EBPs) for women Veterans. We describe the development of the Rapid Implementation Feedback (RIF) report, a pragmatic, team-based approach for the rapid synthesis of qualitative data to aid implementation planning and tailoring, as well as findings from a process evaluation of adopting the RIF report within the EMPOWER 2.0 QUERI.

METHODS

Trained qualitative staff conducted 125 semi-structured pre-implementation interviews with frontline staff, providers, and leadership across 16 VA sites between October 2021 and October 2022. High-priority topic domains informed by the updated Consolidated Framework for Implementation Research were selected in dialogue between EMPOWER 2.0 implementation and evaluation teams, and relevant key points were summarized for each interview to produce a structured RIF report, with emergent findings about each site highlighted in weekly written and verbal communications. Process evaluation was conducted to assess EMPOWER 2.0 team experiences with the RIF report across pre-implementation data collection and synthesis and implementation planning and tailoring.

RESULTS

Weekly RIF updates supported continuous EMPOWER 2.0 team communication around key findings, particularly questions and concerns raised by participating sites related to the three EBPs. Introducing the RIF report into team processes enhanced: team communication; quality and rigor of qualitative data; sensemaking around emergent challenges; understanding of site readiness; and tailoring of REP and EBQI implementation strategies. RIF report findings have facilitated rapid tailoring of implementation planning and rollout, supporting increased responsiveness to sites' needs and concerns.

CONCLUSIONS

The RIF report provides a structured strategy for distillation of time-sensitive findings, continuous team communication amid a complex multi-site implementation effort, and effective tailoring of implementation rollout in real-time. Use of the RIF report may also support trust-building by enhancing responsiveness to sites during pre- and early implementation.

TRIAL REGISTRATION

Enhancing Mental and Physical Health of Women Veterans (NCT05050266); https://clinicaltrials.gov/study/NCT05050266?term=EMPOWER%202.0&rank=1 Date of registration: 09/09/2021.

Impact of Evidence-Based Quality Improvement on Tailoring VA's Patient-Centered Medical Home Model to Women Veterans' Needs

BACKGROUND

Women Veterans' numerical minority, high rates of military sexual trauma, and gender-specific healthcare needs have complicated implementation of comprehensive primary care (PC) under VA's patient-centered medical home model, Patient Aligned Care Teams (PACT).

OBJECTIVE

We deployed an evidence-based quality improvement (EBQI) approach to tailor PACT to meet women Veterans' needs and studied its effects on women's health (WH) care readiness, team-based care, and burnout.

DESIGN

We evaluated EBQI effectiveness in a cluster randomized trial with unbalanced random allocation of 12 VAMCs (8 EBQI vs. 4 control). Clinicians/staff completed web-based surveys at baseline (2014) and 24 months (2016). We adjusted for individual-level covariates (e.g., years at VA) and weighted for non-response in difference-in-difference analyses for readiness and team-based care overall and by teamlet type (mixed-gender PC-PACTs vs. women-only WH-PACTs), as well as post-only burnout comparisons.

PARTICIPANTS

We surveyed all clinicians/staff in general PC and WH clinics.

INTERVENTION

EBQI involved structured engagement of multilevel, multidisciplinary stakeholders at network, VAMC, and clinic levels toward network-specific QI roadmaps. The research team provided QI training, formative feedback, and external practice facilitation, and support for cross-site collaboration calls to VAMC-level QI teams, which developed roadmap-linked projects adapted to local contexts.

MAIN MEASURES

WH care readiness (confidence providing WH care, self-efficacy implementing PACT for women, barriers to providing care for women, gender sensitivity); team-based care (change-readiness, communication, decision-making, PACT-related QI, functioning); burnout.

KEY RESULTS

Overall, EBQI had mixed effects which varied substantively by type of PACT. In PC-PACTs, EBQI increased self-efficacy implementing PACT for women and gender sensitivity, even as it lowered confidence. In contrast, in WH-PACTs, EBQI improved change-readiness, team-based communication, and functioning, and was associated with lower burnout.

CONCLUSIONS

EBQI effectiveness varied, with WH-PACTs experiencing broader benefits and PC-PACTs improving basic WH care readiness. Lower confidence delivering WH care by PC-PACT members warrants further study.

TRIAL REGISTRATION

The data in this paper represent results from a cluster randomized controlled trial registered in ClinicalTrials.gov (NCT02039856).

Telehealth Walking Self-Management for Individuals With Amputation: A Qualitative Study of Therapist Perspectives on Adoption

OBJECTIVE

The aim of this study was to understand therapist-identified factors influencing clinical adoption of a telehealth walking self-management intervention for individuals with lower limb amputation.

METHODS

Semi-structured focus groups were completed with actively practicing physical and occupational therapists treating populations that are medically complex. A qualitative explorative design was employed with conventional content analysis and iterative independent parallel coding using 2 analysts. Themes and subthemes were generated with a consensus building process identifying patterns and collapsing codes to represent participant perspectives.

RESULTS

Thematic saturation was met after 5 focus groups (24 therapists). Therapists were on average 34 years old and predominantly female (n = 19; 79%) physical therapists (n = 17; 71%). Three primary facilitator and barrier themes were identified for intervention adoption: system, therapist, and person. System considerations included telehealth support and interprofessional care coordination. Therapist facilitators included self-management programming that overlapped with standard of care and personalization methods. However, limited behavioral theory training was a therapist level barrier. Finally, person factors such as patient activation could influence both positively and negatively. Person facilitators included social support and barriers included the complex health condition.

CONCLUSION

System, therapist, and person facilitators and barriers must be considered to maximize the adoption of similar telehealth walking self-management interventions and prior to larger scale implementation of the current intervention for individuals with lower limb amputation.

IMPACT

A telehealth walking self-management intervention has potential impact for individuals with lower limb amputation and must be considered in terms of optimizing system, therapist, and person level facilitators and barriers to implementation.

Engaging with Rural Communities for Colorectal Cancer Screening Outreach Using Modified Boot Camp Translation

BACKGROUND

Colorectal cancer (CRC) incidence and mortality are disproportionately high among rural residents and Medicaid enrollees.

OBJECTIVES

To address disparities, we used a modified community engagement approach, Boot Camp Translation (BCT). Research partners, an advisory board, and the rural community informed messaging about CRC outreach and a mailed fecal immunochemical test program.

METHODS

Eligible rural patients (English-speaking and ages 50-74) and clinic staff involved in patient outreach participated in a BCT conducted virtually over two months. We applied qualitative analysis to BCT transcripts and field notes.

RESULTS

Key themes included: the importance of directly communicating about the seriousness of cancer, leveraging close clinic-patient relationships, and communicating the test safety, ease, and low cost.

CONCLUSIONS

Using a modified version of BCT delivered in a virtual format, we were able to successfully capture community input to adapt a CRC outreach program for use in rural settings. Program materials will be tested during a pragmatic trial to address rural CRC screening disparities.

Using Research to Transform Electronic Health Record Modernization: Advancing a VA Partnered Research Agenda to Increase Research Impacts

BACKGROUND

The U.S. Department of Veterans Affairs (VA) is undergoing an enterprise-wide transition from a homegrown electronic health record (EHR) system to a commercial off-the-shelf product. Because of the far-reaching effects of the EHR transformation through all aspects of the healthcare system, VA Health Services Research and Development identified a need to develop a research agenda that aligned with health system priorities so that work may inform evidence-based improvements in implementation processes and outcomes.

OBJECTIVE

The purpose of this paper is to report on the development of a research agenda designed to optimize the EHR transition processes and implementation outcomes in a large, national integrated delivery system.

DESIGN

We used a sequential mixed-methods approach (portfolio assessment, literature review) combined with multi-level stakeholder engagement approach that included research, informatics, and healthcare operations experts in EHR transitions in and outside the VA. Data from each stage were integrated iteratively to identify and prioritize key research areas within and across all stakeholder groups.

PARTICIPANTS

VA informatics researchers, regional VA health system leaders, national VA program office leaders, and external informatics experts with EHR transition experience.

KEY RESULTS

Through three rounds of stakeholder engagement, priority research topics were identified that focused on operations, user experience, patient safety, clinical outcomes, value realization, and informatics innovations.

CONCLUSIONS

The resulting EHR-focused research agenda was designed to guide development and conduct of rigorous research evidence aimed at providing actionable results to address the needs of operations partners, clinicians, clinical staff, patients, and other stakeholders. Continued investment in research and evaluation from both research and operations divisions of VA will be critical to executing the research agenda, ensuring its salience and value to the health system and its end users, and ultimately realizing the promise of this EHR transition.

Costs of implementing a multi-site facilitation intervention to increase access to medication treatment for opioid use disorder

BACKGROUND

The United States has been grappling with the opioid epidemic, which has resulted in over 75,000 opioid-related deaths between April 2020 and 2021. Evidence-based pharmaceutical interventions (buprenorphine, methadone, and naltrexone) are available to reduce opioid-related overdoses and deaths. However, adoption of these medications for opioid use disorder has been stifled due to individual- and system-level barriers. External facilitation is an evidence-based implementation intervention that has been used to increase access to medication for opioid use disorder (MOUD), but the implementation costs of external facilitation have not been assessed. We sought to measure the facility-level direct costs of implementing an external facilitation intervention for MOUD to provide decision makers with estimates of the resources needed to implement this evidence-based program.

METHODS

We performed a cost analysis of the pre-implementation and implementation phases, including an itemization of external facilitation team and local site labor costs. We used labor estimates from the Bureau of Labor and Statistics, and sensitivity analyses were performed using labor estimates from the Veterans Health Administration (VHA) Financial Management System general ledger data.

RESULTS

The average total costs for implementing an external facilitation intervention for MOUD per site was $18,847 (SD 6717) and ranged between $11,320 and $31,592. This translates to approximately $48 per patient with OUD. Sites with more encounters and participants with higher salaries in attendance had higher costs. This was driven mostly by the labor involved in planning and implementation activities. The average total cost of the pre-implementation and implementation activities were $1031 and $17,816 per site, respectively. In the sensitivity analysis, costs for VHA were higher than BLS estimates likely due to higher wages.

CONCLUSIONS

Implementing external facilitation to increase MOUD prescribing may be affordable depending on the payer's budget constraints. Our study reported that there were variations in the time invested at each phase of implementation and the number and type of participants involved with implementing an external facilitation intervention. Participant composition played an important role in total implementation costs, and decision makers will need to identify the most efficient and optimal number of stakeholders to involve in their implementation plans.

Evidence-Based Quality Improvement (EBQI) in the pre-implementation phase: key steps and activities

Background

Evidence-Based Quality Improvement (EBQI) involves researchers and local partners working collaboratively to support the uptake of an evidence-based intervention (EBI). To date, EBQI has not been consistently included in community-engaged dissemination and implementation literature. The purpose of this paper is to illustrate the steps, activities, and outputs of EBQI in the pre-implementation phase.

Methods

The research team applied comparative case study methods to describe key steps, activities, and outputs of EBQI across seven projects. Our approach included: (1) specification of research questions, (2) selection of cases, (3) construction of a case codebook, (4) coding of cases using the codebook, and (5) comparison of cases.

Results

The cases selected included five distinct settings (e.g., correction facilities, community pharmacies), seven EBIs (e.g., nutrition promotion curriculum, cognitive processing therapy) and five unique lead authors. Case examples include both community-embedded and clinically-oriented projects. Key steps in the EBQI process included: (1) forming a local team of partners and experts, (2) prioritizing implementation determinants based on existing literature/data, (3) selecting strategies and/or adaptations in the context of key determinants, (4) specifying selected strategies/adaptations, and (5) refining strategies/adaptations. Examples of activities are included to illustrate how each step was achieved. Outputs included prioritized determinants, EBI adaptations, and implementation strategies.

Conclusions

A primary contribution of our comparative case study is the delineation of various steps and activities of EBQI, which may contribute to the replicability of the EBQI process across other implementation research projects.

Improving primary care team functioning through evidence based quality improvement: A comparative case study

BACKGROUND

Provision of team-based primary care (PC) is associated with improved care quality, but limited empirical evidence guides practices on how to optimize team functioning. We examined how evidence-based quality improvement (EBQI) was used to change PC team processes. EBQI activities were supported by research-clinical partnerships and included multilevel stakeholder engagement, external facilitation, technical support, formative feedback, QI training, local QI development and across-site collaboration to share proven practices.

METHODS

We used a comparative case study in two VA medical centers (Sites A and B) that engaged in EBQI between 2014 and 2016. We analyzed multiple qualitative data sources: baseline and follow-up interviews with key stakeholders and provider team ("teamlet") members (n = 64), and EBQI meeting notes, reports, and supporting materials.

RESULTS

Site A's QI project entailed engaging in structured daily huddles using a huddle checklist and developing a protocol clarifying team member roles and responsibilities; Site B initiated weekly virtual team meetings that spanned two practice locations. Respondents from both sites perceived these projects as improving team structure and staffing, team communications, role clarity, staff voice and personhood, accountability, and ultimately, overall team functioning over time.

CONCLUSION

EBQI enabled local QI teams and other stakeholders to develop and implement innovations to improve PC team processes and characteristics in ways that improved teamlet members' perceptions of team functioning.

IMPLICATIONS

EBQI's multi-level approach may empower staff and facilitate innovation by and within teams, making it an effective implementation strategy for addressing unique practice-based challenges and supporting improvements in team functioning across varied clinical settings.

LEVEL OF EVIDENCE

VI.

Enhancing Mental and Physical Health of Women through Engagement and Retention (EMPOWER) 2.0 QUERI: study protocol for a cluster-randomized hybrid type 3 effectiveness-implementation trial

BACKGROUND

Women Veterans are the fastest-growing segment of Veterans Health Administration (VA) users. The VA has invested heavily in delivering care for women Veterans that is effective, comprehensive, and gender-tailored. However, gender disparities persist in cardiovascular (CV) and diabetes risk factor control, and the rate of perinatal depression among women Veterans is higher than that among civilian women. Challenges such as distance, rurality, negative perception of VA, discrimination (e.g., toward sexual and/or gender minority individuals), and harassment on VA grounds can further impede women's regular use of VA care. Enhancing Mental and Physical Health of Women through Engagement and Retention (EMPOWER) 2.0 builds on work to date by expanding access to evidence-based, telehealth preventive and mental health services for women Veterans with high-priority health conditions in rural and urban-isolation areas.

METHODS

EMPOWER 2.0 will evaluate two implementation strategies, Replicating Effective Practices (REP) and Evidence-Based Quality Improvement (EBQI), in supporting the implementation and sustainment of three evidence-based interventions (Virtual Diabetes Prevention Program; Telephone Lifestyle Coaching Program; and Reach Out, Stay Strong Essentials) focused on preventive and mental health care for women Veterans. We will conduct a mixed-methods implementation evaluation using a cluster-randomized hybrid type 3 effectiveness-implementation trial design to compare the effectiveness of REP and EBQI on improved access to and rates of engagement in telehealth preventive lifestyle and mental health services. Other outcomes of interest include (a) VA performance metrics for telehealth care delivery and related clinical outcomes; (b) progression along the Stages of Implementation Completion; (c) adaptation, sensemaking, and experiences of implementation among multilevel stakeholders; and (d) cost and return on investment. We will also generate implementation playbooks for program partners to support scale-up and spread of these and future evidence-based women's health programs and policies.

DISCUSSION

EMPOWER 2.0 provides a model for mixed-methods hybrid type 3 effectiveness-implementation trial design incorporating evaluation of performance metrics, implementation progress, stakeholder experience, and cost and return on investment, with the ultimate goal of improving access to evidence-based preventive and mental telehealth services for women Veterans with high-priority health conditions.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT05050266 . Registered on 20 September 2021.

Co-Creation in Health 4.0 as a New Solution for a New Era

Previous research on co-creation in healthcare indicates that the use of co-creation in the design process of health solutions influences their greater acceptance and adaptation, resulting in greater efficiency of health services and higher usability of implemented health solutions. Analysis of adaptation and acceptance of new technologies reveals the problem of misunderstanding and the need for more trust in modern tools implemented in the healthcare system. The remedy may be the use of co-creation in the process of developing modern medical products and services. This article's main purpose is to explore the co-creation process in Health 4.0, which is understood as the development of healthcare through the application of methods and tools of the Fourth Industrial Revolution. The literature review provided insights for an analytical framework-the co-creation matrix. We analyzed the case of the Italian medical platform Paginemediche.it to reveal the actors' engagement in co-creation. The results demonstrated different levels of engagement in improving the efficiency of implementing medical and technological solutions. Both theoretical and practical analysis proved that the co-creation matrix helps more precisely define the scale and scope of co-creation in Health 4.0.

Factors That Affect Opioid Quality Improvement Initiatives in Primary Care: Insights from Ten Health Systems

OBJECTIVE

To improve patient safety and pain management, the Centers for Disease Control and Prevention (CDC) released the Guideline for Prescribing Opioids for Chronic Pain (CDC Guideline). Recognizing that issuing a guideline alone is insufficient for transforming practice, CDC supported an Opioid Quality Improvement (QI) Collaborative, consisting of 10 health care systems that represented more than 120 practices across the United States. The research team identified factors related to implementation success using domains described by the integrated Promoting Action on Research Implementation in Health Services (iPARIHS) implementation science framework.

METHODS

Data from interviews, notes from check-in calls, and documents provided by systems were used. The researchers collected data throughout the project through interviews, meeting notes, and documents.

RESULTS

The iPARIHS framework was used to identify factors that affected implementation related to the context, innovation (implementing recommendations from the CDC Guideline), recipient (clinicians), and facilitation (QI team). Contextual characteristics were at the clinic, health system, and broader external context, including staffing and leadership support, previous QI experience, and state laws. Characteristics of the innovation were its adaptability and challenges operationalizing the measures. Recipient characteristics included belief in the importance of the innovation but challenges engaging in the initiative. Finally, facilitation characteristics driving differential outcomes included staffing and available time of the QI team, the ability to make changes, and experience with QI.

CONCLUSION

As health care systems continue to implement the CDC Guideline, these insights can advance successful implementation efforts by describing common implementation challenges and identifying strategies to prepare for and overcome them.

Healthcare quality measures in implementation research: advantages, risks and lessons learned

Implementation studies evaluate strategies to move evidence-based practices into routine clinical practice. Often, implementation scientists use healthcare quality measures to evaluate the integration of an evidence-based clinical practice into real-world healthcare settings. Healthcare quality measures have standardized definitions and are a method to operationalize and monitor guideline-congruent care. Implementation scientists can access existing data on healthcare quality measures through various sources (e.g. operations-calculated), or they can calculate the measures directly from healthcare claims and administrative data (i.e. researcher-calculated). Implementation scientists need a better understanding of the advantages and disadvantages of these methods of obtaining healthcare quality data for designing, planning and executing an implementation study. The purpose of this paper is to describe the advantages, risks and lessons learned when using operations- versus researcher-calculated healthcare quality measures in site selection, implementation monitoring and implementation outcome evaluation. A key lesson learned was that relying solely on operations-calculated healthcare quality measures during an implementation study poses risks to site selection, accurate feedback on implementation progress to stakeholders, and the integrity of study results. A possible solution is using operations-calculated quality measures for monitoring of evidence-based practice uptake and researcher-calculated measures for site section and outcomes evaluation. This approach provides researchers greater control over the data and consistency of the measurement from site selection to outcomes evaluation while still retaining measures that are familiar and understood by key stakeholders whom implementation scientists need to engage in practice change efforts.

Effective engagement of a stakeholder advisory board in severe mental illness (SMI) research: A case study of a clinical trial to improve adherence among people with SMI and hypertension

Objective

Poor adherence to antihypertensive medication occurs in 50-80% of patients. An ongoing randomized controlled trial (RCT) is evaluating a personalized mobile-health intervention in poorly adherent hypertensive persons with bipolar disorder. To enhance efficacy, the ongoing trial elicited guidance from a Stakeholder Advisory Board (SAB) comprised of patients, family members, clinicians, and health system administrators. Our goal is to describe the formation, role, decision-making process, and key contributions of the SAB as a means of demonstrating meaningful community engagement in mental health research.

Methods

Using models and measures from the field of implementation science, eleven SAB members convened across three meetings followed by quantitative surveys that assessed SAB member satisfaction and engagement during the meeting.

Results

Significant suggestions from the SAB included 1) expanding inclusion/exclusion criteria, and 2) operationalizing remote implementation of the RCT. Primary study implementation challenges identified by the SAB were 1) participant difficulty engaging in the mHealth intervention, and 2) identification of procedures for monitoring participant adherence to the RCT protocol and contacting under-engaged participants. Quantitative surveys indicated that all SAB members believed that the objectives of the meetings were clear, perceived that they were able to participate in the discussions, and that they were heard.

Conclusions

Increasing evidence demonstrates the feasibility of engaging with SABs in clinical research and that this process improves intervention design, increases participant engagement, reduces mental health-related stigma, and produces more effective implementation strategies. We encourage future investigators to use an implementation science framework in partnership with SABs to refine their proposed interventions and improve clinical outcomes.

Advancing Pharmacological Treatments for Opioid Use Disorder (ADaPT-OUD): an Implementation Trial in Eight Veterans Health Administration Facilities

BACKGROUND

Identifying effective strategies to improve access to medication treatments for opioid use disorder (MOUD) is imperative. Within the Veterans Health Administration (VHA), provision of MOUD varies significantly, requiring development and testing of implementation strategies that target facilities with low provision of MOUD.

OBJECTIVE

Determine the effectiveness of external facilitation in increasing the provision of MOUD among VHA facilities with low baseline provision of MOUD compared to matched controls.

DESIGN

Pre-post, block randomized study designed to compare facility-level outcomes in a stratified sample of eligible facilities. Four blocks (two intervention facilities in each) were defined by median splits of both the ratio of patients with OUD receiving MOUD and number of patients with OUD not currently receiving MOUD (i.e., number of actionable patients). Intervention facilities participated in a 12-month implementation intervention.

PARTICIPANTS

VHA facilities in the lowest quartile of MOUD provision (35 facilities), eight of which were randomly assigned to participate in the intervention (two per block) with twenty-seven serving as matched controls by block.

INTERVENTION

External facilitation included assessment of local barriers/facilitators, formation of a local implementation team, a site visit for action planning and training/education, cross-facility quarterly calls, monthly coaching calls, and consultation.

MAIN MEASURES

Pre- to post-change in the facility-level ratio of patients with an OUD diagnosis receiving MOUD compared to control facilities.

KEY RESULTS

Intervention facilities significantly increased the ratio of patients with OUD receiving MOUD from an average of 18% at baseline to 30% 1 year later, with an absolute difference of 12% (95% confidence interval [CI]: 6.6%, 17.0%). The difference in differences between intervention and control facilities was 3.0% (95% CI: - 0.2%. 6.7%). The impact of the intervention varied by block, with smaller, less complex facilities more likely to outperform matched controls.

CONCLUSIONS

Intensive external facilitation improved the adoption of MOUD in most low-performing facilities and may enhance adoption beyond other interventions less tailored to individual facility contexts.

Assessing Vietnamese American patient perspectives on population genetic testing in primary care: A community-engaged approach

Achieving health equity in precision medicine remains a critical challenge because of the continued underrepresentation of non-white populations in research and barriers to genetic services. The goal of this study was to explore Vietnamese American (VA) participant views toward incorporating genetics in routine healthcare to better serve the local VA community within an integrated health system offering primary care-based population genetic testing to adults for conditions that could be prevented or mitigated when detected early. We conducted semi-structured interviews from August–September 2021, with 22 individuals receiving primary care who self-identified as Vietnamese or VA, and employed rapid qualitative analysis (RQA) to identify key concepts. Community research team members participated in study design, data collection, RQA, and reporting. Findings from the interviews revealed that several participant perceived challenges to genetic testing, which included lack of information, fear of results impact, cost, and privacy concerns. Participants suggested various ways to overcome some of these barriers, such as decreasing the cost of testing, receiving information from a trusted physician, using preferred education strategies in the community, and having convenient access to testing. Study participants also shared a variety of trusted sources they would seek out for advice on genetic testing. This study with VAs identified barriers, facilitators, and messengers to offering genetic testing in a local healthcare context and demonstrated how community-engaged research coupled with RQA is a promising approach for healthcare institutions as they identify needs and tailor strategies for implementing population genetic screening programs in local ethnic communities.

Leveraging multistakeholder engagement to develop an implementation blueprint for a brief trauma-focused cognitive behavioral therapy in primary care

Objective: The implementation of evidence-based treatments (EBTs) to address posttraumatic stress disorder (PTSD) is a public health priority. Successful EBT implementation requires effective collaboration between multiple stakeholder groups, including hospital leaders, providers, and patients, to build buy-in for this effort. Method: We describe our implementation science approach to meaningful stakeholder engagement, as part of a hybrid type I effectiveness-implementation trial of Skills Training in Affective and Interpersonal Regulation for PTSD treatment in primary care (STAIR-PC) at a large safety net hospital. We used primary care and patient community advisory boards (CABs) to interpret key informant interviews and identify strategies to adapt the intervention to ensure fit with the primary care setting. We documented our stakeholder engagement methodology through comprehensive field notes and minutes from CAB meetings, detailing the focus of meetings, suggestions for intervention and delivery adaptations, decision-making processes, and how disagreements about adaptations between stakeholders were resolved. To support replicability, we specify and operationalize implementation strategies to be used across each implementation phase of the trial. Results: Key strategies involved a) ensuring that research questions are relevant to both patients and clinical providers; b) tailoring interventions that are flexible and adaptable to the needs of the local setting; c) continuous engagement of patients and providers throughout the implementation process; and d) building mutual respect, trust, and credibility between the research team, various provider groups, and patients. Conclusions: Our approach to engaging stakeholders informed an implementation blueprint to guide implementation of EBTs for PTSD in safety net hospital primary care clinics. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Differences in Burnout and Intent to Leave Between Women's Health and General Primary Care Providers in the Veterans Health Administration

BACKGROUND

Although they are a minority of patients served by the Veterans Health Administration (VHA), women Veterans comprise a fast-growing segment of these patients and have unique clinical needs. Women's health primary care providers (WH-PCPs) are specially trained and designated to provide care for women Veterans. Prior work has demonstrated that WH-PCPs deliver better preventative care and have more satisfied patients than PCPs without the WH designation. However, due to unique clinical demands or other factors, WH-PCPs may experience more burnout and intent to leave practice than general PCPs in the VHA.

OBJECTIVE

To examine differences in burnout and intent to leave practice among WH and general PCPs in the VHA.

DESIGN

Multi-level logistic regression analysis of three cross-sectional waves of PCPs within the VHA using the national All Employee Survey and practice data (2017-2019). We modeled outcomes of burnout and intent to leave practice as a function of WH provider designation, gender, and other demographics and practice characteristics, such as support staff ratio, panel size, and setting.

PARTICIPANTS

A total of 7903 primary care providers (5152 general PCPs and 2751 WH-PCPs; response rates: 63.9%, 65.7%, and 67.5% in 2017, 2018, and 2019, respectively).

MAIN MEASURES

Burnout and intent to leave practice.

KEY RESULTS

WH-PCPs were more burned out than general PCPs (unadjusted: 55.0% vs. 46.9%, p<0.001; adjusted: OR=1.29, 95% confidence interval [CI] 1.10-1.55) but did not have a higher intention to leave (unadjusted: 33.4% vs. 32.1%, p=0.27; adjusted: OR=1.07, CI 0.81-1.41). WH-PCPs with intentions to leave were more likely to select the response option of "job-related (e.g., type of work, workload, burnout, boredom)" as their primary reason to leave.

CONCLUSIONS

Burnout is higher among WH-PCPs compared to general PCPs, even after accounting for provider and practice characteristics. More research on causes of and solutions for these differences in burnout is needed.

Stakeholder Perspectives of Attributes and Features of Context Relevant to Knowledge Translation in Health Settings: A Multi-Country Analysis

BACKGROUND

Context is recognized as important to successful knowledge translation (KT) in health settings. What is meant by context, however, is poorly understood. The purpose of the current study was to elicit tacit knowledge about what is perceived to constitute context by conducting interviews with a variety of health system stakeholders internationally so as to compile a comprehensive list of contextual attributes and their features relevant to KT in healthcare.

METHODS

A descriptive qualitative study design was used. Semi-structured interviews were conducted with health system stakeholders (change agents/KT specialists and KT researchers) in four countries: Australia, Canada, the United Kingdom, and the United States. Interview transcripts were analyzed using inductive thematic content analysis in four steps: (1) selection of utterances describing context, (2) coding of features of context, (3) categorizing of features into attributes of context, (4) comparison of attributes and features by: country, KT experience, and role.

RESULTS

A total of 39 interviews were conducted. We identified 66 unique features of context, categorized into 16 attributes. One attribute, Facility Characteristics, was not represented in previously published KT frameworks. We found instances of all 16 attributes in the interviews irrespective of country, level of experience with KT, and primary role (change agent/KT specialist vs. KT researcher), revealing robustness and transferability of the attributes identified. We also identified 30 new context features (across 13 of the 16 attributes).

CONCLUSION

The findings from this study represent an important advancement in the KT field; we provide much needed conceptual clarity in context, which is essential to the development of common assessment tools to measure context to determine which context attributes and features are more or less important in different contexts for improving KT success.

Community engagement to improve access to healthcare: a comparative case study to advance implementation science for transgender health equity

BACKGROUND

Recent calls to action have been made for Implementation Science to attend to health inequities at the intersections of race, gender, and social injustice in the United States. Transgender people, particularly Black and Latina transgender women, experience a range of health inequities and social injustices. In this study, we compared two processes of transgender community engagement in Los Angeles and in Chicago as an implementation strategy to address inequitable access to care; we adapted and extended the Exploration Planning Implementation and Sustainment (EPIS) framework for transgender health equity.

METHODS

A comparative case method and the EPIS framework were used to examine parallel implementation strategies of transgender community engagement to expand access to care. To foster conceptual development and adaptation of EPIS for trans health equity, the comparative case method required detailed description, exploration, and analyses of the community-engagement processes that led to different interventions to expand access. In both cities, the unit of analysis was a steering committee made up of local transgender and cisgender stakeholders.

RESULTS

Both steering committees initiated their exploration processes with World Café-style, transgender community-engaged events in order to assess community needs and structural barriers to healthcare. The steering committees curated activities that amplified the voices of transgender community members among stakeholders, encouraging more effective and collaborative ways to advance transgender health equity. Based on analysis and findings from the Los Angeles town hall, the steering committee worked with a local medical school, extending the transgender medicine curriculum, and incorporating elements of transgender community-engagement. The Chicago steering committee determined from their findings that the most impactful intervention on structural racism and barriers to healthcare access would be to design and pilot an employment program for Black and Latina transgender women.

CONCLUSION

In Los Angeles and Chicago, transgender community engagement guided implementation processes and led to critical insights regarding specific, local barriers to healthcare. The steering committee itself represented an important vehicle for individual-, organizational-, and community-level relationship and capacity building. This comparative case study highlights key adaptations of EPIS toward the formation of an implementation science framework for transgender health equity.

Promising Strategies to Support COVID-19 Vaccination of Healthcare Personnel: Qualitative Insights from the VHA National Implementation

BACKGROUND

In August 2021, up to 30% of Americans were uncertain about taking the COVID-19 vaccine, including some healthcare personnel (HCP).

OBJECTIVE

Our objective was to identify barriers and facilitators of the Veterans Health Administration (VHA) HCP vaccination program.

DESIGN

We conducted key informant interviews with employee occupational health (EOH) providers, using snowball recruitment.

PARTICIPANTS

Participants included 43 VHA EOH providers representing 29 of VHA's regionally diverse healthcare systems.

APPROACH

Thematic analysis elucidated 5 key themes and specific strategies recommended by EOH.

KEY RESULTS

Implementation themes reflected logistics of distribution (supply), addressing any vaccine concerns or hesitancy (demand), and learning health system strategies/approaches for shared learnings. Specifically, themes included the following: (1) use interdisciplinary task forces to leverage diverse skillsets for vaccine implementation; (2) invest in processes and align resources with priorities, including creating detailed processes, addressing time trade-offs for personnel involved in vaccine clinics by suspending everything non-essential, designating process/authority to shift personnel where needed, and proactively involving leaders to support resource allocation/alignment; (3) expect and accommodate vaccine buy-in occurring over time: prepare for some HCP's slow buy-in, align buy-in facilitation with identities and motivation, and encourage word-of-mouth and hyper-local testimonials; (4) overcome misinformation with trustworthy communication: tailor communication to individuals and address COVID vaccines "in every encounter," leverage proactive institutional messaging to reinforce information, and invite bi-directional conversations about any vaccine concerns. A final overarching theme focused on learning health system needs and structures: (5) use existing and newly developed communication channels to foster shared learning across teams and sites.

CONCLUSIONS

Expecting deliberation allows systems to prepare for complex distribution logistics (supply) and make room for conversations that are trustworthy, bi-directional, and identity aligned (demand). Ideally, organizations provide time for conversations that address individual concerns, foster bi-directional shared decision-making, respect HCP beliefs and identities, and emphasize shared identities as healthcare providers.

Applying systems approaches to stakeholder and community engagement and knowledge mobilisation in youth mental health system modelling

Background

There is a significant push to change the trajectory of youth mental ill-health and suicide globally. Ensuring that young people have access to services that meet their individual needs and are easily accessible is a priority. Genuine stakeholder engagement in mental health system design is critical to ensure that system strengthening is likely to be successful within these complex environments. There is limited literature describing engagement processes undertaken by research teams in mental health program implementation and planning. This protocol describes the methods that will be used to engage local communities using systems science methods to mobilize knowledge and action to strengthen youth mental health services.

Methods

Using participatory action research principles, the research team will actively engage with local communities to ensure genuine user-led participatory systems modelling processes and enhance knowledge mobilisation within research sites. Ensuring that culturally diverse and Aboriginal and Torres Strait Islander community voices are included will support this process. A rigorous site selection process will be undertaken to ensure that the community is committed and has capacity to actively engage in the research activities. Stakeholder engagement commences from the site selection process with the aim to build trust between researchers and key stakeholders. The research team will establish a variety of engagement resources and make opportunities available to each site depending on their local context, needs and audiences they wish to target during the process.

Discussion

This protocol describes the inclusive community engagement and knowledge mobilization process for the Right care, first time, where you live research Program. This Program will use an iterative and adaptive approach that considers the social, economic, and political context of each community and attempts to maximise research engagement. A theoretical framework for applying systems approaches to knowledge mobilization that is flexible will enable the implementation of a participatory action research approach. This protocol commits to a rigorous and genuine stakeholder engagement process that can be applied in mental health research implementation.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13033-022-00530-1.

Increasing Engagement of Women Veterans in Health Research

BACKGROUND

Meaningful engagement of patients in health research has the potential to increase research impact and foster patient trust in healthcare. For the past decade, the Veterans Health Administration (VA) has invested in increasing Veteran engagement in research.

OBJECTIVE

We sought the perspectives of women Veterans, VA women's health primary care providers (WH-PCPs), and administrators on barriers to and facilitators of health research engagement among women Veterans, the fastest growing subgroup of VA users.

DESIGN

Semi-structured qualitative telephone interviews were conducted from October 2016 to April 2018.

PARTICIPANTS

Women Veterans (N=31), WH-PCPs (N=22), and administrators (N=6) were enrolled across five VA Women's Health Practice-Based Research Network sites.

APPROACH

Interviews were audio-recorded and transcribed. Consensus-based coding was conducted by two expert analysts.

KEY RESULTS

All participants endorsed the importance of increasing patient engagement in women's health research. Women Veterans expressed altruistic motives as a personal determinant for research engagement, and interest in driving women's health research forward as a stakeholder or research partner. Challenges to engagement included lack of awareness about opportunities, distrust of research, competing priorities, and confidentiality concerns. Suggestions to increase engagement include utilizing VA's patient-facing portals of the electronic health record for outreach, facilitating "warm hand-offs" between researchers and clinic staff, developing an accessible research registry, and communicating the potential research impact for Veterans.

CONCLUSIONS

Participants expressed support for increasing women Veterans' engagement in women's health research and identified feasible ways to foster and implement engagement of women Veterans. Given the unique healthcare needs of women Veterans, engaging them in research could translate to improved care, especially for future generations. Knowledge about how to improve women Veterans' research engagement can inform future VA policy and practice for more meaningful interventions and infrastructure.

Participatory Methods for Systems Modeling of Youth Mental Health: Implementation Protocol

Background

Despite significant investment, mental health issues remain a leading cause of death among young people globally. Sophisticated decision analysis methods are needed to better understand the dynamic and multisector drivers of youth mental health. System modeling can help explore complex issues such as youth mental health and inform strategies to effectively respond to local needs and achieve lasting improvements. The advantages of engaging stakeholders in model development processes have long been recognized; however, the methods for doing so are often not well-described.

Objective

This paper aims to describe the participatory procedures that will be used to support systems modeling for national multisite implementation. The Right Care, First Time, Where You Live research program will focus on regional youth mental health applications of systems modeling in 8 different sites across Australia.

Methods

The participatory model development approach involves an iterative process of engaging with a range of participants, including people with lived experience of mental health issues. Their knowledge of the local systems, pathways, and drivers is combined with the academic literature and data to populate the models and validate their structure. The process centers around 3 workshops where participants interact and actively engage in group model-building activities to define, refine, and validate the systems models. This paper provides a detailed blueprint for the implementation of this process for mental health applications.

Results

The participatory modeling methods described in this paper will be implemented at 2 sites per year from 2022 to 2025. The 8 selected sites have been chosen to capture variations in important factors, including determinants of mental health issues and access to services. Site engagement commenced in August 2021, and the first modeling workshops are scheduled to commence in February 2022.

Conclusions

Mental health system decision makers require tools to help navigate complex environments and leverage interdisciplinary problem-solving. Systems modeling can mobilize data from diverse sources to explore a range of scenarios, including the impact of interventions in different combinations and contexts. Involving stakeholders in the model development process ensures that the model findings are context-relevant and fit-for-purpose to inform decision-making.

International Registered Report Identifier (IRRID)

PRR1-10.2196/32988

Advancing Qualitative Health Research Approaches in Applied Nutrition Research

Applied health research methods are evolving to meet the demands of increasingly complex health research needs. Qualitative health research, focused on individual perspectives of health, wellness, illness, and recovery, has emerged as a unique sub-discipline of this field. With distinct foci, methods, and rules, qualitative health research has the potential to answer applied health research questions to inform practice, education, and policy. Despite this potential, there are challenges to the application of qualitative health research methods in nutrition and dietetics research. These include limited training and mentorship availability for the rigorous application of these methods as well as misaligned goals between the traditional social science-based qualitative approaches and emerging applied nutrition science needs. Recognizing these limitations, this review aims to provide guidance to the nutrition scientist conducting applied qualitative health research. Using nutrition and dietetic examples from the literature, this review defines qualitative health research and advances the EPPiC framework as a tool for constructing structured overarching research questions and introduces four qualitative health research designs (qualitative description, interpretive description, case study, and focused ethnography) relevant to applied nutrition science. This includes guidance on defining the sample, identifying strategies for data collection, analytic techniques, and data reporting. This article is protected by copyright. All rights reserved.

Strategies for effective implementation and scale-up of a multi-level co-designed men's health initiative "Sheds for Life" in Irish Men's Sheds

Sheds for Life is a gender-specific tailored men's health initiative engaging "hard-to-reach" men in the Men's Shed setting in Ireland. It is implemented by multiple stakeholders at individual, provider, organization and systems level and thus multiple contextual factors influence its scalability. This research used established implementation science frameworks to guide participatory research approaches that captured the process and identified facilitators of and barriers to implementation and scale-up. Active recruitment, co-design processes, leadership and stakeholder engagement emerged as key facilitators of implementation. Prominent barriers were institutional capacity and funding. Acceptability, adoption and appropriateness of the initiative were high among stakeholders with sustainability largely contingent on funding and staff resources. Findings make a valuable contribution to knowledge by capturing the process involved in the implementation of a complex multi-level men's health intervention. It provides a "how to" guide of strategies to engage hard-to-reach men with health promotion, the operationalization and application of implementation frameworks in community-based health promotion, and the implementation of health promotion in Men's Sheds. Documented barriers and facilitators that impact implementation of a community-based men's health program are rare and provide a valuable blueprint for practitioners, researchers and policy makers in the field.

Protecting the healthcare workforce during COVID-19: a qualitative needs assessment of employee occupational health in the US national Veterans Health Administration

OBJECTIVE

Early in the COVID-19 pandemic, US Veterans Health Administration (VHA) employee occupational health (EOH) providers were tasked with assuming a central role in coordinating employee COVID-19 screening and clearance for duty, representing entirely novel EOH responsibilities. In a rapid qualitative needs assessment, we aimed to identify learnings from the field to support the vastly expanding role of EOH providers in a national healthcare system.

METHODS

We employed rapid qualitative analysis of key informant interviews in a maximal variation sample on the parameters of job type, rural versus urban and provider gender. We interviewed 21 VHA EOH providers between July and December 2020. This sample represents 15 facilities from diverse regions of the USA (large, medium and small facilities in the Mid-Atlantic; medium sites in the South; large facilities in the West and Pacific Northwest).

RESULTS

Five interdependent needs included: (1) infrastructure to support employee population management, including tools that facilitate infection control measures such as contact tracing (eg, employee-facing electronic health records and coordinated databases); (2) mechanisms for information sharing across settings (eg, VHA listserv), especially for changing policy and protocols; (3) sufficiently resourced staffing using detailing to align EOH needs with human resource capital; (4) connected and resourced local and national leaders; and (5) strategies to support healthcare worker mental health.Our identified facilitators for EOH assuming new challenging and dynamically changing roles during COVID-19 included: (A) training or access to expertise; (B) existing mechanisms for information sharing; (C) flexible and responsive staffing; and (D) leveraging other institutional expertise not previously affiliated with EOH (eg, chaplains to support bereavement).

CONCLUSIONS

Our needs assessment highlights local and system level barriers and facilitators of EOH assuming expanded roles during COVID-19. Integrating changes both within and across systems and with alignment of human capital will enable EOH preparedness for future challenges.

Effects of consumers and health providers working in partnership on health services planning, delivery and evaluation

BACKGROUND

Health services have traditionally been developed to focus on specific diseases or medical specialties. Involving consumers as partners in planning, delivering and evaluating health services may lead to services that are person-centred and so better able to meet the needs of and provide care for individuals. Globally, governments recommend consumer involvement in healthcare decision-making at the systems level, as a strategy for promoting person-centred health services. However, the effects of this 'working in partnership' approach to healthcare decision-making are unclear. Working in partnership is defined here as collaborative relationships between at least one consumer and health provider, meeting jointly and regularly in formal group formats, to equally contribute to and collaborate on health service-related decision-making in real time. In this review, the terms 'consumer' and 'health provider' refer to partnership participants, and 'health service user' and 'health service provider' refer to trial participants. This review of effects of partnership interventions was undertaken concurrently with a Cochrane Qualitative Evidence Synthesis (QES) entitled Consumers and health providers working in partnership for the promotion of person-centred health services: a co-produced qualitative evidence synthesis.

OBJECTIVES

To assess the effects of consumers and health providers working in partnership, as an intervention to promote person-centred health services.

SEARCH METHODS

We searched the CENTRAL, MEDLINE, Embase, PsycINFO and CINAHL databases from 2000 to April 2019; PROQUEST Dissertations and Theses Global from 2016 to April 2019; and grey literature and online trial registries from 2000 until September 2019.

SELECTION CRITERIA

We included randomised controlled trials (RCTs), quasi-RCTs, and cluster-RCTs of 'working in partnership' interventions meeting these three criteria: both consumer and provider participants meet; they meet jointly and regularly in formal group formats; and they make actual decisions that relate to the person-centredness of health service(s).

DATA COLLECTION AND ANALYSIS

Two review authors independently screened most titles and abstracts. One review author screened a subset of titles and abstracts (i.e. those identified through clinical trials registries searches, those classified by the Cochrane RCT Classifier as unlikely to be an RCT, and those identified through other sources). Two review authors independently screened all full texts of potentially eligible articles for inclusion. In case of disagreement, they consulted a third review author to reach consensus. One review author extracted data and assessed risk of bias for all included studies and a second review author independently cross-checked all data and assessments. Any discrepancies were resolved by discussion, or by consulting a third review author to reach consensus. Meta-analysis was not possible due to the small number of included trials and their heterogeneity; we synthesised results descriptively by comparison and outcome. We reported the following outcomes in GRADE 'Summary of findings' tables: health service alterations; the degree to which changed service reflects health service user priorities; health service users' ratings of health service performance; health service users' health service utilisation patterns; resources associated with the decision-making process; resources associated with implementing decisions; and adverse events.

MAIN RESULTS

We included five trials (one RCT and four cluster-RCTs), with 16,257 health service users and more than 469 health service providers as trial participants. For two trials, the aims of the partnerships were to directly improve the person-centredness of health services (via health service planning, and discharge co-ordination). In the remaining trials, the aims were indirect (training first-year medical doctors on patient safety) or broader in focus (which could include person-centredness of health services that targeted the public/community, households or health service delivery to improve maternal and neonatal mortality). Three trials were conducted in high income-countries, one was in a middle-income country and one was in a low-income country. Two studies evaluated working in partnership interventions, compared to usual practice without partnership (Comparison 1); and three studies evaluated working in partnership as part of a multi-component intervention, compared to the same intervention without partnership (Comparison 2). No studies evaluated one form of working in partnership compared to another (Comparison 3). The effects of consumers and health providers working in partnership compared to usual practice without partnership are uncertain: only one of the two studies that assessed this comparison measured health service alteration outcomes, and data were not usable, as only intervention group data were reported. Additionally, none of the included studies evaluating this comparison measured the other primary or secondary outcomes we sought for the 'Summary of findings' table. We are also unsure about the effects of consumers and health providers working in partnership as part of a multi-component intervention compared to the same intervention without partnership. Very low-certainty evidence indicated there may be little or no difference on health service alterations or health service user health service performance ratings (two studies); or on health service user health service utilisation patterns and adverse events (one study each). No studies evaluating this comparison reported the degree to which health service alterations reflect health service user priorities, or resource use. Overall, our confidence in the findings about the effects of working in partnership interventions was very low due to indirectness, imprecision and publication bias, and serious concerns about risk of selection bias; performance bias, detection bias and reporting bias in most studies.

AUTHORS' CONCLUSIONS

The effects of consumers and providers working in partnership as an intervention, or as part of a multi-component intervention, are uncertain, due to a lack of high-quality evidence and/or due to a lack of studies. Further well-designed RCTs with a clear focus on assessing outcomes directly related to partnerships for patient-centred health services are needed in this area, which may also benefit from mixed-methods and qualitative research to build the evidence base.

Applied Rapid Qualitative Analysis to Develop a Contextually Appropriate Intervention and Increase the Likelihood of Uptake

BACKGROUND

Rapid approaches to collecting and analyzing qualitative interview data can accelerate discovery timelines and intervention development while maintaining scientific rigor. We describe the application of these methods to a program designed to improve care coordination between the Veterans Health Administration (VHA) and community providers.

METHODS

Care coordination between VHA and community providers can be challenging in rural areas. The Telehealth-based Coordination of Non-VHA Care (TECNO Care) intervention was designed to improve care coordination among VHA and community providers. To ensure contextually appropriate implementation of TECNO Care, we conducted preimplementation interviews with veterans, VHA administrators, and VHA and community providers involved in community care. Using both a rapid approach and qualitative analysis, an interviewer and 1-2 note-taker(s) conducted interviews.

RESULTS

Over 5 months, 18 stakeholders were interviewed and we analyzed these data to identify how best to deliver TECNO Care. Responses relevant to improving care coordination include health system characteristics; target population; metrics and outcomes; challenges with the current system; and core components. Veterans who frequently visit VHA or community providers and are referred for additional services are at risk for poor outcomes and may benefit from additional care coordination. Using these data, we designed TECNO Care to include information on VHA services and processes, assist in the timely completion of referrals, and facilitate record sharing.

CONCLUSION

Rapid qualitative analysis can inform near real-time intervention development and ensure relevant content creation while setting the stage for stakeholder buy-in. Rigorous and timely analyses support the delivery of contextually appropriate, efficient, high-value patient care.

Patient and surgeon experiences with video visits in plastic surgery-toward a data-informed scheduling triage tool

BACKGROUND

Coronavirus disease 2019 provided the impetus for unprecedented adoption of telemedicine. This study aimed to understand video visit adoption by plastic surgery providers; and patient and surgeon perceptions about its efficacy, value, accessibility, and long-term viability. A secondary aim was to develop the proposed 'Triage Tool for Video Visits in Plastic Surgery' to help determine visit video eligibility.

METHODS

This mixed-methods evaluation assessed provider-level scheduling data from the Division of Plastic and Reconstructive Surgery at Stanford Health Care to quantify telemedicine adoption and semi-structured phone interviews with patients (n = 20) and surgeons (n = 10) to explore stakeholder perspectives on video visits.

RESULTS

During the 13-week period after the local stay-at-home orders due to coronavirus disease 2019, 21.4% of preoperative visits and 45.5% of postoperative visits were performed via video. Video visits were considered acceptable by patients and surgeons in plastic surgery in terms of quality of care but were limited by the inability to perform a physical examination. Interviewed clinicians reported that long-term viability needs to be centered around technology (eg, connection, video quality, etc) and physical examinations. Our findings informed a proposed triage tool to determine the appropriateness of video visits for individual patients that incorporates visit type, anesthesia, case, surgeon's role, and patient characteristics.

CONCLUSION

Video technology has the potential to facilitate and improve preoperative and postoperative patient care in plastic surgery but the following components are needed: patient education on taking high-quality photos; standardized clinical guidelines for conducting video visits; and an algorithm-assisted triage tool to support scheduling.

Implementation Evaluation of a Complex Intervention to Improve Timeliness of Care for Veterans with Transient Ischemic Attack

BACKGROUND

The Protocol-guided Rapid Evaluation of Veterans Experiencing New Transient Neurologic Symptoms (PREVENT) program was designed to address systemic barriers to providing timely guideline-concordant care for patients with transient ischemic attack (TIA).

OBJECTIVE

We evaluated an implementation bundle used to promote local adaptation and adoption of a multi-component, complex quality improvement (QI) intervention to improve the quality of TIA care Bravata et al. (BMC Neurology 19:294, 2019).

DESIGN

A stepped-wedge implementation trial with six geographically diverse sites.

PARTICIPANTS

The six facility QI teams were multi-disciplinary, clinical staff.

INTERVENTIONS

PREVENT employed a bundle of key implementation strategies: team activation; external facilitation; and a community of practice. This strategy bundle had direct ties to four constructs from the Consolidated Framework for Implementation Research (CFIR): Champions, Reflecting & Evaluating, Planning, and Goals & Feedback.

MAIN MEASURES

Using a mixed-methods approach guided by the CFIR and data matrix analyses, we evaluated the degree to which implementation success and clinical improvement were associated with implementation strategies. The primary outcomes were the number of completed implementation activities, the level of team organization and > 15 points improvement in the Without Fail Rate (WFR) over 1 year.

KEY RESULTS

Facility QI teams actively engaged in the implementation strategies with high utilization. Facilities with the greatest implementation success were those with central champions whose teams engaged in planning and goal setting, and regularly reflected upon their quality data and evaluated their progress against their QI plan. The strong presence of effective champions acted as a pre-condition for the strong presence of Reflecting & Evaluating, Goals & Feedback, and Planning (rather than the other way around), helping to explain how champions at the +2 level influenced ongoing implementation.

CONCLUSIONS

The CFIR-guided bundle of implementation strategies facilitated the local implementation of the PREVENT QI program and was associated with clinical improvement in the national VA healthcare system.

TRIAL REGISTRATION

clinicaltrials.gov: NCT02769338.

Increasing Access to Medications for Opioid Use Disorder and Complementary and Integrative Health Services in Primary Care

OBJECTIVES

Evidence-based therapies for opioid use disorder (OUD) and chronic pain, such as medications for OUD (MOUD) and complementary and integrative health (CIH; e.g., acupuncture and meditation) therapies, exist. However, their adoption has been slow, particularly in primary care, due to numerous implementation challenges. We sought to expand the use of MOUD and CIH within primary care by using an evidence-based quality improvement (EBQI) implementation strategy.

METHODS

We used EBQI to engage two facilities in the Veterans Health Administration (VHA) from June 2018 to September 2019. EBQI included multilevel stakeholder engagement, with external facilitators providing technical support, practice facilitation, and routine data feedback. We established a quality improvement (QI) team at each facility with diverse stakeholders (e.g., primary care, addiction, pain, nursing, pharmacy). We met monthly with regional stakeholders to address implementation barriers. We also convened an advisory board to ensure alignment with national priorities.

RESULTS

Pre-implementation interviews indicated facility-level and provider-level barriers to prescribing buprenorphine, including strong primary care provider resistance. Both facilities developed action plans. They both conducted educational meetings (e.g., Grand Rounds, MOUD waiver trainings). Facility A also offered clinical preceptorships for newly trained primary care prescribers. Facility B used mass media and mailings to educate patients about MOUD and CIH options and dashboards to identify potential candidates for MOUD. After 15 months, both facilities increased their OUD treatment rates to the ≥ 90th percentile of VHA medical centers nationally. Exit interviews indicated an attitudinal shift in MOUD delivery in primary care. Stakeholders valued the EBQI process, particularly cross-site collaboration.

IMPLICATIONS

Despite initial implementation barriers, we effectively engaged stakeholders using EBQI strategies. Local QI teams used an assortment of QI interventions and developed tools to catapult their facilities to among the highest performers in VHA OUD treatment.

IMPACTS

EBQI is an effective strategy to partner with stakeholders to implement MOUD and CIH therapies.

Dynamic waitlisted design for evaluating a randomized trial of evidence-based quality improvement of comprehensive women's health care implementation in low-performing VA facilities

Background

Women’s Health Services (WHS) in the Veterans Health Administration (VA) has long partnered with VA researchers to evaluate how VA care is organized for women veterans. This partnership has yielded substantial evidence of (1) variations in women veterans’ access to comprehensive healthcare services that contribute to disparities in quality and patient experience and (2) the positive impacts of gender-specific care models for women veterans’ quality and satisfaction. In an effort to provide support specifically to sites that were low-performing in women’s health, WHS and the VA Quality Enhancement Research Initiative co-funded an effort to roll out and evaluate evidence-based quality improvement (EBQI), an implementation strategy with demonstrated effectiveness in a prior cluster randomized trial in women’s health clinics.

Methods

We will identify 21 low-performing VA facilities through a combination of practice data, VA quality metrics (by gender), and other indicators. In partnership with WHS, an EBQI contractor will deliver the EBQI “package”—local consensus development and priority setting using stakeholder panels, multilevel stakeholder engagement, practice facilitation, local EBQI team training, and formative feedback—to participating sites. We propose a dynamic wait-listed design to evaluate the WHS plans for seven EBQI launches per year over 3 years. The goal is to evaluate (1) barriers and facilitators to achieving delivery of comprehensive women’s health care in low-performing VA facilities; (2) effectiveness of EBQI in supporting low-performing VA facilities to achieve improved practice features (e.g., level of comprehensive services available, care coordination arrangements, Patient Aligned Care Team (PACT) features implemented, environment of care improvements), provider/staff attitudes (e.g., improved gender awareness, women’s health knowledge and practice), quality of care, and patient experience; and (3) contextual factors, local implementation processes, and organizational changes over time.

Discussion

Access to comprehensive women’s health care reduces fragmentation of care, improves patient satisfaction, and results in better patient outcomes. We hypothesize that EBQI implementation will result in changes in leadership awareness and buy-in, multilevel engagement in problem-solving, an enhanced culture of quality improvement, structural changes in care, improved provider/staff attitudes, and better quality and patient experience.

Trial registration

ClinicalTrials.gov, NCT03238417. Registered 3 August 2017. Retrospectively registered, https://clinicaltrials.gov/ct2/show/study/NCT03238417

Models of Arthritis Care: A Systems-level Evaluation of Acceptability as a Dimension of Quality of Care

OBJECTIVE

To describe a systems-level baseline evaluation of central intake (CI) and triage systems in arthritis care within Alberta, Canada. The specific objectives were to (1) describe a process for systems evaluation for the provision of arthritis care; (2) report the findings of the evaluation for different clinical sites that provide arthritis care; and (3) identify opportunities for improving appropriate and timely access based on the findings of the evaluation.

METHODS

The study used a convergent mixed methods design. Surveys and semistructured interviews were the main data collection methods. Participants were recruited through 2 rheumatology clinics and 1 hip and knee clinic providing CI and triage, and included patients, referring physicians, specialists, and clinic staff who experienced CI processes.

RESULTS

A total of 237 surveys were completed by patients (n = 169), referring physicians (n = 50), and specialists (n = 18). Interviews (n = 25) with care providers and patients provided insights to the survey data. Over 95% of referring physicians agreed that the current process of CI was satisfactory. Referring physicians and specialists reported issues with the referral process and perceived support in care for wait-listed patients. Patients reported positive experiences with access and navigation of arthritis care services but expressed concerns around communication and receiving minimal support for self-management of their arthritis before and after receiving specialist care.

CONCLUSION

This baseline evaluation of CI and triage for arthritis care indicates satisfaction with the service, but areas that require further consideration are referral completion, timely waiting lists, and further supporting patients to self-manage their arthritis.

Assessing fidelity to evidence-based quality improvement as an implementation strategy for patient-centered medical home transformation in the Veterans Health Administration

Background

Effective implementation strategies might facilitate patient-centered medical home (PCMH) uptake and spread by targeting barriers to change. Evidence-based quality improvement (EBQI) is a multi-faceted implementation strategy that is based on a clinical-researcher partnership. It promotes organizational change by fostering innovation and the spread of those innovations that are successful. Previous studies demonstrated that EBQI accelerated PCMH adoption within Veterans Health Administration primary care practices, compared with standard PCMH implementation. Research to date has not documented fidelity to the EBQI implementation strategy, limiting usefulness of prior research findings. This paper develops and assesses clinical participants’ fidelity to three core EBQI elements for PCMH (EBQI-PCMH), explores the relationship between fidelity and successful QI project completion and spread (the outcome of EBQI-PCMH), and assesses the role of the clinical-researcher partnership in achieving EBQI-PCMH fidelity.

Methods

Nine primary care practice sites and seven across-sites, topic-focused workgroups participated (2010–2014). Core EBQI elements included leadership-frontlines priority-setting for QI, ongoing access to technical expertise, coaching, and mentoring in QI methods (through a QI collaborative), and data/evidence use to inform QI. We used explicit criteria to measure and assess EBQI-PCMH fidelity across clinical participants. We mapped fidelity to evaluation data on implementation and spread of successful QI projects/products. To assess the clinical-researcher partnership role in EBQI-PCMH, we analyzed 73 key stakeholder interviews using thematic analysis.

Results

Seven of 9 sites and 3 of 7 workgroups achieved high or medium fidelity to leadership-frontlines priority-setting. Fidelity was mixed for ongoing technical expertise and data/evidence use. Longer duration in EBQI-PCMH and higher fidelity to priority-setting and ongoing technical expertise appear correlated with successful QI project completion and spread. According to key stakeholders, partnership with researchers, as well as bi-directional communication between leaders and QI teams and project management/data support were critical to achieving EBQI-PCMH fidelity.

Conclusions

This study advances implementation theory and research by developing measures for and assessing fidelity to core EBQI elements in relationship to completion and spread of QI innovation projects or tools for addressing PCMH challenges. These results help close the gap between EBQI elements, their intended outcome, and the finding that EBQI-PCMH resulted in accelerated adoption of PCMH.

Enhancing patient and organizational readiness for cardiovascular risk reduction among Black and Latinx patients living with HIV: Study protocol

Cardiovascular disease (CVD) is an increasingly important cause of morbidity and mortality among people living with HIV (PLWH) now that HIV is a manageable chronic disease. Identification and treatment of comorbid medical conditions for PLWH, including CVD and its risk factors, typically lack a critical component of care: integrated care for histories of trauma. Experiences of trauma are associated with increased HIV infection, CVD risk, inconsistent treatment adherence, and poor CVD outcomes. To address this deficit among those at greatest risk and disproportionately affected by HIV and trauma-i.e., Black and Latinx individuals-a novel culturally-congruent, evidence-informed care model, "Healing our Hearts, Minds and Bodies" (HHMB), has been designed to address patients' trauma histories and barriers to care, and to prepare patients to engage in CVD risk reduction. Further, in recognition of the need to ensure that PLWH receive guideline-concordant cardiovascular care, implementation strategies have been identified that prepare providers and clinics to address CVD risk among their Black and Latinx PLWH. The focus of this paper is to describe the hybrid Type 2 effectiveness/implementation study design, the goal of which is to increase both patient and organizational readiness to address trauma and CVD risk among 260 Black and Latinx PLWH recruited from two HIV service organizations in Southern California. This study is expected to produce important information regarding the value of the HHMB intervention and implementation processes and strategies designed for use in implementing HHMB and other evidence-informed programs in diverse, resource-constrained treatment settings, including those that serve patients living in deep poverty. Clinical trials registry: NCT04025463.

Reprint of: Qualitative methods in implementation research: An introduction

Qualitative methods are a valuable tool in implementation research because they help to answer complex questions such as how and why efforts to implement best practices may succeed or fail, and how patients and providers experience and make decisions in care. This article orients the novice implementation scientist to fundamentals of qualitative methods and their application in implementation research, describing: 1) implementation-related questions that can be addressed by qualitative methods; 2) qualitative methods commonly used in implementation research; 3) basic sampling and data collection procedures; and 4) recommended practices for data analysis and ensuring rigor. To illustrate qualitative methods decision-making, a case example is provided of a study examining implementation of a primary care-based collaborative care management model for women Veterans with anxiety, depression, and PTSD.

The protocol-guided rapid evaluation of veterans experiencing new transient neurological symptoms (PREVENT) quality improvement program: rationale and methods

BACKGROUND

Transient ischemic attack (TIA) patients are at high risk of recurrent vascular events; timely management can reduce that risk by 70%. The Protocol-guided Rapid Evaluation of Veterans Experiencing New Transient Neurological Symptoms (PREVENT) developed, implemented, and evaluated a TIA quality improvement (QI) intervention aligned with Learning Healthcare System principles.

METHODS

This stepped-wedge trial developed, implemented and evaluated a provider-facing, multi-component intervention to improve TIA care at six facilities. The unit of analysis was the medical center. The intervention was developed based on benchmarking data, staff interviews, literature, and electronic quality measures and included: performance data, clinical protocols, professional education, electronic health record tools, and QI support. The effectiveness outcome was the without-fail rate: the proportion of patients who receive all processes of care for which they are eligible among seven processes. The implementation outcomes were the number of implementation activities completed and final team organization level. The intervention effects on the without-fail rate were analyzed using generalized mixed-effects models with multilevel hierarchical random effects. Mixed methods were used to assess implementation, user satisfaction, and sustainability.

DISCUSSION

PREVENT advanced three aspects of a Learning Healthcare System. Learning from Data: teams examined and interacted with their performance data to explore hypotheses, plan QI activities, and evaluate change over time. Learning from Each Other: Teams participated in monthly virtual collaborative calls. Sharing Best Practices: Teams shared tools and best practices. The approach used to design and implement PREVENT may be generalizable to other clinical conditions where time-sensitive care spans clinical settings and medical disciplines.

TRIAL REGISTRATION

clinicaltrials.gov: NCT02769338 [May 11, 2016].

Qualitative methods in implementation research: An introduction

Qualitative methods are a valuable tool in implementation research because they help to answer complex questions such as how and why efforts to implement best practices may succeed or fail, and how patients and providers experience and make decisions in care. This article orients the novice implementation scientist to fundamentals of qualitative methods and their application in implementation research, describing: 1) implementation-related questions that can be addressed by qualitative methods; 2) qualitative methods commonly used in implementation research; 3) basic sampling and data collection procedures; and 4) recommended practices for data analysis and ensuring rigor. To illustrate qualitative methods decision-making, a case example is provided of a study examining implementation of a primary care-based collaborative care management model for women Veterans with anxiety, depression, and PTSD.

The Mixed-Method 5W2D Approach for Health System Stakeholders Analysis in Quality of Care: An Application to the Moroccan Context

(1) Background: Quality of care (QC) is not only about satisfying patients, but also about satisfying the various health system stakeholders (HSS). This makes it a complex and difficult objective to achieve. This study aims at proposing a methodological framework for identifying HSS, prioritizing them in QC, and analyzing their interrelationships. (2) Methods: The proposed framework is the mixed-method 5W2D approach, which uses a combination of three basic methods: the 5W questioning technique (What, Who, Why, Where, and When), the Delphi method, and the Decision making trial and evaluation laboratory (DEMATEL) technique. It consists of three interdependent phases. First of all, a preliminary list of HSS is established based on a systematic literature review, which is then projected and adapted to the national context using the 5W questioning technique. Secondly, the identified HSS are classified in order according to their influence and impact on QC by employing Delphi method. Thirdly, the interrelationships between HSS are determined and analyzed by applying DEMATEL technique. An application of 5W2D is conducted in the Moroccan context as its health system involves a wide range of stakeholders. (3) Results: Results defined 17 groups of HSS, whose prioritization led to three groups that are at the core of the health system: patients and their families, health personnel, and government. Roles and expectations of these groups regarding QC are divergent and contradictory, which require making trade-offs. The findings of this study intend to guide the development of inclusive strategies and policies that involve key stakeholders for QC assessment and improvement.

The importance of symbolic and engaged participation in evidence-based quality improvement in a complex integrated healthcare system: response to "The science of stakeholder engagement in research"

In this commentary, we respond to the commentary provided by Goodman and Sanders Thompson regarding our paper on multilevel stakeholder engagement in a VA implementation trial of evidence-based quality improvement (EBQI) in women's health primary care. We clarify our overall approach to engagement (comprised of both symbolic and engaged participation, according to the authors' classification rubric), highlighting that symbolic participation is of more import and value than the authors suggest, especially in the context of a hierarchical healthcare system. We contend that the issue of power-and how power matters in stakeholder engagement-needs to be considered in this context rather than in global "community" terms. In response to the authors' call for greater detail, we clarify our planning processes as well as our approach to veteran engagement. We concur with Goodman and Sanders Thompson that the science of stakeholder engagement necessitates a broader understanding of best practices as well as the impact of engagement on implementation outcomes.

Local Leaders' Perspectives on Women Veterans' Health Care: What Would Ideal Look Like?

BACKGROUND

The Veterans Health Administration (VHA) faces challenges in providing comprehensive, gender-sensitive care for women. National policies have led to important advancements, but local leadership also plays a vital role in implementing changes and operationalizing national priorities. In this article, we explore the notions of ideal women veterans' health care articulated by women's health leaders at local VHA facilities and regional networks, with the goal of identifying elements that could inform practice and policy.

METHODS

We conducted semistructured interviews with 86 local and regional women's health leaders at 12 VHA medical centers across four regions. At the conclusion of interviews about women's primary care, participants were asked to imagine "ideal care" for women veterans. Interviews were transcribed and coded using a hybrid inductive/deductive approach.

RESULTS

In describing ideal care, participants commonly touched on whether women veterans should have separate primary care services from men; the need for childcare, expanded reproductive health services, resources, and staffing; geographic accessibility; the value of input from women veterans; the physical appearance of facilities; fostering active interest in women's health across providers and staff; and the relative priority of women's health at the VHA.

CONCLUSIONS

Policy and practice changes to care for women veterans must be mindful of key stakeholders' vision for that care. Specific features of that vision include clinic construction that anticipates a growing patient population, providing childcare and expanded reproductive health services, ensuring adequate support staff, expanding mechanisms to incorporate women veterans' input, and fostering a culture oriented towards women's health at the organizational level.

Tailoring an educational program on the AHRQ Patient Safety Indicators to meet stakeholder needs: lessons learned in the VA

BACKGROUND

Given that patient safety measures are increasingly used for public reporting and pay-for performance, it is important for stakeholders to understand how to use these measures for improvement. The Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) are one particularly visible set of measures that are now used primarily for public reporting and pay-for-performance among both private sector and Veterans Health Administration (VA) hospitals. This trend generates a strong need for stakeholders to understand how to interpret and use the PSIs for quality improvement (QI). The goal of this study was to develop an educational program and tailor it to stakeholders' needs. In this paper, we share what we learned from this program development process.

METHODS

Our study population included key VA stakeholders involved in reviewing performance reports and prioritizing and initiating quality/safety initiatives. A pre-program formative evaluation through telephone interviews and web-based surveys assessed stakeholders' educational needs/interests. Findings from the formative evaluation led to development and implementation of a cyberseminar-based program, which we tailored to stakeholders' needs/interests. A post-program survey evaluated program participants' perceptions about the PSI educational program.

RESULTS

Interview data confirmed that the concepts we had developed for the interviews could be used for the survey. Survey results informed us on what program delivery mode and content topics were of high interest. Six cyberseminars were developed-three of which focused on two content areas that were noted of greatest interest: learning how to use PSIs for monitoring trends and understanding how to interpret PSIs. We also used snapshots of VA PSI reports so that participants could directly apply learnings. Although initial interest in the program was high, actual attendance was low. However, post-program survey results indicated that perceptions about the program were positive.

CONCLUSIONS

Conducting a formative evaluation was a highly important process in program development. The useful information that we collected through the interviews and surveys allowed us to tailor the program to stakeholders' needs and interests. Our experiences, particularly with the formative evaluation process, yielded valuable lessons that can guide others when developing and implementing similar educational programs.