Unique Review Criteria and Patient and Stakeholder Reviewers: Analysis of PCORI's Approach to Research Funding

Increasing Engagement of Women Veterans in Health Research

BACKGROUND

Meaningful engagement of patients in health research has the potential to increase research impact and foster patient trust in healthcare. For the past decade, the Veterans Health Administration (VA) has invested in increasing Veteran engagement in research.

OBJECTIVE

We sought the perspectives of women Veterans, VA women's health primary care providers (WH-PCPs), and administrators on barriers to and facilitators of health research engagement among women Veterans, the fastest growing subgroup of VA users.

DESIGN

Semi-structured qualitative telephone interviews were conducted from October 2016 to April 2018.

PARTICIPANTS

Women Veterans (N=31), WH-PCPs (N=22), and administrators (N=6) were enrolled across five VA Women's Health Practice-Based Research Network sites.

APPROACH

Interviews were audio-recorded and transcribed. Consensus-based coding was conducted by two expert analysts.

KEY RESULTS

All participants endorsed the importance of increasing patient engagement in women's health research. Women Veterans expressed altruistic motives as a personal determinant for research engagement, and interest in driving women's health research forward as a stakeholder or research partner. Challenges to engagement included lack of awareness about opportunities, distrust of research, competing priorities, and confidentiality concerns. Suggestions to increase engagement include utilizing VA's patient-facing portals of the electronic health record for outreach, facilitating "warm hand-offs" between researchers and clinic staff, developing an accessible research registry, and communicating the potential research impact for Veterans.

CONCLUSIONS

Participants expressed support for increasing women Veterans' engagement in women's health research and identified feasible ways to foster and implement engagement of women Veterans. Given the unique healthcare needs of women Veterans, engaging them in research could translate to improved care, especially for future generations. Knowledge about how to improve women Veterans' research engagement can inform future VA policy and practice for more meaningful interventions and infrastructure.

Clinical trials proposed for the VA Cooperative Studies Program: Success rates and factors impacting approval

The process by which funding organizations select among the myriad number of proposals they receive is a matter of significant concern for researchers and the public alike. Despite an extensive literature on the topic of peer review and publications on criteria by which clinical investigations are reviewed, publications analyzing peer review and other processes leading to government funding decisions on large multi-site clinical trials proposals are sparse. To partially address this gap, we reviewed the outcomes of scientific and programmatic evaluation for all letters of intent (LOIs) received by the Department of Veterans Affairs (VA) Cooperative Studies Program (CSP) between July 4, 2008, and November 28, 2016. If accepted, these LOIs represented initial steps towards later full proposals that also underwent scientific peer review. Twenty-two of 87 LOIs were ultimately funded and executed as CSP projects, for an overall success rate of 25%. Most proposals which received a negative decision did so prior to submission of a full proposal. Common reasons for negative scientific review of LOIs included investigator inexperience, perceived lack of major scientific impact, lack of preliminary data and flawed or confused experimental design, while the most common reasons for negative reviews of final proposals included questions of scientific impact and issues of study design, including outcome measures, randomization, and stratification. Completed projects have been published in high impact clinical journals. Findings highlight several factors leading to successfully obtaining funding support for clinical trials. While our analysis is restricted to trials proposed for CSP, the similarities in review processes with those employed by the National Institutes of Health and the Patient Centered Outcomes Research Institute suggest the possibility that they may also be important in a broader context.

It should not require a pandemic to make community engagement in research leadership essential, not optional

Efforts to move community engagement in research from marginalized to mainstream include the NIH requiring community engagement programs in all Clinical and Translational Science Awards (CTSAs). However, the COVID-19 pandemic has exposed how little these efforts have changed the dominant culture of clinical research. When faced with the urgent need to generate knowledge about prevention and treatment of the novel coronavirus, researchers largely neglected to involve community stakeholders early in the research process. This failure cannot be divorced from the broader context of systemic racism in the US that has contributed to Black, Indigenous, and People of Color (BIPOC) communities bearing a disproportionate toll from COVID-19, being underrepresented in COVID-19 clinical trials, and expressing greater hesitancy about COVID-19 vaccination. We call on research funders and research institutions to take decisive action to make community engagement obligatory, not optional, in all clinical and translational research and to center BIPOC communities in this process. Recommended actions include funding agencies requiring all research proposals involving human participants to include a community engagement plan, providing adequate funding to support ongoing community engagement, including community stakeholders in agency governance and proposal reviews, promoting racial and ethnic diversity in the research workforce, and making a course in community engaged research a requirement for Masters of Clinical Research curricula.

Stakeholder engagement in research: a scoping review of current evaluation methods

Aim: Evaluating stakeholder engagement can capture what meaningful engagement in research entails, how it develops, and how it is experienced by all collaborators. We conducted a scoping review of recent approaches for evaluating engagement in research and present a descriptive overview of our findings. Methods: We searched peer-reviewed journal articles published worldwide in English between January 2013 and June 2018. Results: Our final sample consisted of 17 articles. Various approaches for evaluating stakeholder engagement were identified including qualitative approaches, surveys and engagement logs. Discussion & conclusion: We identified evaluation approaches that varied in quality, detail and methods. Valid, systematic and inclusive approaches that are developed with research partners and are inclusive of diverse perspectives are an important area for future research.

Patient Preference Studies During Early Drug Development: Aligning Stakeholders to Ensure Development Plans Meet Patient Needs

Although patient preferences have been studied broadly for marketed products or around the time of submission to authorities and launch, patient preference studies have rarely been used during the early drug development phases. In this paper, we formulate three hypotheses supporting the use of patient preference studies in early product development: (1) integration of the patient perspective into the development process from phase 1 onwards will result in healthcare solutions with outcomes that best address patients' needs; (2) a structured process to build patient-based evidence involving partnerships between patients and other key stakeholders will improve alignment of development activities with the needs of patients; (3) quantitative patient preference research built on robust qualitative insights is necessary to strengthen development decisions in the interests of patients. To illustrate such a structured process, we describe qualitative insights research (social media analysis and online bulletin boards) and quantitative patient preference studies in dry eye disease and non-alcoholic steatohepatitis conducted during early product development by a pharmaceutical company to generate patient-based evidence. The outputs from such early patient preference studies are being used to inform patient reported outcome strategies, clinical development strategies, product design and delivery features, and form the basis for early dialog with regulators, health technology assessment (HTA) bodies and payers to ensure focus and alignment on patient-relevant endpoints. Furthermore, to discuss and theoretically substantiate our hypotheses, we review how different groups and organizations are working to embrace fully the patient perspective in product development and healthcare decision-making. The hypotheses are commensurate with the general trend toward patient-centered healthcare and the activities initiated by regulators, HTA agencies, and patient organizations. We advocate that all healthcare players should actively contribute to aligning on best practices concerning choice of methodologies and engage in multi-stakeholder dialog along the entire product development chain, to realize health technologies that best meet the needs of patients.

Directing research funds to the right research projects: a review of criteria used by research organisations in Australia in prioritising health research projects for funding

OBJECTIVES

Healthcare budgets are limited, and therefore, research funds should be wisely allocated to ensure high-quality, useful and cost-effective research. We aimed to critically review the criteria considered by major Australian organisations in prioritising and selecting health research projects for funding.

METHODS

We reviewed all grant schemes listed on the Australian Competitive Grants Register that were health-related, active in 2017 and with publicly available selection criteria on the funders' websites. Data extracted included scheme name, funding organisation, selection criteria and the relative weight assigned to each criterion. Selection criteria were grouped into five representative domains: relevance, appropriateness, significance, feasibility (including team quality) and cost-effectiveness (ie, value for money).

RESULTS

Thirty-six schemes were included from 158 identified. One-half of the schemes were under the National Health and Medical Research Council. The most commonly used criteria were research team quality and capability (94%), research plan clarity (94%), scientific quality (92%) and research impact (92%). Criteria considered less commonly were existing knowledge (22%), fostering collaboration (22%), research environment (19%), value for money (14%), disease burden (8%) and ethical/moral considerations (3%). In terms of representative domains, relevance was considered in 72% of the schemes, appropriateness in 92%, significance in 94%, feasibility in 100% and cost-effectiveness in 17%. The relative weights for the selection criteria varied across schemes with 5%-30% for relevance, 20%-60% for each appropriateness and significance, 20%-75% for feasibility and 15%-33% for cost-effectiveness.

CONCLUSIONS

In selecting research projects for funding, Australian research organisations focus largely on research appropriateness, significance and feasibility; however, value for money is most often overlooked. Research funding decisions should include an assessment of value for money in order to maximise return on research investment.

Researchers, Patients, and Stakeholders Evaluating Comparative-Effectiveness Research: A Mixed-Methods Study of the PCORI Reviewer Experience

OBJECTIVES

The Patient-Centered Outcomes Research Institute (PCORI) includes patients and stakeholders alongside scientists in reviewing research applications using unique review criteria including patient-centeredness and patient and/or stakeholder engagement. To support extension of this unique collaborative model to other funders, information from the reviewers on the review process is needed to understand how scientists and nonscientists evaluate research proposals together. Thus, this study aimed to describe reviewers' perspectives of the interactions during the in-person review panel; to examine the value and challenges of including scientists, patients, and stakeholders together; and to understand the perceived importance of PCORI's review criteria.

METHODS

This study utilized anonymous, cross-sectional surveys (N = 925 respondents from 5 funding cycles: 470 scientists, 217 patients, 238 stakeholders; survey completion rates by cycle: 70-89%) and group interviews (N = 18).

RESULTS

Reviewers of all types describe PCORI Merit Review as respectful, balanced, and one of reciprocal influence among different reviewer types. Reviewers indicate strong support and value of input from all reviewer types, receptivity to input from others, and the panel chair's incorporation of all views. Patients and stakeholders provide real-world perspectives on importance to patients, research partnership plans, and study feasibility. Challenges included concerns about a lack of technical expertise of patient/stakeholder reviewers and about scientists dominating conversations. The most important criterion for assigning final review scores was technical merit-either alone or in conjunction with patient-centeredness or patient/ stakeholder engagement.

CONCLUSIONS

PCORI Merit Reviewers' self-reports indicate that the perspectives of different reviewer types are influential in panel discussions and Merit Review outcomes.