External community review committee: a new strategy for engaging community stakeholders in research funding decisions

Community Members as Reviewers of Medical Journal Manuscripts: a Randomized Controlled Trial

BACKGROUND

Community members may provide useful perspectives on manuscripts submitted to medical journals.

OBJECTIVE

To determine the impact of community members reviewing medical journal manuscripts.

DESIGN

Randomized controlled trial involving 578 original research manuscripts submitted to two medical journals from June 2018 to November 2021.

PARTICIPANTS

Twenty-eight community members who were trained, supervised, and compensated.

INTERVENTIONS

A total of 289 randomly selected control manuscripts were reviewed by scientific reviewers only. And 289 randomly selected intervention manuscripts were reviewed by scientific reviewers and one community member. Journal editorial teams used all reviews to make decisions about acceptance, revision, or rejection of manuscripts.

MAIN MEASURES

Usefulness of reviews to editors, content of community reviews, and changes made to published articles in response to community reviewer comments.

KEY RESULTS

Editor ratings of community and scientific reviews averaged 3.1 and 3.3, respectively (difference 0.2, 95% confidence interval [CI] 0.1 to 0.3), on a 5-point scale where a higher score indicates a more useful review. Qualitative analysis of the content of community reviews identified two taxonomies of themes: study attributes and viewpoints. Study attributes are the sections, topics, and components of manuscripts commented on by reviewers. Viewpoints are reviewer perceptions and perspectives on the research described in manuscripts and consisted of four major themes: (1) diversity of study participants, (2) relevance to patients and communities, (3) cultural considerations and social context, and (4) implementation of research by patients and communities. A total of 186 community reviewer comments were integrated into 64 published intervention group articles. Viewpoint themes were present more often in 66 published intervention articles compared to 54 published control articles (2.8 vs. 1.7 themes/article, difference 1.1, 95% CI 0.4 to 1.8).

CONCLUSIONS

With training, supervision, and compensation, community members are able to review manuscripts submitted to medical journals. Their comments are useful to editors, address topics relevant to patients and communities, and are reflected in published articles.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03432143.

What works for peer review and decision-making in research funding: a realist synthesis

Introduction

Allocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom.

Methods

Realist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding.

Results

We analysed 96 publications and 36 website sources. Sixty publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations (CMOCs) for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to write and review applications, promoted increased investment into innovation, and lowered cost of panels.

Conclusions

Reports of 50 interventions in different areas of peer review provide useful guidance on ways of solving common issues with the peer review process. Evidence of the broader impact of these interventions on the research ecosystem is still needed, and future research should aim to identify processes that consistently work to improve peer review across funders and research contexts.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41073-022-00120-2.

Building capacity for dissemination and implementation to maximize research impact in a CTSA: The University of Wisconsin story

We report results of an 8-year process of stakeholder engagement aimed at building capacity in Dissemination and Implementation (D&I) research at the University of Wisconsin as part of the National Institutes of Health’s Clinical and Translational Science Award (CTSA). Starting in 2008, annual individual interviews were held with leaders of the Wisconsin CTSA’s community engagement core for strategic planning purposes. Interviews were followed by annual planning meetings that employed a facilitated group decision-making process aimed at identifying and prioritizing gaps in the translational research spectrum. In 2011, the stakeholder engagement process identified D&I as a primary gap limiting overall impact of the institution’s research across the translational spectrum. Since that time, our CTSA has created an array of D&I resources falling into four broad categories: (1) relationship building with D&I partners, (2) D&I skill building, (3) translational research resources, and (4) resources to support D&I activities. Our systematic process of stakeholder engagement has increased the impact of research by providing D&I resources to meet investigator and community needs. CTSAs could engage with leaders of their community engagement cores, which are common to all CTSAs, to adapt or adopt these resources to build D&I capacity.