Project selection in NIH: A natural experiment from ARRA

Scientific prizes and the extraordinary growth of scientific topics

Fast growing scientific topics have famously been key harbingers of the new frontiers of science, yet, large-scale analyses of their genesis and impact are rare. We investigated one possible factor connected with a topic’s extraordinary growth: scientific prizes. Our longitudinal analysis of nearly all recognized prizes worldwide and over 11,000 scientific topics from 19 disciplines indicates that topics associated with a scientific prize experience extraordinary growth in productivity, impact, and new entrants. Relative to matched non-prizewinning topics, prizewinning topics produce 40% more papers and 33% more citations, retain 55% more scientists, and gain 37 and 47% more new entrants and star scientists, respectively, in the first five-to-ten years after the prize. Funding do not account for a prizewinning topic’s growth. Rather, growth is positively related to the degree to which the prize is discipline-specific, conferred for recent research, or has prize money. These findings reveal new dynamics behind scientific innovation and investment.

Scientific revolutions have famously inspired scientists and innovation but large-scale analyses of scientific revolutions in modern science are rare. Here, the authors investigate one possible factor connected with a topic’s extraordinary growth—scientific prizes.

Unique Review Criteria and Patient and Stakeholder Reviewers: Analysis of PCORI's Approach to Research Funding

OBJECTIVE

The Patient-Centered Outcomes Research Institute (PCORI) uses a unique approach to Merit Review that includes patients and stakeholders as reviewers with scientists, and includes unique review criteria (patient-centeredness and active engagement of end users in the research). This study assessed the extent to which different reviewer types influence review scores and funding outcomes, the emphasis placed on technical merit compared to other criteria by a multistakeholder panel, and the impact of the in-person discussion on agreement among different reviewer types.

METHODS

Cross-sectional analysis of administrative data from PCORI online and in-person Merit Review (N = 1312 applications from the five funding cycles from November 2013 to August 2015). Linear and logistic regression models were used to analyze the data.

RESULTS

For all reviewer types, final review scores were associated with at least one review criterion score from each of the three reviewer types. The strongest predictor of final overall scores for all reviewer types was scientists' prediscussion ratings of technical merit. All reviewers' prediscussion ratings of the potential to improve health care and outcomes, and scientists' ratings of technical merit and patient-centeredness, were associated with funding success. For each reviewer type, overall impact scores from the online scoring were changed on at least half of the applications at the in-person panel discussion. Score agreement across reviewer types was greater after panel discussion.

CONCLUSIONS

Scientist, patient, and stakeholder views all contribute to PCORI Merit Review of applications for research funding. Technical merit is critical to funding success but patient and stakeholder ratings of other criteria also influence application disposition.