An openness framework for ex ante evaluation of societal impact of research

Over the last decade, the idea of societal impact resulting from publicly funded research has changed from being a relatively fringe concern related to high-technology entrepreneurship and spin-off companies to becoming an increasingly important public policy concern. This has stimulated academic science policy research to investigate the impact of research and conceptualize how knowledge created in academic contexts can be coupled to real-world problems or needs. Most of the work in this stream of research focuses on ex post impacts, that is, the impacts created by individual research activities or research strands after their completion and likewise there has been much research on ex post impact evaluation. However, ex ante impact evaluations have become increasingly important for funding decisions, but little is known about how to evaluate impact when considering research project proposals. In this article, we propose a conceptual framework to evaluate ex ante impact of research based on the idea that a research proposal is a ‘promise’ to achieve impact. We suggest that evaluators could assess social impact promises by considering two elements, namely the activities that couple their knowledge with non-academic users and the interdependency and consistency of such activities throughout the overall project proposal. We ultimately propose an analytical framework for refining our ‘openness’ conceptual framework in future empirical research.

Evaluation of the arts in performance-based research funding systems: An international perspective

This article provides a comprehensive analysis of the evaluation of the arts within performance-based research funding systems (PRFSs). Previous literature on PRFSs has overlooked the arts and focussed primarily on outputs in relation to the sciences and humanities. We develop a typology of how artistic outputs are evaluated within 10 countries’ PRFSs, operating in Australia, the Czech Republic, Italy, Lithuania, New Zealand, Poland, Portugal, Slovakia, Spain, and the UK, and identify three different types of artistic evaluation systems. The study compares evaluation methods and provides a classification of quality criteria used by evaluation panels. We conclude with a discussion of the challenges specific to different types of systems.

Quis judicabit ipsos judices? A case study on the dynamics of competitive funding panel evaluations

Securing research funding is essential for all researchers. The standard evaluation method for competitive grants is through evaluation by a panel of experts. However, the literature notes that peer review has inherent flaws and is subject to biases, which can arise from differing interpretations of the criteria, the impossibility for a group of reviewers to be experts in all possible topics within their field, and the role of affect. As such, understanding the dynamics at play during panel evaluations is crucial to allow researchers a better chance at securing funding, and also for the reviewers themselves to be aware of the cognitive mechanisms underlying their decision-making. In this study, we conduct a case study based on application and evaluation data for two social sciences panels in a competitive state-funded call in Portugal. Using a mixed-methods approach, we find that qualitative evaluations largely resonate with the evaluation criteria, and the candidate’s scientific output is partially aligned with the qualitative evaluations, but scientometric indicators alone do not significantly influence the candidate’s evaluation. However, the polarity of the qualitative evaluation has a positive influence on the candidate’s evaluation. This paradox is discussed as possibly resulting from the occurrence of a halo effect in the panel’s judgment of the candidates. By providing a multi-methods approach, this study aims to provide insights that can be useful for all stakeholders involved in competitive funding evaluations.