Governing beyond the project: Refocusing innovation governance in emerging science and technology funding

This article analyses how a recent idiom of innovation governance, 'responsible innovation', is enacted in practice, how this shapes innovation processes, and what aspects of innovation are left untouched. Within this idiom, funders typically focus on one point in an innovation system: researchers in projects. However, the more transformational aspirations of responsible innovation are circumscribed by this context. Adopting a mode of critique that assembles, this article considers some alternative approaches to governing the shared trajectories of science, technology, and society. Using the idea of institutional invention to focus innovation governance on four inflection points-agendas, calls, spaces, evaluation-would allow funding organizations and researchers to look 'beyond the project', developing new methods to unpack and reflect on assumed purposes of science, technology, and innovation, and to potentially reconfigure the institutions that condition scientific practice.

Monitoring Open Science as transformative change: Towards a systemic framework

Following a flurry of policies for Open Science (OS), there is now a wave of initiatives to monitor its adoption. However, the great diversity of understandings and activities related to Open Science makes monitoring very challenging. There is a danger that by focusing on what can be readily observed (e.g. publications) many other OS activities are overlooked (e.g. participation), with a potential narrowing of OS scope, streetlight effects, and deviation from the values of OS. Since Open Science can be understood as a systemic transformation of the research system, we have borrowed concepts from Transformative Innovation Policies frameworks which aim at evaluating socio-technical transitions. In accordance with this view of OS as a systemic transformation, we propose that the new monitoring efforts should shift towards: (i) systemic perspectives which considers the various actions related to OS, including policies and outputs (e.g. datasets) but also processes (e.g. participatory events), outcomes (e.g. citizen interest in science) and expected impacts (e.g. better scientific contributions to addressing societal problems); (ii) implementation of monitoring as reflexive learning (rather than accountability or benchmarking); (iii) mapping the directionality of the activities and the values associated with the choices in directions. In summary, a monitoring framework for OS requires a profound change in conventional monitoring practices. The scope should broaden from current focus on outputs (such as publications) towards the processes of connection that make science 'open' (usage, co-creation and dialogue), as well as towards outcomes (changes in practices) and the longer-term impacts that reflect the values and normative commitments of OS.

Evaluative altmetrics: is there evidence for its application to research evaluation?

Introduction

Altmetrics have been demonstrated as a promising tool for analyzing scientific communication on social media. Nevertheless, its application for research evaluation remains underdeveloped, despite the advancement of research in the study of diverse scientific interactions.

Methods

This paper develops a method for applying altmetrics in the evaluation of researchers, focusing on a case study of the Environment/Ecology ESI field publications by researchers at the University of Granada. We considered Twitter as a mirror of social attention, news outlets as media, and Wikipedia as educational, exploring mentions from these three sources and the associated actors in their respective media, contextualizing them using various metrics.

Results

Our analysis evaluated different dimensions such as the type of audience, local attention, engagement generated around the mention, and the profile of the actor. Our methodology effectively provided dashboards that gave a comprehensive view of the different instances of social attention at the author level.

Discussion

The use of altmetrics for research evaluation presents significant potential, as shown by our case study. While this is a novel method, our results suggest that altmetrics could provide valuable insights into the social attention that researchers garner. This can be an important tool for research evaluation, expanding our understanding beyond traditional metrics.

Constraints on Research in Biological and Agricultural Science in Developing Countries: The Example of Latin America

Science is an international effort, receiving contributions from researchers across the globe. The capacity of a country or a region to generate and publish quality research varies greatly according to the location examined. Among the factors that dictate the quantity and quality of scientific research are the availability of infrastructure and human resources, the traditions related to research endeavors, and, most significantly, local governmental support for research. There are several conditions that both individually and cooperatively limit research activities in Latin America, such as insufficient governmental support, a paucity of material and technical resources, heavy teaching loads, the absence of peer networks, and multiple constraints on publication. This commentary has been developed to discuss each of the issues that permit and, more frequently, limit biological and agricultural research endeavors in Latin America.

Evaluación de la investigación científica: mejorando las políticas científicas en Latinoamérica

Este artículo busca identificar criterios e indicadores de evaluación científica, que permitan mejorar la forma en que las agencias de financiación, las instituciones académicas y otros grupos evalúan la calidad e impacto de la investigación. Para ello, se realiza una revisión bibliográfica, basada en artículos sobre políticas de evaluación de la investigación científica y agendas internacionales implementadas en los últimos años (principalmente en el Reino Unido, Estados Unidos, Australia, China y Latinoamérica). Los resultados indican que no existe un solo método de evaluación científica, ya que ningún indicador es absoluto. Cada investigación posee actores distintos que deben ser considerados y se debe valorar la investigación en su contexto. Se recomienda un sistema de evaluación mixto, que incorpore criterios cuantitativos y cualitativos, pero que reconozca los límites y alcances de ambos y también de cada disciplina.

Societal impact of university research in the written press: media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium

Transferring scientific knowledge to non-academic audiences is an essential aspect of the open science agenda, which calls for scholars to pursue a popularization of their research. Accordingly, purposefully introducing scientific insights to the public at large is almost univocally deemed commendable. Indeed, in today’s models of research evaluation, the objects and activities considered are being extended beyond peer-reviewed journal articles to include non-scholarly popular communication. Although altmetrics offer one instrumental way to count some interactions with lay audiences, their reliance on social media makes them susceptible to manipulation, and mostly reflect circulation among niche audiences. In comparison, attention from non-scholarly media like newspapers and magazines seems a more relevant pathway to effectuate societal impact, due to its recognition in qualitative assessment tools and its broad, societal reach. Based on a case study of social scientists’ attention by newspapers and magazines in Flanders (northern Dutch-speaking region of Belgium) in 2019, this paper highlights that frequent participation in the public debate is reserved for high-status researchers only. Results show highly skewed media appearance patterns in both career position and gender, as eight male professors accounted for almost half of all 2019 media attention for social scientists. Because media attention is highly subject-dependent moreover, certain disciplines and fields offer easier pathways to popularization in media than others. Both the open science agenda and research assessment models value presence of researchers in popular media, adding written press attention to existing evaluation assessments however would disproportionately disadvantage early career researchers and exacerbate existing inequalities in academia.

A fairer way to compare researchers at any career stage and in any discipline using open-access citation data

The pursuit of simple, yet fair, unbiased, and objective measures of researcher performance has occupied bibliometricians and the research community as a whole for decades. However, despite the diversity of available metrics, most are either complex to calculate or not readily applied in the most common assessment exercises (e.g., grant assessment, job applications). The ubiquity of metrics like the h-index (h papers with at least h citations) and its time-corrected variant, the m-quotient (h-index ÷ number of years publishing) therefore reflect the ease of use rather than their capacity to differentiate researchers fairly among disciplines, career stage, or gender. We address this problem here by defining an easily calculated index based on publicly available citation data (Google Scholar) that corrects for most biases and allows assessors to compare researchers at any stage of their career and from any discipline on the same scale. Our ε′-index violates fewer statistical assumptions relative to other metrics when comparing groups of researchers, and can be easily modified to remove inherent gender biases in citation data. We demonstrate the utility of the ε′-index using a sample of 480 researchers with Google Scholar profiles, stratified evenly into eight disciplines (archaeology, chemistry, ecology, evolution and development, geology, microbiology, ophthalmology, palaeontology), three career stages (early, mid-, late-career), and two genders. We advocate the use of the ε′-index whenever assessors must compare research performance among researchers of different backgrounds, but emphasize that no single index should be used exclusively to rank researcher capability.

From indicators to indicating interdisciplinarity: A participatory mapping methodology for research communities in-the-making

This article discusses a project under development called “Inventing Indicators of Interdisciplinarity,” as an example of work in methodology development that combines quantitative methods with interpretative approaches in social and cultural research. Key to our project is the idea that Science and Technology Indicators not only have representative value, enabling empirical insight into fields of research and innovation but simultaneously have organizing capacity, as their deployment enables the curation of communities of interpretation. We begin with a discussion of concepts and methods for the analysis of interdisciplinarity in Science and Technology Studies (STS) and scientometrics, stressing that both fields recognize that interdisciplinarity is contested. To make possible a constructive exploration of interdisciplinarity as a contested—and transformative—phenomenon, we sketch out a methodological framework for the development and deployment of “engaging indicators.” We characterize this methodology of indicating as participatory, abductive, interactive, and informed by design, and emphasize that the method is inherently combinatory, as it brings together approaches from scientometrics, STS, and humanities research. In a final section, we test the potential of our approach in a pilot study of interdisciplinarity in AI, and offer reflections on digital mapping as a pathway towards indicating interdisciplinarity.

Exploring why global health needs are unmet by research efforts: the potential influences of geography, industry and publication incentives

Background

It has been well established that research is not addressing health needs in a balanced way — much more research is conducted on diseases with more burden in high-income countries than on those with more burden in lower-income countries. In this study, we explore whether these imbalances persist and inquire about the possible influence of three factors, namely geography, industry and publication incentives.

Methods

We use WHO data on the Global Burden of Disease as a proxy measure of health needs and bibliometric information as a proxy for research efforts. Scientific publications on diseases were collected from MEDLINE using MeSH terms to identify relevant publications. We used Web of Science to collect author affiliations and citation data. We developed a correspondence table between WHO ICD-10 and MeSH descriptors to compare global health needs and research efforts. This correspondence table is available as supplementary material.

Results

Research output is heavily concentrated in high-income countries and is mainly focused on their health needs, resulting in a relative lack of attention to diseases in lower income countries. A new finding is that diseases with a similar burden in high- and middle-income countries are also under-researched, both globally and in relation to disease burden in high- and middle-income countries. Global industrial R&D is found to be very similar to the focus of public research. Diseases more prevalent in high-income countries generate ten-fold more research attention than those in low-income countries. We find no discernible preference towards diseases of high-income countries versus those of low-income countries in the top 25% most prestigious journals. However, in middle-income countries, citation rates are substantially lower for diseases most prevalent in low- and middle-income countries.

Conclusions

From a global perspective, the imbalance between research needs and research efforts persists as most of the research effort concentrates on diseases affecting high-income countries. Both pharmaceutical companies and the public sector also tend to focus on diseases with more burden in high-income countries. Our findings indicate that researchers in middle-income countries receive more citations when researching diseases more prevalent in high-income countries, and this may divert the attention of researchers in these countries from diseases more prevalent in their contexts.