How can impact strategies be developed that better support universities to address twenty-first-century challenges?

To better address twenty-first-century challenges, research institutions often develop and publish research impact strategies, but as a tool, impact strategies are poorly understood. This study provides the first formal analysis of impact strategies from the UK, Canada, Australia, Denmark, New Zealand and Hong Kong, China, and from independent research institutes. Two types of strategy emerged. First, ‘achieving impact’ strategies tended to be bottom-up and co-productive, with a strong emphasis on partnerships and engagement, but they were more likely to target specific beneficiaries with structured implementation plans, use boundary organisations to co-produce research and impact, and recognise impact with less reliance on extrinsic incentives. Second, ‘enabling impact’ strategies were more top-down and incentive-driven, developed to build impact capacity and culture across an institution, faculty or centre, with a strong focus on partnerships and engagement, and they invested in dedicated impact teams and academic impact roles, supported by extrinsic incentives including promotion criteria. This typology offers a new way to categorise, analyse and understand research impact strategies, alongside insights that may be used by practitioners to guide the design of future strategies, considering the limitations of top-down, incentive-driven approaches versus more bottom-up, co-productive approaches.

Research impact as understood by two funders of agricultural research in South Africa

Impact literacy is the understanding of research impact in terms of a junction of three elements: ‘what’, ‘how’ and ‘who’. ‘What’ refers to impact endpoints, that is, the medium- to long-term impacts or even short-term effects that occur during or immediately after research. ‘How’ refers to the understanding of how impact is created, in other words the interactions required. ‘Who’ focuses on which stakeholders need to be integrated into a network of interactions to contribute to impact. This study focused on the ‘what’ of impact, specifically on the understanding of research impact by two funders of agricultural research in South Africa. Members of specialist committees at the two funding organizations were asked, through a survey, to rate several structured items mapped on a research impact classification scheme. They could also provide their own ideas on what impact is. Committee members from both organizations viewed research impact primarily as an effect on the individuals and groups in the industry that they represent. They generally did not consider research impact as an effect on researchers, nor did they place specific emphasis on the societal effects of research. The ‘what’ of research impact was found to imply a series of effects, where the different effects build on each other. Linking these effects to the ‘how’ and ‘who’ of impact will require the construction of impact pathways. Funders should take responsibility for pushing research results to a distal level by planning ‘who’ will do the push, ‘how’ it will be done and managed, and to ‘what’ final aim.

Dinámicas de producción y emisiones modeladas de gases de efecto invernadero en sistemas regionales de producción lechera de Honduras

El objetivo del estudio fue la caracterización productiva y de emisiones modeladas de gases de efecto invernadero (GEI) en 61 sistemas lecheros localizados en cinco regiones de Honduras. Durante las fases inicial (FI) y final (FF), con encuestas aplicadas individualmente a los productores, se identificaron aspectos técnicos y de productividad. Variables numéricas expresadas en Microsoft Excel® permitieron, con el modelo FAO de evaluación ambiental de la ganadería global-interactivo (GLEAM-i, por sus siglas en inglés) de ciclo de vida, estimar emisiones anuales de metano (CH4), óxido nitroso (N2O) y dióxido de carbono (CO2) en cada finca. Cálculos intermedios (GEI/animal) fueron derivados de la modelización GLEAM-i en Excel®. Durante la FI las fincas conjuntamente emitieron 25.038 t CO2 equivalente (CO2-eq), mientras que dichas emisiones fueron 10,5% menores en la FF. Emisiones de GEI/animal (2,85 ± 0,08 t CO2-eq) y de GEI/kg de proteína láctea (96,91 ± 4,50 kg CO2-eq) durante la FI fueron 13 y 21% menores en la FF, respectivamente. Valores de 52,82 ± 1,64 (CH4) y 2,66 ± 0,10 (N2O) kg/animal en la FI fueron 13% y 17% menores en la FF, respectivamente. La región centro suroriente emitió la menor cantidad de CH4 (42,95 ± 2,37 kg/animal) y N2O (1,82 ± 0,15 kg/animal, mientras las regiones occidente y norte experimentaron una reducción del 27% en GEI/kg proteína láctea entre la FI y FF. Se concluyó que la metodología usada identificó los impactos productivos y medioambientales, derivados de alternativas técnicas implementadas en sistemas de producción lechera de Honduras.   

Exploring research impact models: A systematic scoping review

This article explores the models and frameworks developed on “research impact’. We aim to provide a comprehensive overview of related literature through scoping study method. The present research investigates the nature, objectives, approaches, and other main attributes of the research impact models. It examines to analyze and classify models based on their characteristics. Forty-seven studies and 10 reviews published between 1996 and 2020 were included in the analysis. The majority of models were developed for the impact assessment and evaluation purposes. We identified three approaches in the models, namely outcome-based, process-based, and those utilized both of them, among which the outcome-based approach was the most frequently used by impact models and evaluation was considered as the main objective of this group. The process-based ones were mainly adapted from the W.K. Kellogg Foundation logic model and were potentially eligible for impact improvement. We highlighted the scope of processes and other specific features for the recent models. Given the benefits of the process-based approach in enhancing and accelerating the research impact, it is important to consider such approach in the development of impact models. Effective interaction between researchers and stakeholders, knowledge translation, and evidence synthesis are the other possible driving forces contributing to achieve and improve impact.

Science and agriculture: promoting beneficial symbiosis

While fundamental research into key mechanisms and interactions is important, the practical investigations that scientists also undertake have additional considerations, since the results are applicable in the real world but need disseminating in a way that reaches the intended audience. Worldwide, rapid population growth produces multiple pressures on land, meaning agriculture must become more efficient and productive. Other pressures on farmers are also increasing: to meet environmental quality standards, to follow legislation about application of chemical products, to remain financially viable against uncertain markets, and more. Applied research addresses specific aspects, but often reports do not describe local contexts or are too restricted, lacking details that enable an understanding of their wider application. We illustrate from our experience within UK agriculture, with a particular focus on soil, the identification of current shortcomings in many research publications; provide examples of good practice; and make suggestions for how scientists can help agriculturalists use their work to address the global issues currently faced. Specifically, we recommend that communication between science and agricultural communities is nurtured, to improve mutual understanding and facilitate two-way flow of ideas. In scientific publications, provision of as much contextual information as possible, and consideration for climatic/temporal/location influences, will enable investigations and results to be used for maximum practical effect and should increase citations.

Evaluating cancer research impact: lessons and examples from existing reviews on approaches to research impact assessment

BACKGROUND

Performing cancer research relies on substantial financial investment, and contributions in time and effort from patients. It is therefore important that this research has real life impacts which are properly evaluated. The optimal approach to cancer research impact evaluation is not clear. The aim of this study was to undertake a systematic review of review articles that describe approaches to impact assessment, and to identify examples of cancer research impact evaluation within these reviews.

METHODS

In total, 11 publication databases and the grey literature were searched to identify review articles addressing the topic of approaches to research impact assessment. Information was extracted on methods for data collection and analysis, impact categories and frameworks used for the purposes of evaluation. Empirical examples of impact assessments of cancer research were identified from these literature reviews. Approaches used in these examples were appraised, with a reflection on which methods would be suited to cancer research  impact evaluation going forward.

RESULTS

In total, 40 literature reviews were identified. Important methods to collect and analyse data for impact assessments were surveys, interviews and documentary analysis. Key categories of impact spanning the reviews were summarised, and a list of frameworks commonly used for impact assessment was generated. The Payback Framework was most often described. Fourteen examples of impact evaluation for cancer research were identified. They ranged from those assessing the impact of a national, charity-funded portfolio of cancer research to the clinical practice impact of a single trial. A set of recommendations for approaching cancer research impact assessment was generated.

CONCLUSIONS

Impact evaluation can demonstrate if and why conducting cancer research  is worthwhile. Using a mixed methods, multi-category assessment organised within a framework, will provide a robust evaluation, but the ability to perform this type of assessment may be constrained by time and resources. Whichever approach is used, easily measured, but inappropriate metrics should be avoided. Going forward, dissemination of the results of cancer research impact assessments will allow the cancer research community to learn how to conduct these evaluations.

Beyond scientific contribution: Assessment of the societal impact of research and innovation to build a sustainable agri-food sector

Due to the climate change and increased attention toward environmental management issues, the agri-food sector has been extensively relying on research, development, and innovation (RDi) to transform conventional agricultural production into a sustainable and eco-friendly industry. While the academic contribution of research has been relatively easily identified in the literature, the assessment of its societal impact remains underdeveloped. Accordingly, this study employs mixed-method evaluation approaches, mainly ASIRPA framework and Impact Oriented Monitoring (IOM) model to better understand and measure the multi-dimensional impacts of RDi in the agri-food sector in Spain. The objective of this analysis is to identify the impact of research on the society and the ecosystem. An in-depth case study analysis is conducted to examine the "best practices" program to promote sustainable techniques in the rice cultivation. Empirical findings suggest a standardized index to measure the economic, socio-territorial, health, political, capacity building, and environmental impacts, involving the stakeholder-network evaluation. The study highlights important implications for firm management decisions monitoring research uptake and policy design in the agri-food sector.

A participatory method to assess the contribution of agricultural research to societal changes in developing countries

Over the last decade, societal pressure has increased on public research agencies to provide evidence that their research contributes to achieve development goals. To address this challenge, the French Agricultural Research Center for International Development (Cirad), developed an ex-post impact assessment method based on the impact pathway approach, called Impact of Research in the South (ImpresS). The reconstruction of the impact pathway identifies and assesses research outputs, outcomes, and impacts on development over long-time spans, taking into account the contribution of other stakeholders, projects, and contextual factors. By applying mixed methods and participatory approaches, ImpresS involves key actors in assessing the contribution of research to impacts in innovation processes. Such a participatory approach raises, however, questions about the advantages and disadvantages of participation in impact assessment. This article examines whether and how participation affects the results of an evaluation and the methodological challenges this poses. The analysis is based on 13 case studies covering different innovation processes, countries, and time spans. The main results show that participation, combined with triangulation of information including quantitative and qualitative data, strengthens the explanation of the causal relationships among outputs, outcomes, and impacts. It helps reveal a large and diversified list of impacts based on the perception of actors, especially uncovering positive and negative impacts unexpected by researchers. However, participation may render other impacts less visible, especially those related to environmental and political issues. Furthermore, participatory evaluation entangles challenges linked to expectations, divergent perceptions, power dynamics, and social inequality, which must be carefully addressed to provide robust and transparent evaluation results.

Global volatility of public agricultural R&D expenditure

Public investment in agricultural research and development (R&D) is important for global food security and environmental sustainability. Although public agricultural R&D projects are associated with high economic returns, they are characterized by long time horizons and temporal lags. The inherent lag, between when R&D investment takes place and when it comes to fruition, implies that its stability is critical. Existing studies on the stability of public agricultural R&D expenditure are restricted to Sub-Saharan Africa and find evidence of considerable volatility in these expenditures when compared to other developing regions. This chapter extends that analysis to 112 countries by examining the volatility of public agricultural R&D expenditure between 1981 and 2014. We find average global volatility in public agricultural R&D expenditure to be considerable, with the highest levels for Sub-Saharan Africa and the least for South Asia. Although volatility in public agricultural R&D continues to be the highest in low-income countries, it has declined in the 2000s as compared to the previous decade. On the other hand, high-income countries show a steady increase in volatility over time. The co-existence of high R&D volatility with a disproportionately low share in global agricultural R&D for low-income countries is a matter of concern given the adverse impact on agricultural productivity and food security.

Developing an evaluation framework for university-driven technology-based, innovation for inclusive development (UTI4ID) projects

From the evaluation literature related to research and innovation policy, calls have long been made for improved evaluation processes especially where projects and programmes may be evaluated from a systems perspective. Drawing on the literature of innovation systems and innovation for inclusive development (I4ID) this research proposes a state-of-the-art analysis framework, which can be used to evaluate University-driven Technology-based Innovation for Inclusive Development (UTI4ID) projects. In particular, this framework draws on 16 exploratory case studies of UTI4ID that leads to the development of typologies that can be used in future UTI4ID evaluations to better guide such efforts. We thus not only show that advances in the I4ID literature provides an ideal starting point for complexity cognisant evaluations, but also contribute an end-to-end framework to support such evaluations and populate a detailed guide for future UTI4ID projects.

Conceptualizing the societal impact of research in terms of elements of logic models: a survey of researchers in sub-Saharan Africa

This study addressed the conceptualization of the societal impact of research from the perspective of programme evaluation, by focusing on the three ‘result’ elements of logic models: outputs, outcomes, and impact. In research evaluation, the distinction could resemble a difference between product, use, and benefit. The study established whether researchers in sub-Saharan Africa (SSA), South Africa excluded, view societal impact as extending across all three elements or as confined to the last element only. A web survey of 485 SSA researchers was conducted, as researchers from this region are not yet bounded by policy definitions of impact. The survey imposed the three elements of logic models onto five hypothetical descriptions of ‘impactful’ research initiatives. Respondents rated each element in terms of how much it reflects the societal impact of research. For any initiative, use was more likely to be considered a strong example of societal impact compared to a product, but less likely so compared to benefit. Between 23% and 43% of respondents rated all three elements as strong examples of the societal impact of research. Responses were analyzed by SSA region and the research domain and years of research experience of survey participants. An open-ended question about own understandings of societal impact was included as well in the survey. The responses portrayed impact as a (generally) positive effect that contributes to change in the daily life of human kind. The expectation that research should have impact at an almost general level of aggregation could be unique to the SSA context.

Development and validation of a questionnaire to measure research impact

Although questionnaires are widely used in research impact assessment, their metric properties are not well known. Our aim is to test the internal consistency and content validity of an instrument designed to measure the perceived impacts of a wide range of research projects. To do so, we designed a questionnaire to be completed by principal investigators in a variety of disciplines (arts and humanities, social sciences, health sciences, and information and communication technologies). The impacts perceived and their associated characteristics were also assessed. This easy-to-use questionnaire demonstrated good internal consistency and acceptable content validity. However, its metric properties were more powerful in areas such as knowledge production, capacity building and informing policy and practice, in which the researchers had a degree of control and influence. In general, the research projects represented an stimulus for the production of knowledge and the development of research skills. Behavioural aspects such as engagement with potential users or mission-oriented projects (targeted to practical applications) were associated with higher social benefits. Considering the difficulties in assessing a wide array of research topics, and potential differences in the understanding of the concept of ‘research impact’, an analysis of the context can help to focus on research needs. Analyzing the metric properties of questionnaires can open up new possibilities for validating instruments used to measure research impact. Further to the methodological utility of the current exercise, we see a practical applicability to specific contexts where multiple discipline research impact is requires.