Contribution mapping: a method for mapping the contribution of research to enhance its impact

Anticipatory evaluation. How to incorporate an anticipatory technique into a theory-driven evaluation process. Results of application in a case study

In recent years, there has been increased focus on strategic learning from impact analysis, including in the field of science, technology, and innovation. In this paper, we propose combining techniques from two fields of study. Firstly, we adopt the approach of addressing impact through productive interactions between science and society, and secondly, we incorporate an anticipatory dimension by integrating game analysis involving key actors. Through a theory-driven evaluation design, we consider expected impacts as promises of the future. Within an anticipatory perspective, the future can be shaped by the interactions among different actors in the present. In this article, we apply this approach step-by-step to a research institute program in Uruguay. We demonstrate how the achieved results offer strategic insights to the program manager for anticipating and attaining the desired impacts. Additionally, we provide summative inputs for accountability using a flexible technique applicable at any stage of the program life cycle. The article concludes with a discussion of the advantages and disadvantages compared to other techniques, along with lessons learned that may benefit other evaluators seeking to replicate this approach. Furthermore, we explore potential extensions and opportunities for further improvement in this research.

Healing Right Way randomised control trial enhancing rehabilitation services for Aboriginal people with brain injury in Western Australia: translation principles and activities

Background This report provides the theory, method and practice of culturally secure translation and knowledge exchange in the Healing Right Way Clinical Trial (2017-2022), outlining activities to date. Healing Right Way was a stepped wedge cluster randomised controlled trial conducted in Western Australia, aimed at enhancing rehabilitation services and quality of life for Aboriginal Australians following acquired brain injury. The trial translation plan was aspirational and action-oriented, with its implementation iterative and ongoing. Translational activities aimed to inform service and research planning for Aboriginal people with brain injury. Situated in the intercultural space, the work guards against undertaking activities that are monocultural, colonial and appropriating in favour of work that is authentically viewed through the dual lens of whiteness and Aboriginal and Torres Strait Islander ways of knowing, being and doing, and is strengths-based. Methods Three translational and knowledge exchange components were identified, relating to the role of Aboriginal Brain Injury Coordinators, cultural training of hospital staff and the research process itself. Knowledge plans were developed for key audiences, with potential translation products to be monitored for ongoing impact. Results Results demonstrate that translational and knowledge exchange were iteratively embedded throughout the trial life cycle. Data sources included community engagement, partnership meetings and interviews. Activities involved presentations to diverse audiences including bureaucrats, community and participants. Conclusions This report provides a snapshot of the first translation knowledge exchange plan and activities constructed in relation to brain injury rehabilitation services for Aboriginal people. Challenges encountered, as well as successes to date, are discussed.

A scoping review of theories, models and frameworks used or proposed to evaluate knowledge mobilization strategies

BACKGROUND

Evaluating knowledge mobilization strategies (KMb) presents challenges for organizations seeking to understand their impact to improve KMb effectiveness. Moreover, the large number of theories, models, and frameworks (TMFs) available can be confusing for users. Therefore, the purpose of this scoping review was to identify and describe the characteristics of TMFs that have been used or proposed in the literature to evaluate KMb strategies.

METHODS

A scoping review methodology was used. Articles were identified through searches in electronic databases, previous reviews and reference lists of included articles. Titles, abstracts and full texts were screened in duplicate. Data were charted using a piloted data charting form. Data extracted included study characteristics, KMb characteristics, and TMFs used or proposed for KMb evaluation. An adapted version of Nilsen (Implement Sci 10:53, 2015) taxonomy and the Expert Recommendations for Implementing Change (ERIC) taxonomy (Powell et al. in Implement Sci 10:21, 2015) guided data synthesis.

RESULTS

Of the 4763 search results, 505 were retrieved, and 88 articles were eligible for review. These consisted of 40 theoretical articles (45.5%), 44 empirical studies (50.0%) and four protocols (4.5%). The majority were published after 2010 (n = 70, 79.5%) and were health related (n = 71, 80.7%). Half of the studied KMb strategies were implemented in only four countries: Canada, Australia, the United States and the United Kingdom (n = 42, 47.7%). One-third used existing TMFs (n = 28, 31.8%). According to the adapted Nilsen taxonomy, process models (n = 34, 38.6%) and evaluation frameworks (n = 28, 31.8%) were the two most frequent types of TMFs used or proposed to evaluate KMb. According to the ERIC taxonomy, activities to "train and educate stakeholders" (n = 46, 52.3%) were the most common, followed by activities to "develop stakeholder interrelationships" (n = 23, 26.1%). Analysis of the TMFs identified revealed relevant factors of interest for the evaluation of KMb strategies, classified into four dimensions: context, process, effects and impacts.

CONCLUSIONS

This scoping review provides an overview of the many KMb TMFs used or proposed. The results provide insight into potential dimensions and components to be considered when assessing KMb strategies.

What makes knowledge translation work in practice? Lessons from a demand-driven and locally led project in Cameroon, Jordan and Nigeria

BACKGROUND

Over the years, the knowledge translation (KT) field has moved from promoting linearized models to embracing the importance of interaction and learning. Likewise, there is now increased attention on the transfer of KT approaches to new environments. Some scholars, however, have warned that ideas about transferability still hinge on linear thinking and doing. In the current study, we therefore sought to use a more reflexive approach to KT and to study how actors align KT approaches with their local environments.

METHODS

Our (auto) ethnographic study took place in a wider KT project. This project intended to combine three components: (1) co-organizing demand-driven, locally led and embedded KT cycles in Cameroon, Jordan, and Nigeria, (2) building upon established KT methods and (3) equipping and empowering local teams. We conducted 63 semi-structured interviews with key KT actors, observed 472 h of KT practices, and collected a paper trail of documents. At the same time, we also compiled project exchanges, such as project documents, plans, protocols, field notes, meeting notes and an archive of (email) correspondence between project members. We analysed all data abductively.

RESULTS

We show that there were numerous moments where the design of our project indeed enabled us to align with local practices and needs. Yet this often did not suffice, and the project design sometimes conflicted with other logics and values. By analysing these tensions, we want to show that doing KT work which acts upon different values and knowledges and is sensitive towards the different effects that it produces demands both structuring projects in a specific way and requires significant alignment work of KT actors in practice.

CONCLUSIONS

We show that practising KT more reflexively relies on two important conditions. First, KT projects have to be structured with sufficient discretionary space. Second, even though the structure of a project is important, there will be continuous need for alignment work. It is important to facilitate such alignment work and to further support it. In the discussion of this paper, we therefore articulate three design principles and three sensitivities. These elements can be used to make future KT projects more reflexive and sensitive to (social) complexity.

"And when will you install the new water pump?": disconcerted reflections on how to be a 'good' Global Health scholar

BACKGROUND

While critique on Global Health is not new, recent years show a surge of criticism on the field's colonial legacy and practices specifically. Such accounts argue that despite Global Health's strive for universality and equity in health, its activities regularly produce the opposite. The epistemic privileging of Northern academics and scientific method, further augmented by how Global Health funding is arranged, paints a picture of a fragmented field in which 'doing good' has become a normatively laden and controversial term. It is specifically this controversy that we seek to unpack in this paper: what does it take to be a 'good' Global Health scholar?

RESULTS

We used Helen Verran's notion of 'disconcertment' to analyse three auto-ethnographic vignettes of Robert's Global Health 'fieldwork'. We illustrate that disconcertment, a bodily and personalised experience of unease and conflicting feelings, may serve as an important diagnostic of conflicting imperatives in Global Health. Robert's fieldwork was entangled with incongruous imperatives which he constantly had to navigate through and that repeatedly produced disconcertment. The contribution that we seek to make here is that such disconcertment is not something to defuse or ignore, but to take seriously and stay with instead.

CONCLUSION

Staying with the disconcertment serves as a starting point for conversations about 'doing good' in Global Health fieldwork and creates opportunity for making Global Health teaching and projects more reflexive. The paper thereby positions itself in discussions about fair collaborations between the Global North and South and our analysis offers a set of considerations that can be used by Northern scholars to critically reflect on their own role within Global Health.

The disconnect between researcher ambitions and reality in achieving impact in the Earth & Environmental Sciences – author survey

Background: There is an increasing desire for research to provide solutions to the grand challenges facing our global society, such as those expressed in the UN SDGs (“real-world impact”). Herein, we undertook an author survey to understand how this desire influenced the choice of research topic, choice of journal, and preferred type of impact. Methods: We conducted a survey of authors who had published in >100 of our Earth & Environmental Science journals. The survey was sent to just under 60,000 authors and we received 2,695 responses (4% response rate).   Results: Respondents indicated that the majority of their research (74%) is currently concerned with addressing urgent global needs, whilst 90% of respondents indicated that their work either currently contributed to meeting real-world problems or that it would be a priority for them in the future; however, the impetus for this research focus seems to be altruistic researcher desire, rather than incentives or support from publishers, funders, or their institutions. Indeed, when contextualised within existing reward and incentive structures, respondents indicated that citations or downloads were more important to them than contributing to tackling real-world problems. Conclusions: At present, it seems that the laudable and necessary ambition of researchers in the Earth & Environmental Sciences to contribute to the tackling of real-world problems, such as those included in the UN SDGs, is seemingly being lost amidst the realities of being a researcher, owing to the prioritisation of other forms of impact, such as citations and downloads.

Research impact as understood by two funders of agricultural research in South Africa

Impact literacy is the understanding of research impact in terms of a junction of three elements: ‘what’, ‘how’ and ‘who’. ‘What’ refers to impact endpoints, that is, the medium- to long-term impacts or even short-term effects that occur during or immediately after research. ‘How’ refers to the understanding of how impact is created, in other words the interactions required. ‘Who’ focuses on which stakeholders need to be integrated into a network of interactions to contribute to impact. This study focused on the ‘what’ of impact, specifically on the understanding of research impact by two funders of agricultural research in South Africa. Members of specialist committees at the two funding organizations were asked, through a survey, to rate several structured items mapped on a research impact classification scheme. They could also provide their own ideas on what impact is. Committee members from both organizations viewed research impact primarily as an effect on the individuals and groups in the industry that they represent. They generally did not consider research impact as an effect on researchers, nor did they place specific emphasis on the societal effects of research. The ‘what’ of research impact was found to imply a series of effects, where the different effects build on each other. Linking these effects to the ‘how’ and ‘who’ of impact will require the construction of impact pathways. Funders should take responsibility for pushing research results to a distal level by planning ‘who’ will do the push, ‘how’ it will be done and managed, and to ‘what’ final aim.

What funders are doing to assess the impact of their investments in health and biomedical research

As pressures to maximize research funding grow, biomedical research funders are increasingly tasked with demonstrating the long-term and real-world impacts of their funded research investments. Over the past three decades, research impact assessments (RIA) have emerged as an important tool for analysing the impacts of research by incorporating logic models, frameworks and indicators to track measures of knowledge production, capacity-building, development of research products, adoption of research into clinical guidelines and policies, and the realization of health, economic and social benefits. While there are currently several models for RIA within the literature, less attention has been paid to how funders can practically select and implement a RIA model to demonstrate the impacts of their own research portfolios. In this paper, a literature review was performed to understand (1) which research funders have performed RIAs of their research portfolios to date; (2) how funders have designed their assessments, including the models and tools they have used; (3) what challenges to and facilitators of success have funders found when adopting the RIA model to their own portfolio; and (4) who participates in the assessments. Forty-four papers from both published and grey literature were found to meet the review criteria and were examined in detail. There is a growing culture of RIA among funders, and included papers spanned a diverse set of funders from 10 countries or regions. Over half of funders (59.1%) used a framework to conduct their assessment, and a variety of methods for collecting impact data were reported. Issues of methodological rigour were observed across studies in the review, and this was related to numerous challenges funders faced in designing timely RIAs with quality impact data. Over a third of articles (36.4%) included input from stakeholders, yet only one article reported surveying patients and members of the public as part of the assessment. To advance RIA among funders, we offer several recommendations for increasing the methodological rigour of RIAs and suggestions for future research, and call for a careful reflection of the voices needed in an impact assessment to ensure that RIAs are having a meaningful impact on patients and the public.

The disconnect between researcher ambitions and reality in achieving impact in the Earth & Environmental Sciences – author survey

Background: There is an increasing desire for research to provide solutions to the grand challenges facing our global society, such as those expressed in the UN SDGs (“real-world impact”). Herein, we undertook an author survey to understand how this desire influenced the choice of research topic, choice of journal, and preferred type of impact. Methods: We conducted a survey of authors who had published in >100 of our Earth & Environmental Science journals. The survey was sent to just under 60,000 authors and we received 2,695 responses (4% response rate).   Results: Respondents indicated that the majority of their research (74%) is currently concerned with addressing urgent global needs, whilst 90% of respondents indicated that their work either currently contributed to meeting real-world problems or that it would be a priority for them in the future; however, the impetus for this research focus seems to be altruistic researcher desire, rather than incentives or support from publishers, funders, or their institutions. Indeed, when contextualised within existing reward and incentive structures, respondents indicated that citations or downloads were more important to them than contributing to tackling real-world problems. Conclusions: At present, it seems that the laudable and necessary ambition of researchers in the Earth & Environmental Sciences to contribute to the tackling of real-world problems, such as those included in the UN SDGs, is seemingly being lost amidst the realities of being a researcher, owing to the prioritisation of other forms of impact, such as citations and downloads.

Collective health research assessment: developing a tool to measure the impact of multistakeholder research initiatives

Background

The need to more collaboratively measure the impact of health research and to do so from multidimensional perspectives has been acknowledged. A scorecard was developed as part of the Collective Research Impact Framework (CRIF), to engage stakeholders in the assessment of the impacts of health research and innovations. The purpose of this study was to describe the developmental process of the MULTI-ACT Master Scorecard (MSC) and how it can be used as a workable tool for collectively assessing future responsible research and innovation measures.

Methods

An extensive review of the health research impact literature and of multistakeholder initiatives resulted in a database of 1556 impact indicators. The MSC was then cocreated by engaging key stakeholders and conducting semi-structured interviews of experts in the field.

Results

The MSC consists of five accountability dimensions: excellence, efficacy, economic, social and patient-reported outcomes. The tool contains 125 potential indicators, classified into 53 impact measurement aspects that are considered the most relevant topics for multistakeholder research and innovation initiatives when assessing their impact on the basis of their mission and their stakeholders’ interests. The scorecard allows the strategic management of multistakeholder research initiatives to demonstrate their impact on people and society. The value of the tool is that it is comprehensive, customizable and easy to use.

Conclusions

The MSC is an example of how the views of society can be taken into account when research impacts are assessed in a more sustainable and balanced way. The engagement of patients and other stakeholders is an integral part of the CRIF, facilitating collaborative decision-making in the design of policies and research agendas. In policy making, the collective approach allows the evaluation perspective to be extended to the needs of society and towards responsible research and innovation. Multidimensionality makes research and innovations more responsive to systemic challenges, and developing more equitable and sustainable health services.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-022-00856-9.

Strategies to promote evidence use for health programme improvement: learning from the experiences of embedded implementation research teams in Latin America and the Caribbean

Background

To achieve global health targets, innovative approaches are needed to strengthen the implementation of efficacious interventions. New approaches in implementation research that bring together health system decision-makers alongside researchers to collaboratively design, produce and apply research evidence are gaining traction. Embedded implementation research (EIR) approaches led by decision-maker principal investigators (DM PIs) appear promising in this regard. Our aim is to describe the strategies study teams employ in the post-research phase of EIR to promote evidence-informed programme or policy improvement.

Methods

We conducted a prospective, comparative case study of an EIR initiative in Bolivia, Colombia and Dominican Republic. Guided by a conceptual framework on EIR, we used semi-structured key informant interviews (n = 51) and document reviews (n = 20) to examine three decision-maker-led study teams (“cases”). Focusing on three processes (communication/dissemination, stakeholder engagement with evidence, integrating evidence in decision-making) and the main outcome (enacting improvements), we used thematic analysis to identify associated strategies and enabling or hindering factors.

Results

Across cases, we observed diverse strategies, shaped substantially by whether the DM PI was positioned to lead the response to study findings within their sphere of work. We found two primary change pathways: (1) DM PIs implement remedial measures directly, and (2) DM PIs seek to influence other stakeholders to respond to study findings. Throughout the post-research phase, EIR teams adapted research use strategies based on the evolving context.

Conclusions

EIR led by well-positioned DM PIs can facilitate impactful research translation efforts. We draw lessons around the importance of (1) understanding DM PI positionality, (2) ongoing assessment of the evolving context and stakeholders and (3) iterative adaptation to dynamic, uncertain circumstances. Findings may guide EIR practitioners in planning and conducting fit-for-purpose and context-sensitive strategies to advance the use of evidence for programme improvement.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-022-00834-1.

A Research Pathway Model for evaluating the implementation of practice-based research: The case of self-management health innovations

This study explores the evaluation of research pathways of self-management health innovations from discovery to implementation in the context of practice-based research. The aim is to understand how a new process model for evaluating practice-based research provides insights into the implementation success of innovations. Data were collected from nine research projects in the Netherlands. Through document analysis and semi-structured interviews, we analysed how the projects start, evolve, and contribute to the healthcare practice. Building on previous research evaluation approaches to monitor knowledge utilization, we developed a Research Pathway Model. The model’s process character enables us to include and evaluate the incremental work required throughout the lifespan of an innovation project and it helps to foreground that innovation continues during implementation in real-life settings. We found that in each research project, pathways are followed that include activities to explore a new solution, deliver a prototype and contribute to theory. Only three projects explored the solution in real life and included activities to create the necessary changes for the solutions to be adopted. These three projects were associated with successful implementation. The exploration of the solution in a real-life environment in which users test a prototype in their own context seems to be a necessary research activity for the successful implementation of self-management health innovations.

Looking for evidence of research impact and use: A qualitative study of an Australian research-policy system

Current assessments of research impact have been criticized for capturing what can be easily counted not what actually counts. To empirically examine this issue, we approached measuring research impact from two directions, tracing forwards from research and backwards from policy, within a defined research-policy system (childhood obesity prevention research and policy in New South Wales, Australia from 2000 to 2015). The forward tracing research impact assessment component traced a sample of 148 local research projects forward to examine their policy impacts. Of the projects considered, 16% had an impact on local policy and for a further 19%, decision-makers were aware of the research, but there was no evidence it influenced policy decisions. The backward tracing component of the study included an analysis of research use across three policy initiatives. It provided a more nuanced understanding of the relative influence of research on policy. Both direct uses of specific research and indirect uses of research incorporated as broader bodies of knowledge were evident. Measuring research impact from both directions captured the diverse ways that research was used in decision-making. Our findings illustrate complexities in the assessment process and in real-life policymaking trajectories. They highlight the role that timing of assessment plays in perception of impacts and difficulties attributing longer-term impacts to specific research. This study supports the use of models where politics and complex system dynamics shape knowledge and its influence on decision-making, rather than research being the primary driver for policy change.

Exploring research impact models: A systematic scoping review

This article explores the models and frameworks developed on “research impact’. We aim to provide a comprehensive overview of related literature through scoping study method. The present research investigates the nature, objectives, approaches, and other main attributes of the research impact models. It examines to analyze and classify models based on their characteristics. Forty-seven studies and 10 reviews published between 1996 and 2020 were included in the analysis. The majority of models were developed for the impact assessment and evaluation purposes. We identified three approaches in the models, namely outcome-based, process-based, and those utilized both of them, among which the outcome-based approach was the most frequently used by impact models and evaluation was considered as the main objective of this group. The process-based ones were mainly adapted from the W.K. Kellogg Foundation logic model and were potentially eligible for impact improvement. We highlighted the scope of processes and other specific features for the recent models. Given the benefits of the process-based approach in enhancing and accelerating the research impact, it is important to consider such approach in the development of impact models. Effective interaction between researchers and stakeholders, knowledge translation, and evidence synthesis are the other possible driving forces contributing to achieve and improve impact.

The production of scientific and societal value in research evaluation: a review of societal impact assessment methods

Over the past two decades, several methods have been developed to evaluate the societal impact of research. Compared to the practical development of the field, the conceptual development is relatively weak. This review article contributes to the latter by elucidating the theoretical aspects of the dominant methods for evaluating societal impact of research, in particular, their presuppositions about the relationship between scientific and societal value of research. We analyse 10 approaches to the assessment of the societal impact of research from a constructivist perspective. The methods represent different understandings of knowledge exchange, which can be understood in terms of linear, cyclical, and co-production models. In addition, the evaluation methods use a variety of concepts for the societal value of research, which suggest different relationships with scientific value. While some methods rely on a clear and explicit distinction between the two types of value, other methods, in particular Evaluative Inquiry, ASIRPA, Contribution Mapping, Public Value Mapping, and SIAMPI, consider the mechanisms for producing societal value integral to the research process. We conclude that evaluation methods must balance between demarcating societal value as a separate performance indicator for practical purposes and doing justice to the (constructivist) science studies’ findings about the integration of scientific and societal value of research. Our analytic comparison of assessment methods can assist research evaluators in the conscious and responsible selection of an approach that fits with the object under evaluation. As evaluation actively shapes knowledge production, it is important not to use oversimplified concepts of societal value.

Employing the Payback Framework to Assess Implementation Science Research Utilization: Lessons From the USAID's PEPFAR HIV/AIDS Implementation Science Awards

BACKGROUND

Stakeholders question whether implementation science (IS) is successful in conducting rigorous science that expedites the inclusion of health innovations into policies and accelerates the pace and scale of health service delivery into clinical and public health practice. Using the Payback Framework (PF) for research utilization (RU), we assessed the impact of USAID's IS investment on a subset of studies examining HIV prevention, care, and treatment.

SETTING

Selected USAID-funded IS awards implemented between 2012 and 2017 in 9 sub-Saharan African countries.

METHODS

A modified version of a RU framework, the PF, was applied to 10 USAID-funded IS awards. A semistructured, self-administered/interviewer-administered questionnaire representing operational items for the 5 categories of the modified PF was used to describe the type and to quantify the level of payback achieved. The raw score was tallied within and across the 5 PF categories, and the percentage of "payback" achieved by category was tabulated. Distribution of payback scores was summarized by tertiles.

RESULTS

Knowledge production had the highest level of payback (75%), followed by benefits to future research (70%), benefits to policy (45%), benefits to health and the health system (18%), and broader economic benefits (5%).

CONCLUSIONS

All awards achieved some level of knowledge production and benefits to future research, but translation to policy and programs was low and variable. We propose the use of policy, health system, and economic monitoring indicators of RU throughout the research process to increase IS studies' impact on health practice, programs, and policy.

Methods for mapping the impact of social sciences and humanities—A literature review

This article explores the current literature on ‘research impact’ in the social sciences and humanities (SSH). By providing a comprehensive review of available literature, drawing on national and international experiences, we take a systematic look at the impact agenda within SSH. The primary objective of this article is to examine key methodological components used to assess research impact comparing the advantages and disadvantages of each method. The study finds that research impact is a highly complex and contested concept in the SSH literature. Drawing on the strong methodological pluralism emerging in the literature, we conclude that there is considerable room for researchers, universities, and funding agencies to establish impact assessment tools directed towards specific missions while avoiding catch-all indicators and universal metrics.

Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?

Societal impact of research does not occur primarily as unexpected extraordinary incidents of particularly useful breakthroughs in science. It is more often a result of normal everyday interactions between organizations that need to create, exchange, and make use of new knowledge to further their goals. We use the distinctions between normal and extraordinary societal impact and between organizational- and individual-level activities and responsibilities to discuss how science–society relations can better be understood, evaluated, and improved by focusing on the organizations that typically interact in a specific domain of research.

The impact of patient-reported outcome (PRO) data from clinical trials: a systematic review and critical analysis

Background

Patient-reported outcomes (PROs) are commonly collected in clinical trials and should provide impactful evidence on the effect of interventions on patient symptoms and quality of life. However, it is unclear how PRO impact is currently realised in practice. In addition, the different types of impact associated with PRO trial results, their barriers and facilitators, and appropriate impact metrics are not well defined. Therefore, our objectives were: i) to determine the range of potential impacts from PRO clinical trial data, ii) identify potential PRO impact metrics and iii) identify barriers/facilitators to maximising PRO impact; and iv) to examine real-world evidence of PRO trial data impact based on Research Excellence Framework (REF) impact case studies.

Methods

Two independent investigators searched MEDLINE, EMBASE, CINAHL+, HMIC databases from inception until December 2018. Articles were eligible if they discussed research impact in the context of PRO clinical trial data. In addition, the REF 2014 database was systematically searched. REF impact case studies were included if they incorporated PRO data in a clinical trial.

Results

Thirty-nine publications of eleven thousand four hundred eighty screened met the inclusion criteria. Nine types of PRO trial impact were identified; the most frequent of which centred around PRO data informing clinical decision-making. The included publications identified several barriers and facilitators around PRO trial design, conduct, analysis and report that can hinder or promote the impact of PRO trial data. Sixty-nine out of two hundred nine screened REF 2014 case studies were included. 12 (17%) REF case studies led to demonstrable impact including changes to international guidelines; national guidelines; influencing cost-effectiveness analysis; and influencing drug approvals.

Conclusions

PRO trial data may potentially lead to a range of benefits for patients and society, which can be measured through appropriate impact metrics. However, in practice there is relatively limited evidence demonstrating directly attributable and indirect real world PRO-related research impact. In part, this is due to the wider challenges of measuring the impact of research and PRO-specific issues around design, conduct, analysis and reporting. Adherence to guidelines and multi-stakeholder collaboration is essential to maximise the use of PRO trial data, facilitate impact and minimise research waste.

Trial registration

Systematic Review registration PROSPERO CRD42017067799.

Trends in Scientific Reports on Cartilage Bioprinting: Scoping Review

Background

Satisfactory therapeutic strategies for cartilaginous lesion repair do not yet exist. This creates a challenge for surgeons and biomedical engineers and leads them to investigate the role of bioprinting and tissue engineering as viable treatments through orthopedic surgery, plastic surgery, and otorhinolaryngology. Recent increases in related scientific literature suggest that bioprinted cartilage may develop into a viable solution.

Objective

The objectives of this review were to (1) synthesize the scientific advances published to date, (2) identify unresolved technical problems regarding human application, and (3) identify more effective ways for the scientific community to transfer their findings to clinicians.

Methods

This scoping review considered articles published between 2009 and 2019 that were identified through searching PubMed, Scopus, Web of Science, and Google Scholar. Arksey and O'Malley’s five-step framework was used to delimit and direct the initial search results, from which we established the following research questions: (1) What do authors of current research say about human application? (2) What necessary technical improvements are identified in the research? (3) On which issues do the authors agree? and (4) What future research priorities emerge in the studies? We used the Cohen kappa statistic to validate the interrater reliability.

Results

The 13 articles included in the review demonstrated the feasibility of cartilage bioprinting in live animal studies. Some investigators are already considering short-term human experimentation, although technical limitations still need to be resolved. Both the use and manufacturing process of stem cells need to be standardized, and a consensus is needed regarding the composition of hydrogels. Using on-site printing strategies and predesigned implants may allow techniques to adapt to multiple situations. In addition, the predictive capacity of implant behavior may lead to optimal results.

Conclusions

Cartilage bioprinting for surgical applications is nearing its initial use in humans. Current research suggests that surgeons will soon be able to replace damaged tissue with bioprinted material.

Envisioning and shaping translation of knowledge into action: A comparative case-study of stakeholder engagement in the development of a European tobacco control tool

Stakeholder engagement in health policy research is often said to increase 'research impact', but the active role of stakeholders in creating impact remains underexplored. We explored how stakeholders shaped the translation of health policy research into action. Our comparative case-study tracked a European research project that aimed to transfer an existing tobacco control return on investment tool. That project also aimed to increase its impact by engaging with stakeholders in further developing the tool. We conducted semi-structured interviews, using an actor-scenario mapping approach. Actor-scenarios can be seen as relational descriptions of a future world. We mapped the scenarios by asking stakeholders to describe who and what would play a role in the tool's utilisation. Our results show that stakeholders envisioned disparate futures for the tool. Some scenarios were specific, whereas most were generic projections of abstract potential users and responsibilities. We show how stakeholders mobilised elements of context, such as legislative support and agricultural practice, that would affect the tool's use. We conclude that stakeholders shape knowledge translation processes by continuously putting forth explicit or implicit scenarios about the future. Mapping actor-scenarios may help in aligning knowledge production with utilisation. Insights into potential roles and responsibilities could be fed back in research projects with the aim of increasing the likelihood that the study results may be used.

Using the Contribution Analysis Approach to Evaluate Science Impact: A Case Study of the National Institute for Occupational Safety and Health

Interest from Congress, executive branch leadership, and various other stakeholders for greater accountability in government continues to gain momentum today with government-wide efforts. However, measuring the impact of research programs has proven particularly difficult. Cause and effect linkages between research findings and changes to morbidity and mortality are difficult to prove. To address this challenge, the National Institute for Occupational Safety and Health program evaluators used a modified version of contribution analysis (CA) to evaluate two research programs. CA proved to be a useful framework for assessing research impact, and both programs received valuable, actionable feedback. Although there is room to further refine our approach, this was a promising step toward moving beyond bibiliometrics to more robust assessment of research impact.

Embracing complexity and uncertainty to create impact: exploring the processes and transformative potential of co-produced research through development of a social impact model

The potential use, influence and impact of health research is seldom fully realised. This stubborn problem has caused burgeoning global interest in research aiming to address the implementation ‘gap’ and factors inhibiting the uptake of scientific evidence. Scholars and practitioners have questioned the nature of evidence used and required for healthcare, highlighting the complex ways in which knowledge is formed, shared and modified in practice and policy. This has led to rapid expansion, expertise and innovation in the field of knowledge mobilisation and funding for experimentation into the effectiveness of different knowledge mobilisation models. One approach gaining prominence involves stakeholders (e.g. researchers, practitioners, service users, policy-makers, managers and carers) in the co-production, and application, of knowledge for practice, policy and research (frequently termed integrated knowledge translation in Canada). Its popularity stems largely from its potential to address dilemmas inherent in the implementation of knowledge generated using more reductionist methods. However, despite increasing recognition, demands for co-produced research to illustrate its worth are becoming pressing while the means to do so remain challenging. This is due not only to the diversity of approaches to co-production and their application, but also to the ways through which different stakeholders conceptualise, measure, reward and use research. While research co-production can lead to demonstrable benefits such as policy or practice change, it may also have more diffuse and subtle impact on relationships, knowledge sharing, and in engendering culture shifts and research capacity-building. These relatively intangible outcomes are harder to measure and require new emphases and tools. This opinion paper uses six Canadian and United Kingdom case studies to explore the principles and practice of co-production and illustrate how it can influence interactions between research, policy and practice, and benefit diverse stakeholders. In doing so, we identify a continuum of co-production processes. We propose and illustrate the use of a new ‘social model of impact’ and framework to capture multi-layered and potentially transformative impacts of co-produced research. We make recommendations for future directions in research co-production and impact measurement.

In the eye of the beholder: to make global health estimates useful, make them more socially robust

A plethora of new development goals and funding institutions have greatly increased the demand for internationally comparable health estimates in recent years, and have brought important new players into the field of health estimate production. These changes have rekindled debates about the validity and legitimacy of global health estimates. This paper draws on country case studies and personal experience to support our opinion that the production and use of estimates are deeply embedded in specific social, economic, political and ideational contexts, which differ at different levels of the global health architecture.

Broadly, most global health estimates tend to be made far from the local contexts in which the data upon which they are based are collected, and where the results of estimation processes must ultimately be used if they are to make a difference to the health of individuals. Internationally standardised indicators are necessary, but they are no substitute for data that meet local needs, and that fit with local ideas of what is credible and useful. In other words, data that are both technically and socially robust for those who make key decisions about health.

We suggest that greater engagement of local actors (and local data) in the formulation, communication and interpretation of health estimates would increase the likelihood that these data will be used by those most able to translate them into health gains for the longer term. Besides strengthening national information systems, this requires ongoing interaction, building trust and establishing a communicative infrastructure. Local capacities to use knowledge to improve health must be supported.

Assessing the influence of knowledge translation platforms on health system policy processes to achieve the health millennium development goals in Cameroon and Uganda: a comparative case study

There is a scarcity of empirical data on the influence of initiatives supporting evidence-informed health system policy-making (EIHSP), such as the knowledge translation platforms (KTPs) operating in Africa. To assess whether and how two KTPs housed in government-affiliated institutions in Cameroon and Uganda have influenced: (1) health system policy-making processes and decisions aiming at supporting achievement of the health millennium development goals (MDGs); and (2) the general climate for EIHSP. We conducted an embedded comparative case study of four policy processes in which Evidence Informed Policy Network (EVIPNet) Cameroon and Regional East African Community Health Policy Initiative (REACH-PI) Uganda were involved between 2009 and 2011. We combined a documentary review and semi structured interviews of 54 stakeholders. A framework-guided thematic analysis, inspired by scholarship in health policy analysis and knowledge utilization was used. EVIPNet Cameroon and REACH-PI Uganda have had direct influence on health system policy decisions. The coproduction of evidence briefs combined with tacit knowledge gathered during inclusive evidence-informed stakeholder dialogues helped to reframe health system problems, unveil sources of conflicts, open grounds for consensus and align viable and affordable options for achieving the health MDGs thus leading to decisions. New policy issue networks have emerged. The KTPs indirectly influenced health policy processes by changing how interests interact with one another and by introducing safe-harbour deliberations and intersected with contextual ideational factors by improving access to policy-relevant evidence. KTPs were perceived as change agents with positive impact on the understanding, acceptance and adoption of EIHSP because of their complementary work in relation to capacity building, rapid evidence syntheses and clearinghouse of policy-relevant evidence. This embedded case study illustrates how two KTPs influenced policy decisions through pathways involving policy issue networks, interest groups interaction and evidence-supported ideas and how they influenced the general climate for EIHSP.

Assessing the policy and practice impact of an international policy initiative: the State of the World's Midwifery 2014

BACKGROUND

Understanding how policies lead to changes in health systems and in practice helps policymakers and researchers to intervene more successfully. Yet identifying all the possible changes that occur as a result of a new policy is challenging not only methodologically and logistically, as limited resources are available to conduct indefinite evaluations, but also theoretically, as a complete mapping and attribution of post-hoc changes requires a full understanding of the mechanisms underpinning all change. One option is to identify possible changes across a number of policy impact domains.

METHODS

Using a Policy Impact Framework, we brought together data from media, documents and interviews to identify changes to midwifery policy, practice and provision, following the launch of a new global policy initiative, the State of the World's Midwifery (SoWMy 2014) report published in 2014. We used these identified impacts to develop a map of the mechanisms underpinning these changes.

RESULTS

SoWMy 2014 contributed to a number of changes at national levels, including increased status of midwifery within national governments, improved curricula and training opportunities for midwives, and improved provision of and access to midwifery-led care. These contributions were attributed to SoWMy 2014 via mechanisms such as stakeholder interaction and acquisition of government support, holding national and international dissemination and training events, and a perceived global momentum around supporting midwifery provision. Policy initiatives of this kind can lead to changes in national and international policy dialogue and practice. We identify factors and mechanisms that are likely to increase the scope and scale of these changes, at contextual, national and global levels.

CONCLUSIONS

Identifying changes following a policy using a policy impact framework can help researchers and policymakers understand why policies have the effect they do. This is important information for those wishing to increase the effectiveness of future policies and interventions.

Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Background

Measuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research.

Methods

Keyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework.

Results

We identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilised forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change.

Conclusions

The Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0310-4) contains supplementary material, which is available to authorized users.

Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme

BACKGROUND

This report reviews approaches and tools for measuring the impact of research programmes, building on, and extending, a 2007 review.

OBJECTIVES

(1) To identify the range of theoretical models and empirical approaches for measuring the impact of health research programmes; (2) to develop a taxonomy of models and approaches; (3) to summarise the evidence on the application and use of these models; and (4) to evaluate the different options for the Health Technology Assessment (HTA) programme.

DATA SOURCES

We searched databases including Ovid MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library from January 2005 to August 2014.

REVIEW METHODS

This narrative systematic literature review comprised an update, extension and analysis/discussion. We systematically searched eight databases, supplemented by personal knowledge, in August 2014 through to March 2015.

RESULTS

The literature on impact assessment has much expanded. The Payback Framework, with adaptations, remains the most widely used approach. It draws on different philosophical traditions, enhancing an underlying logic model with an interpretative case study element and attention to context. Besides the logic model, other ideal type approaches included constructionist, realist, critical and performative. Most models in practice drew pragmatically on elements of several ideal types. Monetisation of impact, an increasingly popular approach, shows a high return from research but relies heavily on assumptions about the extent to which health gains depend on research. Despite usually requiring systematic reviews before funding trials, the HTA programme does not routinely examine the impact of those trials on subsequent systematic reviews. The York/Patient-Centered Outcomes Research Institute and the Grading of Recommendations Assessment, Development and Evaluation toolkits provide ways of assessing such impact, but need to be evaluated. The literature, as reviewed here, provides very few instances of a randomised trial playing a major role in stopping the use of a new technology. The few trials funded by the HTA programme that may have played such a role were outliers.

DISCUSSION

The findings of this review support the continued use of the Payback Framework by the HTA programme. Changes in the structure of the NHS, the development of NHS England and changes in the National Institute for Health and Care Excellence's remit pose new challenges for identifying and meeting current and future research needs. Future assessments of the impact of the HTA programme will have to take account of wider changes, especially as the Research Excellence Framework (REF), which assesses the quality of universities' research, seems likely to continue to rely on case studies to measure impact. The HTA programme should consider how the format and selection of case studies might be improved to aid more systematic assessment. The selection of case studies, such as in the REF, but also more generally, tends to be biased towards high-impact rather than low-impact stories. Experience for other industries indicate that much can be learnt from the latter. The adoption of researchfish^® (researchfish Ltd, Cambridge, UK) by most major UK research funders has implications for future assessments of impact. Although the routine capture of indexed research publications has merit, the degree to which researchfish will succeed in collecting other, non-indexed outputs and activities remains to be established.

LIMITATIONS

There were limitations in how far we could address challenges that faced us as we extended the focus beyond that of the 2007 review, and well beyond a narrow focus just on the HTA programme.

CONCLUSIONS

Research funders can benefit from continuing to monitor and evaluate the impacts of the studies they fund. They should also review the contribution of case studies and expand work on linking trials to meta-analyses and to guidelines.

FUNDING

The National Institute for Health Research HTA programme.

Towards fair and effective North-South collaboration: realising a programme for demand-driven and locally led research

BACKGROUND

At the turn of the 90s, studies showed that health research contributed little to health and development in low- and middle-income countries because it was oriented towards international priorities and dominated by researchers from the North. A new approach to North-South collaboration was required that would support demand-driven and locally led research in the South. The aim of this study was to analyse the development and functioning of a programme for demand-driven and locally led research in Ghana that was supported by a North-South collaboration.

METHODS

For this mixed-method case study, we combined document analysis, key informant interviews and observation of programme events.

RESULTS

The development of the research programme started with constructing a sponsorship constellation in the Netherlands. After highlighting the problems with traditional research collaboration, an advisory council formulated a vision for a more equal and effective approach to North-South collaboration. Together with Ghanaian partners, this vision was turned into a proposal for a Ghanaian-led programme for demand-driven and locally led research, which was funded by the Netherlands government. Research priority setting showed that the Ghanaian research needs were very different from the priorities of foreign funders and researchers. After a slow start, the number of locally submitted proposals increased from 13 in 2001 to 94 in 2005, revealing the existence of a substantial, but partly latent reservoir of research capacity. In total, 79 studies were funded. An impact evaluation showed that the results of the majority of the studies were used to contribute to action. Despite its success, the research programme came to an end in 2008 after the sponsorship constellation in the Netherlands fell apart.

CONCLUSION

Our study shows that realising a programme for demand-driven and locally led research in the South provides an effective approach to North-South collaboration in which results are used and local capacities and institutions are strengthened.

'Knowledge for better health' revisited - the increasing significance of health research systems: a review by departing Editors-in-Chief

How can nations organise research investments to obtain the best bundle of knowledge and the maximum level of improved health, spread as equitably as possible? This question was the central focus of a major initiative from WHO led by Prof Tikki Pang, which resulted in a range of developments, including the publication of a conceptual framework for national health research systems - Knowledge for better health - in 2003, and in the founding of the journal Health Research Policy and Systems (HARPS). As Editors-in-Chief of the journal since 2006, we mark our retirement by tracking both the progress of the journal and the development of national health research systems. HARPS has maintained its focus on a range of central themes that are key components of a national health research system in any country. These include building capacity to conduct and use health research, identifying appropriate priorities, securing funds and allocating them accountably, producing scientifically valid research outputs, promoting the use of research in polices and practice in order to improve health, and monitoring and evaluating the health research system. Some of the themes covered in HARPS are now receiving increased attention and, for example, with the assessment of research impact and development of knowledge translation platforms, the journal has covered their progress throughout that expansion of interest. In addition, there is increasing recognition of new imperatives, including the importance of promoting gender equality in health research if benefits are to be maximised. In this Editorial, we outline some of the diverse and developing perspectives considered within each theme, as well as considering how they are held together by the growing desire to build effective health research systems in all countries.From 2003 until mid-June 2017, HARPS published 590 articles on the above and related themes, with authors being located in 76 countries. We present quantitative data tracing the journal's growth and the increasing external recognition of its role. We thank the many colleagues who have kindly contributed to the journal's success, and finish on an exciting note by welcoming the new Editors-in-Chief who will take HARPS forward.

Assessing the impact of healthcare research: A systematic review of methodological frameworks

BACKGROUND

Increasingly, researchers need to demonstrate the impact of their research to their sponsors, funders, and fellow academics. However, the most appropriate way of measuring the impact of healthcare research is subject to debate. We aimed to identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix.

METHODS AND FINDINGS

Two independent investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the common concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Twenty-four unique methodological frameworks were identified, addressing 5 broad categories of impact: (1) 'primary research-related impact', (2) 'influence on policy making', (3) 'health and health systems impact', (4) 'health-related and societal impact', and (5) 'broader economic impact'. These categories were subdivided into 16 common impact subgroups. Authors of the included publications proposed 80 different metrics aimed at measuring impact in these areas. The main limitation of the study was the potential exclusion of relevant articles, as a consequence of the poor indexing of the databases searched.

CONCLUSIONS

The measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise research benefit, and to help minimise research waste. This review provides a collective summary of existing methodological frameworks for research impact, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

Research impact of systems-level long-term care research: a multiple case study

BACKGROUND

Traditional reporting of research outcomes and impacts, which tends to focus on research product publications and grant success, does not capture the value, some contributions, or the complexity of research projects. The purpose of this study was to understand the contributions of five systems-level research projects as they were unfolding at the Bruyère Centre for Learning, Research and Innovation (CLRI) in long-term care (LTC) in Ottawa, Ontario, Canada. The research questions were, (1) How are partnerships with research end-users (policymakers, administrators and other public/private organisations) characterised? (2) How have interactions with the CLRI Management Committee and Steering Committee influenced the development of research products? (3) In what way have other activities, processes, unlinked actors or organisations been influenced by the research project activities?

METHODS

The study was guided by Kok and Schuit's concept of research impacts, using a multiple case study design. Data were collected through focus groups and interviews with research teams, a management and a steering committee, research user partners, and unlinked actors. Documents were collected and analysed for contextual background.

RESULTS

Cross-case analysis revealed four major themes: (1) Benefits and Perceived Tensions: Working with Partners; (2) Speaking with the LTC Community: Interactions with the CLRI Steering Committee; (3) The Knowledge Broker: Interactions with the Management Committee; and (4) All Forms of Research Contributions.

CONCLUSIONS

Most contributions were focused on interactions with networks and stimulating important conversations in the province about LTC issues. These contributions were well-supported by the Steering and Management Committees' research-to-action platform, which can be seen as a type of knowledge brokering model. It was also clear that researcher-user partnerships were beneficial and important.

Research for Policy (R4P): development of a reflection tool for researchers to improve knowledge utilization

Background

To improve knowledge utilization in policymaking, alignment between researchers and policymakers during knowledge production is essential, but difficult to maintain. In three previously reported case studies, we extensively evaluated complex research projects commissioned by policymakers to investigate how alignment is achieved in a research process and to discover ways to enhance knowledge contributions to health policy. In the present study, we investigated how the findings of these three research projects could be integrated into a practical tool for researchers to enhance their contribution to evidence-based policy.

Methods

A cross-case analysis was conducted to integrate the findings of the evaluation of the three research projects and to identify important alignment areas in these projects. By means of an iterative process, we prepared a tool that includes reflection questions for researchers. The “Research for Policy” tool was tested with input from the project managers of three new research projects. Based on the findings, the final version of the Research for Policy tool was prepared.

Results

By cross-case analysis of the three case studies, the following important alignment areas were identified: the goal, quality, relevance, timing, and presentation of research, the tasks and authorities of actors, the consultative structure and vertical alignment within organizations, and the organizational environment.

The project managers regarded the Research for Policy tool as a useful checklist for addressing the important alignment areas in a research project. Based on their feedback, the illustrative examples from the case studies were added to the reflection questions. The project managers suggested making the tool accessible not only to researchers but also to policymakers. The format of the Research for Policy tool was further adjusted to users’ needs by adding clickable links.

Conclusions

Alignment between research and policymaking requires continuous efforts and a clear understanding of process issues in the research project. The Research for Policy tool offers practical alignment guidance and facilitates reflection on process issues, which supports researchers in aligning with policymakers and in acting in a context-sensitive way.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-016-0496-1) contains supplementary material, which is available to authorized users.

Obstacles and Enablers on the Way towards Integrated Physical Activity Policies for Childhood Obesity Prevention: An Exploration of Local Policy Officials' Views

Background. Limited physical activity (PA) is a risk factor for childhood obesity. In Netherlands, as in many other countries worldwide, local policy officials bear responsibility for integrated PA policies, involving both health and nonhealth domains. In practice, its development seems hampered. We explore which obstacles local policy officials perceive in their effort. Methods. Fifteen semistructured interviews were held with policy officials from health and nonhealth policy domains, working at strategic, tactic, and operational level, in three relatively large municipalities. Questions focused on exploring perceived barriers for integrated PA policies. The interviews were deductively coded by applying the Behavior Change Ball framework. Findings. Childhood obesity prevention appeared on the governmental agenda and all officials understood the multicausal nature. However, operational officials had not yet developed a tradition to develop integrated PA policies due to insufficient boundary-spanning skills and structural and cultural differences between the domains. Tactical level officials did not sufficiently support intersectoral collaboration and strategic level officials mainly focused on public-private partnerships. Conclusion. Developing integrated PA policies is a bottom-up innovation process that needs to be supported by governmental leaders through better guiding organizational processes leading to such policies. Operational level officials can assist in this by making progress in intersectoral collaboration visible.

Contributions of knowledge products to health policy: a case study on the Public Health Status and Forecasts Report 2010

Background: The Dutch Public Health Status and Forecasts report (PHSF Report) integrates research data and identifies future trends affecting public health in the Netherlands. To investigate how PHSF contributions to health policy can be enhanced, we analysed the development process whereby the PHSF Report for 2010 was produced (PHSF-2010). Method: To collect data, a case study approach was used along the lines of Contribution Mapping including analysis of documents from the PHSF-2010 process and interviews with actors involved. All interviews were recorded and transcribed ad verbatim and coded using an inductive code list. Results: The PHSF-2010 process included activities aimed at alignment between researchers and policy-makers, such as informal meetings. However, we identified three issues that are easily overlooked in knowledge development, but provide suggestions for enhancing contributions: awareness of divergent; continuously changing actor scenarios; vertical alignment within organizations involved and careful timing of draft products to create early adopters. Conclusion: To enhance the contributions made by an established public health report, such as the PHSF Report, it is insufficient to raise the awareness of potential users. The knowledge product must be geared to policy-makers’ needs and must be introduced into the scenarios of actors who may be less familiar. The demand for knowledge product adaptations has to be considered. This requires continuous alignment efforts in all directions: horizontal and vertical, external and internal. The findings of this study may be useful to researchers who aim to enhance the contributions of their knowledge products to health policy.

Research impact: a narrative review

Impact occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base. Its mechanisms are complex and reflect the multiple ways in which knowledge is generated and utilised. Much progress has been made in measuring both the outcomes of research and the processes and activities through which these are achieved, though the measurement of impact is not without its critics. We review the strengths and limitations of six established approaches (Payback, Research Impact Framework, Canadian Academy of Health Sciences, monetisation, societal impact assessment, UK Research Excellence Framework) plus recently developed and largely untested ones (including metrics and electronic databases). We conclude that (1) different approaches to impact assessment are appropriate in different circumstances; (2) the most robust and sophisticated approaches are labour-intensive and not always feasible or affordable; (3) whilst most metrics tend to capture direct and proximate impacts, more indirect and diffuse elements of the research-impact link can and should be measured; and (4) research on research impact is a rapidly developing field with new methodologies on the horizon.

Which health research gets used and why? An empirical analysis of 30 cases

BACKGROUND

While health research is considered essential for improving health worldwide, it remains unclear how it is best organized to contribute to health. This study examined research that was part of a Ghanaian-Dutch research program that aimed to increase the likelihood that results would be used by funding research that focused on national research priorities and was led by local researchers. The aim of this study was to map the contribution of this research to action and examine which features of research and translation processes were associated with the use of the results.

METHODS

Using Contribution Mapping, we systematically examined how 30 studies evolved and how results were used to contribute to action. We combined interviews with 113 purposively selected key informants, document analysis and triangulation to map how research and translation processes evolved and contributions to action were realized. After each case was analysed separately, a cross-case analysis was conducted to identify patterns in the association between features of research processes and the use of research.

RESULTS

The results of 20 of the 30 studies were used to contribute to action within 12 months. The priority setting and proposal selection process led to the funding of studies which were from the outset closely aligned with health sector priorities. Research was most likely to be used when it was initiated and conducted by people who were in a position to use their results in their own work. The results of 17 out of 18 of these user-initiated studies were translated into action. Other features of research that appeared to contribute to its use were involving potential key users in formulating proposals and developing recommendations.

CONCLUSIONS

Our study underlines the importance of supporting research that meets locally-expressed needs and that is led by people embedded in the contexts in which results can be used. Supporting the involvement of health sector professionals in the design, conduct and interpretation of research appears to be an especially worthwhile investment.

Establishing research priorities for malaria elimination in the context of the emergency response to artemisinin resistance framework-the Cambodian approach

Background

Countries of the greater Mekong subregion have made a transition from malaria control to an aim for falciparum and vivax malaria elimination. The elimination of falciparum malaria will have to be achieved against a background of increasing artemisinin and multi-drug resistance. This ambitious goal requires an operational research (OR) agenda that addresses the dynamic challenges encountered on the path to elimination, which will need to be flexible and developed in close relation with the cambodian national programme for parasitology, entomology and malaria control (CNM). In Cambodia, a number of meetings with stakeholders were convened by the CNM and emergency response to artemisinin resistance (ERAR) hub, producing an initial list of priority OR topics. The process and outcome of these meetings are described, which could serve as a template for other countries in the region.

Methods

A landscaping exercise was conducted to gather all past, on-going and planned malaria focussed OR activities conducted by the cambodian research consortium in Cambodia and categorized according to research theme. The six themes included (1) malaria epidemiology, surveillance and response, (2) malaria case management, (3) malaria vector control, (4) malaria behavioural issues, (5) malaria clinical studies, and (6) other vector-borne diseases (dengue, neglected tropical diseases, soil-transmitted helminths). The different themes were discussed in small focus groups, which made an initial prioritization list which was then presented to a plenary group for further discussion. This produced a list of research questions ranked according to priority.

Results

OR priorities produced by the thematic groups were discussed in the plenary meeting and given a priority score by group voting. A list of 17 OR questions were developed, finalized and listed, which included questions on surveillance, active case detection and treatment efficacy.

Conclusion

This paper describes ERAR’s work on supporting Cambodia’s transition to malaria elimination by identifying national operational research priorities. ERAR has initiated and currently plays a critical role in the development of country specific research agendas for malaria elimination. The first example of this has been the described exercise in Cambodia, which could serve a template for setting OR priorities in the wider region.

SCALS: a fourth-generation study of assisted living technologies in their organisational, social, political and policy context

INTRODUCTION

Research to date into assisted living technologies broadly consists of 3 generations: technical design, experimental trials and qualitative studies of the patient experience. We describe a fourth-generation paradigm: studies of assisted living technologies in their organisational, social, political and policy context. Fourth-generation studies are necessarily organic and emergent; they view technology as part of a dynamic, networked and potentially unstable system. They use co-design methods to generate and stabilise local solutions, taking account of context.

METHODS AND ANALYSIS

SCALS (Studies in Co-creating Assisted Living Solutions) consists (currently) of 5 organisational case studies, each an English health or social care organisation striving to introduce technology-supported services to support independent living in people with health and/or social care needs. Treating these cases as complex systems, we seek to explore interdependencies, emergence and conflict. We employ a co-design approach informed by the principles of action research to help participating organisations establish, refine and evaluate their service. To that end, we are conducting in-depth ethnographic studies of people's experience of assisted living technologies (micro level), embedded in evolving organisational case studies that use interviews, ethnography and document analysis (meso level), and exploring the wider national and international context for assisted living technologies and policy (macro level). Data will be analysed using a sociotechnical framework developed from structuration theory.

ETHICS AND DISSEMINATION

Research ethics approval for the first 4 case studies has been granted. An important outcome will be lessons learned from individual co-design case studies. We will document the studies' credibility and rigour, and assess the transferability of findings to other settings while also recognising unique aspects of the contexts in which they were generated. Academic outputs will include a cross-case analysis and progress in theory and method of fourth-generation assisted living technology research. We will produce practical guidance for organisations, policymakers, designers and service users.

The Cambodia Research Consortium: expediting research for malaria elimination with the emergency response to artemisinin resistance framework

This commentary offers insight into how to best address barriers that may hinder the translation of malaria research findings into policy. It also proposes viable methods of implementing these policies in Cambodia. Currently, a wide range of malaria research is being conducted by in-country stakeholders, including Cambodia's National Programme for Parasitology, Entomology and Malaria Control's (CNM), non-governmental organizations, and academic institutions. Coordinating research amongst these partners, as well as within the Ministry of Health, is a challenge. Results are rarely disseminated widely and seldom inform programme and policy decisions. CNM and its research partners have severely limited access to each other's databases. This lack of accessibility, timeliness, engagement and cooperation between CNM and its partners greatly impacts overall research efficiency in this field, and is stifling innovation both within and beyond CNM. Cambodia has set a goal to eradicate all forms of malaria by 2030. As countries approach the elimination phase, there is a greater need for sharing research-generated evidence amongst partners, in order to ensure that appropriate and impactful activities are conducted. The Cambodian Research Consortium was established to serve as a framework for partners, stakeholders and researchers to share research projects, information and results, and to promote the goals of CNM. The sharing of malaria research results will help to inform prevention, control and elimination activities in the country. It will also determine and address the country's operational research needs, and could potentially become a framework model to be used in other countries aiming to transition from malaria control to elimination.

Research impact: neither quick nor easy

Greenhalgh and Fahy's paper about the 2014 Research Excellence Framework provides insights into the challenges of assessing research impact. Future research assessment exercises should consider how best to include measurement of indirect and non-linear impact and whether efforts in knowledge transfer and co-production should be explicitly recognised. Greenhalgh and Fahy's findings also demonstrate that the structure of the assessment exercise can privilege certain kinds of research and may therefore miss some research that has a high impact on policy and practice. There are a growing number of courses, tools, and funding models to assist researchers in making an impact, although as yet there is little evidence about whether these approaches work in practice. Please see related article: http://www.biomedcentral.com/1741-7015/13/232.

Research impact in the community-based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework

BACKGROUND

The 2014 UK Research Excellence Framework (REF2014) generated a unique database of impact case studies, each describing a body of research and impact beyond academia. We sought to explore the nature and mechanism of impact in a sample of these.

METHODS

The study design was manual content analysis of a large sample of impact case studies (producing mainly quantitative data), plus in-depth interpretive analysis of a smaller sub-sample (for qualitative detail), thereby generating both breadth and depth. For all 162 impact case studies submitted to sub-panel A2 in REF2014, we extracted data on study design(s), stated impacts and audiences, mechanisms of impact, and efforts to achieve impact. We analysed four case studies (selected as exemplars of the range of approaches to impact) in depth, including contacting the authors for their narratives of impact efforts.

RESULTS

Most impact case studies described quantitative research (most commonly, trials) and depicted a direct, linear link between research and impact. Research was said to have influenced a guideline in 122 case studies, changed policy in 88, changed practice in 84, improved morbidity in 44 and reduced mortality in 25. Qualitative and participatory research designs were rare, and only one case study described a co-production model of impact. Eighty-two case studies described strong and ongoing linkages with policymakers, but only 38 described targeted knowledge translation activities. In 40 case studies, no active efforts to achieve impact were described. Models of good implementation practice were characterised by an ethical commitment by researchers, strong institutional support and a proactive, interdisciplinary approach to impact activities.

CONCLUSION

REF2014 both inspired and documented significant efforts by UK researchers to achieve impact. But in contrast with the published evidence on research impact (which depicts much as occurring indirectly through non-linear mechanisms), this sub-panel seems to have captured mainly direct and relatively short-term impacts one step removed from patient outcomes. Limited impacts on morbidity and mortality, and researchers' relatively low emphasis on the processes and interactions through which indirect impacts may occur, are concerns. These findings have implications for multi-stakeholder research collaborations such as UK National Institute for Health Research Collaborations for Leadership in Applied Health Research and Care, which are built on non-linear models of impact.

Conditions for sustainability of Academic Collaborative Centres for Public Health in the Netherlands: a mixed methods design

BACKGROUND

Contemporary research should increasingly be carried out in the context of application. Nowotny called this new form of knowledge production Mode-2. In line with Mode-2 knowledge production, the Dutch government in 2006 initiated the so-called Academic Collaborative Centres (ACC) for Public Health. The aim of these ACCs is to build a regional, sustainable knowledge-sharing network to deliver socially robust knowledge. The present study aims to highlight the enabling and constraining push and pull factors of these ACCs in order to assess whether the ACCs are able to build and strengthen a sustainable integrated organizational network between public health policy, practice, and research.

METHODS

Our empirical analysis builds on a mixed methods design. Quantitative data was derived from records of a survey sent to all 11 ACCs about personnel investments, number and nature of projects, and earning power. Qualitative data was derived from 21 in-depth interviews with stakeholders involved. The interviews were tape-recorded, transcribed, and manually coded as favourable or unfavourable pull or push factors.

RESULTS

The extra funding appeared to be the most enabling push factor. The networks secured external grants for about 150 short- and long-term Mode-2 knowledge production projects in the past years. Enabling pull factors improved, especially the number of policy-driven short-term research projects. Exchange agents were able to constructively deal with the constraining push factors, like university's publication pressure and budget limitations. However, the constraining pull factors like local government's involvement and their low demand for scientific evidence were difficult to overcome.

CONCLUSIONS

A clear improvement of the organizational networks was noticed whereby the ACC's were pushed rather than pulled. Efforts are needed to increase the demand for scientific and socially robust evidence from policymakers and to resolve the regime differences between the research and policy systems, in order to make the bidirectionality of the links sustainable.

Enhancing the contribution of research to health care policy-making: a case study of the Dutch Health Care Performance Report

OBJECTIVES

The Dutch Health Care Performance Report, issued by the National Institute of Public Health and the Environment, aims to monitor health care performance in The Netherlands. Both the National Institute and the Ministry of Health wish to increase the contribution of the Report to health care policy-making. Our aim was to identify ways to achieve that.

METHOD

We used contribution mapping as a theoretical framework that recognizes alignment of research as crucial to managing contributions to policy-making. To investigate which areas need alignment efforts by researchers and/or policy-makers, we interviewed National Institute researchers and policy-makers from the Ministry of Health and assessed the process for developing the 2010 Report.

RESULTS

We identified six areas where alignment is specifically relevant for enhancing the contributions of future versions of the Dutch Health Care Performance Report: well-balanced information for different ministerial directorates; backstage work; double role actors; reports of other knowledge institutes; data collection/generation and presentation forms.

CONCLUSION

The contribution of health care performance reporting to policy-making is complex and requires continuous alignment efforts between researchers and policy-makers. These efforts should form an inseparable part of health care performance reporting and although this demands considerable resources, it is worth considering since it may pay back in better contributions to policy-making.