The development of ORACLe: a measure of an organisation's capacity to engage in evidence-informed health policy

Exploring evidence use and capacity for health services management and planning in Swiss health administrations: A mixed-method interview study

BACKGROUND

Health administrations require evidence, meaning robust information, data, and research, on health services and systems. Little is known about the resources and processes available within administrations to support evidence-informed policymaking. This study assessed Swiss health administrations' capacity for evidence use and investigated civil servants' needs and perspectives regarding the role and use of evidence in health services management and planning.

METHODS

In this mixed-method study, we interviewed civil servants from Swiss German-speaking cantonal health administrations. We quantitatively assessed administrations' organization-level capacity by applying six structured interviews using an existing measurement tool (ORACLe). Individual-level needs and perspectives regarding evidence use and capacity were qualitatively explored with twelve in-depth interviews that were analyzed using the framework method.

FINDINGS

Respondents indicated moderate evidence-use capacity in all administrations. Administrations displayed a similar pattern of high and low capacity in specific capacity areas, generally with considerable variation within administrations. Most administrations indicated high capacity for producing or commissioning evidence and close relationships with research. They showed limited capacity in the documentation of processes and availability of tools, programs, or training opportunities. Administrations place the responsibility for engagement with evidence at the level of individual civil servants rather than at the organizational level. Although administrations highly value evidence-informed policymaking and consider it vital to effective health services management and planning, they face significant constraints in accessing evidence-specific resources and receive little organizational support. Administrations rely on external capacity to compensate for these limitations and engage with evidence pragmatically.

CONCLUSION

Our findings indicate moderate and improvable capacity for evidence use in Swiss health administrations that place limited value on organizational support. Besides strengthening organizational support, leadership buy-in, particular staff needs, and balancing the implementation of specific measures with the provision of more general resources should be considered to unlock the potential of strengthened engagement with evidence.

Factors that influence evidence-informed meso-level regional primary health care planning: a qualitative examination and conceptual framework

BACKGROUND

Evidence-informed primary health care (PHC) planning in decentralised, meso-level regional organisations has received little research attention. In this paper we examine the factors that influence planning within this environment, and present a conceptual framework.

METHODS

We employed mixed methods: case studies of five Australian Primary Health Networks (PHNs), involving 29 primary interviews and secondary analysis of 38 prior interviews; and analysis of planning documents from all 31 PHNs. The analysis was informed by a WHO framework of evidence-informed policy-making, and institutional theory.

RESULTS

Influential actors included federal and state/territory governments, Local Health Networks, Aboriginal Community Controlled Health Organisations, local councils, public hospitals, community health services, and providers of allied health, mental health and aged care services. The federal government was most influential, constraining PHNs' planning scope, time and funding. Other external factors included: the health service landscape; local socio-demographic and geographic characteristics; (neoliberal) ideology; interests and politics; national policy settings and reforms; and system reorganisation. Internal factors included: organisational structure; culture, values and ideology; various capacity factors; planning processes; transition history; and experience. The additional regional layer of context adds to the complexity of planning.

CONCLUSIONS

Like national health policy-making, meso-level PHC planning occurs in a complex environment, but with additional regional factors and influences. We have developed a conceptual framework of the meso-level PHC planning environment, which can be employed by similar regional organisations to elucidate influential factors, and develop strategies and tools to promote transparent, evidence-informed PHC planning for better health outcomes.

Individual and institutional capacity-building for evidence-informed health policy-making in Iran: a mix of local and global evidence

Background

Providing valid evidence to policy-makers is a key factor in the development of evidence-informed policy-making (EIPM). This study aims to review interventions used to promote researchers’ and knowledge-producing organizations’ knowledge and skills in the production and translation of evidence to policy-making and explore the interventions at the individual and institutional level in the Iranian health system to strengthen EIPM.

Methods

The study was conducted in two main phases: a systematic review and a qualitative study. First, to conduct the systematic review, the PubMed and Scopus databases were searched. Quality appraisal was done using the Joanna Briggs Institute checklists. Second, semi-structured interviews and document review were used to collect local data. Purposive sampling was used and continued until data saturation. A qualitative content analysis approach was used for data analysis.

Results

From a total of 11,514 retrieved articles, 18 papers were eligible for the analysis. Based on the global evidence, face-to-face training workshops for researchers was the most widely used intervention for strengthening researchers’ capacity regarding EIPM. Target audiences in almost all of the training programmes were researchers. Setting up joint training sessions that helped empower researchers in understanding the needs of health policy-makers had a considerable effect on strengthening EIPM. Based on the local collected evidence, the main interventions for individual and institutional capacity-building were educational and training programmes or courses related to the health system, policy-making and policy analysis, and research cycle management. To implement the individual and institutional interventions, health system planners and authorities and the community were found to have a key role as facilitating factors.

Conclusion

The use of evidence-based interventions for strengthening research centres, such as training health researchers on knowledge translation and tackling institutional barriers that can prevent well-trained researchers from translating their knowledge, as well as the use of mechanisms and networks for effective interactions among policy-makers at the macro and meso (organizational) level and the research centre, will be constructive for individual and institutional capacity-building. The health system needs to strengthen its strategic capacity to facilitate an educational and training culture in order to motivate researchers in producing appropriate evidence for policy-makers.

Policymakers' Research Capacities, Engagement, and Use of Research in Public Health Policymaking

The use of research in public health policymaking is one of the prerequisites for successfully implemented health policies which have better population health as an outcome. This policy process is influenced by the actors involved under the policy umbrella, with inter-related contextual factors and specific structural and institutional circumstances. Our study investigates how policymakers’ research capacities influence the use of research in the health policy process and identify areas where capacity-building interventions give the most meaning and impact. Furthermore, we investigate policymakers’ research engagement and use this to inform public health policy in the public sector in Denmark. We collect and report data using Seeking, Engaging with, and Evaluation Research (SEER) methodology. Policymakers are reported to have research capacity, but it is questionable how those competences have actually been used in policymaking. Decision-makers were often not aware or did not know about the existing organizational tools and systems for research engagement and use and two third of respondents had not been part of any research activities or had any collaboration with researchers. Overall, research use in public health policymaking and evaluation was limited. As a conclusion, we propose that capacity-building interventions for increasing research use and collaboration in EIPM should be context-oriented, measurable, and sustainable in developing individual and organizational competences.

Policy options to increase motivation for improving evidence-informed health policy-making in Iran

BACKGROUND

Current incentive programmes are not sufficient to motivate researchers and policy-makers to use research evidence in policy-making. We conducted a mixed-methods design to identify context-based policy options for strengthening motivations among health researchers and policy-makers to support evidence-informed health policy-making (EIHP) in Iran.

METHODS

This study was conducted in 2019 in two phases. In the first phase, we conducted a scoping review to extract interventions implemented or proposed to strengthen motivations to support EIHP. Additionally, we employed a comparative case study design for reviewing the performance evaluation (PE) processes in Iran and other selected countries to determine the current individual and organizational incentives to encourage EIHP. In the second phase, we developed two policy briefs and then convened two policy dialogues, with 12 and 8 key informants, respectively, where the briefs were discussed. Data were analysed using manifest content analysis in order to propose contextualized policy options.

RESULTS

The policy options identified to motivate health researchers and policy-makers to support EIHP in Iran were: revising the criteria of academic PE; designing appropriate incentive programmes for nonacademic researchers; developing an indicator for the evaluation of research impact on policy-making or health outcomes; revising the current policies of scientific journals; revising existing funding mechanisms; presenting the knowledge translation plan when submitting a research proposal, as a mandatory condition; encouraging and supporting mechanisms for increasing interactions between policy-makers and researchers; and revising some administrative processes (e.g. managers and staff PEs; selection, appointment, and changing managers and reward mechanisms).

CONCLUSIONS

The current individual or organizational incentives are mainly focused on publications, rather than encouraging researchers and policy-makers to support EIHP. Relying more on incentives that consider the other impacts of research (e.g. impacts on health system and policy, or health outcomes) is recommended. These incentives may encourage individuals and organizations to be more involved in conducting research evidence, resulting in promoting EIHP.

TRIAL REGISTRATION

NA.

Examining knowledge translation in blood donor research: A review of vasovagal reaction literature

BACKGROUND

Knowledge translation focuses on the transfer of research findings into policy and practice. To provide insight into the state of knowledge translation in blood donor research, we undertook a rapid review of a key research area in the field with high potential for translation, vasovagal reactions (VVRs). We examined the number and nature of VVR-related studies to determine the availability of research evidence, and mapped the included articles along the research-to-practice trajectory using the Knowledge to Action framework.

STUDY DESIGN AND METHODS

PubMed, PsycINFO, CINAHL, and EMBASE were searched for peer-reviewed journal articles from inception to October 2019 using the terms blood don* AND vasovagal OR faint* OR syncope.

RESULTS

A total of 176 articles met our inclusion criteria. Studies relating to VVRs increased substantially from 1942 to 2019, with 84% published in the last 20 years. Articles were predominately observation (non-intervention) studies (117; 66%), followed by intervention (knowledge inquiry) studies (31; 18%) and review (knowledge synthesis) studies (20; 11%). The evidence from intervention research was limited, with 14 strategies tested in 31 studies and often by the same research groups. Only 5 (3%) implementation and evaluation studies were found; all focused on evaluating the effects of a newly introduced intervention on VVR rates through uncontrolled or cross-sectional study designs.

DISCUSSION

VVR research is in the early stages of knowledge translation. More intervention research is needed to provide a robust evidence base as well as more published implementation research to share knowledge of translating research into policy and practice.

Assessing organisational capacity for evidence-informed health policy and planning: an adaptation of the ORACLe tool for Australian primary health care organizations

Background

Many nations have established primary health care (PHC) organizations that conduct PHC planning for defined geographical areas. The Australian Government established Primary Health Networks (PHNs) in 2015 to develop and commission PHC strategies to address local needs. There has been little written about the capacity of such organizations for evidence-informed planning, and no tools have been developed to assess this capacity, despite their potential to contribute to a comprehensive effective and efficient PHC sector.

Methods

We adapted the ORACLe tool, originally designed to examine evidence-informed policy-making capacity, to examine organizational capacity for evidence-informed planning in meso-level PHC organizations, using PHNs as an example. Semi-structured interviews were conducted with 14 participants from five PHNs, using the ORACLe tool, and scores assigned to responses, in seven domains of capacity.

Results

There was considerable variation between PHNs and capacity domains. Generally, higher capacity was demonstrated in regard to mechanisms which could inform planning through research, and support relationships with researchers. PHNs showed lower capacity for evaluating initiatives, tools and support for staff, and staff training.

Discussion and conclusions

We critique the importance of weightings and scope of some capacity domains in the ORACLe tool. Despite this, with some minor modifications, we conclude the ORACLe tool can identify capacity strengths and limitations in meso-level PHC organizations. Well-targeted capacity development enables PHC organizations’ strategies to be better informed by evidence, for optimal impact on PHC and population health outcomes.

The EVITA framework for evidence-based mental health policy agenda setting in low- and middle-income countries

The burden of mental illness is excessive, but many countries lack evidence-based policies to improve practice. Mental health research evidence translation into policymaking is a 'wicked problem', often failing despite a robust evidence base. In a recent systematic review, we identified a gap in frameworks on agenda setting and actionability, and pragmatic, effective tools to guide action to link research and policy are needed. Responding to this gap, we developed the new EVITA 1.1 (EVIdence To Agenda setting) conceptual framework for mental health research-policy interrelationships in low- and middle-income countries (LMICs). We (1) drafted a provisional framework (EVITA 1.0); (2) validated it for specific applicability to mental health; (3) conducted expert in-depth interviews to (a) validate components and mechanisms and (b) assess intelligibility, functionality, relevance, applicability and effectiveness. To guide interview validation, we developed a simple evaluation framework. (4) Using deductive framework analysis, we coded and identified themes and finalized the framework (EVITA 1.1). Theoretical agenda-setting elements were added, as targeting the policy agenda-setting stage was found to lead to greater policy traction. The framework was validated through expert in-depth interviews (n = 13) and revised. EVITA 1.1 consists of six core components [advocacy coalitions, (en)actors, evidence generators, external influences, intermediaries and political context] and four mechanisms (capacity, catalysts, communication/relationship/partnership building and framing). EVITA 1.1 is novel and unique because it very specifically addresses the mental health research-policy process in LMICs and includes policy agenda setting as a novel, effective mechanism. Based on a thorough methodology, and through its specific design and mechanisms, EVITA has the potential to improve the challenging process of research evidence translation into policy and practice in LMICs and to increase the engagement and capacity of mental health researchers, policy agencies/planners, think tanks, NGOs and others within the mental health research-policy interface. Next, EVITA 1.1 will be empirically tested in a case study.

A literature review of knowledge translation and partnership research training programs for health researchers

BACKGROUND

Researchers and policy-makers are increasingly working together with the goal of creating research that is focused on solving real-world problems; however, knowledge translation (KT) activities, and the partnerships they often require, can be challenging. The aim of this review is to determine the extent of the literature on training programs designed to improve researcher competency in KT and to describe existing training methods that may be used by those hoping to build capacity for partnership research.

METHODS

MEDLINE, EMBASE, PsycINFO and CINAHL were searched for peer review articles published between January 2000 and July 2019. Studies were eligible for inclusion in the review if they described the development of, curriculum for, or evaluation of KT and/or partnership research training programs. Data extraction included information on evaluation methods, outcomes and implications as well as the format, aims and themes of each capacity-building program.

RESULTS

The review identified nine published articles that met inclusion criteria - four papers described training events, two papers described participant experiences of specific learning sessions within a larger training course, two papers described part time secondments for KT capacity-building and one paper described a plan for KT training embedded within an existing research training course. All programs were delivered face-to-face, all included practical skills-building opportunities, and all employed multiple learning modalities such as seminars and small group discussions. Evaluation of the training programs was primarily conducted through qualitative interviews or feedback surveys.

CONCLUSION

To date, few KT training initiatives have been described in the literature and none of these have been rigorously evaluated. The present review offers insights into the planning, development and participant experiences associated with the small number of training initiatives that have been described. There is insufficient evidence available at present to identify the most effective models for training researchers in KT and partnership skills.

Organisational factors that facilitate research use in public health policy-making: a scoping review

Background

Although important syntheses and theoretical works exist in relation to understanding the organisational factors that facilitate research use, these contributions differ in their scope and object of study as well as their theoretical underpinnings. Therefore, from an exploratory angle, it may be useful to map out the current literature on organisational factors of research use in public health policy-making when revisiting existing theories and frameworks to gain further theoretical insights.

Methods

Herein, a scoping review technique and thematic content analysis were used to bring together findings from both synthesised and empirical studies of different types to map out the organisational factors that facilitate research use in public health policy-making.

Results

A total of 14 reviews and 40 empirical studies were included in the analysis. These were thematically coded and the intra-organisational factors reported as enabling research use were examined. Five main categories of organisational factors that advance research use in policy organisations – (1) individual factors, (2) the management of research integration, (3) organisational systems and infrastructures of research use, (4) institutional structures and rules for policy-making, and (5) organisational characteristics – were derived as well as 18 subcategories and a total of 64 specific factors, where 27 factors were well supported by research.

Conclusions

Using a scoping review methodology, the intra-organisational factors influencing research use in policy-making (including individual factors) were systematically mapped and the theories applied in this area of research were assessed. The review findings confirm the importance of an intra-organisational perspective when exploring research use, showing that many organisational factors are critical facilitators of research use but also that many factors and mechanisms are understudied. The synthesis shows a lack of studies on politicians and the need for more theoretically founded research. Despite increased efforts to update the existing evidential and theoretical basis of research use, we still need frameworks that combine different approaches and theories to help us grasp the complex organisational mechanisms that facilitate research use in policy settings.

How was research engaged with and used in the development of 131 policy documents? Findings and measurement implications from a mixed methods study

Background

Much has been written about the use of evidence in policy; however, there is still little known about whether and how research is engaged with and used in policy development or the impact of reported barriers and facilitators. This paper aims to (1) describe the characteristics of 131 policy documents, (2) describe the ways in which research was engaged with (e.g. was searched for, appraised or generated) and used (e.g. to clarify understanding, persuade others or inform a policy) in the development of these policy documents, and (3) identify the most commonly reported barriers and facilitators and describe their association with research engagement and use.

Methods

Six health policy and program development agencies based in Sydney, Australia, contributed four recently finalised policy documents for consideration over six measurement periods. Structured, qualitative interviews were conducted with the policymakers most heavily involved in developing each of the 131 policy documents. Interviews covered whether and how research was engaged with and used in the development of the policy product and any barriers or facilitators related to this. Interviews were scored using the empirically validated SAGE tool and thematically analysed. Descriptive statistics were calculated for all key variables and comparisons made between agencies. Multiple regression analyses were used to estimate the impact of specific barriers and facilitators on research engagement and use.

Results

Our data shows large variations between policy agencies in the types of policy documents produced and the characteristics of these documents. Nevertheless, research engagement and use was generally moderate across agencies. A number of barriers and facilitators to research use were identified. No barriers were significantly associated with any aspects of research engagement or use. Access to consultants and relationships with researchers were both associated with increased research engagement but not use. Thus, access to consultants and relationships with researchers may increase the extent and quality of the evidence considered in policy development.

Conclusions

Our findings suggest that those wishing to develop interventions and programs designed to improve the use of evidence in policy agencies might usefully target increasing access to consultants and relationships with researchers in order to increase the extent and quality of the research considered, but that a greater consideration of context might be required to develop strategies to increase evidence use.

Electronic supplementary material

The online version of this article (10.1186/s13012-019-0886-2) contains supplementary material, which is available to authorized users.

Increasing the capacity of policy agencies to use research findings: a stepped-wedge trial

BACKGROUND

This paper describes the trial of a novel intervention, Supporting Policy In health with evidence from Research: an Intervention Trial (SPIRIT). It examines (1) the feasibility of delivering this kind of programme in practice; (2) its acceptability to participants; (3) the impact of the programme on the capacity of policy agencies to engage with research; and (4) the engagement with and use of research by policy agencies.

METHODS

SPIRIT was a multifaceted, highly tailored, stepped-wedge, cluster-randomised, trial involving six health policy agencies in Sydney, Australia. Agencies were randomly allocated to one of three start dates to receive the 1-year intervention programme. SPIRIT included audit, feedback and goal setting; a leadership programme; staff training; the opportunity to test systems to facilitate research use in policies; and exchange with researchers. Outcome measures were collected at each agency every 6 months for 30 months.

RESULTS

Participation in SPIRIT was associated with significant increases in research use capacity at staff and agency levels. Staff reported increased confidence in research use skills, and agency leaders reported more extensive systems and structures in place to support research use. Self-report data suggested there was also an increase in tactical research use among agency staff. Given the relatively small numbers of participating agencies and the complexity of their contexts, findings suggest it is possible to effect change in the way policy agencies approach the use of research. This is supported by the responses on the other trial measures; while these were not statistically significant, on 18 of the 20 different measures used, the changes observed were consistent with the hypothesised intervention effect (that is, positive impacts).

CONCLUSIONS

As an early test of an innovative approach, SPIRIT has demonstrated that it is possible to increase research engagement and use in policy agencies. While more work is needed to establish the replicability and generalisability of these findings, this trial suggests that building staff skills and organisational structures may be effective in increasing evidence use.

Discrete Choice Experiments in Health Economics: Past, Present and Future

OBJECTIVES

Discrete choice experiments (DCEs) are increasingly advocated as a way to quantify preferences for health. However, increasing support does not necessarily result in increasing quality. Although specific reviews have been conducted in certain contexts, there exists no recent description of the general state of the science of health-related DCEs. The aim of this paper was to update prior reviews (1990-2012), to identify all health-related DCEs and to provide a description of trends, current practice and future challenges.

METHODS

A systematic literature review was conducted to identify health-related empirical DCEs published between 2013 and 2017. The search strategy and data extraction replicated prior reviews to allow the reporting of trends, although additional extraction fields were incorporated.

RESULTS

Of the 7877 abstracts generated, 301 studies met the inclusion criteria and underwent data extraction. In general, the total number of DCEs per year continued to increase, with broader areas of application and increased geographic scope. Studies reported using more sophisticated designs (e.g. D-efficient) with associated software (e.g. Ngene). The trend towards using more sophisticated econometric models also continued. However, many studies presented sophisticated methods with insufficient detail. Qualitative research methods continued to be a popular approach for identifying attributes and levels.

CONCLUSIONS

The use of empirical DCEs in health economics continues to grow. However, inadequate reporting of methodological details inhibits quality assessment. This may reduce decision-makers' confidence in results and their ability to act on the findings. How and when to integrate health-related DCE outcomes into decision-making remains an important area for future research.

Development of measurable indicators to enhance public health evidence-informed policy-making

BACKGROUND

Ensuring health policies are informed by evidence still remains a challenge despite efforts devoted to this aim. Several tools and approaches aimed at fostering evidence-informed policy-making (EIPM) have been developed, yet there is a lack of availability of indicators specifically devoted to assess and support EIPM. The present study aims to overcome this by building a set of measurable indicators for EIPM intended to infer if and to what extent health-related policies are, or are expected to be, evidence-informed for the purposes of policy planning as well as formative and summative evaluations.

METHODS

The indicators for EIPM were developed and validated at international level by means of a two-round internet-based Delphi study conducted within the European project 'REsearch into POlicy to enhance Physical Activity' (REPOPA). A total of 82 researchers and policy-makers from the six European countries (Denmark, Finland, Italy, the Netherlands, Romania, the United Kingdom) involved in the project and international organisations were asked to evaluate the relevance and feasibility of an initial set of 23 indicators developed by REPOPA researchers on the basis of literature and knowledge gathered from the previous phases of the project, and to propose new indicators.

RESULTS

The first Delphi round led to the validation of 14 initial indicators and to the development of 8 additional indicators based on panellists' suggestions; the second round led to the validation of a further 11 indicators, including 6 proposed by panellists, and to the rejection of 6 indicators. A total of 25 indicators were validated, covering EIPM issues related to human resources, documentation, participation and monitoring, and stressing different levels of knowledge exchange and involvement of researchers and other stakeholders in policy development and evaluation.

CONCLUSION

The study overcame the lack of availability of indicators to assess if and to what extent policies are realised in an evidence-informed manner thanks to the active contribution of researchers and policy-makers. These indicators are intended to become a shared resource usable by policy-makers, researchers and other stakeholders, with a crucial impact on fostering the development of policies informed by evidence.

Frameworks for embedding a research culture in allied health practice: a rapid review

BACKGROUND

Although allied health clinicians play a key role in the provision of healthcare, embedding a culture of research within public and private health systems remains a challenge. In this rapid review we critically evaluate frameworks for embedding research into routine allied health practice, as the basis for high quality, safe, efficient and consumer-focused care.

METHODS

A rapid review (PROSPERO: CRD42017075699) was conducted to evaluate frameworks designed to create and embed research in the health sector. Included were full-text, English-language, peer-reviewed publications or Government reports of frameworks that could inform the implementation of an allied health research framework. Eight electronic databases and four government websites were searched, using search terms such as models, frameworks and research capacity-building. Two independent researchers conducted all review stages and used content and thematic analysis to interpret the results.

RESULTS

Sixteen framework papers were finally included. Content analysis identified 44 system and regulatory level items that informed the research frameworks, 125 healthcare organisation items and 76 items relating to individual clinicians. Thematic analysis identified four major themes. Firstly, sustainable change requires allied health research policies, regulation, governance and organisational structures that support and value evidence-based practice. Secondly, research capability, receptivity, advocacy and literacy of healthcare leaders and managers are key to successful research implementation. Third, organisational factors that facilitate a research culture include dedicated staff research positions, time allocated to research, mentoring, professional education and research infrastructure. When healthcare agencies had strong partnerships with universities and co-located research leaders, research implementation was strengthened. Finally, individual attributes of clinicians, such as their research skills and capabilities, motivation, and participation in research teams, are essential to embedding research into practice.

CONCLUSION

Theoretical frameworks were identified that informed processes to embed a culture of allied health research into healthcare services. Research-led and evidence-informed allied health practice enables optimisation of workforce capability and high-quality care.

Organisational capacity and its relationship to research use in six Australian health policy agencies

There are calls for policymakers to make greater use of research when formulating policies. Therefore, it is important that policy organisations have a range of tools and systems to support their staff in using research in their work. The aim of the present study was to measure the extent to which a range of tools and systems to support research use were available within six Australian agencies with a role in health policy, and examine whether this was related to the extent of engagement with, and use of research in policymaking by their staff. The presence of relevant systems and tools was assessed via a structured interview called ORACLe which is conducted with a senior executive from the agency. To measure research use, four policymakers from each agency undertook a structured interview called SAGE, which assesses and scores the extent to which policymakers engaged with (i.e., searched for, appraised, and generated) research, and used research in the development of a specific policy document. The results showed that all agencies had at least a moderate range of tools and systems in place, in particular policy development processes; resources to access and use research (such as journals, databases, libraries, and access to research experts); processes to generate new research; and mechanisms to establish relationships with researchers. Agencies were less likely, however, to provide research training for staff and leaders, or to have evidence-based processes for evaluating existing policies. For the majority of agencies, the availability of tools and systems was related to the extent to which policymakers engaged with, and used research when developing policy documents. However, some agencies did not display this relationship, suggesting that other factors, namely the organisation's culture towards research use, must also be considered.

Does engagement predict research use? An analysis of The Conversation Annual Survey 2016

The impact of research on the world beyond academia has increasingly become an area of focus in research performance assessments internationally. Impact assessment is expected to incentivise researchers to increase engagement with industry, government and the public more broadly. Increased engagement is in turn expected to increase translation of research so decision-makers can use research to inform development of policies, programs, practices, processes, products, and other mechanisms, through which impact can be realised. However, research has shown that various factors affect research use, and evidence on 'what works' to increase decision-makers' use of research is limited. The Conversation is an open access research communication platform, published under Creative Commons licence, which translates research into news articles to engage a general audience, aiming to improve understanding of current issues and complex social problems. To identify factors that predict use of academic research and expertise reported in The Conversation, regression analyses were performed using The Conversation Australia 2016 Annual Survey data. A broad range of factors predicted use, with engagement actions being the most common. Interestingly, different types of engagement actions predicted different types of use. This suggests that to achieve impact through increased engagement, a deeper understanding of how and why different engagement actions elicit different types of use is needed. Findings also indicate The Conversation is overcoming some of the most commonly identified barriers to the use of research: access, relevance, actionable outcomes, and timeliness. As such, The Conversation offers an effective model for providing access to and communicating research in a way that enables use, a necessary precursor to achieving research impact.

Policymakers' experience of a capacity-building intervention designed to increase their use of research: a realist process evaluation

BACKGROUND

An intervention's success depends on how participants interact with it in local settings. Process evaluation examines these interactions, indicating why an intervention was or was not effective, and how it (and similar interventions) can be improved for better contextual fit. This is particularly important for innovative trials like Supporting Policy In health with Research: an Intervention Trial (SPIRIT), where causal mechanisms are poorly understood. SPIRIT was testing a multi-component intervention designed to increase the capacity of health policymakers to use research.

METHODS

Our mixed-methods process evaluation sought to explain variation in observed process effects across the six agencies that participated in SPIRIT. Data collection included observations of intervention workshops (n = 59), purposively sampled interviews (n = 76) and participant feedback forms (n = 553). Using a realist approach, data was coded for context-mechanism-process effect configurations (retroductive analysis) by two authors.

RESULTS

Intervention workshops were very well received. There was greater variation of views regarding other aspects of SPIRIT such as data collection, communication and the intervention's overall value. We identified nine inter-related mechanisms that were crucial for engaging participants in these policy settings: (1) Accepting the premise (agreeing with the study's assumptions); (2) Self-determination (participative choice); (3) The Value Proposition (seeing potential gain); (4) 'Getting good stuff' (identifying useful ideas, resources or connections); (5) Self-efficacy (believing 'we can do this!'); (6) Respect (feeling that SPIRIT understands and values one's work); (7) Confidence (believing in the study's integrity and validity); (8) Persuasive leadership (authentic and compelling advocacy from leaders); and (9) Strategic insider facilitation (local translation and mediation). These findings were used to develop tentative explanatory propositions and to revise the programme theory.

CONCLUSION

This paper describes how SPIRIT functioned in six policy agencies, including why strategies that worked well in one site were less effective in others. Findings indicate a complex interaction between participants' perception of the intervention, shifting contextual factors, and the form that the intervention took in each site. Our propositions provide transferable lessons about contextualised areas of strength and weakness that may be useful in the development and implementation of similar studies.

Does knowledge brokering improve the quality of rapid review proposals? A before and after study

BACKGROUND

Rapid reviews are increasingly being used to help policy makers access research in short time frames. A clear articulation of the review's purpose, questions, scope, methods and reporting format is thought to improve the quality and generalisability of review findings. The aim of the study is to explore the effectiveness of knowledge brokering in improving the perceived clarity of rapid review proposals from the perspective of potential reviewers. To conduct the study, we drew on the Evidence Check program, where policy makers draft a review proposal (a pre knowledge brokering proposal) and have a 1-hour session with a knowledge broker, who re-drafts the proposal based on the discussion (a post knowledge brokering proposal).

METHODS

We asked 30 reviewers who had previously undertaken Evidence Check reviews to examine the quality of 60 pre and 60 post knowledge brokering proposals. Reviewers were blind to whether the review proposals they received were pre or post knowledge brokering. Using a six-point Likert scale, reviewers scored six questions examining clarity of information about the review's purpose, questions, scope, method and format and reviewers' confidence that they could meet policy makers' needs. Each reviewer was allocated two pre and two post knowledge brokering proposals, randomly ordered, from the 60 reviews, ensuring no reviewer received a pre and post knowledge brokering proposal from the same review.

RESULTS

The results showed that knowledge brokering significantly improved the scores for all six questions addressing the perceived clarity of the review proposal and confidence in meeting policy makers' needs; with average changes of 0.68 to 1.23 from pre to post across the six domains.

CONCLUSIONS

This study found that knowledge brokering increased the perceived clarity of information provided in Evidence Check rapid review proposals and the confidence of reviewers that they could meet policy makers' needs. Further research is needed to identify how the knowledge brokering process achieves these improvements and to test the applicability of the findings in other rapid review programs.

Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research

Background

Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers’ capacity to engage with and use research.

Methods

We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales).

Results

Response rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal.

Conclusions

Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-016-0162-8) contains supplementary material, which is available to authorized users.

Building health research systems: WHO is generating global perspectives, and who's celebrating national successes?

In 2016, England’s National Institute for Health Research (NIHR) celebrated its tenth anniversary as an innovative national health research system with a focus on meeting patients’ needs. This provides a good opportunity to reflect on how the creation of the NIHR has greatly enhanced important work, started in 1991, to develop a health research system in England that is embedded in the National Health Service.

In 2004, WHO identified a range of functions that a national health research system should undertake to improve the health of populations. Health Research Policy and Systems (HRPS) has taken particular interest in the pioneering developments in the English health research system, where the comprehensive approach has covered most, if not all, of the functions identified by WHO. Furthermore, several significant recent developments in thinking about health research are relevant for the NIHR and have informed accounts of its achievements. These include recognition of the need to combat waste in health research, which had been identified as a global problem in successive papers in the Lancet, and an increasing emphasis on demonstrating impact. Here, pioneering evaluation of United Kingdom research, conducted through the impact case studies of the Research Excellence Framework, is particularly important. Analyses informed by these and other approaches identified many aspects of NIHR’s progress in combating waste, building and sustaining research capacity, creating centres of research excellence linked to leading healthcare institutions, developing research networks, involving patients and others in identifying research needs, and producing and adopting research findings that are improving health outcomes.

The NIHR’s overall success, and an analysis of the remaining problems, might have lessons for other systems, notwithstanding important advances in many countries, as described in papers in HRPS and elsewhere. WHO’s recently established Global Observatory for Health Research and Development provides an opportunity to promote some of these lessons. To inform its work, the Observatory is sponsoring a thematic series of papers in HRPS focusing on health research issues such as funding flows, priority setting, capacity building, utilisation and equity. While important papers on these have been published, this series is still open to new submissions.