How hard can it be to include research evidence and evaluation in local health policy implementation? Results from a mixed methods study

Exploring evidence use and capacity for health services management and planning in Swiss health administrations: A mixed-method interview study

BACKGROUND

Health administrations require evidence, meaning robust information, data, and research, on health services and systems. Little is known about the resources and processes available within administrations to support evidence-informed policymaking. This study assessed Swiss health administrations' capacity for evidence use and investigated civil servants' needs and perspectives regarding the role and use of evidence in health services management and planning.

METHODS

In this mixed-method study, we interviewed civil servants from Swiss German-speaking cantonal health administrations. We quantitatively assessed administrations' organization-level capacity by applying six structured interviews using an existing measurement tool (ORACLe). Individual-level needs and perspectives regarding evidence use and capacity were qualitatively explored with twelve in-depth interviews that were analyzed using the framework method.

FINDINGS

Respondents indicated moderate evidence-use capacity in all administrations. Administrations displayed a similar pattern of high and low capacity in specific capacity areas, generally with considerable variation within administrations. Most administrations indicated high capacity for producing or commissioning evidence and close relationships with research. They showed limited capacity in the documentation of processes and availability of tools, programs, or training opportunities. Administrations place the responsibility for engagement with evidence at the level of individual civil servants rather than at the organizational level. Although administrations highly value evidence-informed policymaking and consider it vital to effective health services management and planning, they face significant constraints in accessing evidence-specific resources and receive little organizational support. Administrations rely on external capacity to compensate for these limitations and engage with evidence pragmatically.

CONCLUSION

Our findings indicate moderate and improvable capacity for evidence use in Swiss health administrations that place limited value on organizational support. Besides strengthening organizational support, leadership buy-in, particular staff needs, and balancing the implementation of specific measures with the provision of more general resources should be considered to unlock the potential of strengthened engagement with evidence.

Local implementation of public health policies revealed by the COVID-19 crisis: the French case

BACKGROUND

Improving health system performance depends on the quality of health policy implementation at the local level. However, in general, the attention of researchers is mainly directed towards issues of health policy design and evaluation rather than implementation at the local level. The management of the COVID-19 crisis, especially in Europe, has particularly highlighted the complexity of implementing health policies, decided at the national or supranational level, at the local level.

METHODS

We conducted 23 semi-structured interviews with the main stakeholders in the management of the COVID-19 crisis in the second largest French region in order to identify the different actors and modes of coordination of the local implementation of health policies that this crisis management illustrates in a very visible way. Our methodology is complemented by a content analysis of the main guidelines and decisions related to this implementation.

RESULTS

The analysis of these data allows us to identify three levels of implementation of health policies at the local level (administrative, organizational and operational). Interviews also reveal the existence of different types of coordination specific to each of these levels of local implementation of health policies. These results then make it possible to identify important managerial avenues for promoting global coordination of these three levels of implementation.

CONCLUSIONS

Although research on health services emphasizes the existence of several levels of local implementation of health policies, it offers little in the way of definition or characterization of these levels. The identification in this study of the three levels of local implementation of health policies and their specific forms of coordination contribute to a more precise characterization of this implementation in order to promote, in practice, its global coordination.

Application of Mixed Methods in Health Services Management Research: A Systematic Review

Mixed methods research (MMR) is versatile, pragmatic, and adaptable to constraints and opportunities during a research process. Although MMR has gain popularity in health services management research, little is known about how the research approach has been used and the quality of research. We conducted a systematic review of 198 MMR articles published in selected U.S.-based and international health services management journals from 2000 through 2018 to examine the extent of MMR application and scientific rigor. Results showed limited, yet increasing, use of MMR and a high degree of correspondence between MMR designs and study purposes. However, most articles did not clearly justify using MMR designs and the reporting of method details and research integration were inadequate in a significant portion of publications. We propose a checklist to assist the preparation and review of MMR manuscripts. Additional implications and recommendations to improve transparency, rigor, and quality in MMR are discussed.

South Australian Medicines Evaluation Panel in review: providing evidence-based guidance on the use of high-cost medicines in the South Australian public health system

Objective The South Australian Medicines Evaluation Panel (SAMEP) was established in 2011 to make evidence-based recommendations on the funding of high-cost medicines in South Australian public hospitals via a high-cost medicines formulary. SAMEP represents one component of South Australia's process for state-based health technology assessment (HTA). The aim of this study was to describe the experience of SAMEP in the context of Australia's complex governance model for hospital-based care. Methods A retrospective review was conducted of the SAMEP process and outcomes of medicine evaluations. Decision summaries and meeting minutes were reviewed and reflected upon by the authors to explore the views of the SAMEP membership regarding the function of the committee and state-based HTA more broadly. Results SAMEP has reviewed 29 applications, with 14 (48%) listed on the high-cost medicines formulary. Three applications have been the subject of outcome review and confirm expectations of patient benefit. Conclusion Retrospective review of the committee experience suggests that state-based HTA as operationalised by SAMEP is feasible, provides greater equity of access to high-cost medicines in the South Australian public hospital system and allows for access with evidence development. What is known about the topic? State-based hospital funders often need to make decisions on the provision of high-cost medicines for which there is no national guidance or subsidy. Little published information exists about state-based approaches to medicines evaluation and reimbursement within public hospitals in Australia. What does this paper add? The South Australian experience demonstrates a method for states and territories to tackle the challenges of providing evidence-based access to high-cost medicines in Australian public hospitals. What are the implications for practitioners? This paper provides information for other jurisdictions considering state-based approaches to medicines evaluation and contributes to the broader literature about state-based HTA in Australia.

Scoping Review of Access to Emergency Contraception for Sexual Assault Victims in Emergency Departments in the United States

OBJECTIVE

We conducted a scoping review of the literature to assess the prevalence of hospital policies regarding emergency contraception (EC) and the frequency that U.S. emergency departments (EDs) provide EC to sexual assault survivors.

STUDY DESIGN

We searched PubMed, Embase, the Cochrane Library, and CINAHL from inception to January 2019 for studies that assessed access to EC for sexual assault survivors in EDs. We included English-language studies that surveyed ED staff at U.S. hospitals and reported the weighted mean of the percentage of hospitals with an EC policy for sexual assault survivors and the percentage that provided EC counseling, offered or dispensed EC onsite. Secondary outcomes were rates of testing and prophylaxis for sexually transmitted infections (STIs) and HIV.

RESULTS

From 390 articles retrieved, 14 studies met the inclusion criteria; all studies were published between 2000 and 2013. Eligible studies surveyed staff at 3,314 hospitals. Overall, 60% (weighted mean) of hospitals had a policy on EC, 75% provided EC counseling, 44% offered EC, and 62% had EC available to dispense onsite. Four studies reported secondary outcomes: 81% of hospitals provided STI testing, 84% provided STI prophylaxis, 64% provided HIV testing, and 53% provided HIV prophylaxis.

CONCLUSION

Existing literature demonstrates significant variability in EC policies and practices for sexual assault survivors in U.S. hospital EDs. Future research should assess whether EC access for survivors has improved in ED settings as well as evaluate persistent or new barriers to access.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Protocol: mixed-methods study of how implementation of US state medical cannabis laws affects treatment of chronic non-cancer pain and adverse opioid outcomes

BACKGROUND

Thirty-three US states and Washington, D.C., have enacted medical cannabis laws allowing patients with chronic non-cancer pain to use cannabis, when recommended by a physician, to manage their condition. However, clinical guidelines do not recommend cannabis for treatment of chronic non-cancer pain due to limited and mixed evidence of effectiveness. How state medical cannabis laws affect delivery of evidence-based treatment for chronic non-cancer pain is unclear. These laws could lead to substitution of cannabis in place of clinical guideline-discordant opioid prescribing, reducing risk of opioid use disorder and overdose. Conversely, state medical cannabis laws could lead to substitution of cannabis in place of guideline-concordant treatments such as topical analgesics or physical therapy. This protocol describes a mixed-methods study examining the implementation and effects of state medical cannabis laws on treatment of chronic non-cancer pain. A key contribution of the study is the examination of how variation in state medical cannabis laws' policy implementation rules affects receipt of chronic non-cancer pain treatments.

METHODS

The study uses a concurrent-embedded design. The primary quantitative component of the study employs a difference-in-differences design using a policy trial emulation approach. Quantitative analyses will evaluate state medical cannabis laws' effects on treatment for chronic non-cancer pain as well as on receipt of treatment for opioid use disorder, opioid overdose, cannabis use disorder, and cannabis poisoning among people with chronic non-cancer pain. Secondary qualitative and survey methods will be used to characterize implementation of state medical cannabis laws through interviews with state leaders and representative surveys of physicians who treat, and patients who experience, chronic non-cancer pain in states with medical cannabis laws.

DISCUSSION

This study will examine the effects of medical cannabis laws on patients' receipt of guideline-concordant non-opioid, non-cannabis treatments for chronic non-cancer pain and generate new evidence on the effects of state medical cannabis laws on adverse opioid outcomes. Results will inform the dynamic policy environment in which numerous states consider, enact, and/or amend medical cannabis laws each year.

Risk factors affecting maternal health outcomes in Rivers State of Nigeria: Towards the PRISMA model

Existing research suggests that Nigeria accounts for about 23% of the world's maternal mortality ratio, with negative impacts on women's wellbeing and the country's socio-economic development. The underlying risk factors can be categorized into political influences, poor access to healthcare, inadequate utilization of health facilities, poor family planning support and complex pregnancy-related illness. Yet, the complex interrelations amongst the factors makes it difficult to ascertain the riskiest ones that affect women's reproduction and child death, with the existing intervening strategies failing to address the problem. This study identifies maternal health risk factors and prioritizes their management in Rivers State of Nigeria, using the Prevention and Recovery Information System for Monitoring and Analysis (PRISMA) model. Taking a quantitative turn, we applied exploratory factor analysis to analyze 174 returned questionnaires from healthcare professionals working in Rivers State and used the results to establish relationships between maternal health risk factors, prioritizing the riskiest factors. The outcomes indicate that the PRISMA model provides an effective framework for identifying and managing maternal mortality risks that can enable healthcare experts and managers to address the avoidable risk factors and mitigate the unavoidable patient-related risk factors in Nigeria. The implications for theory, practice and policy are discussed.

Rugby Fans in Training New Zealand (RUFIT-NZ): protocol for a randomized controlled trial to assess the effectiveness and cost-effectiveness of a healthy lifestyle program for overweight men delivered through professional rugby clubs in New Zealand

Background

A healthy lifestyle program that appeals to, and supports, obese New Zealand (NZ) European, Māori (indigenous) and Pasifika men to achieve weight loss is urgently needed. In Scotland, Football Fans in Training (FFIT), a weight management and healthy lifestyle program for overweight and obese men aged 35–65 years , delivered by community coaching staff at professional football clubs, has been shown to be beneficial and cost-effective. A pilot program inspired by FFIT but delivered by professional rugby clubs in NZ (n = 96) was shown to be effective in weight loss, improved physiological outcomes, and adherence to healthy lifestyle behaviors in overweight and obese men. The objective of this trial is to determine the effectiveness and cost-effectiveness of the Rugby Fans in Training New Zealand (RUFIT-NZ) program.

Methods

A pragmatic, two-arm, multi-center, randomized controlled trial involving 308 overweight and obese men aged 30–65 years, randomized to either an intervention group (n = 154) or a wait-list control group (n = 154). The intervention-group participated in the 12-week RUFIT-NZ program, a gender-sensitized, healthy lifestyle intervention adapted to the environment and cultural diversity of NZ and delivered through professional rugby clubs. Participants in the intervention group undergo physical training sessions, in addition to workshop-based sessions to learn about nutrition, physical activity, sleep, sedentary behavior, and a range of behavior-change strategies for sustaining a healthier lifestyle. The control group receives the program after 52 weeks. The primary outcome is change in body weight from baseline to 52 weeks. Secondary outcomes include change in body weight at 12 weeks; waist circumference, blood pressure, fitness, and lifestyle behaviors at 12 and 52 weeks; and cost-effectiveness. A process evaluation informed by the RE-AIM framework will evaluate potential implementation of RUFIT-NZ as an ongoing program in NZ after the trial.

Discussion

This trial will investigate the effectiveness and cost-effectiveness of the RUFIT-NZ program in overweight and obese NZ men.

Trial registration

Australia New Zealand Clinical Trials Registry, ACTRN12619000069156. Registered on 18 January 2019, according to the World Health Organization Trial Registration Data Set. Universal Trial Number, U1111-1245-0645.

The role of scientific evidence in decisions to adopt complex innovations in cancer care settings: a multiple case study in Nova Scotia, Canada

Background

Health care delivery and outcomes can be improved by using innovations (i.e., new ideas, technologies, and practices) supported by scientific evidence. However, scientific evidence may not be the foremost factor in adoption decisions and is rarely sufficient. The objective of this study was to examine the role of scientific evidence in decisions to adopt complex innovations in cancer care.

Methods

Using an explanatory, multiple case study design, we examined the adoption of complex innovations in five purposively sampled cases in Nova Scotia, Canada. Data were collected via documents and key informant interviews. Data analysis involved an in-depth analysis of each case, followed by a cross-case analysis to develop theoretically informed, generalizable knowledge on the role of scientific evidence in innovation adoption that may be applied to similar settings and contexts.

Results

The analyses identified key concepts alongside important caveats and considerations. Key concepts were (1) scientific evidence underpinned the adoption process, (2) evidence from multiple sources informed decision-making, (3) decision-makers considered three key issues when making decisions, and (4) champions were essential to eventual adoption. Caveats and considerations related to the presence of urgent problems and short-term financial pressures and minimizing risk.

Conclusions

The findings revealed the different types of issues decision-makers consider while making these decisions and why different sources of evidence are needed in these processes. Future research should examine how different types of evidence are legitimized and why some types are prioritized over others.

Electronic supplementary material

The online version of this article (10.1186/s13012-019-0859-5) contains supplementary material, which is available to authorized users.

Key components of knowledge transfer and exchange in health services research: Findings from a systematic scoping review

AIMS

To identify the key common components of knowledge transfer and exchange in existing models to facilitate practice developments in health services research.

BACKGROUND

There are over 60 models of knowledge transfer and exchange designed for various areas of health care. Many of them remain untested and lack guidelines for scaling-up of successful implementation of research findings and of proven models ensuring that patients have access to optimal health care, guided by current research.

DESIGN

A scoping review was conducted in line with PRISMA guidelines. Key components of knowledge transfer and exchange were identified using thematic analysis and frequency counts.

DATA SOURCES

Six electronic databases were searched for papers published before January 2015 containing four key terms/variants: knowledge, transfer, framework, health care.

REVIEW METHODS

Double screening, extraction and coding of the data using thematic analysis were employed to ensure rigour. As further validation stakeholders' consultation of the findings was performed to ensure accessibility.

RESULTS

Of the 4,288 abstracts, 294 full-text articles were screened, with 79 articles analysed. Six key components emerged: knowledge transfer and exchange message, Stakeholders and Process components often appeared together, while from two contextual components Inner Context and the wider Social, Cultural and Economic Context, with the wider context less frequently considered. Finally, there was little consideration of the Evaluation of knowledge transfer and exchange activities. In addition, specific operational elements of each component were identified.

CONCLUSIONS

The six components offer the basis for knowledge transfer and exchange activities, enabling researchers to more effectively share their work. Further research exploring the potential contribution of the interactions of the components is recommended.

Sustainability in Health care by Allocating Resources Effectively (SHARE) 11: reporting outcomes of an evidence-driven approach to disinvestment in a local healthcare setting

Background

This is the final paper in a thematic series reporting a program of Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. The SHARE Program was established to explore a systematic, integrated, evidence-based organisation-wide approach to disinvestment in a large Australian health service network. This paper summarises the findings, discusses the contribution of the SHARE Program to the body of knowledge and understanding of disinvestment in the local healthcare setting, and considers implications for policy, practice and research.

Discussion

The SHARE program was conducted in three phases. Phase One was undertaken to understand concepts and practices related to disinvestment and the implications for a local health service and, based on this information, to identify potential settings and methods for decision-making about disinvestment. The aim of Phase Two was to implement and evaluate the proposed methods to determine which were sustainable, effective and appropriate in a local health service. A review of the current literature incorporating the SHARE findings was conducted in Phase Three to contribute to the understanding of systematic approaches to disinvestment in the local healthcare context.

SHARE differed from many other published examples of disinvestment in several ways: by seeking to identify and implement disinvestment opportunities within organisational infrastructure rather than as standalone projects; considering disinvestment in the context of all resource allocation decisions rather than in isolation; including allocation of non-monetary resources as well as financial decisions; and focusing on effective use of limited resources to optimise healthcare outcomes.

Conclusion

The SHARE findings provide a rich source of new information about local health service decision-making, in a level of detail not previously reported, to inform others in similar situations. Multiple innovations related to disinvestment were found to be acceptable and feasible in the local setting. Factors influencing decision-making, implementation processes and final outcomes were identified; and methods for further exploration, or avoidance, in attempting disinvestment in this context are proposed based on these findings. The settings, frameworks, models, methods and tools arising from the SHARE findings have potential to enhance health care and patient outcomes.

Electronic supplementary material

The online version of this article (10.1186/s12913-018-3172-0) contains supplementary material, which is available to authorized users.

Evaluation Tool for Assessing a Newly Implemented Massive Transfusion Protocol

Exsanguination requires massive blood product replacement and termination of the bleeding source to prevent hemorrhagic shock and death. Massive transfusion protocols (MTPs) are algorithms that allow the health care team to quickly stabilize the bleeding patient and guide blood product administration. However, no national MTP guidelines or a standardized evaluation tool exist for collecting and reporting MTP-related data. The purpose of this article is to describe an original MTP evaluation tool, how it was used, barriers encountered, and a framework for reporting the MTP evaluation data. The evidence-based Broxton MTP Evaluation Tool was developed to evaluate the use of a newly implemented MTP via a retrospective review of electronic medical records (EMRs). Although the instrument itself worked well, barriers were encountered while reviewing the EMRs for the MTP evaluation. These barriers included no institutional entity was charged with tracking MTP activations, no searchable database was established to collect data concerning the MTP-activated patients, and no standard location in the EMR was designated for documenting the MTP activation. When devising protocols such as an MTP, a priori strategies should be developed for its implementation, documentation, and evaluation. Research is needed to determine best practices for evaluating an MTP to ensure positive patient outcomes with this protocol.

Organisational capacity and its relationship to research use in six Australian health policy agencies

There are calls for policymakers to make greater use of research when formulating policies. Therefore, it is important that policy organisations have a range of tools and systems to support their staff in using research in their work. The aim of the present study was to measure the extent to which a range of tools and systems to support research use were available within six Australian agencies with a role in health policy, and examine whether this was related to the extent of engagement with, and use of research in policymaking by their staff. The presence of relevant systems and tools was assessed via a structured interview called ORACLe which is conducted with a senior executive from the agency. To measure research use, four policymakers from each agency undertook a structured interview called SAGE, which assesses and scores the extent to which policymakers engaged with (i.e., searched for, appraised, and generated) research, and used research in the development of a specific policy document. The results showed that all agencies had at least a moderate range of tools and systems in place, in particular policy development processes; resources to access and use research (such as journals, databases, libraries, and access to research experts); processes to generate new research; and mechanisms to establish relationships with researchers. Agencies were less likely, however, to provide research training for staff and leaders, or to have evidence-based processes for evaluating existing policies. For the majority of agencies, the availability of tools and systems was related to the extent to which policymakers engaged with, and used research when developing policy documents. However, some agencies did not display this relationship, suggesting that other factors, namely the organisation's culture towards research use, must also be considered.

Sustainability in Health care by Allocating Resources Effectively (SHARE) 8: developing, implementing and evaluating an evidence dissemination service in a local healthcare setting

BACKGROUND

This is the eighth in a series of papers reporting Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. The SHARE Program was a systematic, integrated, evidence-based program for disinvestment within a large Australian health service. One of the aims was to explore methods to deliver existing high quality synthesised evidence directly to decision-makers to drive decision-making proactively. An Evidence Dissemination Service (EDS) was proposed. While this was conceived as a method to identify disinvestment opportunities, it became clear that it could also be a way to review all practices for consistency with current evidence. This paper reports the development, implementation and evaluation of two models of an in-house EDS.

METHODS

Frameworks for development of complex interventions, implementation of evidence-based change, and evaluation and explication of processes and outcomes were adapted and/or applied. Mixed methods including a literature review, surveys, interviews, workshops, audits, document analysis and action research were used to capture barriers, enablers and local needs; identify effective strategies; develop and refine proposals; ascertain feedback and measure outcomes.

RESULTS

Methods to identify, capture, classify, store, repackage, disseminate and facilitate use of synthesised research evidence were investigated. In Model 1, emails containing links to multiple publications were sent to all self-selected participants who were asked to determine whether they were the relevant decision-maker for any of the topics presented, whether change was required, and to take the relevant action. This voluntary framework did not achieve the aim of ensuring practice was consistent with current evidence. In Model 2, the need for change was established prior to dissemination, then a summary of the evidence was sent to the decision-maker responsible for practice in the relevant area who was required to take appropriate action and report the outcome. This mandatory governance framework was successful. The factors influencing decisions, processes and outcomes were identified.

CONCLUSION

An in-house EDS holds promise as a method of identifying disinvestment opportunities and/or reviewing local practice for consistency with current evidence. The resource-intensive nature of delivery of the EDS is a potential barrier. The findings from this study will inform further exploration.

A research utilisation framework for informing global health and development policies and programmes

A shift in the culture and practice of health and development research is required to maximise the real-world use of evidence by non-academic or non-research-oriented audiences. Many frameworks have been developed to guide and measure the research utilisation process, yet none have been widely applied. Some frameworks are simplified to an unrealistic linear representation while others are rendered overly complex and unusable in an attempt to capture all aspects of the research utilisation process. Additionally, many research utilisation frameworks have focused on the policy development process or within a clinical setting, with less application of the translation process at the programme level. In response to this gap – and drawing from over a decade of experience implementing research utilisation strategies – we developed a simple, four-phase framework to guide global health and development efforts that seek to apply evidence to policies and programmes. We present a detailed description of each phase in our framework, with examples of its relevance and application illustrated through our own case study experiences in global health. We believe the utility of this framework extends beyond the health sector and is relevant for maximising use of evidence to achieve the Sustainable Development Goals.

Text4Heart II - improving medication adherence in people with heart disease: a study protocol for a randomized controlled trial

Background

Cardiac rehabilitation (CR) is an essential component of contemporary management for patients with coronary heart disease, including following an acute coronary syndrome (ACS). CR typically involves education and support to assist people following an ACS to make lifestyle changes and prevent subsequent events. Despite its benefits, uptake and participation in tradition CR programs is low. The use of mobile technologies (mHealth) offers the potential to improve reach, access, and delivery of CR support. We aim to determine the effectiveness and cost-effectiveness of a text-messaging intervention (Text4Heart II) to improve adherence to medication and lifestyle change in addition to usual care in people following an ACS. A second aim is to use the RE-AIM framework to inform the potential implementation of Text4Heart II within health services in New Zealand.

Methods

Text4Heart II is a two-arm, parallel, superiority randomized controlled trial conducted in two large metropolitan hospitals in Auckland, New Zealand. Three hundred and thirty participants will be randomized to either a 24-week theory- and evidence-based personalized text message program to support self-management in addition to usual CR, or usual CR alone (control). Outcomes are assessed at 6 and 12 months. The primary outcome is the proportion of participants adhering to medication at 6 months as measured by dispensed records. Secondary outcomes include medication adherence at 12 months, the proportion of participants adhering to self-reported healthy behaviors (physical activity, fruit and vegetable consumption, moderating alcohol intake and smoking status) measured using a composite health behavior score, self-reported medication adherence, cardiovascular risk factors (lipids, blood pressure), readmissions and related hospital events at 6 and 12 months. A cost-effectiveness analysis will also be conducted. Using the RE-AIM framework, we will determine uptake and sustainability of the intervention.

Discussion

The Text4Heart II trial will determine the effectiveness of a text-messaging intervention to improve adherence to medication and lifestyle behaviors at both 6 and 12 months. Using the RE-AIM framework this trial will provide much needed data and insight into the potential implementation of Text4Heart II. This trial addresses many limitations/criticisms of previous mHealth trials; it builds on our Text4Heart pilot trial, it is adequately powered, has sufficient duration to elicit behavior change, and the follow-up assessments (6 and 12 months) are long enough to determine the sustained effect of the intervention.

Trial registration

Australian New Zealand Clinical Trials Registry, ID: ACTRN12616000422426. Registered retrospectively on 1 April 2016. 

Electronic supplementary material

The online version of this article (10.1186/s13063-018-2468-z) contains supplementary material, which is available to authorized users.

Evidence use in decision-making on introducing innovations: a systematic scoping review with stakeholder feedback

BACKGROUND

A range of evidence informs decision-making on innovation in health care, including formal research findings, local data and professional opinion. However, cultural and organisational factors often prevent the translation of evidence for innovations into practice. In addition to the characteristics of evidence, it is known that processes at the individual level influence its impact on decision-making. Less is known about the ways in which processes at the professional, organisational and local system level shape evidence use and its role in decisions to adopt innovations.

METHODS

A systematic scoping review was used to review the health literature on innovations within acute and primary care and map processes at the professional, organisational and local system levels which influence how evidence informs decision-making on innovation. Stakeholder feedback on the themes identified was collected via focus groups to test and develop the findings.

RESULTS

Following database and manual searches, 31 studies reporting primary qualitative data met the inclusion criteria: 24 were of sufficient methodological quality to be included in the thematic analysis. Evidence use in decision-making on innovation is influenced by multi-level processes (professional, organisational, local system) and interactions across these levels. Preferences for evidence vary by professional group and health service setting. Organisations can shape professional behaviour by requiring particular forms of evidence to inform decision-making. Pan-regional organisations shape innovation decision-making at lower levels. Political processes at all levels shape the selection and use of evidence in decision-making.

CONCLUSIONS

The synthesis of results from primary qualitative studies found that evidence use in decision-making on innovation is influenced by processes at multiple levels. Interactions between different levels shape evidence use in decision-making (e.g. professional groups and organisations can use local systems to validate evidence and legitimise innovations, while local systems can tailor or frame evidence to influence activity at lower levels). Organisational leaders need to consider whether the environment in which decisions are made values diverse evidence and stakeholder perspectives. Further qualitative research on decision-making practices that highlights how and why different types of evidence come to count during decisions, and tracks the political aspects of decisions about innovation, is needed.

Sustainability in Health care by Allocating Resources Effectively (SHARE) 9: conceptualising disinvestment in the local healthcare setting

BACKGROUND

This is the ninth in a series of papers reporting a program of Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. The disinvestment literature has broadened considerably over the past decade; however there is a significant gap regarding systematic, integrated, organisation-wide approaches. This debate paper presents a discussion of the conceptual aspects of disinvestment from the local perspective.

DISCUSSION

Four themes are discussed: Terminology and concepts, Motivation and purpose, Relationships with other healthcare improvement paradigms, and Challenges to disinvestment. There are multiple definitions for disinvestment, multiple concepts underpin the definitions and multiple alternative terms convey these concepts; some definitions overlap and some are mutually exclusive; and there are systematic discrepancies in use between the research and practice settings. Many authors suggest that the term 'disinvestment' should be avoided due to perceived negative connotations and propose that the concept be considered alongside investment in the context of all resource allocation decisions and approached from the perspective of optimising health care. This may provide motivation for change, reduce disincentives and avoid some of the ethical dilemmas inherent in other disinvestment approaches. The impetus and rationale for disinvestment activities are likely to affect all aspects of the process from identification and prioritisation through to implementation and evaluation but have not been widely discussed. A need for mechanisms, frameworks, methods and tools for disinvestment is reported. However there are several health improvement paradigms with mature frameworks and validated methods and tools that are widely-used and well-accepted in local health services that already undertake disinvestment-type activities and could be expanded and built upon. The nature of disinvestment brings some particular challenges for policy-makers, managers, health professionals and researchers. There is little evidence of successful implementation of 'disinvestment' projects in the local setting, however initiatives to remove or replace technologies and practices have been successfully achieved through evidence-based practice, quality and safety activities, and health service improvement programs.

CONCLUSIONS

These findings suggest that the construct of 'disinvestment' may be problematic at the local level. A new definition and two potential approaches to disinvestment are proposed to stimulate further research and discussion.

Sustainability in Health care by Allocating Resources Effectively (SHARE) 10: operationalising disinvestment in a conceptual framework for resource allocation

Background

This is the tenth in a series of papers reporting a program of Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. After more than a decade of research, there is little published evidence of active and successful disinvestment. The paucity of frameworks, methods and tools is reported to be a factor in the lack of success. However there are clear and consistent messages in the literature that can be used to inform development of a framework for operationalising disinvestment. This paper, along with the conceptual review of disinvestment in Paper 9 of this series, aims to integrate the findings of the SHARE Program with the existing disinvestment literature to address the lack of information regarding systematic organisation-wide approaches to disinvestment at the local health service level.

Discussion

A framework for disinvestment in a local healthcare setting is proposed. Definitions for essential terms and key concepts underpinning the framework have been made explicit to address the lack of consistent terminology. Given the negative connotations of the word ‘disinvestment’ and the problems inherent in considering disinvestment in isolation, the basis for the proposed framework is ‘resource allocation’ to address the spectrum of decision-making from investment to disinvestment. The focus is positive: optimising healthcare, improving health outcomes, using resources effectively.

The framework is based on three components: a program for decision-making, projects to implement decisions and evaluate outcomes, and research to understand and improve the program and project activities. The program consists of principles for decision-making and settings that provide opportunities to introduce systematic prompts and triggers to initiate disinvestment. The projects follow the steps in the disinvestment process. Potential methods and tools are presented, however the framework does not stipulate project design or conduct; allowing application of any theories, methods or tools at each step. Barriers are discussed and examples illustrating constituent elements are provided.

Conclusions

The framework can be employed at network, institutional, departmental, ward or committee level. It is proposed as an organisation-wide application, embedded within existing systems and processes, which can be responsive to needs and priorities at the level of implementation. It can be used in policy, management or clinical contexts.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-017-2506-7) contains supplementary material, which is available to authorized users.

Sustainability in Health care by Allocating Resources Effectively (SHARE) 7: supporting staff in evidence-based decision-making, implementation and evaluation in a local healthcare setting

BACKGROUND

This is the seventh in a series of papers reporting Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. The SHARE Program was a systematic, integrated, evidence-based program for resource allocation within a large Australian health service. It aimed to facilitate proactive use of evidence from research and local data; evidence-based decision-making for resource allocation including disinvestment; and development, implementation and evaluation of disinvestment projects. From the literature and responses of local stakeholders it was clear that provision of expertise and education, training and support of health service staff would be required to achieve these aims. Four support services were proposed. This paper is a detailed case report of the development, implementation and evaluation of a Data Service, Capacity Building Service and Project Support Service. An Evidence Service is reported separately.

METHODS

Literature reviews, surveys, interviews, consultation and workshops were used to capture and process the relevant information. Existing theoretical frameworks were adapted for evaluation and explication of processes and outcomes.

RESULTS

Surveys and interviews identified current practice in use of evidence in decision-making, implementation and evaluation; staff needs for evidence-based practice; nature, type and availability of local health service data; and preferred formats for education and training. The Capacity Building and Project Support Services were successful in achieving short term objectives; but long term outcomes were not evaluated due to reduced funding. The Data Service was not implemented at all. Factors influencing the processes and outcomes are discussed.

CONCLUSION

Health service staff need access to education, training, expertise and support to enable evidence-based decision-making and to implement and evaluate the changes arising from those decisions. Three support services were proposed based on research evidence and local findings. Local factors, some unanticipated and some unavoidable, were the main barriers to successful implementation. All three proposed support services hold promise as facilitators of EBP in the local healthcare setting. The findings from this study will inform further exploration.

Sustainability in Health care by Allocating Resources Effectively (SHARE) 6: investigating methods to identify, prioritise, implement and evaluate disinvestment projects in a local healthcare setting

BACKGROUND

This is the sixth in a series of papers reporting Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. The SHARE program was established to investigate a systematic, integrated, evidence-based approach to disinvestment within a large Australian health service. This paper describes the methods employed in undertaking pilot disinvestment projects. It draws a number of lessons regarding the strengths and weaknesses of these methods; particularly regarding the crucial first step of identifying targets for disinvestment.

METHODS

Literature reviews, survey, interviews, consultation and workshops were used to capture and process the relevant information. A theoretical framework was adapted for evaluation and explication of disinvestment projects, including a taxonomy for the determinants of effectiveness, process of change and outcome measures. Implementation, evaluation and costing plans were developed.

RESULTS

Four literature reviews were completed, surveys were received from 15 external experts, 65 interviews were conducted, 18 senior decision-makers attended a data gathering workshop, 22 experts and local informants were consulted, and four decision-making workshops were undertaken. Mechanisms to identify disinvestment targets and criteria for prioritisation and decision-making were investigated. A catalogue containing 184 evidence-based opportunities for disinvestment and an algorithm to identify disinvestment projects were developed. An Expression of Interest process identified two potential disinvestment projects. Seventeen additional projects were proposed through a non-systematic nomination process. Four of the 19 proposals were selected as pilot projects but only one reached the implementation stage. Factors with potential influence on the outcomes of disinvestment projects are discussed and barriers and enablers in the pilot projects are summarised.

CONCLUSION

This study provides an in-depth insight into the experience of disinvestment in one local healthcare service. To our knowledge, this is the first paper to report the process of disinvestment from identification, through prioritisation and decision-making, to implementation and evaluation, and finally explication of the processes and outcomes.

Sustainability in Health care by Allocating Resources Effectively (SHARE) 5: developing a model for evidence-driven resource allocation in a local healthcare setting

Background

This is the fifth in a series of papers reporting Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. This paper synthesises the findings from Phase One of the SHARE Program and presents a model to be implemented and evaluated in Phase Two. Monash Health, a large healthcare network in Melbourne Australia, sought to establish an organisation-wide systematic evidence-based program for disinvestment. In the absence of guidance from the literature, the Centre for Clinical Effectiveness, an in-house ‘Evidence Based Practice Support Unit’, was asked to explore concepts and practices related to disinvestment, consider the implications for a local health service and identify potential settings and methods for decision-making.

Methods

Mixed methods were used to capture the relevant information. These included literature reviews; online questionnaire, interviews and structured workshops with a range of stakeholders; and consultation with experts in disinvestment, health economics and health program evaluation. Using the principles of evidence-based change, the project team worked with health service staff, consumers and external experts to synthesise the findings from published literature and local research and develop proposals, frameworks and plans.

Results

Multiple influencing factors were extracted from these findings. The implications were both positive and negative and addressed aspects of the internal and external environments, human factors, empirical decision-making, and practical applications. These factors were considered in establishment of the new program; decisions reached through consultation with stakeholders were used to define four program components, their aims and objectives, relationships between components, principles that underpin the program, implementation and evaluation plans, and preconditions for success and sustainability. The components were Systems and processes, Disinvestment projects, Support services, and Program evaluation and research. A model for a systematic approach to evidence-based resource allocation in a local health service was developed.

Conclusion

A robust evidence-based investigation of the research literature and local knowledge with a range of stakeholders resulted in rich information with strong consistent messages. At the completion of Phase One, synthesis of the findings enabled development of frameworks and plans and all preconditions for exploration of the four main aims in Phase Two were met.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-017-2208-1) contains supplementary material, which is available to authorized users.

Sustainability in health care by allocating resources effectively (SHARE) 3: examining how resource allocation decisions are made, implemented and evaluated in a local healthcare setting

Background

This is the third in a series of papers reporting a program of Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. Leaders in a large Australian health service planned to establish an organisation-wide, systematic, integrated, evidence-based approach to disinvestment. In order to introduce new systems and processes for disinvestment into existing decision-making infrastructure, we aimed to understand where, how and by whom resource allocation decisions were made, implemented and evaluated. We also sought the knowledge and experience of staff regarding previous disinvestment activities.

Methods

Structured interviews, workshops and document analysis were used to collect information from multiple sources in an environmental scan of decision-making systems and processes. Findings were synthesised using a theoretical framework.

Results

Sixty-eight respondents participated in interviews and workshops. Eight components in the process of resource allocation were identified: Governance, Administration, Stakeholder engagement, Resources, Decision-making, Implementation, Evaluation and, where appropriate, Reinvestment of savings. Elements of structure and practice for each component are described and a new framework was developed to capture the relationships between them. A range of decision-makers, decision-making settings, type and scope of decisions, criteria used, and strengths, weaknesses, barriers and enablers are outlined. The term ‘disinvestment’ was not used in health service decision-making. Previous projects that involved removal, reduction or restriction of current practices were driven by quality and safety issues, evidence-based practice or a need to find resource savings and not by initiatives where the primary aim was to disinvest. Measuring resource savings is difficult, in some situations impossible. Savings are often only theoretical as resources released may be utilised immediately by patients waiting for beds, clinic appointments or surgery. Decision-making systems and processes for resource allocation are more complex than assumed in previous studies.

Conclusion

There is a wide range of decision-makers, settings, scope and type of decisions, and criteria used for allocating resources within a single institution. To our knowledge, this is the first paper to report this level of detail and to introduce eight components of the resource allocation process identified within a local health service.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-017-2207-2) contains supplementary material, which is available to authorized users.

Sustainability in Health care by Allocating Resources Effectively (SHARE) 2: identifying opportunities for disinvestment in a local healthcare setting

Background

This is the second in a series of papers reporting a program of Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. Rising healthcare costs, continuing advances in health technologies and recognition of ineffective practices and systematic waste are driving disinvestment of health technologies and clinical practices that offer little or no benefit in order to maximise outcomes from existing resources. However there is little information to guide regional health services or individual facilities in how they might approach disinvestment locally. This paper outlines the investigation of potential settings and methods for decision-making about disinvestment in the context of an Australian health service.

Methods

Methods include a literature review on the concepts and terminology relating to disinvestment, a survey of national and international researchers, and interviews and workshops with local informants. A conceptual framework was drafted and refined with stakeholder feedback.

Results

There is a lack of common terminology regarding definitions and concepts related to disinvestment and no guidance for an organisation-wide systematic approach to disinvestment in a local healthcare service.

A summary of issues from the literature and respondents highlight the lack of theoretical knowledge and practical experience and provide a guide to the information required to develop future models or methods for disinvestment in the local context.

A conceptual framework was developed. Three mechanisms that provide opportunities to introduce disinvestment decisions into health service systems and processes were identified. Presented in order of complexity, time to achieve outcomes and resources required they include 1) Explicit consideration of potential disinvestment in routine decision-making, 2) Proactive decision-making about disinvestment driven by available evidence from published research and local data, and 3) Specific exercises in priority setting and system redesign.

Conclusion

This framework identifies potential opportunities to initiate disinvestment activities in a systematic integrated approach that can be applied across a whole organisation using transparent, evidence-based methods. Incorporating considerations for disinvestment into existing decision-making systems and processes might be achieved quickly with minimal cost; however establishment of new systems requires research into appropriate methods and provision of appropriate skills and resources to deliver them.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-017-2211-6) contains supplementary material, which is available to authorized users.

Sustainability in health care by allocating resources effectively (SHARE) 4: exploring opportunities and methods for consumer engagement in resource allocation in a local healthcare setting

BACKGROUND

This is the fourth in a series of papers reporting a program of Sustainability in Health care by Allocating Resources Effectively (SHARE) in a local healthcare setting. Healthcare decision-makers have sought to improve the effectiveness and efficiency of services through removal or restriction of practices that are unsafe or of little benefit, often referred to as 'disinvestment'. A systematic, integrated, evidence-based program for disinvestment was being established within a large Australian health service network. Consumer engagement was acknowledged as integral to this process. This paper reports the process of developing a model to integrate consumer views and preferences into an organisation-wide approach to resource allocation.

METHODS

A literature search was conducted and interviews and workshops were undertaken with health service consumers and staff. Findings were drafted into a model for consumer engagement in resource allocation which was workshopped and refined.

RESULTS

Although consumer engagement is increasingly becoming a requirement of publicly-funded health services and documented in standards and policies, participation in organisational decision-making is not widespread. Several consistent messages for consumer engagement in this context emerged from the literature and consumer responses. Opportunities, settings and activities for consumer engagement through communication, consultation and participation were identified within the resource allocation process. Sources of information regarding consumer values and perspectives in publications and locally-collected data, and methods to use them in health service decision-making, were identified. A model bringing these elements together was developed.

CONCLUSION

The proposed model presents potential opportunities and activities for consumer engagement in the context of resource allocation.

The use of evidence in English local public health decision-making: a systematic scoping review

BACKGROUND

Public health decision-making structures in England have transformed since the implementation of reforms in 2013, with responsibility for public health services and planning having shifted from the "health" boundary to local authority (LA; local government) control. This transformation may have interrupted flows of research evidence use in decision-making and introduced a new political element to public health decision-making. For generators of research evidence, understanding and responding to this new landscape and decision-makers' evidence needs is essential.

METHODS

We conducted a systematic scoping review of the literature, drawing upon four databases and undertaking manual searching and citation tracking. Included studies were English-based, published in 2010 onwards, and were focused on public health decision-making, including the utilisation or underutilisation of research evidence use, in local (regional or sub-regional) areas. All studies presented empirical findings collected through primary research methods or through the reanalysis of existing primary data.

RESULTS

From a total of 903 records, 23 papers from 21 studies were deemed to be eligible and were included for further data extraction. Three clear trends in evidence use were identified: (i) the primacy of local evidence, (ii) the important role of local experts in providing evidence and knowledge, and (iii) the high value placed on local evaluation evidence despite the varying methodological rigour. Barriers to the use of research evidence included issues around access and availability of applicable research evidence, and indications that the use of evidence could be perceived as a bureaucratic process. Two new factors resulting from reforms to public health structures were identified that potentially changed existing patterns of research evidence use and decision-making requirements: (i) greater emphasis among public health practitioners on the perceived uniqueness of LA areas and structures following devolution of public health into LAs and (ii) challenges introduced in responding to higher levels of local political accountability.

CONCLUSIONS

There is a need to better understand and respond to the evidence needs of decision-makers working in public health and to work more collaboratively in developing solutions to the underutilisation of research evidence in decision-making.

Preparing the sexual health workforce to deliver integrated services: is education the answer? A qualitative study exploring the impact of sexual health education on developing integrated policy and practice

Aim This study aimed to explore the ability of sexual health nurses working in the South West of England, to implement new learning within existing sexual health service delivery models. Drawing on Lipsky's account of street-level bureaucracy to conceptualise policy implementation, the impact of workforce learning on the development of integrated services across this region of the United Kingdom was assessed.

BACKGROUND

In order to achieve the United Nations' goal of universal access to sexual health, it is essential for reproductive and sexual health, including HIV provision, to integrate into a single service. This integration requires a commitment to collaboration by service commissioners and an alignment of principles and values across sexual health and contraceptive services. UK health policy has embraced this holistic agenda but moves towards integrating historically separate clinical services, has presented significant workforce development challenges and influenced policy success.

METHODS

Employing a qualitative approach, the study included data from semi-structured telephone interviews and focus groups, and longitudinal data from pre- and post-intervention surveys, collected between September 2013 and September 2015. Data were collected from 88 nurses undertaking a workforce development programme and six of their service managers. Data were analysed using thematic analysis to identify consistent themes. Findings Nurses confirmed the role of new learning in enabling them to negotiate the political landscape but expressed frustration at their lack of agency in the integration agenda, exposing a clear dichotomy between the intentions of policy and the reality of practice. Nevertheless, using high levels of professional judgement and discretion practitioners managed the incongruence between policy and practice in order to deliver integrated services in the interests of patients. Workforce education, while essential for the transition to the delivery of integrated services, was insufficient to fulfil the sexual health agenda without a strengthening of public health.

Getting the most out of knowledge and innovation transfer agents in health care: a qualitative study

Background

Knowledge and innovation transfer (KIT) is recognised internationally as a complex, dynamic process that is difficult to embed in organisations. There is growing use of health service–academic–industry collaborations in the UK, with knowledge brokers linking producers with the users of knowledge and innovation.

Aim

Focusing on KIT ‘agent’ roles within Academic Health Science Networks in England and Partnerships in Wales, we show how individual dispositions, processes and content contribute to desired outcomes.

Methods

We studied the KIT intentions of all Academic Health Science Networks in England, and the South East Wales Academic Health Science Partnership. Using a qualitative case study design, we studied the work of 13 KIT agents purposively sampled from five networks, by collecting data from observation of meetings, documentation, KIT agent audio-diaries, and semistructured interviews with KIT agents, their line managers and those they supported (‘Links’). We also used a consensus method in a meeting of experts (nominal group technique) to discuss the measurement of outcomes of KIT agent activity.

Findings

The case study KIT agents were predominantly from a clinical background with differing levels of experience and expertise, with the shared aim of improving services and patient care. Although outside of recognised career structures, the flexibility afforded to KIT agents to define their role was an enabler of success. Other helpful factors included (1) time and resources to devote to KIT activity; (2) line manager support and a team to assist in the work; and (3) access and the means to use data for improvement projects. The organisational and political context could be challenging. KIT agents not only tackled local barriers such as siloed working, but also navigated shifting regional and national policies. Board-level support for knowledge mobilisation together with a culture of reflection (listening to front-line staff), openness to challenges and receptivity to research all enabled KIT agents to achieve desired outcomes. Nominal group findings underscored the importance of relating measures to specific intended outcomes. However, the case studies highlighted that few measures were employed by KIT agents and their managers. Using social marketing theory helped to show linkages between processes, outcomes and impact, and drew attention to how KIT agents developed insight into their clients’ needs and tailored work accordingly.

Limitations

Level of KIT agent participation varied; line managers and Links were interviewed only once; and outcomes were self-reported.

Conclusions

Social marketing theory provided a framework for analysing KIT agent activity. The preparatory work KIT agents do in listening, understanding local context and building relationships enabled them to develop ‘insight’ and adapt their ‘offer’ to clients to achieve desired outcomes.

Future work

The complexity of the role and the environment in which it is played out justifies more research on KIT agents. Suggestions include (1) longitudinal study of career pathways; (2) how roles are negotiated within teams and how competing priorities are managed; (3) how success is measured; (4) the place of improvement methodologies within KIT work; (5) the application of social marketing theory to comparative study of similar roles; and (6) patients as KIT agents.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Barriers and facilitators to evaluation of health policies and programs: Policymaker and researcher perspectives

Our research sought to identify the barriers and facilitators experienced by policymakers and evaluation researchers in the critical early stages of establishing an evaluation of a policy or program. We sought to determine the immediate barriers experienced at the point of initiating or commissioning evaluations and how these relate to broader system factors previously identified in the literature. We undertook 17 semi-structured interviews with a purposive sample of senior policymakers (n=9) and senior evaluation researchers (n=8) in Australia. Six themes were consistently raised by participants: political influence, funding, timeframes, a 'culture of evaluation', caution over anticipated results, and skills of policy agency staff. Participants also reflected on the dynamics of policy-researcher relationships including different motivations, physical and conceptual separation of the policy and researcher worlds, intellectual property concerns, and trust. We found that political and system factors act as macro level barriers to good evaluation practice that are manifested as time and funding constraints and contribute to organisational cultures that can come to fear evaluation. These factors then fed into meso and micro level factors. The dynamics of policy-researcher relationship provide a further challenge to evaluating government policies and programs.

Study protocol: DEcisions in health Care to Introduce or Diffuse innovations using Evidence (DECIDE)

BACKGROUND

A range of evidence informs healthcare decision-making, from formal research findings to 'soft intelligence' or local data, as well as practical experience or tacit knowledge. However, cultural and organisational factors often prevent the translation of such evidence into practice. Using a multi-level framework, this project will analyse how interactions between the evidence available and processes at the micro (individual/group) and meso (organisational/system) levels influence decisions to introduce or diffuse innovations in acute and primary care within the National Health Service in the UK.

METHODS/DESIGN

This study will use a mixed methods design, combining qualitative and quantitative methods, and involves four interdependent work streams: (1) rapid evidence synthesis of relevant literature with stakeholder feedback; (2) in-depth case studies of 'real-world' decision-making in acute and primary care; (3) a national survey and discrete choice experiment; and (4) development of guidance for decision-makers and evaluators to support the use of evidence in decision-making.

DISCUSSION

This study will enhance the understanding of decision-makers' use of diverse forms of evidence. The findings will provide insights into how and why some evidence does inform decisions to introduce healthcare innovations, and why barriers persist in other cases. It will also quantify decision-makers' preferences, including the 'tipping point' of evidence needed to shift stakeholders' views. Practical guidance will be shared with healthcare decision-makers and evaluators on uses of evidence to enable the introduction and diffusion of innovation.

The development of ORACLe: a measure of an organisation's capacity to engage in evidence-informed health policy

Background

Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations’ research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations’ capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership).

Method

We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe.

Results

The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4).

Conclusion

We utilised qualitative and quantitative methods to develop a system to assess and score organisations’ capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-015-0069-9) contains supplementary material, which is available to authorized users.

Development, implementation and evaluation of an evidence-based program for introduction of new health technologies and clinical practices in a local healthcare setting

Background

This paper reports the process of establishing a transparent, accountable, evidence-based program for introduction of new technologies and clinical practices (TCPs) in a large Australian healthcare network. Many countries have robust evidence-based processes for assessment of new TCPs at national level. However many decisions are made by local health services where the resources and expertise to undertake health technology assessment (HTA) are limited and a lack of structure, process and transparency has been reported.

Methods

An evidence-based model for process change was used to establish the program. Evidence from research and local data, experience of health service staff and consumer perspectives were incorporated at each of four steps: identifying the need for change, developing a proposal, implementation and evaluation. Checklists assessing characteristics of success, factors for sustainability and barriers and enablers were applied and implementation strategies were based on these findings. Quantitative and qualitative methods were used for process and outcome evaluation. An action research approach underpinned ongoing refinement to systems, processes and resources.

Results

A Best Practice Guide developed from the literature and stakeholder consultation identified seven program components: Governance, Decision-Making, Application Process, Monitoring and Reporting, Resources, Administration, and Evaluation and Quality Improvement. The aims of transparency and accountability were achieved. The processes are explicit, decisions published, outcomes recorded and activities reported. The aim of ascertaining rigorous evidence-based information for decision-making was not achieved in all cases. Applicants proposing new TCPs provided the evidence from research literature and local data however the information was often incorrect or inadequate, overestimating benefits and underestimating costs. Due to these limitations the initial application process was replaced by an Expression of Interest from applicants followed by a rigorous HTA by independent in-house experts.

Conclusion

The program is generalisable to most health care organisations. With one exception, the components would be achievable with minimal additional resources; the lack of skills and resources required for HTA will limit effective application in many settings. A toolkit containing details of the processes and sample materials is provided to facilitate replication or local adaptation by those wishing to establish a similar program.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-1178-4) contains supplementary material, which is available to authorized users.

Evidence-informed policy formulation and implementation: a comparative case study of two national policies for improving health and social care in Sweden

Background

Evidence has come to play a central role in health policymaking. However, policymakers tend to use other types of information besides research evidence. Most prior studies on evidence-informed policy have focused on the policy formulation phase without a systematic analysis of its implementation. It has been suggested that in order to fully understand the policy process, the analysis should include both policy formulation and implementation. The purpose of the study was to explore and compare two policies aiming to improve health and social care in Sweden and to empirically test a new conceptual model for evidence-informed policy formulation and implementation.

Methods

Two concurrent national policies were studied during the entire policy process using a longitudinal, comparative case study approach. Data was collected through interviews, observations, and documents. A Conceptual Model for Evidence-Informed Policy Formulation and Implementation was developed based on prior frameworks for evidence-informed policymaking and policy dissemination and implementation. The conceptual model was used to organize and analyze the data.

Results

The policies differed regarding the use of evidence in the policy formulation and the extent to which the policy formulation and implementation phases overlapped. Similarities between the cases were an emphasis on capacity assessment, modified activities based on the assessment, and a highly active implementation approach relying on networks of stakeholders. The Conceptual Model for Evidence-Informed Policy Formulation and Implementation was empirically useful to organize the data.

Conclusions

The policy actors’ roles and functions were found to have a great influence on the choices of strategies and collaborators in all policy phases. The Conceptual Model for Evidence-Informed Policy Formulation and Implementation was found to be useful. However, it provided insufficient guidance for analyzing actors involved in the policy process, capacity-building strategies, and overlapping policy phases. A revised version of the model that includes these aspects is suggested.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-015-0359-1) contains supplementary material, which is available to authorized users.

The gap between available knowledge and its use in clinical psychiatry

OBJECTIVE

The time span between knowledge becoming available and its integration into daily clinical routine is lengthy. This phenomenon is explored in this study.

METHOD

We used the outcomes of our activities for investigating and strengthening the research-based activities to improve physical health in the routines of clinical psychiatric wards as examples for our analyses.

RESULTS

The time span between new knowledge becoming available and its implementation into general clinical treatment is very long. However, a shortening of this time span is seen through active leadership backup and clinical research experience among psychiatrists and staff in the wards. In particular, the involvement of medical students interested in clinical research activities seems to have a positive impact.

CONCLUSION

Academia needs to be re-implemented into clinical psychiatry. Staff with research experience is needed in all professions to increase evidence-based practice. Leaders must take responsibility for implementing new knowledge into the routines of the department and must support staff in these activities on a daily basis.

The DECIDE evidence to recommendation framework adapted to the public health field in Sweden

Organizations worldwide compile results from scientific studies, and grade the evidence of interventions, in order to assist policy makers. However, quality of evidence alone is seldom sufficient to make a recommendation. The Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) framework aims to facilitate decision making and to improve dissemination and implementation of recommendations in the healthcare and public health sector. The aim of this study was to investigate whether the DECIDE framework is applicable in the public health field in Sweden. The DECIDE framework was presented and discussed in interviews with stakeholders and governmental organizations and tested in panels. Content analyses were performed. In general, the informants were positive to the DECIDE framework. However, two questions, the first regarding individual autonomy and the second regarding method sustainability, were by the stakeholders felt to be missing in the framework. The importance of the composition of the DECIDE stakeholder panel was lifted by the informants, as was the significant role of the chair. Further, the informants raised concerns about the general lack of research evidence based on RCT design regarding universal methods in the public health sector. Finally, the local, regional and national levels' responsibility for dissemination and implementation of recommendations were lifted by the informants. The DECIDE framework might be useful as a tool for dissemination and implementation of recommendations in the public health field in Sweden. Important questions for further research are whether these findings are suitable for other public health topics and in other public health settings.

The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy

The recent proliferation of strategies designed to increase the use of research in health policy (knowledge exchange) demands better application of contemporary conceptual understandings of how research shapes policy. Predictive models, or action frameworks, are needed to organise existing knowledge and enable a more systematic approach to the selection and testing of intervention strategies. Useful action frameworks need to meet four criteria: have a clearly articulated purpose; be informed by existing knowledge; provide an organising structure to build new knowledge; and be capable of guiding the development and testing of interventions. This paper describes the development of the SPIRIT Action Framework. A literature search and interviews with policy makers identified modifiable factors likely to influence the use of research in policy. An iterative process was used to combine these factors into a pragmatic tool which meets the four criteria. The SPIRIT Action Framework can guide conceptually-informed practical decisions in the selection and testing of interventions to increase the use of research in policy. The SPIRIT Action Framework hypothesises that a catalyst is required for the use of research, the response to which is determined by the capacity of the organisation to engage with research. Where there is sufficient capacity, a series of research engagement actions might occur that facilitate research use. These hypotheses are being tested in ongoing empirical work.

Using conjoint analysis to develop a system to score research engagement actions by health decision makers

Background

Effective use of research to inform policymaking can be strengthened by policymakers undertaking various research engagement actions (e.g., accessing, appraising, and applying research). Consequently, we developed a thorough measurement and scoring tool to assess whether and how policymakers undertook research engagement actions in the development of a policy document. This scoring tool breaks down each research engagement action into its key ‘subactions’ like a checklist. The primary aim was to develop the scoring tool further so that it assigned appropriate scores to each subaction based on its effectiveness for achieving evidence-informed policymaking. To establish the relative effectiveness of these subactions, we conducted a conjoint analysis, which was used to elicit the opinions and preferences of knowledge translation experts.

Method

Fifty-four knowledge translation experts were recruited to undertake six choice surveys. Respondents were exposed to combinations of research engagement subactions called ‘profiles’, and rated on a 1–9 scale whether each profile represented a limited (1–3), moderate (4–6), or extensive (7–9) example of each research engagement action. Generalised estimating equations were used to analyse respondents’ choice data, where a utility coefficient was calculated for each subaction. A large utility coefficient indicates that a subaction was influential in guiding experts’ ratings of extensive engagement with research.

Results

The calculated utilities were used as the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of engaging with research: searching academic literature databases, obtaining systematic reviews and peer-reviewed research, appraising relevance by verifying its applicability to the policy context, appraising quality by evaluating the validity of the method and conclusions, engaging in thorough collaborations with researchers, and undertaking formal research projects to inform the policy in question.

Conclusions

We have generated an empirically-derived and context-sensitive method of measuring and scoring the extent to which policymakers engaged with research to inform policy development. The scoring system can be used by organisations to quantify staff research engagement actions and thus provide them with insights into what types of training, systems, and tools might improve their staff’s research use capacity.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-015-0013-z) contains supplementary material, which is available to authorized users.