Using conjoint analysis to develop a system to score research engagement actions by health decision makers

From bench to policy: a critical analysis of models for evidence-informed policymaking in healthcare

Background

The use of research evidence in policy making is a complex and challenging process that has a long history in various fields, especially in healthcare. Different terms and concepts have been used to describe the relationship between research and policy, but they often lack clarity and consensus. To address this gap, several strategies and models have been proposed to facilitate evidence informed policy making and to identify the key factors and mechanisms involved. This study aims to critically review the existing models of evidence informed policy making (EIPM) in healthcare and to assess their strengths and limitations.

Method

A systematic search and review conducted to identify and critically assess EIPM models in healthcare. We searched PubMed, Web of Science and Scopus databases as major electronic databases and applied predefined inclusion criteria to select the models. We also checked the citations of the included models to find other scholars' perspectives. Each model was described and critiqued each model in detail and discussed their features and limitations.

Result

Nine models of EIPM in healthcare were identified. While models had some strengths in comprehension, flexibility and theoretical foundations, analysis also identified limitations including: presupposing rational policymaking; lacking alternatives for time-sensitive situations; not capturing policy complexity; neglecting unintended effects; limited context considerations; inadequate complexity concepts; limited collaboration guidance; and unspecified evidence adaptations.

Conclusion

The reviewed models provide useful frameworks for EIPM but need further improvement to address their limitations. Concepts from sociology of knowledge, change theory and complexity science can enrich the models. Future EIPM models should better account for the complexity of research-policy relationships and provide tailored strategies based on the policy context.

A Virtual 3D Dynamic Model of Caries Lesion Progression as a Learning Object for Caries Detection Training and Teaching: Video Development Study

BACKGROUND

In the last decade, 3D virtual models have been used for educational purposes in the health sciences, specifically for teaching human anatomy and pathology. These models provide an opportunity to didactically visualize key spatial relations that can be poorly understood when taught by traditional educational approaches. Caries lesion detection is a crucial process in dentistry that has been reported to be difficult to learn. One especially difficult aspect is linking clinical characteristics of the different severity stages with their histological features, which is fundamental for treatment decision-making.

OBJECTIVE

This project was designed to develop a virtual 3D digital model of caries lesion formation and progression to aid the detection of lesions at different severity stages as a potential complement to traditional lectures.

METHODS

Pedagogical planning, including identification of objectives, exploration of the degree of difficulty of caries diagnosis-associated topics perceived by dental students and lecturers, review of the literature regarding key concepts, and consultation of experts, was performed prior to constructing the model. An educational script strategy was created based on the topics to be addressed (dental tissues, biofilm stagnation areas, the demineralization process, caries lesion progression on occlusal surfaces, clinical characteristics related to different stages of caries progression, and histological correlations). Virtual 3D models were developed using the Virtual Man Project and refined using multiple 3D software applications. In the next phase, computer graphic modelling and previsualization were executed. After that, the video was revised and edited based on suggestions. Finally, explanatory subtitles were generated, the models were textured and rendered, and voiceovers in 3 languages were implemented.

RESULTS

We developed a 6-minute virtual 3D dynamic video in 3 languages (English, Spanish, and Brazilian Portuguese) intended for dentists and dental students to support teaching and learning of caries lesion detection. The videos were made available on YouTube; to date, they have received more than 100,000 views.

CONCLUSIONS

Complementary pedagogical tools are valuable to support cariology education. This tool will be further tested in terms of utility and usability as well as user satisfaction in achieving the proposed objectives in specific contexts.

Experimental design issues in choice-based conjoint applied to patient choice in healthcare

Choice-based conjoint (CBC) is used to understand how individuals develop preferences for decision alternatives. When decision alternatives can be described in terms of attributes, researchers want to determine the value respondents attach to various attribute levels. Popular in psychology, marketing, economics and other areas, CBC is now finding applications in healthcare to understand patient choice in healthcare policy, drug development, doctor-patient communications, etc. However, a lack of standard methodologies has served as a barrier to its use in healthcare. Therefore, there is a need to identify good research practices for CBC in healthcare. We review recent advances in CBC such as Pareto optimal choice sets, information per profile and reducing choice set sizes, as applied to patient choice.

How was research engaged with and used in the development of 131 policy documents? Findings and measurement implications from a mixed methods study

Background

Much has been written about the use of evidence in policy; however, there is still little known about whether and how research is engaged with and used in policy development or the impact of reported barriers and facilitators. This paper aims to (1) describe the characteristics of 131 policy documents, (2) describe the ways in which research was engaged with (e.g. was searched for, appraised or generated) and used (e.g. to clarify understanding, persuade others or inform a policy) in the development of these policy documents, and (3) identify the most commonly reported barriers and facilitators and describe their association with research engagement and use.

Methods

Six health policy and program development agencies based in Sydney, Australia, contributed four recently finalised policy documents for consideration over six measurement periods. Structured, qualitative interviews were conducted with the policymakers most heavily involved in developing each of the 131 policy documents. Interviews covered whether and how research was engaged with and used in the development of the policy product and any barriers or facilitators related to this. Interviews were scored using the empirically validated SAGE tool and thematically analysed. Descriptive statistics were calculated for all key variables and comparisons made between agencies. Multiple regression analyses were used to estimate the impact of specific barriers and facilitators on research engagement and use.

Results

Our data shows large variations between policy agencies in the types of policy documents produced and the characteristics of these documents. Nevertheless, research engagement and use was generally moderate across agencies. A number of barriers and facilitators to research use were identified. No barriers were significantly associated with any aspects of research engagement or use. Access to consultants and relationships with researchers were both associated with increased research engagement but not use. Thus, access to consultants and relationships with researchers may increase the extent and quality of the evidence considered in policy development.

Conclusions

Our findings suggest that those wishing to develop interventions and programs designed to improve the use of evidence in policy agencies might usefully target increasing access to consultants and relationships with researchers in order to increase the extent and quality of the research considered, but that a greater consideration of context might be required to develop strategies to increase evidence use.

Electronic supplementary material

The online version of this article (10.1186/s13012-019-0886-2) contains supplementary material, which is available to authorized users.

Increasing the capacity of policy agencies to use research findings: a stepped-wedge trial

Background

This paper describes the trial of a novel intervention, Supporting Policy In health with evidence from Research: an Intervention Trial (SPIRIT). It examines (1) the feasibility of delivering this kind of programme in practice; (2) its acceptability to participants; (3) the impact of the programme on the capacity of policy agencies to engage with research; and (4) the engagement with and use of research by policy agencies.

Methods

SPIRIT was a multifaceted, highly tailored, stepped-wedge, cluster-randomised, trial involving six health policy agencies in Sydney, Australia. Agencies were randomly allocated to one of three start dates to receive the 1-year intervention programme. SPIRIT included audit, feedback and goal setting; a leadership programme; staff training; the opportunity to test systems to facilitate research use in policies; and exchange with researchers. Outcome measures were collected at each agency every 6 months for 30 months.

Results

Participation in SPIRIT was associated with significant increases in research use capacity at staff and agency levels. Staff reported increased confidence in research use skills, and agency leaders reported more extensive systems and structures in place to support research use. Self-report data suggested there was also an increase in tactical research use among agency staff. Given the relatively small numbers of participating agencies and the complexity of their contexts, findings suggest it is possible to effect change in the way policy agencies approach the use of research. This is supported by the responses on the other trial measures; while these were not statistically significant, on 18 of the 20 different measures used, the changes observed were consistent with the hypothesised intervention effect (that is, positive impacts).

Conclusions

As an early test of an innovative approach, SPIRIT has demonstrated that it is possible to increase research engagement and use in policy agencies. While more work is needed to establish the replicability and generalisability of these findings, this trial suggests that building staff skills and organisational structures may be effective in increasing evidence use.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0408-8) contains supplementary material, which is available to authorized users.

Audience segmentation to disseminate behavioral health evidence to legislators: an empirical clustering analysis

Background

Elected officials (e.g., legislators) are an important but understudied population in dissemination research. Audience segmentation is essential in developing dissemination strategies that are tailored for legislators with different characteristics, but sophisticated audience segmentation analyses have not been conducted with this population. An empirical clustering audience segmentation study was conducted to (1) identify behavioral health (i.e., mental health and substance abuse) audience segments among US state legislators, (2) identify legislator characteristics that are predictive of segment membership, and (3) determine whether segment membership is predictive of support for state behavioral health parity laws.

Methods

Latent class analysis (LCA) was used. Data were from a multi-modal (post-mail, e-mail, telephone) survey of state legislators fielded in 2017 (N = 475). Nine variables were included in the LCA (e.g., perceptions of behavioral health treatment effectiveness, mental illness stigma). Binary logistic regression tested associations between legislator characteristics (e.g., political party, gender, ideology) and segment membership. Multi-level logistic regression assessed the predictive validity of segment membership on support for parity laws. A name was developed for each segment that captured its most salient features.

Results

Three audience segments were identified. Budget-oriented skeptics with stigma (47% of legislators) had the least faith in behavioral health treatment effectiveness, had the most mental illness stigma, and were most influenced by budget impact. This segment was predominantly male, Republican, and ideologically conservative. Action-oriented supporters (24%) were most likely to have introduced a behavioral health bill, most likely to identify behavioral health issues as policy priorities, and most influenced by research evidence. This was the most politically and ideologically diverse segment. Passive supporters (29%) had the greatest faith in treatment effectiveness and the least stigma, but were also least likely to have introduced a behavioral health bill. Segment membership was a stronger predictor of support for parity laws than almost all other legislator characteristics.

Conclusions

State legislators are a heterogeneous audience when it comes to behavioral health. There is a need to develop and test behavioral health evidence dissemination strategies that are tailored for legislators in different audience segments. Empirical clustering approaches to audience segmentation are a potentially valuable tool for dissemination science.

Electronic supplementary material

The online version of this article (10.1186/s13012-018-0816-8) contains supplementary material, which is available to authorized users.

A systematic review of frameworks for the interrelationships of mental health evidence and policy in low- and middle-income countries

BACKGROUND

The interrelationships between research evidence and policy-making are complex. Different theoretical frameworks exist to explain general evidence-policy interactions. One largely unexplored element of these interrelationships is how evidence interrelates with, and influences, policy/political agenda-setting. This review aims to identify the elements and processes of theories, frameworks and models on interrelationships of research evidence and health policy-making, with a focus on actionability and agenda-setting in the context of mental health in low- and middle-income countries (LMICs).

METHODS

A systematic review of theories was conducted based on the BeHeMOTh search method, using a tested and refined search strategy. Nine electronic databases and other relevant sources were searched for peer-reviewed and grey literature. Two reviewers screened the abstracts, reviewed full-text articles, extracted data and performed quality assessments. Analysis was based on a thematic analysis. The included papers had to present an actionable theoretical framework/model on evidence and policy interrelationships, such as knowledge translation or evidence-based policy, specifically target the agenda-setting process, focus on mental health, be from LMICs and published in English.

RESULTS

From 236 publications included in the full text analysis, no studies fully complied with our inclusion criteria. Widening the focus by leaving out 'agenda-setting', we included ten studies, four of which had unique conceptual frameworks focusing on mental health and LMICs but not agenda-setting. The four analysed frameworks confirmed research gaps from LMICs and mental health, and a lack of focus on agenda-setting. Frameworks and models from other health and policy areas provide interesting conceptual approaches and lessons with regards to agenda-setting.

CONCLUSION

Our systematic review identified frameworks on evidence and policy interrelations that differ in their elements and processes. No framework fulfilled all inclusion criteria. Four actionable frameworks are applicable to mental health and LMICs, but none specifically target agenda-setting. We have identified agenda-setting as a research theory gap in the context of mental health knowledge translation in LMICs. Frameworks from other health/policy areas could offer lessons on agenda-setting and new approaches for creating policy impact for mental health and to tackle the translational gap in LMICs.

Organisational capacity and its relationship to research use in six Australian health policy agencies

There are calls for policymakers to make greater use of research when formulating policies. Therefore, it is important that policy organisations have a range of tools and systems to support their staff in using research in their work. The aim of the present study was to measure the extent to which a range of tools and systems to support research use were available within six Australian agencies with a role in health policy, and examine whether this was related to the extent of engagement with, and use of research in policymaking by their staff. The presence of relevant systems and tools was assessed via a structured interview called ORACLe which is conducted with a senior executive from the agency. To measure research use, four policymakers from each agency undertook a structured interview called SAGE, which assesses and scores the extent to which policymakers engaged with (i.e., searched for, appraised, and generated) research, and used research in the development of a specific policy document. The results showed that all agencies had at least a moderate range of tools and systems in place, in particular policy development processes; resources to access and use research (such as journals, databases, libraries, and access to research experts); processes to generate new research; and mechanisms to establish relationships with researchers. Agencies were less likely, however, to provide research training for staff and leaders, or to have evidence-based processes for evaluating existing policies. For the majority of agencies, the availability of tools and systems was related to the extent to which policymakers engaged with, and used research when developing policy documents. However, some agencies did not display this relationship, suggesting that other factors, namely the organisation's culture towards research use, must also be considered.

Development of an index to define overall disease severity in IBD

BACKGROUND AND AIM

Disease activity for Crohn's disease (CD) and UC is typically defined based on symptoms at a moment in time, and ignores the long-term burden of disease. The aims of this study were to select the attributes determining overall disease severity, to rank the importance of and to score these individual attributes for both CD and UC.

METHODS

Using a modified Delphi panel, 14 members of the International Organization for the Study of Inflammatory Bowel Diseases (IOIBD) selected the most important attributes related to IBD. Eighteen IOIBD members then completed a statistical exercise (conjoint analysis) to create a relative ranking of these attributes. Adjusted utilities were developed by creating proportions for each level within an attribute.

RESULTS

For CD, 15.8% of overall disease severity was attributed to the presence of mucosal lesions, 10.9% to history of a fistula, 9.7% to history of abscess and 7.4% to history of intestinal resection. For UC, 18.1% of overall disease severity was attributed to mucosal lesions, followed by 14.0% for impact on daily activities, 11.2% C reactive protein and 10.1% for prior experience with biologics. Overall disease severity indices were created on a 100-point scale by applying each attribute's average importance to the adjusted utilities.

CONCLUSIONS

Based on specialist opinion, overall CD severity was associated more with intestinal damage, in contrast to overall UC disease severity, which was more dependent on symptoms and impact on daily life. Once validated, disease severity indices may provide a useful tool for consistent assessment of overall disease severity in patients with IBD.

Promoting evidence informed policymaking for maternal and child health in Nigeria: lessons from a knowledge translation workshop

Background: Knowledge translation (KT) is a process that ensures that research evidence gets translated into policy and practice. In Nigeria, reports indicate that research evidence rarely gets into policymaking process. A major factor responsible for this is lack of KT capacity enhancement mechanisms. The objective of this study was to improve KT competence of an implementation research team (IRT), policymakers and stakeholders in maternal and child health to enhance evidence-informed policymaking.

Methods: This study employed a “before and after” design, modified as an intervention study. The study was conducted in Bauchi, north-eastern Nigeria. A three-day KT training workshop was organized and 15 modules were covered including integrated and end-of-grant KT; KT models, measures, tools and strategies; priority setting; managing political interference; advocacy and consensus building/negotiations; inter-sectoral collaboration; policy analysis, contextualization and legislation. A 4-point Likert scale pre-/post-workshop questionnaires were administerd to evaluate the impact of the training, it was designed in terms of extent of adequacy; with “grossly inadequate” representing 1 point, and “very adequate” representing 4 points.

Results: A total of 45 participants attended the workshop. There was a noteworthy improvement in the participants’ understanding of KT processes and strategies. The range of the preworkshop mean of participants knowledge of modules taught was from 2.04-2.94, the range for the postworkshop mean was from 3.10–3.70 on the 4-point Likert scale. The range of percentage increase in mean for participants’ knowledge at the end of the workshop was from 13.3%–55.2%.

Conclusion: The outcome of this study suggests that using a KT capacity building programme e.g., workshop, health researchers, policymakers and other stakeholders can acquire capacity and skill that will facilitate evidence-to-policy link.

Preliminary testing of the reliability and feasibility of SAGE: a system to measure and score engagement with and use of research in health policies and programs

Background

Few measures of research use in health policymaking are available, and the reliability of such measures has yet to be evaluated. A new measure called the Staff Assessment of Engagement with Evidence (SAGE) incorporates an interview that explores policymakers’ research use within discrete policy documents and a scoring tool that quantifies the extent of policymakers’ research use based on the interview transcript and analysis of the policy document itself. We aimed to conduct a preliminary investigation of the usability, sensitivity, and reliability of the scoring tool in measuring research use by policymakers.

Methods

Nine experts in health policy research and two independent coders were recruited. Each expert used the scoring tool to rate a random selection of 20 interview transcripts, and each independent coder rated 60 transcripts. The distribution of scores among experts was examined, and then, interrater reliability was tested within and between the experts and independent coders. Average- and single-measure reliability coefficients were computed for each SAGE subscales.

Results

Experts’ scores ranged from the limited to extensive scoring bracket for all subscales. Experts as a group also exhibited at least a fair level of interrater agreement across all subscales. Single-measure reliability was at least fair except for three subscales: Relevance Appraisal, Conceptual Use, and Instrumental Use. Average- and single-measure reliability among independent coders was good to excellent for all subscales. Finally, reliability between experts and independent coders was fair to excellent for all subscales.

Conclusions

Among experts, the scoring tool was comprehensible, usable, and sensitive to discriminate between documents with varying degrees of research use. Secondly, the scoring tool yielded scores with good reliability among the independent coders. There was greater variability among experts, although as a group, the tool was fairly reliable. The alignment between experts’ and independent coders’ ratings indicates that the independent coders were scoring in a manner comparable to health policy research experts. If the present findings are replicated in a larger sample, end users (e.g. policy agency staff) could potentially be trained to use SAGE to reliably score research use within their agencies, which would provide a cost-effective and time-efficient approach to utilising this measure in practice.

Electronic supplementary material

The online version of this article (10.1186/s13012-017-0676-7) contains supplementary material, which is available to authorized users.

Policymakers' experience of a capacity-building intervention designed to increase their use of research: a realist process evaluation

BACKGROUND

An intervention's success depends on how participants interact with it in local settings. Process evaluation examines these interactions, indicating why an intervention was or was not effective, and how it (and similar interventions) can be improved for better contextual fit. This is particularly important for innovative trials like Supporting Policy In health with Research: an Intervention Trial (SPIRIT), where causal mechanisms are poorly understood. SPIRIT was testing a multi-component intervention designed to increase the capacity of health policymakers to use research.

METHODS

Our mixed-methods process evaluation sought to explain variation in observed process effects across the six agencies that participated in SPIRIT. Data collection included observations of intervention workshops (n = 59), purposively sampled interviews (n = 76) and participant feedback forms (n = 553). Using a realist approach, data was coded for context-mechanism-process effect configurations (retroductive analysis) by two authors.

RESULTS

Intervention workshops were very well received. There was greater variation of views regarding other aspects of SPIRIT such as data collection, communication and the intervention's overall value. We identified nine inter-related mechanisms that were crucial for engaging participants in these policy settings: (1) Accepting the premise (agreeing with the study's assumptions); (2) Self-determination (participative choice); (3) The Value Proposition (seeing potential gain); (4) 'Getting good stuff' (identifying useful ideas, resources or connections); (5) Self-efficacy (believing 'we can do this!'); (6) Respect (feeling that SPIRIT understands and values one's work); (7) Confidence (believing in the study's integrity and validity); (8) Persuasive leadership (authentic and compelling advocacy from leaders); and (9) Strategic insider facilitation (local translation and mediation). These findings were used to develop tentative explanatory propositions and to revise the programme theory.

CONCLUSION

This paper describes how SPIRIT functioned in six policy agencies, including why strategies that worked well in one site were less effective in others. Findings indicate a complex interaction between participants' perception of the intervention, shifting contextual factors, and the form that the intervention took in each site. Our propositions provide transferable lessons about contextualised areas of strength and weakness that may be useful in the development and implementation of similar studies.

The development of ORACLe: a measure of an organisation's capacity to engage in evidence-informed health policy

Background

Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations’ research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations’ capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership).

Method

We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe.

Results

The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4).

Conclusion

We utilised qualitative and quantitative methods to develop a system to assess and score organisations’ capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-015-0069-9) contains supplementary material, which is available to authorized users.

Using conjoint analysis to develop a system of scoring policymakers' use of research in policy and program development

Background

The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers’ research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts.

Methods

Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1–3), moderate (4–6), or extensive (7–9) example of research use. Generalised Estimating Equations were used to analyse respondents’ choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts’ ratings of extensive research use.

Results

Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker’s organisation.

Conclusions

We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-015-0022-y) contains supplementary material, which is available to authorized users.