A systematic review of coverage decision-making on health technologies-evidence from the real world

The reimbursement process in three national healthcare systems: variation in time to reimbursement of pembrolizumab for metastatic non-small cell lung cancer

In this article, we focus on the reimbursement process, and as an example, characterize the time to reimbursement of pembrolizumab, a PD-1 immune checkpoint inhibitor for treatment of metastatic NSCLC from publicly available websites, in three different healthcare systems: The National Institute for Health and Care Excellence (NICE) in the UK, the Pharmaceutical Benefits Advisory Committee (PBAC) in Australia, and the National Advisory Committee for the Basket of Health Services in Israel, all who have publicly funded health systems which include drug coverage. Our study found that there are substantial differences in time to reimbursement of pembrolizumab for the same conditions in different countries, with NICE and The National Advisory Committee for the Basket of Health Services in Israel approving one condition at the same time, Israel approving two conditions earlier than NICE, and PBAC lagging behind for every condition. These differences could be due to the differences in health policy systems and the many factors that affect reimbursement. Comparing the reimbursement process between different countries can highlight the challenges facing their health systems in early adoption of new treatments.

A novel drug selection decision support model based on real-world medical data by the hybrid entropic weight TOPSIS method

BACKGROUND

The medicine selection method is a critical and challenging issue in medical insurance decision-making.

OBJECTIVES

This study proposed a real-world data-based multi-criteria decision analysis (MCDA) model with a hybrid entropic weight Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) algorithms to select satisfactory drugs.

METHODS

The evaluation index includes two levels: primary criteria and sub-criteria. Firstly, we proposed six primary criteria to form the value health framework. The primary criteria's weights were derived from the policymakers' questionnaire. Meanwhile, clinically relevant sub-criteria were derived from high-quality (screened by GRADE scores) clinical-research literature. Their weights are determined by the entropy weight (EW) algorithm. Secondly, we split the primary criteria into six mini-EW-TOPSIS models. Then, we obtained six ideal closeness degree scores (ICDS) for each candidate drug. Thirdly, we get the total utility score by linear weighting the ICDS. The higher the utility score, the higher the ranking.

RESULTS

A national multicenter real-world case study of the ranking of four generic antibiotics validated the proposed model. This model is verified by comparative experiments and sensitivity analysis. The whole ranking model was consistent and reliable. Based on these results, medical policymakers can intuitively and easily understand the characteristics of each drug to facilitate follow-up drug policy-making.

CONCLUSION

The ranking algorithm combines the objective characteristics of medicine and policy makers' opinions, which can improve the applicability of the results. This model can help decision-makers, clinicians, and related researchers better understand the drug assessment process.

Health Technology Assessment Development in Vietnam: A Qualitative Study of Current Progress, Barriers, Facilitators, and Future Strategies

INTRODUCTION

To make more efficient use of limited resources, Vietnam incorporated health technology assessment (HTA) into the decision-making process for the health insurance benefit package in 2014. We evaluated progress in HTA institutionalization in Vietnam based on the theoretical framework developed by the National Institute for Health and Care Excellence and the Health Intervention and Technology Assessment Program, identified negative and conducive factors for HTA development, and finally suggested policy recommendations that fit the Vietnamese context.

METHODS

Semi-structured in-depth qualitative interviews were conducted between December 2017 and March and April 2018 with a purposive sample of 24 stakeholders involved in decision-making for health insurance reimbursement. We employed thematic analysis to examine themes within the data.

RESULTS

Despite a variety of activities (e.g., training and advising/mentoring) and a substantial level of output (e.g., policy statements, focal points assigned, and case studies/demonstration projects), Vietnam has not yet reached the policy decision stage based on HTA with scientific integrity and active stakeholder participation. Most respondents, except some clinicians, supported the use of HTA. The lack of capacity of human resources in the government sector and academia, the limited data infrastructure, the absence of guidelines, the government's interest in immediate budget-saving, and public resistance were identified as barriers to the advancement of HTA.

CONCLUSIONS

A structured data repository, guidelines based on the Vietnamese context for both policy decision-making at the central level and daily clinical decision-making at the micro-level, and integration of a participatory process into HTA are suggested as priorities for HTA institutionalization in Vietnam.

Guidelines that use the GRADE approach often fail to provide complete economic information for recommendations: A systematic survey

OBJECTIVE

Little is known about how developers and panel members report cost and cost effectiveness considerations in GRADE guideline Evidence-to-Decision (EtD) frameworks. A systematic survey was conducted to explore approaches and factors contributing to variability in economic information reporting.

STUDY DESIGN AND SETTING

Guideline organization websites were systematically searched to create a convenience sample of guidelines. Reviewers screened published EtD frameworks and generated frequencies of reporting approaches. We used thematic analysis to summarize factors related to variability of economic information reporting.

RESULTS

We included 142 guidelines. The overall rate of reporting economic information was high (91%); however, there was variability across completion of predefined EtD Likert-type judgments (70%), noting information as not identified across EtD framework domains (57%), and providing remarks to justify recommendations (38%). Six themes contributing to variability emerged, related to: intervention, population, payor, provider, healthcare resource use, and economic model building factors. Only 2 guidelines performed a GRADE certainty appraisal of economic outcomes.

CONCLUSION

Completing predefined EtD Likert-type judgments, specifically reporting a literature review approach, study selection criteria and economic model building limitations, as well as linking these to recommendation justification remarks are potential areas for improved use, adoption and adaptation of recommendation, and transparency of GRADE EtD frameworks.

Value judgment of new medical treatments: Societal and patient perspectives to inform priority setting in The Netherlands

BACKGROUND

In many countries, medical interventions are reimbursed on the basis of recommendations made by advisory boards and committees that apply multiple criteria in their assessment procedures. Given the diversity of these criteria, it is difficult to find common ground to determine what information is required for setting priorities.

OBJECTIVE

To investigate whether society and patients share the same interests and views concerning healthcare priorities.

METHODS

We applied a framework of discrete choice models in which respondents were presented with judgmental tasks to elicit their preferences. They were asked to choose between two hypothetical scenarios of patients receiving a new treatment. The scenarios graphically presented treatment outcomes and patient characteristics. Responses were collected through an online survey administered among respondents from the general population (N = 1,253) and patients (N = 1,389) and were analyzed using conditional logit and mixed logit models.

RESULTS

The respondents' preferences regarding new medical treatments revealed that they attached the most relative importance to additional survival years, age at treatment, initial health condition, and the cause of disease. Minor differences in the relative importance assigned to three criteria: age at treatment, initial health, and cause of disease were found between the general population and patient samples. Health scenarios in which patients had higher initial health-related quality of life (i.e., a lower burden of disease) were favored over those in which patients' initial health-related quality of life was lower.

CONCLUSIONS

Overall, respondents within the general population expressed preferences that were similar to those of the patients. Therefore, priority-setting studies that are based on the perspectives of the general population may be useful for informing decisions on reimbursement and other types of priority-setting processes in health care. Incorporating the preferences of the general population may simultaneously increase public acceptance of these decisions.

Disagreement on cancer drug decisions in Europe

OBJECTIVES

Despite the efforts of the European Union (EU) to promote voluntary cooperation among Health Technology Assessment (HTA) agencies, different reimbursement decisions for the same drug are made across European countries. The aim of this paper is to compare the agreement of cancer drug reimbursement decisions using inter-rater reliability measures.

METHODS

This study is based on primary data on 161 cancer drug reimbursement decisions from nine European countries from 2002 to 2014. To achieve our goal, we use two measures to analyze agreement, in other words, congruency: (i) percentage of agreement and (ii) the κ score.

RESULTS

One main conclusion can be drawn from the analysis. There is a weak to medium agreement among cancer drug decisions in the European countries analyzed (based on the percentage of agreement and the κ score). England and Scotland show the highest consistency between the two measures, showing a medium agreement. These results are in line with previous literature on the congruency of HTA decisions.

CONCLUSIONS

This paper contributes to the HTA literature, by highlighting the extent of weak to medium agreement among cancer decisions in Europe.

[GRADE: Evidence to Decision (EtD) framework for coverage decisions]

BACKGROUND

Coverage decisions are decisions by third party payers about whether and how much to pay for technologies or services, and under what conditions. Given their complexity, a systematic and transparent approach is needed. The DECIDE project, a GRADE working group initiative funded by the European Union, has developed GRADE Evidence to Decision (EtD) frameworks for different types of decisions, including coverage ones.

METHODS

We used an iterative approach, including brainstorming to generate ideas, consultation with stakeholders, user testing, and pilot testing of the framework.

RESULTS

The general structure of the EtD includes formulation of the question, an assessment using 12 criteria, and conclusions. Criteria that are relevant for coverage decisions are similar to those for clinical recommendations from a population perspective. Important differences between the two include the decision-making processes, accountability, and the nature of the judgments that need to be made for some criteria. Although cost-effectiveness is a key consideration when making coverage decisions, it may not be the determining factor. Strength of recommendation is not directly linked to the type of coverage decisions, but when there are important uncertainties, it may be possible to cover an intervention for a subgroup, in the context of research, with price negotiation, or with restrictions.

CONCLUSION

The EtD provides a systematic and transparent approach for making coverage decisions. It helps ensure consideration of key criteria that determine whether a technology or service should be covered and that judgments are informed by the best available evidence.

Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making

Background

Decision-making in public health and health policy is complex and requires careful deliberation of many and sometimes conflicting normative and technical criteria. Several approaches and tools, such as multi-criteria decision analysis, health technology assessments and evidence-to-decision (EtD) frameworks, have been proposed to guide decision-makers in selecting the criteria most relevant and appropriate for a transparent decision-making process. This study forms part of the development of the WHO-INTEGRATE EtD framework, a framework rooted in global health norms and values as reflected in key documents of the World Health Organization and the United Nations system. The objective of this study was to provide a comprehensive overview of criteria used in or proposed for real-world decision-making processes, including guideline development, health technology assessment, resource allocation and others.

Methods

We conducted an overview of systematic reviews through a combination of systematic literature searches and extensive reference searches. Systematic reviews reporting criteria used for real-world health decision-making by governmental or non-governmental organization on a supranational, national, or programme level were included and their quality assessed through a bespoke critical appraisal tool. The criteria reported in the reviews were extracted, de-duplicated and sorted into first-level (i.e. criteria), second-level (i.e. sub-criteria) and third-level (i.e. decision aspects) categories. First-level categories were developed a priori using a normative approach; second- and third-level categories were developed inductively.

Results

We included 36 systematic reviews providing criteria, of which one met all and another eleven met at least five of the items of our critical appraisal tool. The criteria were subsumed into 8 criteria, 45 sub-criteria and 200 decision aspects. The first-level of the category system comprised the following seven substantive criteria: “Health-related balance of benefits and harms”; “Human and individual rights”; “Acceptability considerations”; “Societal considerations”; “Considerations of equity, equality and fairness”; “Cost and financial considerations”; and “Feasibility and health system considerations”. In addition, we identified an eight criterion “Evidence”.

Conclusion

This overview of systematic reviews provides a comprehensive overview of criteria used or suggested for real-world health decision-making. It also discusses key challenges in the selection of the most appropriate criteria and in seeking to implement a fair decision-making process.

Why do health technology assessment drug reimbursement recommendations differ between countries? A parallel convergent mixed methods study

Using quantitative and qualitative research designs, respectively, two studies investigated why countries make different health technology assessment (HTA) drug reimbursement recommendations. Building on these, the objective of this study was to (a) develop a conceptual framework integrating the factors explaining these decisions, (b) explore their relationship and (c) assess if they are congruent, complementary or discrepant. A parallel convergent mixed methods design was used. Countries included in both previous studies were selected (England, Sweden, Scotland and France). A conceptual framework that integrated and organised the factors explaining the decisions from the two studies was developed. Relationships between factors were explored and illustrated through case studies. The framework distinguishes macro-level factors from micro-level ones. Only two of the factors common to both studies were congruent, while two others reached discrepant conclusions (stakeholder input and external review of the evidence processes). The remaining factors identified within one or both studies were complementary. Bringing together these findings contributed to generating a more complete picture of why countries make different HTA recommendations. Results were mostly complementary, explaining and enhancing each other. We conclude that differences often result from a combination of factors, with an important component relating to what occurs during the deliberative process.

Paradigms in operation: explaining pharmaceutical benefit assessment outcomes in England and Germany

Health technology assessments (HTAs) are used as a policy tool to appraise the clinical value, or cost effectiveness, of new medicines to inform reimbursement decisions in health care. As HTA organisations have been established in different countries, it has become clear that the outcomes of medicine appraisals can vary from country to country, even though the same scientific evidence in the form of randomised controlled trials is available. The extant literature explains such variations with reference to institutional variables and administrative rules. However, little research has been conducted to advance the theoretical understanding of how variations in HTA outcomes might be explained. This paper compares cases of HTA in England and Germany using insights from Kuhn (1962, The Structure of Scientific Revolutions, 2nd edn. Chicago: The University of Chicago Press) and Hall (1993, Policy paradigms, social learning, and the state: the case of economic policymaking in Britain. Comparative Politics 25, 275-296) to demonstrate how policy paradigms can explain the outcomes of HTA processes. The paper finds that HTA outcomes are influenced by a combination of logical issues that require reasoning within a paradigm, and institutional and political issues that speak to the interaction between ideational and interest-based variables. It sets out an approach that advances the theoretical explanation of divergent HTA outcomes, and offers an analytical basis on which to assess current and future policy changes in HTA.

What is driving HTA decision-making? Evidence from cancer drug reimbursement decisions from 6 European countries

BACKGROUND

Decisions on the reimbursement of the same cancer drugs are different across European countries, but empirical work on the reasons behind these differences has been scarce. The main objective of this paper is to make a methodological contribution to existing research, specifically by outlining the systematic process of analysis to address such questions and determining the factors that might lead to different drug reimbursement decisions, and to explore its application in the field of oncology.

METHODS

Reimbursement decisions on cancer drugs in six European countries (Belgium, England, Poland, Portugal, Scotland, and Sweden) between 2006 and 2014 were included in the study. A taxonomy was developed, comprising two groups of variables (system-level and product-specific) and an econometric model was specified (multilevel mixed-effects ordered probit).

RESULTS

Only one in six evaluations in the sample reach the same reimbursement recommendation. Most health system variables were not determinants of a higher or lower probability of a positive reimbursement recommendation. However, the probability of reimbursement was higher when a drug was considered cost-effective by NICE/SMC and when there was a financial Managed Entry Agreement. This work also demonstrated a possible econometric approach for analysing differences in reimbursement decisions and contributes a structured approach for collecting and preparing data for such analyses.

CONCLUSIONS

Drug reimbursement decisions can be analysed in detail along a set of factors that are related to each decision. This information is essential, not only for understanding why a particular drug is accepted in one country and not in another but also when trying to implement a new HTA system or reform an existing one. This analysis provides policy makers and stakeholders with a model that enables a better understanding of the factors that drive HTA decisions and is adaptable to answer similar questions. Moreover, the data collection limitations encountered and described in this work shed light on the need for greater accessibility and transparency in HTA systems and regarding HTA outcomes.

Developing Evidence to Decision Frameworks and an Interactive Evidence to Decision Tool for Making and Using Decisions and Recommendations in Health Care

Evidence-informed health care decisions and recommendations need to be made systematically and transparently. Mediating technology can help manage boundaries between groups making decisions and target audiences, enhancing salience, credibility, and legitimacy for all. This article describes the development of the Evidence to Decision (EtD) framework and an interactive tool to create and use frameworks (iEtD) to support communication in decision making. Methods: Using a human-centered design approach, we created prototypes employing a broad range of methods to iteratively develop EtD framework content and iEtD tool functionality. Results: We developed tailored EtD frameworks for making evidence-informed decisions and recommendations about clinical practice interventions, diagnostic and screening tests, coverage, and health system and public health options. The iEtD tool provides functionality for preparing frameworks, using them in group discussions, and publishing output for implementation or adaption. EtD and iEtD are intuitive and useful for producers and users of frameworks, and flexible for use across different types of topics, decisions, and organizations. They bring valued structure to panel discussions and transparency to published output. Conclusion: EtD and iEtD can resolve some of the challenges inherent in multicriteria, multistakeholder decision systems. They are freely available online for all to use at https://ietd.epistemonikos.org/ and https://gradepro.org.

Contextual Factors Influencing Cost and Quality Decisions in Health and Care: A Structured Evidence Review and Narrative Synthesis

BACKGROUND

Decisions affecting cost and quality are taken across health and care but investigation of the mediating role of context in these is in its infancy. This paper presents a synthesis of the evidence on the contextual factors that influence 'decisions of value' - defined as those characterised by having a significant and demonstrable impact on both quality and resources - in health and care. The review considers the full range of resource/quality decisions and synthesises knowledge on the contextual drivers of these.

METHODS

The method involved structured evidence review and narrative synthesis. Literature was identified through searches of electronic databases (HMIC, Medline, Embase, CINAHL, NHS Evidence, Cochrane, Web of Knowledge, ABI Inform/Proquest), journal and bibliography hand-searching and snowball searching using citation analysis. Structured data extraction was performed drawing out descriptive information and content against review aims and questions. Data synthesis followed a thematic approach in accordance with the varied nature of the retrieved literature.

RESULTS

Twenty-one literature items reporting 14 research studies and seven literature reviews met the inclusion criteria. The review shows that in health and care contexts, research into decisions of value in health and care is in its infancy and contains wide variation in approach and remit. The evidence is drawn from a range of service and country settings and this reduces generalisability or transferability of findings. An area of relative strength in the published evidence is inquiry into factors influencing coverage and commissioning decisions in health care systems. Allocative decisions have therefore been more consistently researched than technical decisions. We use Pettigrew's (1985) distinction between inner and outer context to structure analysis of the range of factors reported as being influential. These include: evidence/information, organisational culture and governance regimes, and; economic and political conditions.

CONCLUSION

Decisions of value in health and care are subject to range of intersecting influences that often lead to a departure from narrow notions of rational decision-making. Future research should pay greater attention to the relatively under-explored area of technical, as opposed to allocative, decision-making.

[GRADE: Evidence to Decision (EtD) frameworks - a systematic and transparent approach to making well informed healthcare choices. 1: Introduction]

In healthcare, the processes, criteria, and evidence that decision makers use to reach their judgments often remain unclear. Decision makers sometimes neglect important criteria, give undue weight to criteria, or do not use the best available evidence to inform their judgments. Thus, the GRADE (Grading of Recommendations Assessment, Development and Evaluation) working group developed a system to support transparent decision making. The purpose of the Evidence-to-Decision (EtD) framework is to help people use evidence in a structured and transparent way to inform decisions in the context of clinical recommendations, coverage decisions, and health system, or public health recommendations and decisions. EtD frameworks include the formulation of the question, an assessment of the evidence, and drawing conclusions. EtD frameworks inform users of recommendations about judgments that were made and the evidence supporting these judgments by making the basis for decisions transparent to target audiences. EtD frameworks also facilitate dissemination of recommendations.

Value judgment of health interventions from different perspectives: arguments and criteria

Background

The healthcare sector is evolving while life expectancy is increasing. These trends put greater pressure on healthcare resources, prompt healthcare reforms, and demand transparent arguments and criteria to assess the overall value of health interventions. There is no consensus on the core criteria by which to value and prioritize interventions, and individual stakeholders might value specific elements differently. The present study is based on a literature review that retrieved the most widely recognized arguments and criteria used in decision-making. The aim was to compile a smaller set of arguments and criteria that would seem most relevant to different stakeholders.

Methods

A literature review was performed in Medline and EMBASE. The initial search retrieved over 2000 articles and documents of relevant committees. A selection was made based on their reference to healthcare, policy issues, or social justice. Finally, 84 papers were included. Data extraction took place after appraisal of the articles. A full table was made, including all arguments and criteria found; next, identical or largely overlapping arguments were excluded. The remaining arguments and criteria were assessed for relevance and a reduced set was compiled.

Results

The final set included 25 arguments and criteria, categorized by type (clinical, social justice, ethical, and policy). For each argument and criterion, relevance to stakeholders was scored on three levels (not, partly, and completely relevant).

Conclusions

Many arguments and criteria play a role in making value judgments on health interventions, but not all are relevant to all interventions. Moreover, they may interact with each other. A viable way to deal with interacting and possibly conflicting arguments and criteria might be to arrange public discussions that would evoke different stakeholders’ perspectives.

Revealed and Stated Preferences of Decision Makers for Priority Setting in Health Technology Assessment: A Systematic Review

BACKGROUND

There is much interest from stakeholders in understanding how health technology assessment (HTA) committees make national funding decisions for health technologies. A growing literature has analysed past decisions by committees (revealed preference, RP studies) and hypothetical decisions by committee members (stated preference, SP studies) to identify factors influencing decisions and assess their importance.

OBJECTIVES

A systematic review of the literature was undertaken to provide insight into committee preferences for these factors (after controlling for other factors) and the methods used to elicit them.

METHODS

Ovid Medline, Embase, Econlit and Web of Science were searched from inception to 11 May 2017. Included studies had to have investigated factors considered by HTA committees and to have conducted multivariate analysis to identify the effect of each factor on funding decisions. Factors were classified as being important based on statistical significance, and their impact on decisions was compared using marginal effects.

RESULTS

Twenty-three RP and four SP studies (containing 42 analyses) of 14 HTA committees met the inclusion criteria. Although factors were defined differently, the SP literature generally found clinical efficacy, cost-effectiveness and equity factors (such as disease severity) were each important to the Pharmaceutical Benefits Advisory Committee (PBAC), the National Institute for Health and Care Excellence (NICE) and the All Wales Medicines Strategy Group. These findings were supported by the RP studies of the PBAC, but not the other committees, which found funding decisions by these and other committees were mostly influenced by the acceptance of the clinical evidence and, where applicable, cost-effectiveness. Trust in the evidence was very important for decision makers, equivalent to reducing the incremental cost-effectiveness ratio (cost per quality-adjusted life-year) by A$38,000 (Australian dollars) for the PBAC and £15,000 for NICE.

CONCLUSIONS

This review found trust in the clinical evidence and, where applicable, cost-effectiveness were important for decision makers. Many methodological differences likely contributed to the diversity in some of the other findings across studies of the same committee. Further work is needed to better understand how competing factors are valued by different HTA committees.

[GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction]

Clinicians, guideline developers, and policymakers sometimes neglect important criteria, give undue weight to criteria, and do not use the best available evidence to inform their judgments. Explicit and transparent systems for decision making can help to ensure that all important criteria are considered and that decisions are informed by the best available research evidence. The GRADE Working Group has developed Evidence to Decision (EtD) frameworks for the different type of recommendations or decisions. The purpose of EtD frameworks is to help people use evidence in a structured and transparent way to inform decisions in the context of clinical recommendations, coverage decisions, and health system or public health recommendations and decisions. EtD frameworks have a common structure that includes formulation of the question, an assessment of the evidence, and drawing conclusions, though there are some differences between frameworks for each type of decision. EtD frameworks inform users about the judgments that were made and the evidence supporting those judgments by making the basis for decisions transparent to target audiences. EtD frameworks also facilitate dissemination of recommendations and enable decision makers in other jurisdictions to adopt recommendations or decisions, or adapt them to their context. This article is a translation of the original article published in British Medical Journal. The EtD frameworks are currently used in the Clinical Practice Guideline Programme of the Spanish National Health System, co-ordinated by GuíaSalud.

Pharmaceutical Pricing in Germany: How Is Value Determined within the Scope of AMNOG?

OBJECTIVES

To analyze how value is determined within the scope of the German Pharmaceutical Restructuring Act, which came into effect in 2011.

METHODS

Using data from all pharmaceuticals that had undergone assessment, appraisal, and price negotiations in Germany before June 30, 2016, we applied generalized linear model regression to analyze the impact of added benefit on the difference between negotiated prices and the prices of comparators. Data were extracted from the Federal Joint Committee's appraisals and price databases. We specified added benefit in various ways. In all models, we controlled for additional criteria such as size of patient population, European price levels, and whether the comparators were generic.

RESULTS

Our regression results confirmed the descriptive results, with price premiums reflecting the extent of added benefit as appraised by the Federal Joint Committee. On the substance level, an added benefit was associated with an increase in price premium of 227.2% (P < 0.001) compared with no added benefit. Moreover, we saw increases in price premium of 377.5% (P < 0.001), 90.0% (P < 0.001), and 336.8% (P < 0.001) for added benefits that were "considerable," "minor," and "not quantifiable," respectively. Beneficial effects on mortality were associated with the greatest price premium (624.3%; P < 0.001), followed by such effects on morbidity (174.7%; P < 0.001) and adverse events (93.1%; P = 0.019).

CONCLUSIONS

Price premiums, or "value," are driven by health gain, the share of patients benefiting from a pharmaceutical, European price levels, and whether comparators are generic. No statement can be made, however, about the appropriateness of the level of price premiums.

GRADE EVIDENCE TO DECISION (EtD) FRAMEWORK FOR COVERAGE DECISIONS

Objectives: Coverage decisions are decisions by third party payers about whether and how much to pay for technologies or services, and under what conditions. Given their complexity, a systematic and transparent approach is needed. The DECIDE (Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence) Project, a GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group initiative funded by the European Union, has developed GRADE Evidence to Decision (EtD) framework for different types of decisions, including coverage ones.

Methods: We used an iterative approach, including brainstorming to generate ideas, consultation with stakeholders, user testing, and pilot testing of the framework.

Results: The general structure of the EtD includes formulation of the question, an assessment using twelve criteria, and conclusions. Criteria that are relevant for coverage decisions are similar to those for clinical recommendations from a population perspective. Important differences between the two include the decision-making processes, accountability, and the nature of the judgments that need to be made for some criteria. Although cost-effectiveness is a key consideration when making coverage decisions, it may not be the determining factor. Strength of recommendation is not directly linked to the type of coverage decisions, but when there are important uncertainties, it may be possible to cover an intervention for a subgroup, in the context of research, with price negotiation, or with restrictions.

Conclusions: The EtD provides a systematic and transparent approach for making coverage decisions. It helps ensure consideration of key criteria that determine whether a technology or service should be covered and that judgments are informed by the best available evidence.

The bare necessities? A realist review of necessity argumentations used in health care coverage decisions

CONTEXT

Policy makers and insurance companies decide on coverage of care by both calculating (cost-) effectiveness and assessing the necessity of coverage.

AIM

To investigate argumentations pertaining to necessity used in coverage decisions made by policy makers and insurance companies, as well as those argumentations used by patients, authors, the public and the media.

METHODS

This study is designed as a realist review, adhering to the RAMESES quality standards. Embase, Medline and Web of Science were searched and 98 articles were included that detailed necessity-based argumentations.

RESULTS

We identified twenty necessity-based argumentation types. Seven are only used to argue in favour of coverage, five solely for arguing against coverage, and eight are used to argue both ways. A positive decision appears to be facilitated when patients or the public set the decision on the agenda. Moreover, half the argumentation types are only used by patients, authors, the public and the media, whereas the other half is also used by policy makers and insurance companies. The latter group is more accepted and used in more different countries.

CONCLUSION

The majority of necessity-based argumentation types is used for either favouring or opposing coverage, and not for both. Patients, authors, the public and the media use a broader repertoire of argumentation types than policy makers and insurance companies.

Developing a decision support system to link health technology assessment (HTA) reports to the health system policies in Iran

The recent increase of 'Health Technology Assessment' (HTA)-related activities in Iran has necessitated the clarification of policy-making process based on the HTA reports. This study aimed to develop a Decision Support System (DSS) in order to adopt evidence-informed policies regarding health technologies in Iran. The study can be classified as Health Policy and Systems Research. A core panel of seven experts conducted two separate reviews of relevant literature for: 1- Determining the potential technology-related policies. 2- Listing the criteria influencing those policy decisions. The policies and criteria were separately discussed and subsequently rated for appropriateness and necessity during two expert meetings in 2013. In the next step, The 'Discrete Choice Experiment' (DCE) method was employed to develop the DSS for the final technology-related policies. Accordingly, the core panel members independently rated the appropriateness of each policy for 30 virtual technologies based on the random values assigned to all the criteria for each technology. The obtained data for each policy were separately analysed using stepwise regression model, resulting in a minimal set of independent and statistically significant criteria contributing in the experts' judgments about the appropriateness of that policy. The obtained regression coefficients were used as the relative weights of the different levels of the final criteria of any policy statement, shaping the decision support scoring tool for each policy. The study has outlined 64 policy decisions under 7 macro policy areas concerning a health technology. Also, 34 criteria used for making those policy decisions have been organized within a portfolio. DCE, using stepwise regression, resulted in 64 scoring tools shaping the DSS for all HTA-related policies. Both the results and methodology of the study may serve as a guide for policy makers (researchers), particularly in low and middle income countries, in developing decision aids for their own context-specific HTA-related policies.

Preferences for care towards the end of life when decision-making capacity may be impaired: A large scale cross-sectional survey of public attitudes in Great Britain and the United States

Background

There is continuing public debate about treatment preferences at the end of life, and the acceptability and legal status of treatments that sustain or end life. However, most surveys use binary yes/no measures, and little is known about preferences in neurological disease when decision-making capacity is lost, as most studies focus on cancer. This study investigates changes in public preferences for care towards the end of life, with a focus on measures to sustain or end life.

Methods

Large-scale international public opinion surveys using a six-stage patient vignette, respondents chose a level of intervention for each stage as health and decision-making capacity deteriorated. Cross-sectional representative samples of the general public in Great Britain and the USA (N = 2016). Primary outcome measure: changes in respondents’ preferences for care, measured on a four-point scale designed before data collection. The scale ranged from: maintaining life at all costs; to intervention with agreement; to no intervention; to measures for ending life.

Results

There were no significant differences between GB and USA. Preference for measures to sustain life at all costs peaked at short-term memory loss (30.2%, n = 610). Respondents selecting ‘measures to help me die peacefully’ increased from 3.9% to 37.0% as the condition deteriorated, with the largest increase occurring when decision-making capacity was lost (10.3% to 23.0%). Predictors of choosing ‘measures to help me die peacefully’ at any stage were: previous personal experience (OR = 1.34, p<0.010), and older age (OR = 1.09 per decade, p<0.010). Negative predictors: living with children (OR = 0.72, p<0.010) and being of “black” race/ethnicity (OR = 0.45, p<0.001).

Conclusions

Public opinion was uniform between GB and USA, but markedly heterogeneous. Despite contemporaneous capacitous consent providing an essential legal safeguard in most jurisdictions, there was a high prevalence of preference for “measures to end my life peacefully” when decision-making capacity was compromised, which increased as dementia progressed. In contrast, a significant number chose preservation of life at all costs, even in end stage dementia. It is challenging to respect the longstanding values of people with dementia concerning either the inviolability of life or personal autonomy, whilst protecting those without decision-making capacity.

Stakeholders apply the GRADE evidence-to-decision framework to facilitate coverage decisions

OBJECTIVES

Coverage decisions are complex and require the consideration of many factors. A well-defined, transparent process could improve decision-making and facilitate decision-maker accountability.

STUDY DESIGN AND SETTING

We surveyed key US-based stakeholders regarding their current approaches for coverage decisions. Then, we held a workshop to test an evidence-to-decision (EtD) framework for coverage based on the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) criteria.

RESULTS

A total of 42 individuals (including 19 US stakeholders as well as international health policymakers and GRADE working group members) attended the workshop. Of the 19 stakeholders, 14 (74%) completed the survey before the workshop. Almost all of their organizations (13 of 14; 93%) used systematic reviews for coverage decision-making; few (2 of 14; 14%) developed their own evidence synthesis; a majority (9 of 14; 64%) rated the certainty of evidence (using various systems); almost all (13 of 14; 93%) denied formal consideration of resource use; and half (7 of 14; 50%) reported explicit criteria for decision-making. At the workshop, stakeholders successfully applied the EtD framework to four case studies and provided narrative feedback, which centered on contextual factors affecting coverage decisions in the United States, the need for reliable data on subgroups of patients, and the challenge of decision-making without formal consideration of resource use.

CONCLUSION

Stakeholders successfully applied the EtD framework to four case studies and highlighted contextual factors affecting coverage decisions and affirmed its value. Their input informed the further development of a revised EtD framework, now publicly available (http://gradepro.org/).

Health Technology Assessment (HTA) Case Studies: Factors Influencing Divergent HTA Reimbursement Recommendations in Australia, Canada, England, and Scotland

OBJECTIVES

To evaluate the national regulatory, health technology assessment (HTA), and reimbursement pathways for public health care in Australia, Canada, England, and Scotland, to compare initial Canadian national HTA recommendations with the initial decisions of the other HTA agencies, and to identify factors for differing national HTA recommendations between the four HTA agencies.

METHODS

Information from the public domain was used to develop a regulatory process map for each jurisdiction and to compare the HTA agencies' reimbursement recommendations. Medicines that were reviewed by all four agencies and received a negative recommendation from only one agency were selected as case studies.

RESULTS

All four countries have a national HTA agency. Their reimbursement recommendations are guided by both clinical efficacy and cost-effectiveness, and the necessity for patient input. Their activities, however, vary because of different mandates and their unique political, social, and population needs. All have an implicit or explicit quality-adjusted life-year threshold. The seven divergent case studies demonstrate examples in which new medicine-indication pairs have been rejected because of uncertainties surrounding a range of factors including cost-effectiveness, comparator choice, clinical benefit, safety, trial design, and submission timing.

CONCLUSIONS

The four HTA agencies selected for inclusion in this study share common factors, including a focus on clinical efficacy and cost-effectiveness in their decision-making processes. The differences in recommendations could be considered to be due to an individual agency's approach to risk perception, and the comparator choice used in clinical and cost-effectiveness studies.

The influence of cost-per-DALY information in health prioritisation and desirable features for a registry: a survey of health policy experts in Vietnam, India and Bangladesh

Background

Economic evaluation has been implemented to inform policy in many areas, including coverage decisions, technology pricing, and the development of clinical practice guidelines. However, there are barriers to evidence-based policy in low- and middle-income countries (LMICs) that include limited stakeholder awareness, resources and data availability, as well as the lack of capacity to conduct country-specific economic evaluations. This study aims to survey health policy experts’ opinions on barriers to use of cost-effectiveness data in these settings and to obtain their advice on how to make a new cost-per-DALY database being developed by Tufts Medical Center more relevant to LMICs. It also identifies the factors influencing transferability.

Methods

In-depth interviews were conducted with 32 participants, including policymakers, technical advisors, and researchers in Health Ministries, universities and non-governmental organisations in Bangladesh, India (New Delhi, Tamil Nadu and Karnataka) and Vietnam.

Results

The survey revealed that, in all settings, the use of cost-effectiveness information in policy development is lacking, owing to limited knowledge among policymakers and inadequate human resources with health economics expertise in the government sector. Furthermore, researchers in universities do not have close connections with health authorities. In India and Vietnam, the demand for evidence to inform coverage decisions tends to increase as the countries are moving towards universal health coverage. The informants in all countries argue that cost-effectiveness data are useful for decision-makers; however, most of them do not perform data searches by themselves but rely on the information provided by the technical advisor counterparts. Most interviewees were familiar with using evidence from other countries and were also aware of the influences of contextual elements as a limitation of transferability. Finally, strategies to promote the newly developed database include training on basic economic evaluation for policymakers and researchers, and effective communication programs, with support from reputable global agencies.

Conclusions

Although cost-effectiveness information is recognised as essential in resource allocation, there are several impediments in the generation and use of such evidence to inform priority setting in LMICs. As such, the Cost-per-DALY database should be well-designed and introduced with appropriate promotion strategies so that it will be helpful in real-world policymaking.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-016-0156-6) contains supplementary material, which is available to authorized users.

Evaluating alignment between Canadian Common Drug Review reimbursement recommendations and provincial drug plan listing decisions: an exploratory study

BACKGROUND

The CADTH Common Drug Review was established in 2002 to prepare national health technology assessment reports to guide listing decisions for 18 participating drug plans. The aim of this study was to compare the nonmandatory recommendations from the Common Drug Review in Canada with the listing decisions of provincial payers to determine alignment.

METHODS

We identified the recommendations issued by the Common Drug Review from Jan. 1, 2009, to Jan. 1, 2015, and compared these with the listing decisions of 3 provincial public payers (Alberta, British Columbia and Ontario) that participate in the Common Drug Review and the recommendations from Quebec.

RESULTS

We identified 174 medicine-indication pairs in CADTH Common Drug Review reports issued from Jan. 1, 2009, to Jan. 1, 2015; 110 of these met the inclusion criterion. Among the 110 medicine-indication pairs, listing decisions were available for 95 in Alberta, 102 in Quebec, 104 in Ontario and 106 in BC. There was moderate to substantial agreement between provincial listing decisions and Common Drug Review recommendations: 74.5% (κ = 0.47, 95% confidence interval [CI] 0.31-0.64) for Quebec, 78.8% (κ = 0.56, 95% CI 0.41-0.72) for Ontario, 78.9% (κ = 0.58, 95% CI 0.42-0.74) for Alberta and 81.1% (κ = 0.62, 95% CI 0.47-0.77) for BC.

INTERPRETATION

Our study showed moderate to substantial agreement between Common Drug Review recommendations and provincial listing decisions. Future studies can build on this research by evaluating the concordance between Common Drug Review recommendations and listing decisions of all participating federal, provincial and territorial drug plans.

Using CART to Identify Thresholds and Hierarchies in the Determinants of Funding Decisions

There is much interest in understanding decision-making processes that determine funding outcomes for health interventions. We use classification and regression trees (CART) to identify cost-effectiveness thresholds and hierarchies in the determinants of funding decisions. The hierarchical structure of CART is suited to analyzing complex conditional and nonlinear relationships. Our analysis uncovered hierarchies where interventions were grouped according to their type and objective. Cost-effectiveness thresholds varied markedly depending on which group the intervention belonged to: lifestyle-type interventions with a prevention objective had an incremental cost-effectiveness threshold of $2356, suggesting that such interventions need to be close to cost saving or dominant to be funded. For lifestyle-type interventions with a treatment objective, the threshold was much higher at $37,024. Lower down the tree, intervention attributes such as the level of patient contribution and the eligibility for government reimbursement influenced the likelihood of funding within groups of similar interventions. Comparison between our CART models and previously published results demonstrated concurrence with standard regression techniques while providing additional insights regarding the role of the funding environment and the structure of decision-maker preferences.

Developing an evidence-based methodological framework to systematically compare HTA coverage decisions: A mixed methods study

Health Technology Assessment (HTA) often results in different coverage recommendations across countries for a same medicine despite similar methodological approaches. This paper develops and pilots a methodological framework that systematically identifies the reasons for these differences using an exploratory sequential mixed methods research design. The study countries were England, Scotland, Sweden and France. The methodological framework was built around three stages of the HTA process: (a) evidence, (b) its interpretation, and (c) its influence on the final recommendation; and was applied to two orphan medicinal products. The criteria accounted for at each stage were qualitatively analyzed through thematic analysis. Piloting the framework for two medicines, eight trials, 43 clinical endpoints and seven economic models were coded 155 times. Eighteen different uncertainties about this evidence were coded 28 times, 56% of which pertained to evidence commonly appraised and 44% to evidence considered by only some agencies. The poor agreement in interpreting this evidence (κ=0.183) was partly explained by stakeholder input (ns=48 times), or by agency-specific risk (nu=28 uncertainties) and value preferences (noc=62 "other considerations"), derived through correspondence analysis. Accounting for variability at each stage of the process can be achieved by codifying its existence and quantifying its impact through the application of this framework. The transferability of this framework to other disease areas, medicines and countries is ensured by its iterative and flexible nature, and detailed description.

Which factors may determine the necessary and feasible type of effectiveness evidence? A mixed methods approach to develop an instrument to help coverage decision-makers

OBJECTIVES

Reimbursement decisions require evidence of effectiveness and, in general, a blinded randomised controlled trial (RCT) is the preferred study design to provide it. However, there are situations where a cohort study, or even patient series, can be deemed acceptable. The aim of this study was to develop an instrument that first examines which study characteristics of a blinded RCT are necessary, and then, if particular characteristics are considered necessary, examines whether these characteristics are feasible.

DESIGN

We retrospectively studied 22 interventions from 20 reimbursement reports concerning medical specialist care made by the Dutch National Health Care Institute (ZIN) to identify any factors that influenced the necessity and feasibility of blinded RCTs, and their constituent study characteristics, that is, blinding, randomisation and a control group. A literature review was performed to identify additional factors. Additional expertise was included by interviewing eight experts in epidemiology, medicine and ethics. The resulting instrument was called the FIT instrument (Feasible Information Trajectory), and was prospectively validated using three consecutive reimbursement reports.

RESULTS

(Blinded) RCT evidence was lacking in 5 of 11 positive reimbursement decisions and 3 of 11 negative decisions. In the reimbursement reports, we found no empirical evidence supporting situations where a blinded RCT is unnecessary. The literature also revealed few arguments against the necessity of a blinded RCT. In contrast, many factors influencing the feasibility of randomisation, a control group and blinding, were found in the reimbursement reports and the literature; for example, when a patient population is too small or when an intervention is common practice, randomisation will be hindered.

CONCLUSIONS

Policy regarding the necessity and feasibility of different types of evidence of effectiveness would benefit from systematic guidance. The FIT instrument has the potential to support transparent, reproducible and well-founded decisions on appropriate evidence of effectiveness in medical specialist care.

Reimbursement of orphan drugs in Belgium: what (else) matters?

Background

Most orphan drugs do not meet traditional standards of cost-effectiveness. Yet, most orphan drugs are reimbursed, which implies that other factors are taken into account at the time of reimbursement. To increase accountability of decision-makers, there is a need for more transparency in the factors that play a role in reimbursement decisions of orphan drugs. Therefore, the aim of this study is to use a combination of qualitative research methods to examine which official and non-official factors influence reimbursement decisions for orphan drugs in Belgium.

Methods

Six semi-structured interviews with past or present members of the Drug Reimbursement Committee (DRC) were performed with a view to obtaining an overview of the potential factors influencing reimbursement. Additionally, these presence of these factors was assessed in the reimbursement dossiers of all orphan drugs (n = 64) for which an application for reimbursement was submitted to the National Institute for Health and Disability Insurance in Belgium between January 2002 and July 2013.

Results

Different official (i.e. therapeutic value, budget impact, price and impact in clinical practice) and non-official factors (i.e. pricing and reimbursement in other countries, interference by patient organisations and experts, arguments related to quality of branded drug versus compounding, media attention, innovative character, economic importance, ethical arguments and the political climate) may have influenced past reimbursement decisions for orphan drugs in Belgium.

Discussion

The identification of factors influencing orphan drug reimbursement is a crucial step in the development of a transparent and consistent framework which will guide future decision-making for reimbursement of orphan drugs.

Electronic supplementary material

The online version of this article (doi:10.1186/s13023-014-0139-z) contains supplementary material, which is available to authorized users.

Public funding of pharmaceuticals in The Netherlands: investigating the effect of evidence, process and context on CVZ decision-making

The College Voor Zorgverzekeringen (CVZ) provides guidance to the Dutch healthcare system on funding and use of new pharmaceutical technologies. This study examined the impact of evidence, process and context factors on CVZ decisions in 2004-2009. A data set of CVZ decisions pertaining to pharmaceutical technologies was created, including 29 variables extracted from published information. A three-category outcome variable was used, defined as the decision to 'recommend', 'restrict' or 'not recommend' a technology. Technologies included in list 1A/1B or on the expensive drug list were considered recommended; those included in list 2 or for which patient co-payment is required were considered restricted; technologies not included on any reimbursement list were classified as 'not recommended'. Using multinomial logistic regression, the relative contribution of explanatory variables on CVZ decisions was assessed. In all, 244 technology appraisals (256 technologies) were analysed, with 51%, of technologies recommended, 33% restricted and 16% not recommended by CVZ for funding. The multinomial model showed significant associations (p ≤ 0.10) between CVZ outcome and several variables, including: (1) use of an active comparator and demonstration of statistical superiority of the primary endpoint in clinical trials, (2) pharmaceutical budget impact associated with introduction of the technology, (3) therapeutic indication and (4) prevalence of the target population. Results confirm the value of a comprehensive and multivariate approach to understanding CVZ decision-making.

Early Benefit Assessment of Pharmaceuticals in Germany

Background. Since 2011, when the German Pharmaceutical Market Restructuring Act (AMNOG) came into effect, newly licensed pharmaceuticals must demonstrate an added benefit over a comparator treatment to be reimbursed at a value greater than the reference price. Evidence submitted by manufacturers is assessed by the Institute for Quality and Efficiency in Health Care (IQWiG) and subsequently appraised by the German Federal Joint Committee (FJC) as part of so-called early benefit assessments (EBA). This study aims to explain the decisions made, clarify the roles of the parties (manufacturers, IQWiG, FJC) involved, and guide manufacturers in developing future submissions by analyzing 42 EBAs concluded since January 2011. Methods. We developed a variable list representing the essential components of the EBA: the rating decisions of manufacturers, IQWiG, and the FJC regarding each pharmaceutical’s added benefit; the characteristics of the pharmaceutical; the characteristics of the EBA process; the types of evidence submitted; the methods used to generate evidence; and the pharmaceutical’s maximum possible budget impact. We used Cohen’s kappa to analyze agreement between the rating decisions of the different parties. The chi-square test and bivariate regression were used to identify associations between components of the EBA process and the rating decisions of the FJC. Results. We observed a low level of agreement between manufacturers and the FJC (kappa = 0.21; 95% CI 0.107–0.31) and a substantial level of agreement between IQWiG and the FJC (kappa = 0.64; 95% CI 0.451–0.827) in their rating decisions. The characteristics of the EBA process—for example, duration of the process ( P = 0.357), participation in the official hearing ( P = 0.227), and the pharmaceutical’s budget impact ( P = 0.725)—did not have a significant effect on the rating decisions of the FJC. There was, however, an association between the type of evidence submitted and the FJC’s rating decision when the manufacturer’s dossier reported outcomes related to morbidity ( P = 0.009) or adverse events ( P < 0.001) but not mortality ( P = 0.718) or quality of life ( P = 0.783). Conclusions. While the FJC tends to disagree with the rating of benefit by manufacturers, it softens IQWiG’s decisions, potentially to make the final outcome more acceptable. Concerns voiced that the FJC might be exceeding its statutory authority by taking cost or procedural considerations into account appear to be unfounded. Choosing appropriate evidence to submit for each endpoint remains a challenge, as submission of health outcomes evidently influences decisions.

Link between process and appraisal in coverage decisions: an analysis with structural equation modeling

BACKGROUND

To achieve fair-coverage decision making, both material criteria and criteria of procedural justice have been proposed. The relationship between these is still unclear.

OBJECTIVE

To analyze hypotheses underlying the assumption that more assessment, transparency, and participation have a positive impact on the reasonableness of coverage decisions.

METHODS

We developed a structural equation model in which the process components were considered latent constructs and operationalized by a set of observable indicators. The dependent variable "reasonableness" was defined by the relevance of clinical, economic, and other ethical criteria in technology appraisal (as opposed to appraisal based on stakeholder lobbying). We conducted an Internet survey among conference participants familiar with coverage decisions of third-party payers in industrialized countries between 2006 and 2011. Partial least squares path modeling (PLS-PM) was used, which allows analyzing small sample sizes without distributional assumptions. Data on 97 coverage decisions from 15 countries and 40 experts were used for model estimation.

RESULTS

Stakeholder participation (regression coefficient [RC] =0.289; P = 0.005) and scientific rigor of assessment (RC = 0.485; P < 0.001) had a significant influence on the construct of reasonableness. The path from transparency to reasonableness was not significant (RC = 0.289; P = 0.358). For the reasonableness construct, a considerable share of the variance was explained (R (2) = 0.44). Biases from missing data and nesting effects were assessed through sensitivity analyses. Limitations. The results are limited by a small sample size and the overrepresentation of some decision makers.

CONCLUSIONS

Rigorous assessment and intense stakeholder participation appeared effective in promoting reasonable decision making, whereas the influence of transparency was not significant. A sound evidence base seems most important as the degree of scientific rigor of assessment had the strongest effect.

Transparency vs. closed-door policy: do process characteristics have an impact on the outcomes of coverage decisions? A statistical analysis

The aim of this study was to analyze influences of process- and technology-related characteristics on the outcomes of coverage decisions. Using survey data on 77 decisions from 13 countries, we examined whether outcomes differ by 14 variables that describe components of decision-making processes and the technology. We analyzed the likelihood of committees covering a technology, i.e. positive (including partial coverage) vs. negative coverage decisions. We performed non-parametric univariate tests and binomial logistic regression with a stepwise variable selection procedure. We identified a negative association between a positive decision and whether the technology is a prescribed medicine (p=0.0097). Other significant influences on a positive decision outcome included one disease area (p=0.0311) and whether a technology was judged to be (cost-)effective (p<0.0001). The first estimation of the logistic regression yielded a quasi-complete separation for technologies that were clearly judged (cost-)effective. In uncertain decisions, a higher number of stakeholders involved in voting (odds ratio=2.52; p=0.03) increased the likelihood of a positive outcome. The results suggest that decisions followed the lines of evidence-based decision-making. Despite claims for transparent and participative decision-making, the phase of evidence generation seemed most critical as decision-makers usually adopted the assessment recommendations. We identified little impact of process configurations.