Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework

Development of a decision support tool to compare diagnostic strategies for establishing the herd status for infectious diseases: An example with Salmonella Dublin infection in dairies

The diagnosis of infectious diseases at herd level can be challenging as different stakeholders can have conflicting priorities. The current study proposes a "proof of concept" of an approach that considers a reasonable number of criteria to rank plausible diagnostic strategies using multi-criteria decision analysis (MCDA) methods. The example of Salmonella Dublin diagnostic in Québec dairy herds is presented according to two epidemiological contexts: (i) in herds with no history of S. Dublin infection and absence of clinical signs, (ii) in herds with a previous history of infection, but absence of clinical signs at the moment of testing. Multiple multiparty exchanges were conducted to determine: 1) stakeholders' groups; 2) the decision problem; 3) solutions to the problem (options) or diagnostic strategies to be ordered; 4) criteria and indicators; 5) criteria weights; 6) the construction of a performance matrix for each option; 7) the multi-criteria analyses using the visual preference ranking organization method for enrichment of evaluations approach; 8) the sensitivity analyses, and 9) the final decision. A total of nine people from four Québec's organizations (the dairy producers provincial association along with the DHI company, the ministry of agriculture, the association of veterinary practitioners, and experts in epidemiology) composed the MCDA team. The decision problem was "What is the optimal diagnostic strategy for establishing the status of a dairy herd for S. Dublin infection when there are no clinical signs of infection?". Fourteen diagnostic strategies composed of the three following parameters were considered: 1) biological samples (bulk tank milk or blood from 10 heifers aged over three months); 2) sampling frequencies (one to three samples collection visits); 3) case definitions to conclude to a positive status using imperfect milk- or blood-ELISA tests. The top-ranking diagnostic strategy was the same in the two contexts: testing the bulk tank milk and the blood samples, all samples collected during one visit and the herd being assigned a S. Dublin positive status if one sample is ELISA-positive. The final decision favored the top-ranking option for both contexts. This MCDA approach and its application to S. Dublin infection in dairy herds allowed a consensual, rational, and transparent ranking of feasible diagnostic strategies while taking into account the diagnostic tests accuracy, socio-economic, logistic, and perception considerations of the key actors in the dairy industry. This promising tool can be applied to other infectious diseases that lack a well-established diagnostic procedure to define a herd status.

Development of a Value Assessment Framework for Pediatric Health Technologies Using Multicriteria Decision Analysis: Expanding the Value Lens for Funding Decision Making

OBJECTIVES

A health technology assessment (HTA) does not systematically account for the circumstances and needs of children and youth. To supplement HTA processes, we aimed to develop a child-tailored value assessment framework using a multicriteria decision analysis approach.

METHODS

We constructed a multicriteria-decision-analysis-based model in multiple phases to create the Comprehensive Assessment of Technologies for Child Health (CATCH) framework. Using a modified Delphi process with stakeholders having broad disciplinary and geographic variation (N = 23), we refined previously generated criteria and developed rank-based weights. We established a criterion-pertinent scoring rubric for assessing incremental benefits of new drugs. Three clinicians independently assessed comprehension by pilotscoring 9 drugs. We then validated CATCH for 2 childhood cancer therapies through structured deliberation with an expert panel (N = 10), obtaining individual scores, consensus scores, and verbal feedback. Analyses included descriptive statistics, thematic analysis, exploratory disagreement indices, and sensitivity analysis.

RESULTS

The modified Delphi process yielded 10 criteria, based on absolute importance/relevance and agreed importance (median disagreement indices = 0.34): Effectiveness, Child-specific Health-related Quality of Life, Disease Severity, Unmet Need, Therapeutic Safety, Equity, Family Impacts, Life-course Development, Rarity, and Fair Share of Life. Pilot scoring resulted in adjusted criteria definitions and more precise score-scaling guidelines. Validation panelists endorsed the framework's key modifiers of value. Modes of their individual prescores aligned closely with deliberative consensus scores.

CONCLUSIONS

We iteratively developed a value assessment framework that captures dimensions of child-specific health and nonhealth gains. CATCH could improve the richness and relevance of HTA decision making for children in Canada and comparable health systems.

Treatment Preference for Alzheimer's Disease: A Multicriteria Decision Analysis with Caregivers, Neurologists, and Payors

INTRODUCTION

Alzheimer's disease (AD) is a chronic neurodegenerative disorder associated with a high burden of illness. New therapies under development include agents that target amyloid-beta (Aβ), a key component in AD pathogenesis. Understanding the decision-making process for new AD drugs would help determine if such therapies should be adopted by society. Multicriteria decision analysis (MCDA) was applied to three key stakeholder groups to assess treatment alternatives for AD based on a multitude of decision trade-offs covering main components of care.

METHODS

AD caregivers (n = 117), neurologists (n = 90), and payors (n = 90) from the USA received an online survey. The decision problem was broken down into four decision criterion and 12 subcriteria for two treatment scenarios: an Aβ-targeted therapy vs. the standard of care (SOC). Respondents were asked to indicate how much they preferred one option over another on a scale from 1 (equal preference) to 9 (high preference) based on each criterion and subcriterion. The decision criteria and subcriteria were weighted and presented as partial utility scores (pUS), with higher scores suggesting an increased preference for that decision-making component.

RESULTS

Caregivers and payors applied the highest value to need for intervention (mean pUS = 0.303 and 0.259) and clinical outcomes (mean pUS = 0.286 and 0.377). In contrast, neurologists placed the highest value on clinical outcomes and types of benefits (mean pUS = 0.436 and 0.248). When decision subcriteria were examined, efficacy (mean pUS = 0.115, 0.219, and 0.166) and the type of patient benefits (mean pUS = 0.135, 0.178, and 0.126) were among the most valued by caregivers, neurologists, and payors.

CONCLUSION

All groups placed the highest value on drug efficacy and types of benefit derived by patients. In contrast, cost implications were among the least important aspects in their decision-making.

Inter-sectoral prioritization of climate technologies: insights from a Technology Needs Assessment for mitigation in Brazil

Technological development is key for national strategies to cope with the Paris Agreement's goals. Technology Needs Assessments (TNAs) aim to identify, prioritize, and diffuse climate change mitigation and/or adaptation technologies in developing countries. Their methodology includes a multi-criteria decision analysis (MCDA) framework but, although many countries already conducted a TNA, literature lacks discussions on country-specific processes for a TNA, as it usually follows a one-size-fits-all approach. This paper provides empirical evidence on the importance of country-driven processes that help shaping international programmes into country-specific needs and capabilities. It presents lessons learned from a tailored process for identification, prioritization, and selection of mitigation technologies in the scope of a TNA project for Brazil, an exceptional case of a developing country with strong capacity in integrated assessment modelling (IAM) scenarios for guiding its climate strategies. A previous IAM scenario result allowed pre-selecting technologies in six key economic sectors, while other TNAs prioritized no more than three. This allowed the elaboration of an overall ranking from the MCDA, in contrast to sectoral rankings that are mostly employed in other countries' TNAs. The overall ranking serves not only as a basis for the selection of priority technologies but also provides information on the integrated innovations framework for climate technologies in the country. Further specific findings of the tailored Brazilian TNA approach are discussed in the paper in order to call for the importance that a technology transfer project should not only be country-driven but also conducted through a country-specific process.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s11027-022-10025-6.

Multicriteria decision analysis in health care decision in oncology: a systematic review

INTRODUCTION

Multicriteria decision analysis (MCDA) has been used to inform health decisions in health technology assessments (HTA) processes. This is particularly important to complex treatment decisions in oncology.

AREAS COVERED

Five databases (PubMed, EMBASE, LILACS, Web of Science and CRD's NHS Economic Evaluation Database) were searched for studies comparing health technologies in oncology, involving the concept MCDA. The ISPOR MCDA Good Practices Guidelines were used to assess the reporting quality. Study selection, appraisal, and data extraction were performed by two reviewers. Fifteen studies were included. The main decision problem was related to health technology assessment of cancer treatments. Clinicians and public health experts were the most frequent stakeholders. The most frequently included criteria comprised therapeutic benefit, and socio-economic impact. Value measurement approach, direct rating techniques, and additive model for aggregation were used in most studies. Uncertainty analysis revealed the impact of posology and costs on the studies' results. All studies showed some level of overlapping decision criteria.

EXPERT OPINION

There is considerable diversity of methods in MCDA for healthcare decision-making in oncology. The evidence presented can serve as a resource when considering which stakeholders, criteria, and techniques to include in future MCDA studies in oncology.

Model of Choice Photovoltaic Panels Considering Customers’ Expectations

Photovoltaic electricity generation is key to achieving deep decarbonization with a high degree of electrification. It is predicted that the energy sector will reduce carbon dioxide by producing electricity mainly from photovoltaic (PV) power. Although dynamic development of the implementation of photovoltaic panels has been observed, their choice considering customer specificity is still a problem. Therefore, the purpose of this study is to propose the model of choice photovoltaic panels considering customers’ expectations. It can support the choice of a photovoltaic panel of a certain quality (satisfaction of concrete customer) in combination with the cost of its purchase. The proposed model includes acquiring and then processing customers’ expectations into technical criteria, while simultaneously considering the weighting of these criteria. It is realized in a standardized way, i.e., the zero-unitarization method (MUZ), after which normalized values of the quality of the photovoltaic panels’ criteria are obtained. In turn, the quality of these products is estimated by the weighted sum model (WSM) and then integrated with purchase cost in qualitative cost analysis (AKJ). As a result, using the scale of relative states, it is possible to categorize customer satisfaction from indicating qualitative cost and selecting the photovoltaic panel expected by customers (the most satisfactory). The effectiveness of the model was demonstrated by a sensitivity analysis, after which the key PV criteria were indicated. The proposed model is intended for any entity who selects a photovoltaic panel for customers. The computerization of calculations may contribute to its utilitarian dissemination.

Value-based decision-making for orphan drugs with multiple criteria decision analysis: burosumab for the treatment of X-linked hypophosphatemia

OBJECTIVE

Burosumab is an orphan medicinal product (OMP) approved in Europe for the treatment of X-linked hypophosphatemia (XLH). The aim of this study was to assess the value of burosumab versus conventional therapy for the treatment of paediatric XLH, through a multi-criteria decision analysis (MCDA) framework for health technology assessment (HTA) of OMPs in Portugal.

METHODS

The MCDA framework considered 14 criteria related to disease burden, therapeutic value and economic burden. A multidisciplinary panel of national stakeholders participated in a two-phase exercise. In the first phase, relative weights and part-worth utilities for the criteria and their levels were elicited and a reimbursement likelihood function was calibrated through adaptive conjoint analysis. In the second phase, burosumab and conventional therapy were assessed against the criteria, providing a global value score (0-100) and reimbursement likelihood (0-100%) for both.

RESULTS

Of the 14 criteria, disease burden, therapeutic value and economic burden criteria represented 27.29%, 57.17% and 15.53% of the total weight in the decision, respectively. All disease burden and some therapeutic value criteria, typically not included in traditional HTA, represented 47.88% of the total weight. Burosumab was unanimously considered superior to conventional therapy, with an average (range) global value score of 84.96 (82.48-86.54) against 48.06 (43.37-57.68), and reimbursement likelihood of 97.50% (96.78%-98.32%) against 43.66% (31.48%-68.73%), respectively.

CONCLUSIONS

MCDA represents a powerful tool in HTA decision-making for OMPs. The results of this MCDA acknowledge burosumab as a disease-modifying drug, deemed superior to conventional therapy for the treatment of paediatric XLH.

Development and Practical Application of a Multiple-Criteria Decision Analysis Framework on Respiratory Inhalers: Is It Always Useful in the MOH Malaysia Medicines Formulary Listing Context?

Objectives. The current health technology assessment used to evaluate respiratory inhalers is associated with limitations that have necessitated the development of an explicit formulary decision-making framework to ensure balance between the accessibility, value, and affordability of medicines. This study aimed to develop a multiple-criteria decision analysis (MCDA) framework, apply the framework to potential and currently listed respiratory inhalers in the Ministry of Health Medicines Formulary (MOHMF), and analyze the impacts of applying the outputs, from the perspective of listing and delisting medicines in the formulary. Methods. The overall methodology of the framework development adhered to the recommendations of the ISPOR MCDA Emerging Good Practices Task Force. The MCDA framework was developed using Microsoft Excel 2010 and involved all relevant stakeholders. The framework was then applied to 27 medicines, based on data gathered from the highest levels of available published evidence, pharmaceutical companies, and professional opinions. The performance scores were analyzed using the additive model. The end values were then deliberated by an expert committee. Results. A total of eight main criteria and seven subcriteria were determined by the stakeholders. The economic criterion was weighted at 30%. Among the noneconomic criteria, "patient suitability" was weighted the highest. Based on the MCDA outputs, the expert committee recommended one potential medicine (out of three; 33%) be added to the MOHMF and one existing medicine (out of 24; 4%) be removed/delisted from the MOHMF. The other existing medicines remained unchanged. Conclusions. Although this framework was useful for deciding to add new medicines to the formulary, it appears to be less functional and impactful for the removal/delisting existing medicines from the MOHMF. The generalizability of this conclusion to other formulations remains to be confirmed.

An application of analytic network process model in supporting decision making to address pharmaceutical shortage

Background

The present study aimed to develop an Analytic Network Process (ANP) model to assist policymakers in identifying and prioritizing allocation indicators, which are being used or should be used to distribute drugs in short supply among different provinces.

Methods

The model encompasses the interactions between various indicators and efficiency, equity, and effectiveness paradigms. Accordingly, a set of clusters and elements, which were associated with the allocation of drugs in short supply in Iran’s pharmaceutical system, were detected to develop the model and were then compared in pairs in terms of a specified factor to show the priorities.

Results

Equity had the highest priority (0.459) following by Efficiency (0.37), and Effectiveness (0.171). The 4 most important allocation indicator were “number of prescriptions” (0.26) and “total bed occupancy rate” (0.19) related to equity, “total population” (0.21) in efficiency and “the burden of rare and incurable disease” (0.07) in effectiveness paradigm.

Conclusions

The capability to overcome inefficient resource allocation patterns caused by both oversupply and undersupply derived from historic resource allocation may be highly limited in the absence of the need indicators. The quality of the decision is related to a careful balancing act of the three paradigms which represents roughly the triple aim of public healthcare systems: clinical improvement (effectiveness), population health improvement (equity and access), and reducing cost (economic aspects -efficiency).

What's important in AAC decision making for children? Evidence from a best-worst scaling survey

The choice of which AAC device to provide for a child can have long lasting consequences, but little is known about the decision-making of AAC professionals who make recommendations in this context. A survey was conducted with AAC professionals using best-worst scaling methodology examining what characteristics of children and attributes of AAC devices are considered most important in decision-making. A total of 19 child characteristics and 18 device attributes were selected by the authors from lists generated from literature reviews and from focus groups with AAC professionals, people who use AAC, and other stakeholders. The characteristics and attributes were used to develop two best-worst scaling surveys that were administered to 93 AAC professionals based in the UK. The relative importance of characteristics/attributes was estimated using statistical modelling. Child characteristics related to language and communication, cognitive and learning abilities, and personality traits were generally found to be more important than physical features. Communication, language, and interface-related AAC device attributes were generally more important than hardware and physical attributes. Respondent demographics (e.g., experience, professional background) did not seem to influence the importance assigned to device characteristics or attributes. Findings may inform both future quantitative research into decision-making and efforts to improve decision-making in practice.

Can reflective multicriteria be the new paradigm for healthcare decision-making? The EVIDEM journey

Background

Multiple technologies, procedures and programs call for fairly-based decisions for prioritization of healthcare interventions. There is a diversity of perspectives of what constitutes a legitimate decision, which depends on both the process and the reasoning applied. Current approaches focus on technical aspects while methods to support alignment of decisions with the compassionate impetus of healthcare systems is lacking.

Methods

The framework was developed based on an analysis of the foundations of healthcare systems, the reasoning underlying decisions and fair processes. The concept of reflective multicriteria was created: it assumes that decisionmakers guided by a generic interpretative frame rooted in the compassionate impetus of healthcare systems, can sharpen their reasoning, raise awareness of their motivation and increase legitimacy of decisions. The initial framework was made available through a not for profit organization (the EVIDEM Collaboration, 2006–2017) to stimulate its development with thought leaders and stakeholders in an open source philosophy. Development was tailored to the real-life needs of decisionmakers and drew on several domains of knowledge including healthcare ethics, evidenced-based medicine, health economics, health technology assessment and multicriteria approaches.

Results

The 10th edition framework builds on four dimensions: (1) the universal impetus of healthcare systems, (2) reasoning, values and ethics, (3) evidence and knowledge on interventions, and (4) a transformative process. Mathematical aspects of the framework are designed to help clarify, express and share individual reasoning; this non-conventional use of numbers requires a cultural change and needs to be phased in slowly. The framework includes four tools for easy adaptation and operationalization: (a) concepts and operationalization, (b) adapt and pilot, (c) evidence matrix, (d) mathematical representation of reasoning. Application is useful throughout all types of healthcare interventions, for all levels of decision, and across the globe.

Conclusion

By clarifying their reasoning while keeping decisionmakers aware of the impetus of healthcare systems, reflective multicriteria provides an effective approach to increase the legitimacy of decisions. Beyond a tool, reflective multicriteria pioneered by EVIDEM is geared to transform our vision of the value of healthcare interventions and how they might contribute to relevant, equitable and sustainable healthcare systems.

Electronic supplementary material

The online version of this article (10.1186/s12962-018-0116-9) contains supplementary material, which is available to authorized users.

Multicriteria decision analysis (MCDA) in health care: a systematic review of the main characteristics and methodological steps

Background

The health area is one of the most affected systems on the perspective of decision-making with multiobjectives, thus becoming prone to errors in the final solution, however, multicriteria decision analysis (MDCA) appears as an aid tool for this process decision-making. Therefore,the present study aims to analyze and synthesize articles found in the literature, involing MCDA in health care, evaluating general issues and methodological aspects, structuring them in a single work.

Methods

Surveys in the bibliographic databases SCOPUS and PUBMED indicated 1852 documents on the subject, however after a careful verificatios, 66 studies were selected to be analyzed completely. The data extracted from the included articles were organized into a spreadsheet for the preparation of analysis, and the technique used was descriptive statistics.

Results

It was possible to identify a growth trend in the application of the MCDA in the health area, but no dominance was identified in relation to the authors of the publication and the periodicals where they are published, but some countries stood out in terms of the number of published researches, such as: Canada and Turkey. In defining the decision problem, and in defining criteria, the “literature” presented the greatest demand for those who wish to structure their decision problem. Finally, it was verified by the analysis of the problem, that the MCDA to solve the problems of ranking has comprehensive application and that there is a greater incidence in the use of the AHP and Logic methods Fuzzy.

Conclusion

With this, it is possible to observe, through the data of this review, that more than the multicriteria methods, the multicriteria decision model has been highlighted, also in the health area. In addition, the study can guide new applications and techniques using MCDA in the health care.

Applying Reflective Multicriteria Decision Analysis (MCDA) to Patient-Clinician Shared Decision-Making on the Management of Gastroenteropancreatic Neuroendocrine Tumors (GEP-NET) in the Spanish Context

Introduction

Unresectable, well-differentiated nonfunctioning gastroenteropancreatic neuroendocrine tumors (GEP-NETs) can be monitored (watchful waiting, WW) or treated with systemic therapy such as somatostatin analogues (SSAs) to delay progression. We applied a reflective multicriteria decision analysis (MCDA) shared-decision framework (previously developed for the USA) to explore what matters to Spanish patients and clinicians considering GEP-NET treatment options.

Methods

The EVIDEM-derived framework was updated and adapted to the Spanish context. During a Chatham House session, five patients and six physicians assigned criteria weights using hierarchical point allocation and direct rating scale (alternative analysis). Informed by synthesized evidence embedded in the framework, participants scored how each criterion favored SSA treatment (reference case lanreotide) or WW and shared insights and knowledge. Weights and scores were combined into value contributions (norm. weight × score/5), which were added across criteria to derive the relative benefit–risk balance (RBRB, scale − 1 to + 1). Exploratory comparisons to US study findings were performed.

Results

Focusing on intervention outcomes (effectiveness, patient-reported, and safety), the mean RBRB favored treatment over WW (+ 0.32 ± 0.24), with the largest contributions from progression-free survival (+ 0.11 ± SD 0.07), fatal adverse events (+ 0.06 ± SD 0.08), and impact on HRQoL (+ 0.04 ± SD 0.04). Consideration of modulating criteria (type of benefit, need, costs, evidence, and feasibility) increased the RBRB to + 0.50 ± 0.14, with type of therapeutic benefit (+ 0.10 ± SD 0.08) and quality of evidence (+ 0.08 ± SD 0.06) contributing most towards treatment. Alternative weighting yielded similar results. Results were broadly comparable to those derived from the US study.

Conclusion

The multicriteria framework helped Spanish patients and clinicians identify and express what matters to them. The approach is transferable across decision-making contexts.

Funding

IPSEN Pharma.

Electronic supplementary material

The online version of this article (10.1007/s12325-018-0745-6) contains supplementary material, which is available to authorized users.

Applying a Multicriteria Decision Analysis (MCDA) Approach to Elicit Stakeholders' Preferences in Italy: The Case of Obinutuzumab for Rituximab-Refractory Indolent Non-Hodgkin Lymphoma (iNHL)

BACKGROUND

Healthcare decision makers need to make trade-offs between different elements of value of new treatments. Multicriteria decision analysis (MCDA) provides a framework that can help decision makers to understand stakeholders' preferences and be explicit about the trade-offs that are being made.

OBJECTIVE

The objective of this study was to use MCDA to obtain preferences and views on decision criteria across three stakeholder groups (patients, clinicians and payers) in Italy and to use these to assess the performance of obinutuzumab for rituximab-refractory indolent non-Hodgkin lymphoma (iNHL).

METHODS

We used EVIDEM V3.0, an MCDA framework, and collected participants' preferences via an online survey and structured meetings.

RESULTS

Patients and clinicians expressed a preference for interventions targeting severe conditions. Payers expressed preference for treatments targeting areas with an unmet need, which are cheaper than the comparator, and with high-quality evidence. Obinutuzumab in combination with bendamustine, compared with bendamustine alone, received high positive scores for the criteria 'disease severity' and 'type of therapeutic benefit' by all three groups, and negative scores on the economic-related criteria, according to all stakeholder groups.

CONCLUSIONS

MCDA can be used to elicit the views of different stakeholder groups and has the potential to structure and inform reimbursement decisions.

Using multi-criteria decision analysis to appraise orphan drugs: a systematic review

INTRODUCTION

Multi-criteria decision analysis (MCDA) could potentially solve current methodological difficulties in the appraisal of orphan drugs.

AREAS COVERED

We provide an overview of the existing evidence regarding the use of MCDA in the appraisal of orphan drugs worldwide. Three databases (Pubmed, Embase, Web of Science) were searched for English, French and German literature published between January 2000 and April 2017. Full-text articles were supplemented with conference abstracts. A total of seven articles and six abstracts were identified.

EXPERT COMMENTARY

The literature suggests that MCDA is increasingly being used in the context of appraising orphan drugs. It has shown itself to be a flexible approach with the potential to assist in decision-making regarding reimbursement for orphan drugs. However, further research regarding its application must be conducted.

Development of a Framework Based on Reflective MCDA to Support Patient-Clinician Shared Decision-Making: The Case of the Management of Gastroenteropancreatic Neuroendocrine Tumors (GEP-NET) in the United States

Introduction

Well- or moderately differentiated gastroenteropancreatic neuroendocrine tumors (GEP-NETs) are often slow-growing, and some patients with unresectable, asymptomatic, non-functioning tumors may face the choice between watchful waiting (WW), or somatostatin analogues (SSA) to delay progression. We developed a comprehensive multi-criteria decision analysis (MCDA) framework to help patients and physicians clarify their values and preferences, consider each decision criterion, and support communication and shared decision-making.

Methods

The framework was adapted from a generic MCDA framework (EVIDEM) with patient and clinician input. During a workshop, patients and clinicians expressed their individual values and preferences (criteria weights) and, on the basis of two scenarios (treatment vs WW; SSA-1 [lanreotide] vs SSA-2 [octreotide]) with evidence from a literature review, expressed how consideration of each criterion would impact their decision in favor of either option (score), and shared their knowledge and insights verbally and in writing.

Results

The framework included benefit-risk criteria and modulating factors, such as disease severity, quality of evidence, costs, and constraints. Overall and progression-free survival being most important, criteria weights ranged widely, highlighting variations in individual values and the need to share them. Scoring and considering each criterion prompted a rich exchange of perspectives and uncovered individual assumptions and interpretations. At the group level, type of benefit, disease severity, effectiveness, and quality of evidence favored treatment; cost aspects favored WW (scenario 1). For scenario 2, most criteria did not favor either option.

Conclusions

Patients and clinicians consider many aspects in decision-making. The MCDA framework provided a common interpretive frame to structure this complexity, support individual reflection, and share perspectives.

Funding

Ipsen Pharma.

Electronic supplementary material

The online version of this article (10.1007/s12325-017-0653-1) contains supplementary material, which is available to authorized users.

DRUG EVALUATION AND DECISION MAKING IN CATALONIA: DEVELOPMENT AND VALIDATION OF A METHODOLOGICAL FRAMEWORK BASED ON MULTI-CRITERIA DECISION ANALYSIS (MCDA) FOR ORPHAN DRUGS

OBJECTIVES

The aim of this study was to adapt and assess the value of a Multi-Criteria Decision Analysis (MCDA) framework (EVIDEM) for the evaluation of Orphan drugs in Catalonia (Catalan Health Service).

METHODS

The standard evaluation and decision-making procedures of CatSalut were compared with the EVIDEM methodology and contents. The EVIDEM framework was adapted to the Catalan context, focusing on the evaluation of Orphan drugs (PASFTAC program), during a Workshop with sixteen PASFTAC members. The criteria weighting was done using two different techniques (nonhierarchical and hierarchical). Reliability was assessed by re-test.

RESULTS

The EVIDEM framework and methodology was found useful and feasible for Orphan drugs evaluation and decision making in Catalonia. All the criteria considered for the development of the CatSalut Technical Reports and decision making were considered in the framework. Nevertheless, the framework could improve the reporting of some of these criteria (i.e., "unmet needs" or "nonmedical costs"). Some Contextual criteria were removed (i.e., "Mandate and scope of healthcare system", "Environmental impact") or adapted ("population priorities and access") for CatSalut purposes. Independently of the weighting technique considered, the most important evaluation criteria identified for orphan drugs were: "disease severity", "unmet needs" and "comparative effectiveness", while the "size of the population" had the lowest relevance for decision making. Test-retest analysis showed weight consistency among techniques, supporting reliability overtime.

CONCLUSIONS

MCDA (EVIDEM framework) could be a useful tool to complement the current evaluation methods of CatSalut, contributing to standardization and pragmatism, providing a method to tackle ethical dilemmas and facilitating discussions related to decision making.

Appraising the holistic value of Lenvatinib for radio-iodine refractory differentiated thyroid cancer: A multi-country study applying pragmatic MCDA

BACKGROUND

The objective of the study was to reveal through pragmatic MCDA (EVIDEM) the contribution of a broad range of criteria to the value of the orphan drug lenvatinib for radioiodine refractory differentiated thyroid cancer (RR-DTC) in country-specific contexts.

METHODS

The study was designed to enable comprehensive appraisal (12 quantitative, 7 qualitative criteria) in the current disease context (watchful waiting, sorafenib) of France, Italy and Spain. Data on the value of lenvatinib was collected from diverse stakeholders during country-specific panels and included: criteria weights (individual and social values); performance scores (judgments on evidence-collected through MCDA systematic review); qualitative impacts of contextual criteria; and verbal and written insights structured by criteria. The value contribution of each criterion was calculated and uncertainty explored.

RESULTS

Comparative effectiveness, Quality of evidence (Spain and Italy) and Disease severity (France) received the greatest weights. Four criteria contributed most to the value of lenvatinib, reflecting its superior Comparative effectiveness (16-22% of value), the severity of RR-DTC (16-22%), significant unmet needs (14-21%) and robust evidence (14-20%). Contributions varied by comparator, country and individuals, highlighting the importance of context and consultation. Results were reproducible at the group level. Impacts of contextual criteria varied across countries reflecting different health systems and cultural backgrounds. The MCDA process promoted sharing stakeholders' knowledge on lenvatinib and insights on context.

CONCLUSIONS

The value of lenvatinib was consistently positive across diverse therapeutic contexts. MCDA identified the aspects contributing most to value, revealed rich contextual insights, and helped participants express and explicitly tackle ethical trade-offs inherent to balanced appraisal and decisionmaking.

The Value of Medicines: A Crucial but Vague Concept

Health Technology Assessment is increasingly used to evaluate the value of healthcare products and to prioritize resources; however, defining exactly what value is and how it should be measured remains a challenge. In this article, we report the results of a literature review, focusing on nine European countries, with the aim of investigating how value is defined from the perspective of different stakeholders, how definitions of value are used, and how value is incorporated into decision making. Only three articles were identified that presented definitions of value, and there was no single shared definition of value in healthcare, which appears to be a highly subjective concept. The majority of the countries investigated combine clinical assessment with economic evaluation to make reimbursement recommendations; the quality-adjusted life-year is the most commonly used measure of value but does not capture broader aspects of value that may be important to patients and healthcare systems. We describe the use of value-based pricing and multi-criteria decision analysis, two approaches to the incorporation of broader aspects of value into decision making. Overall, we have identified considerable variation in how a product's value is defined by different stakeholders. Although a universal understanding of value in healthcare is important, it is clear that current definitions are insufficient, potentially leading to inconsistent reimbursement decisions. Ultimately, the establishment of clearer policies for defining and measuring value in healthcare is needed, and is likely to lead to improvements in the consistency of decision making.

Experimental measurement of preferences in health and healthcare using best-worst scaling: an overview

Best-worst scaling (BWS), also known as maximum-difference scaling, is a multiattribute approach to measuring preferences. BWS aims at the analysis of preferences regarding a set of attributes, their levels or alternatives. It is a stated-preference method based on the assumption that respondents are capable of making judgments regarding the best and the worst (or the most and least important, respectively) out of three or more elements of a choice-set. As is true of discrete choice experiments (DCE) generally, BWS avoids the known weaknesses of rating and ranking scales while holding the promise of generating additional information by making respondents choose twice, namely the best as well as the worst criteria. A systematic literature review found 53 BWS applications in health and healthcare. This article expounds possibilities of application, the underlying theoretical concepts and the implementation of BWS in its three variants: 'object case', 'profile case', 'multiprofile case'. This paper contains a survey of BWS methods and revolves around study design, experimental design, and data analysis. Moreover the article discusses the strengths and weaknesses of the three types of BWS distinguished and offered an outlook. A companion paper focuses on special issues of theory and statistical inference confronting BWS in preference measurement.

Using Best-Worst Scaling to Investigate Preferences in Health Care

INTRODUCTION

Best-worst scaling (BWS) is becoming increasingly popular to elicit preferences in health care. However, little is known about current practice and trends in the use of BWS in health care. This study aimed to identify, review and critically appraise BWS in health care, and to identify trends over time in key aspects of BWS.

METHODS

A systematic review was conducted, using Medline (via Pubmed) and EMBASE to identify all English-language BWS studies published up until April 2016. Using a predefined extraction form, two reviewers independently selected articles and critically appraised the study quality, using the Purpose, Respondents, Explanation, Findings, Significance (PREFS) checklist. Trends over time periods (≤2010, 2011, 2012, 2013, 2014 and 2015) were assessed further.

RESULTS

A total of 62 BWS studies were identified, of which 26 were BWS object case studies, 29 were BWS profile case studies and seven were BWS multi-profile case studies. About two thirds of the studies were performed in the last 2 years. Decreasing sample sizes and decreasing numbers of factors in BWS object case studies, as well as use of less complicated analytical methods, were observed in recent studies. The quality of the BWS studies was generally acceptable according to the PREFS checklist, except that most studies did not indicate whether the responders were similar to the non-responders.

CONCLUSION

Use of BWS object case and BWS profile case has drastically increased in health care, especially in the last 2 years. In contrast with previous discrete-choice experiment reviews, there is increasing use of less sophisticated analytical methods.

Measuring patients' priorities using the Analytic Hierarchy Process in comparison with Best-Worst-Scaling and rating cards: methodological aspects and ranking tasks

BACKGROUND

Identifying patient priorities and preference measurements have gained importance as patients claim a more active role in health care decision making. Due to the variety of existing methods, it is challenging to define an appropriate method for each decision problem. This study demonstrates the impact of the non-standardized Analytic Hierarchy Process (AHP) method on priorities, and compares it with Best-Worst-Scaling (BWS) and ranking card methods.

METHODS

We investigated AHP results for different Consistency Ratio (CR) thresholds, aggregation methods, and sensitivity analyses. We also compared criteria rankings of AHP with BWS and ranking cards results by Kendall's tau b.

RESULTS

The sample for our decision analysis consisted of 39 patients with rare diseases and mean age of 53.82 years. The mean weights of the two groups of CR ≤ 0.1 and CR ≤ 0.2 did not differ significantly. For the aggregation by individual priority (AIP) method, the CR was higher than for aggregation by individual judgment (AIJ). In contrast, the weights of AIJ were similar compared to AIP, but some criteria's rankings differed. Weights aggregated by geometric mean, median, and mean showed deviating results and rank reversals. Sensitivity analyses showed instable rankings. Moderate to high correlations between the rankings resulting from AHP and BWS.

LIMITATIONS

Limitations were the small sample size and the heterogeneity of the patients with different rare diseases.

CONCLUSION

In the AHP method, the number of included patients is associated with the threshold of the CR and choice of the aggregation method, whereas both directions of influence could be demonstrated. Therefore, it is important to implement standards for the AHP method. The choice of method should depend on the trade-off between the burden for participants and possibilities for analyses.

Patient-Focused Benefit-Risk Analysis to Inform Regulatory Decisions: The European Union Perspective

Regulatory decisions are often based on multiple clinical end points, but the perspectives used to judge the relative importance of those end points are predominantly those of expert decision makers rather than of the patient. However, there is a growing awareness that active patient and public participation can improve decision making, increase acceptance of decisions, and improve adherence to treatments. The assessment of risk versus benefit requires not only information on clinical outcomes but also value judgments about which outcomes are important and whether the potential benefits outweigh the harms. There are a number of mechanisms for capturing the input of patients, and regulatory bodies within the European Union are participating in several initiatives. These can include patients directly participating in the regulatory decision-making process or using information derived from patients in empirical studies as part of the evidence considered. One promising method that is being explored is the elicitation of "patient preferences." Preferences, in this context, refer to the individual's evaluation of health outcomes and can be understood as statements regarding the relative desirability of a range of treatment options, treatment characteristics, and health states. Several methods for preference measurement have been proposed, and pilot studies have been undertaken to use patient preference information in regulatory decision making. This article describes how preferences are currently being considered in the benefit-risk assessment context, and shows how different methods of preference elicitation are used to support decision making within the European context.

Multiple Criteria Decision Analysis for Health Care Decision Making--Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force

Health care decisions are complex and involve confronting trade-offs between multiple, often conflicting objectives. Using structured, explicit approaches to decisions involving multiple criteria can improve the quality of decision making. A set of techniques, known under the collective heading, multiple criteria decision analysis (MCDA), are useful for this purpose. In 2014, ISPOR established an Emerging Good Practices Task Force. The task force's first report defined MCDA, provided examples of its use in health care, described the key steps, and provided an overview of the principal methods of MCDA. This second task force report provides emerging good-practice guidance on the implementation of MCDA to support health care decisions. The report includes: a checklist to support the design, implementation and review of an MCDA; guidance to support the implementation of the checklist; the order in which the steps should be implemented; illustrates how to incorporate budget constraints into an MCDA; provides an overview of the skills and resources, including available software, required to implement MCDA; and future research directions.

Can the EVIDEM Framework Tackle Issues Raised by Evaluating Treatments for Rare Diseases: Analysis of Issues and Policies, and Context-Specific Adaptation

BACKGROUND

The multiplicity of issues, including uncertainty and ethical dilemmas, and policies involved in appraising interventions for rare diseases suggests that multicriteria decision analysis (MCDA) based on a holistic definition of value is uniquely suited for this purpose. The objective of this study was to analyze and further develop a comprehensive MCDA framework (EVIDEM) to address rare disease issues and policies, while maintaining its applicability across disease areas.

METHODS

Specific issues and policies for rare diseases were identified through literature review. Ethical and methodological foundations of the EVIDEM framework v3.0 were systematically analyzed from the perspective of these issues, and policies and modifications of the framework were performed accordingly to ensure their integration.

RESULTS

Analysis showed that the framework integrates ethical dilemmas and issues inherent to appraising interventions for rare diseases but required further integration of specific aspects. Modification thus included the addition of subcriteria to further differentiate disease severity, disease-specific treatment outcomes, and economic consequences of interventions for rare diseases. Scoring scales were further developed to include negative scales for all comparative criteria. A methodology was established to incorporate context-specific population priorities and policies, such as those for rare diseases, into the quantitative part of the framework. This design allows making more explicit trade-offs between competing ethical positions of fairness (prioritization of those who are worst off), the goal of benefiting as many people as possible, the imperative to help, and wise use of knowledge and resources. It also allows addressing variability in institutional policies regarding prioritization of specific disease areas, in addition to existing uncertainty analysis available from EVIDEM.

CONCLUSION

The adapted framework measures value in its widest sense, while being responsive to rare disease issues and policies. It provides an operationalizable platform to integrate values, competing ethical dilemmas, and uncertainty in appraising healthcare interventions.

Making Good Decisions in Healthcare with Multi-Criteria Decision Analysis: The Use, Current Research and Future Development of MCDA

Healthcare decision making is usually characterized by a low degree of transparency. The demand for transparent decision processes can be fulfilled only when assessment, appraisal and decisions about health technologies are performed under a systematic construct of benefit assessment. The benefit of an intervention is often multidimensional and, thus, must be represented by several decision criteria. Complex decision problems require an assessment and appraisal of various criteria; therefore, a decision process that systematically identifies the best available alternative and enables an optimal and transparent decision is needed. For that reason, decision criteria must be weighted and goal achievement must be scored for all alternatives. Methods of multi-criteria decision analysis (MCDA) are available to analyse and appraise multiple clinical endpoints and structure complex decision problems in healthcare decision making. By means of MCDA, value judgments, priorities and preferences of patients, insurees and experts can be integrated systematically and transparently into the decision-making process. This article describes the MCDA framework and identifies potential areas where MCDA can be of use (e.g. approval, guidelines and reimbursement/pricing of health technologies). A literature search was performed to identify current research in healthcare. The results showed that healthcare decision making is addressing the problem of multiple decision criteria and is focusing on the future development and use of techniques to weight and score different decision criteria. This article emphasizes the use and future benefit of MCDA.

[Involving patients, the insured and the general public in healthcare decision making]

No doubt, the public should be involved in healthcare decision making, especially when decision makers from politics and self-government agencies are faced with the difficult task of setting priorities. There is a general consensus on the need for a stronger patient centeredness, even in HTA processes, and internationally different ways of public participation are discussed and tested in decision making processes. This paper describes how the public can be involved in different decision situations, and it shows how preference measurement methods are currently being used in an international context to support decision making. It distinguishes between different levels of decision making on health technologies: approval, assessment, pricing, and finally utilization. The range of participation efforts extends from qualitative surveys of patients' needs (Citizen Councils of NICE in the UK) to science-based documentation of quantitative patient preferences, such as in the current pilot projects of the FDA in the US and the EMA at the European level. Possible approaches for the elicitation and documentation of preference structures and trade-offs in relation to alternate health technologies are decision aids, such as multi-criteria decision analysis (MCDA), that provide the necessary information for weighting and prioritizing decision criteria.