A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions

Design of a multiple criteria decision analysis framework for prioritizing high-impact health technologies in a regional health service

OBJECTIVES

This study aims to develop a framework for establishing priorities in the regional health service of Murcia, Spain, to facilitate the creation of a comprehensive multiple criteria decision analysis (MCDA) framework. This framework will aid in decision-making processes related to the assessment, reimbursement, and utilization of high-impact health technologies.

METHOD

Based on the results of a review of existing frameworks for MCDA of health technologies, a set of criteria was proposed to be used in the context of evaluating high-impact health technologies. Key stakeholders within regional healthcare services, including clinical leaders and management personnel, participated in a focus group (n = 11) to discuss the proposed criteria and select the final fifteen. To elicit the weights of the criteria, two surveys were administered, one to a small sample of healthcare professionals (n = 35) and another to a larger representative sample of the general population (n = 494).

RESULTS

The responses obtained from health professionals in the weighting procedure exhibited greater consistency compared to those provided by the general public. The criteria more highly weighted were "Need for intervention" and "Intervention outcomes." The weights finally assigned to each item in the multicriteria framework were derived as the equal-weighted sum of the mean weights from the two samples.

CONCLUSIONS

A multi-attribute function capable of generating a composite measure (multicriteria) to assess the value of high-impact health interventions has been developed. Furthermore, it is recommended to pilot this procedure in a specific decision context to evaluate the efficacy, feasibility, usefulness, and reliability of the proposed tool.

Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013-2023)

Uncertainty estimation in healthcare involves quantifying and understanding the inherent uncertainty or variability associated with medical predictions, diagnoses, and treatment outcomes. In this era of Artificial Intelligence (AI) models, uncertainty estimation becomes vital to ensure safe decision-making in the medical field. Therefore, this review focuses on the application of uncertainty techniques to machine and deep learning models in healthcare. A systematic literature review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our analysis revealed that Bayesian methods were the predominant technique for uncertainty quantification in machine learning models, with Fuzzy systems being the second most used approach. Regarding deep learning models, Bayesian methods emerged as the most prevalent approach, finding application in nearly all aspects of medical imaging. Most of the studies reported in this paper focused on medical images, highlighting the prevalent application of uncertainty quantification techniques using deep learning models compared to machine learning models. Interestingly, we observed a scarcity of studies applying uncertainty quantification to physiological signals. Thus, future research on uncertainty quantification should prioritize investigating the application of these techniques to physiological signals. Overall, our review highlights the significance of integrating uncertainty techniques in healthcare applications of machine learning and deep learning models. This can provide valuable insights and practical solutions to manage uncertainty in real-world medical data, ultimately improving the accuracy and reliability of medical diagnoses and treatment recommendations.

How is return on investment from quality improvement programmes conceptualised by mental healthcare leaders and why: a qualitative study

BACKGROUND

Return on Investment (ROI), whereby the ratio of costs to benefits is assessed, is encouraged in-order to justify the value of Quality Improvement (QI) programmes. We previously performed a literature review to develop a ROI conceptual framework for QI programmes. We concluded that, QI-ROI is conceptualised as any monetary and non-monetary benefit. In the current study, we explored if this finding is shared by mental healthcare leaders. We also investigated the stability of this conceptualisation against influencing factors and potential for disinvestment.

METHODS

We performed qualitative interviews with leaders in an NHS mental health organisation. There were 16 participants: nine board members and seven senior leaders. The interviews were held online via Microsoft Teams and lasted an hour on average. We performed deductive-inductive analysis to seek data from our initial ROI framework and any new data.

RESULTS

We found that in mental healthcare, QI-ROI is also conceptualised as any valued monetary and non-monetary benefits. There was a strong emphasis on benefits to external partners and a de-emphasis of benefit monetisation. This conceptualisation was influenced by the 1) perceived mandates to improve quality and manage scarce resources, 2) expectations from QI, 3) health and social care values, 4) ambiguity over expectations, and 5) uncertainty over outcomes. Uncertainty, ambiguity, and potential for disinvestment posed a threat to the stability of this conceptualisation but did not ultimately change it. Health and social care values supported maintaining the QI-ROI as any benefit, with a focus on patients and staff outcomes. Socio-political desires to improve quality were strong drivers for QI investment.

CONCLUSION

Mental healthcare leaders primarily conceptualise QI-ROI as any valued benefit. The inclusion of externalised outcomes which are hard to attribute may be challenging. However, mental healthcare services do collaborate with external partners. The de-emphases of benefit monetisation may also be controversial due to the need for financial accountability. Mental healthcare leaders recognise the importance of efficiency savings. However, they raised concerns over the legitimacy and utility of traditional ROI as a tool for assessing QI value. Further research is needed to bring more clarity on these aspects of the QI-ROI concept.

The Use of Multicriteria Decision Analysis to Support Decision Making in Healthcare: An Updated Systematic Literature Review

OBJECTIVES

Multicriteria decision analysis (MCDA) is increasingly used for decision making in healthcare. However, its application in different decision-making contexts is still unclear. This study aimed to provide a comprehensive review of MCDA studies performed to inform decisions in healthcare and to summarize its application in different decision contexts.

METHODS

We updated a systematic review conducted in 2013 by searching Embase, MEDLINE, and Google Scholar for MCDA studies in healthcare, published in English between August 2013 and November 2020. We also expanded the search by reviewing grey literature found via Trip Medical Database and Google, published between January 1990 and November 2020. A comprehensive template was developed to extract information about the decision context, criteria, methods, stakeholders involved, and sensitivity analyses conducted.

RESULTS

From the 4295 identified studies, 473 studies were eligible for full-text review after assessing titles and abstracts. Of those, 228 studies met the inclusion criteria and underwent data extraction. The use of MCDA continues to grow in healthcare literature, with most of the studies (49%) informing priority-setting decisions. Safety, cost, and quality of care delivery are the most frequently used criteria, although there are considerable differences across decision contexts. Almost half of the MCDA studies used the linear additive model whereas scales and the analytical hierarchy process were the most used techniques for scoring and weighting, respectively. Not all studies report on each one of the MCDA steps, consider axiomatic properties, or justify the methods used.

CONCLUSIONS

A guide on how to conduct and report MCDA that acknowledges the particularities of the different decision contexts and methods needs to be developed.

Regulatory requirements and optimization of multiple criteria decision analysis to quantify the benefit-risk assessment of medical devices

INTRODUCTION

Worldwide medical device regulatory authorities increasingly rely on the benefit-risk ratio for decision-making. However, current benefit-risk assessment (BRA) methods are mostly descriptive, not quantitative.

AREAS COVERED

We aimed to summarize the regulatory requirements of BRA, discuss the feasibility of adopting multiple criteria decision analysis (MCDA), and explore factors for optimizing the MCDA for quantitative BRA of devices.

EXPERT OPINION

Regulatory organizations emphasize BRA in their guidance, and some recommend user-friendly worksheets to conduct qualitative/descriptive BRA. The MCDA is considered one of the most useful and relevant quantitative BRA methods by pharmaceutical regulatory agencies and the industry; the International Society for Pharmacoeconomics and Outcomes Research summarized the principles and good practice guidance of MCDA. We recommend optimizing the MCDA by considering the following unique characteristics of the device BRA: using data from state of the art as a control and clinical data from post-market surveillance and literature; considering the device's diverse characteristics when selecting controls; assigning weight according to type, magnitude/severity, and duration of benefits and risks; and including physician and patient opinions in the MCDA. This article is the first to explore using MCDA for device BRA and might lead to a novel quantitative BRA method for devices.

Determination of phage load and administration time in simulated occurrences of antibacterial treatments

The use of phages as antibacterials is becoming more and more common in Western countries. However, a successful phage-derived antibacterial treatment needs to account for additional features such as the loss of infective virions and the multiplication of the hosts. The parameters critical inoculation size (V F ) and failure threshold time (T F ) have been introduced to assure that the viral dose (V ϕ) and administration time (T ϕ) would lead to the extinction of the targeted bacteria. The problem with the definition of V F and T F is that they are non-linear equations with two unknowns; thus, obtaining their explicit values is cumbersome and not unique. The current study used machine learning to determine V F and T F for an effective antibacterial treatment. Within these ranges, a Pareto optimal solution of a multi-criterial optimization problem (MCOP) provided a pair of V ϕ and T ϕ to facilitate the user's work. The algorithm was tested on a series of in silico microbial consortia that described the outgrowth of a species at high cell density by another species initially present at low concentration. The results demonstrated that the MCOP-derived pairs of V ϕ and T ϕ could effectively wipe out the bacterial target within the context of the simulation. The present study also introduced the concept of mediated phage therapy, where targeting booster bacteria might decrease the virulence of a pathogen immune to phagial infection and highlighted the importance of microbial competition in attaining a successful antibacterial treatment. In summary, the present work developed a novel method for investigating phage/bacteria interactions that can help increase the effectiveness of the application of phages as antibacterials and ease the work of microbiologists.

Effect of COVID-19 on Selected Characteristics of Life Satisfaction Reflected in a Fuzzy Model

The general goal of the research in this article is to devise an algorithm for assessing overall life satisfaction—a term often referred to as Quality of Life (QoL). It is aggregated to its own proposition, called personal life usual satisfaction (PLUS). An important assumption here is that the model is based on already known and commonly used solutions, such as medical (psychological and physiotherapeutic) questionnaires. Thanks to this, the developed solution allows us to obtain a synergy effect from the existing knowledge, without the need to design new, complicated procedures. Fuzzy multivariate characterization of life satisfaction presents a challenge for a complete analysis of the phenomenon. The complexity of description using multiple scales, including linguistic, requires additional computational solutions, as presented in this paper. The detailed aim of this study is twofold: (1) to develop a fuzzy model reflecting changes in life satisfaction test scores as influenced by the corona virus disease 2019 (COVID-19) pandemic, and (2) to develop guidelines for further research on more advanced models that are clinically useful. Two groups affected by professional burnout to different degrees were analyzed toward life satisfaction twice (pre- and during pandemy): a study group (physiotherapists, n=25) and a reference group (computer scientists, n=25). The Perceived Stress Score (PSS10), Maslach Burnout Inventory (MBI), Satisfaction with Life Scale (SWLS), and Nordic Musculoskeletal Questionnaire (NMQ) were used. The resultant model is based on a hierarchical fuzzy system. The novelty of the proposed approach lies in the combination of the use of data from validated clinimetric tests with the collection of data from characteristic time points and the way in which they are analyzed using fuzzy logic through transparent and scalable hierarchical models. To date, this approach is unique and has no equivalent in the literature. Thanks to the hierarchical structure, the evaluation process can be defined as a modular construction, which increases transparency and makes the whole procedure more flexible.

Multiple Criteria Decision Analysis (MCDA) for evaluating cancer treatments in hospital-based health technology assessment: The Paraconsistent Value Framework

Background

In recent years, the potential of multi-criteria decision analysis (MCDA) in the health field has been discussed widely. However, most MCDA methodologies have given little attention to the aggregation of different stakeholder individual perspectives.

Objective

To illustrate how a paraconsistent theory-based MCDA reusable framework, designed to aid hospital-based Health Technology Assessment (HTA), could be used to aggregate individual expert perspectives when valuing cancer treatments.

Methods

An MCDA methodological process was adopted based on paraconsistent theory and following ISPOR recommended steps in conducting an MCDA study. A proof-of-concept exercise focusing on identifying and assessing the global value of first-line treatments for metastatic colorectal cancer (mCRC) was conducted to foster the development of the MCDA framework.

Results

On consultation with hospital-based HTA committee members, 11 perspectives were considered in an expert panel: medical oncology, oncologic surgery, radiotherapy, palliative care, pharmacist, health economist, epidemiologist, public health expert, health media expert, pharmaceutical industry, and patient advocate. The highest weights were assigned to the criteria “overall survival” (mean 0.22), “burden of disease” (mean 0.21) and “adverse events” (mean 0.20), and the lowest weights were given to “progression-free survival” and “cost of treatment” (mean 0.18 for both). FOLFIRI and mFlox scored the highest global value score of 0.75, followed by mFOLFOX6 with a global value score of 0.71. mIFL was ranked last with a global value score of 0.62. The paraconsistent analysis (para-analysis) of 6 first-line treatments for mCRC indicated that FOLFIRI and mFlox were the appropriate options for reimbursement in the context of this study.

Conclusion

The Paraconsistent Value Framework is proposed as a step beyond the current MCDA practices, in order to improve means of dealing with individual expert perspectives in hospital-based HTA of cancer treatments.

Optimisation of maintenance in delivery systems for cytostatic medicines

BACKGROUND

The real-world application of maintenance in organisations brings together a number of maintenance policies in order to achieve the desired availability, efficiency and profitability. However, the literature mostly chooses a single maintenance policy, and so the decision process is not suited to the real conditions in the company to which it is applied. Our study takes a combination of maintenance policies as alternatives, and so conforms to the actual practice of maintenance in organisations. Furthermore, it introduces the possibility of including extra spare parts, or outsourcing maintenance policies. Although the selection of maintenance policies has been applied to many kinds of business and of machine, there is almost no instance of its application to hospitals, and it has never been applied to delivery systems for cytostatic drugs.

METHODS

The model uses the fuzzy Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), which is recognised as being highly suitable for solving group decision-making problems in a fuzzy environment. Fuzzy set theory is also considered to be more proficient than crisp numbers for handling the ambiguity, imprecisions, data scarcity, and uncertainty inherent in decisions made by human beings. The judgements required were obtained from a decision group comprising the heads of facilities maintenance, maintenance of medical equipment, health and safety at work, environment, and programming-admission. The group also included care staff; specifically, the heads of the main clinical services, and the medical supervisors. The model includes original criteria, such as Quality of health care, which measures impact on care as a function of mean availability of each alternative. It also considers Impact on hospital management via the criteria: Working environment in the organisation and Impact on health care; the former criterion measures equality among care services in the hospital, while the latter assesses the effect on regional health cover. The model was built using real data obtained from a state hospital in Spain. The model can also be easily applied to other national and international healthcare organisations, providing weights specific to the criteria. These are produced by a decision group from each healthcare organisation and the alternatives are updated in accordance with what is considered important in each hospital.

RESULTS

The results obtained from the model recommend changing the alternative that is currently in use, Corrective and Preventive Maintenance, to Corrective and Preventive Maintenance plus two spare hoods. This alternative would lead to an availability of 1 (the highest possible) in the systems for preparing personalised cytotoxic drugs, and so the quality of service is therefore very high. Additionally, it could offer services to all the users of the hospital, and also offer cover in the preparation of cytotoxic medicines to other hospitals in the catchment area.

CONCLUSIONS

The results suggest the possibility that improvements to the support and logistical systems, which include maintenance, traditionally held to have no effect on quality of care, may be key to improving care quality, but also in reducing risk to patients, care and non-care staff, and the environment.

Simulation-based multi-criteria decision making: an interactive method with a case study on infectious disease epidemics

Whenever a system needs to be operated by a central decision making authority in the presence of two or more conflicting goals, methods from multi-criteria decision making can help to resolve the trade-offs between these goals. In this work, we devise an interactive simulation-based methodology for planning and deciding in complex dynamic systems subject to multiple objectives and parameter uncertainty. The outline intermittently employs simulation models and global sensitivity analysis methods in order to facilitate the acquisition of system-related knowledge throughout the iterations. Moreover, the decision maker participates in the decision making process by interactively adjusting control variables and system parameters according to a guiding analysis question posed for each iteration. As a result, the overall decision making process is backed up by sensitivity analysis results providing increased confidence in terms of reliability of considered decision alternatives. Using the efficiency concept of Pareto optimality and the sensitivity analysis method of Sobol' sensitivity indices, the methodology is then instantiated in a case study on planning and deciding in an infectious disease epidemic situation similar to the 2020 coronavirus pandemic. Results show that the presented simulation-based methodology is capable of successfully addressing issues such as system dynamics, parameter uncertainty, and multi-criteria decision making. Hence, it represents a viable tool for supporting decision makers in situations characterized by time dynamics, uncertainty, and multiple objectives.

Economic Evaluation of New Models of Care: Does the Decision Change Between Cost-Utility Analysis and Multi-Criteria Decision Analysis?

OBJECTIVES

To experiment with new approaches of collaboration in healthcare delivery, local authorities implement new models of care. Regarding the local decision context of these models, multi-criteria decision analysis (MCDA) may be of added value to cost-utility analysis (CUA), because it covers a wider range of outcomes. This study compares the 2 methods using a side-by-side application.

METHODS

A new Dutch model of care, Primary Care Plus (PC+), was used as a case study to compare the results of CUA and MCDA. Data of patients referred to PC+ or care-as-usual were retrieved by questionnaires and administrative databases with a 3-month follow-up. Propensity score matching together with generalized linear regression models was used to reduce confounding. Univariate and probabilistic sensitivity analyses were performed to explore uncertainty in the results.

RESULTS

Although both methods indicated PC+ as the dominant alternative, complementary differences were observed. MCDA provided additional evidence that PC+ improved access to care (standardized performance score of 0.742 vs 0.670) and that improvement in health-related quality of life was driven by the psychological well-being component (standardized performance score of 0.710 vs 0.704). Furthermore, MCDA estimated the budget required for PC+ to be affordable in addition to preferable (€521.42 per patient). Additionally, MCDA was less sensitive to the utility measures used.

CONCLUSIONS

MCDA may facilitate an auditable and transparent evaluation of new models of care by providing additional information on a wider range of outcomes and incorporating affordability. However, more effort is needed to increase the usability of MCDA among local decision makers.

Multi-criteria Decision Analysis Software in Healthcare Priority Setting: A Systematic Review

OBJECTIVE

The objectives of this systematic review were to identify studies using Multi-Criteria Decision Analysis (MCDA) software tools to support health prioritisation processes and describe the technical capabilities of the MCDA software tools identified.

METHODS

First, a systematic literature review was conducted in the MEDLINE, EMBASE, Web of Science, EconLit and Cochrane databases in July 2019 to identify studies that have used MCDA software for priority setting in health-related problems. Second, the MCDA software tools found in the review were downloaded (full versions, where freely available, and trial versions otherwise) and tested to extract their key technical characteristics.

RESULTS

Nine studies were included, from which seven different software tools, 1000minds^®, M-MACBETH, Socio Technical Allocation of Resources (STAR), Strategic Multi-Attribute Ranking Tool (SMART), Visual PROMETHEE, EVIDEM and the Prioritisation Framework, were identified. These software tools differed in terms of the operating systems (including web interface), MCDA technique(s) available for use, visualisation features, and the capability to perform Value for Money (VfM) and sensitivity analyses.

CONCLUSIONS

The use of MCDA software in prioritisation processes has a number of advantages such as inclusion of several types of stakeholders and the ability to analyse a greater number of alternatives and criteria and perform real-time sensitivity analyses. Proprietary software (i.e. software with licensing fees) seemed to have more features than freely available software. However, this field is still developing, with only a few studies where MCDA software was used to support health priority setting and opportunity costs not explicitly captured in many software tools.

A prospective decision-impact study incorporating Breast Cancer Index into extended endocrine therapy decision-making

Aim: To prospectively assess the impact of gene expression-based assay Breast Cancer Index (BCI) on extended endocrine therapy (EET) decision-making. Patients & methods: The BCI-tested samples from primary tumors (Stage I–III, hormone receptor positive breast cancer, >3.5 year endocrine therapy). Patients and physicians completed questionnaires on EET preferences and decision conflict. Using these data, a fact-based economic model was developed to project the cost impact of BCI. Results: The BCI results affected treatment recommendations for 42/141 patients (overall mean, 62 year; 83% postmenopausal; 63% Stage I). Patient decision conflict decreased pre- to post-test. The BCI-related projected net savings (US$5190/patient) was robust under sensitivity analysis. Conclusion: Incorporating BCI into clinical practice meaningfully impacted physician EET recommendations and decreased patient decision conflict, with projected cost savings.

Comparison of weighting methods used in multicriteria decision analysis frameworks in healthcare with focus on low- and middle-income countries

AIM

Criteria weighting is a key element of multicriteria decision analysis that is becoming extensively used in healthcare decision-making. In our narrative review we describe the advantages and disadvantages of various weighting methods.

METHODS

An assessment of the eight identified primary criteria weighting methods was compiled on domains including their resource requirements, and potential for bias.

RESULTS

In general, we found more complex methods to have less potential for bias; however, resource intensity and general participant burden is greater for these methods.

CONCLUSION

The selection of the most appropriate method depends on the decision-making context. The simple multiattribute rating technique (SMART) method combined with swing-weighting technique and the analytic hierarchy process methods may be the most feasible approaches for low- and middle-income countries.

Development of multi-criteria decision analysis (MCDA) framework for off-patent pharmaceuticals - an application on improving tender decision making in Indonesia

Background

Off-patent pharmaceuticals (OPPs) hold vital importance in meeting public health objectives, especially in developing countries where resources are limited. OPPs are comprised of off-patent originals, branded generics and unbranded generics; nonetheless, these products are not identical and often there are differences in their equivalence, manufacturing quality standards and reliability of supply. This necessitates reconsideration of the lowest price policy objective in pharmaceutical decision making. The aim of this study was to develop a Multi-Criteria Decision Analysis (MCDA) framework through a pilot workshop to inform the national procurement of OPPs in Indonesia.

Methods

An initial list of potentially relevant criteria was identified based on previous work and a literature review. In a 2-day pilot policy workshop, twenty local experts representing different stakeholder groups and decision-making bodies selected the final criteria, approved the scoring function for each criterion, and assigned weights to each criterion.

Results

An MCDA framework was proposed for OPP drug decision making in developing countries, which included price and 8 non-price criteria. Based on the pilot policy workshop 6 + 1 criteria were considered relevant for Indonesia: pharmaceutical price (40% weight), manufacturing quality (18.8%), equivalence with the reference product (12.2%), product stability and drug formulation (12.2%), reliability of drug supply (8.4%), real world clinical or economic outcomes, such as adherence or non-drug costs (4.2%) and pharmacovigilance (3.6%).

Conclusions

According to the pilot policy workshop, other criteria apart from price need to be strengthened in the tendering process. The introduction of additional criteria for OPP procurement in an MCDA framework creates incentives for manufacturers to invest into improved manufacturing standards, equivalence proof, product quality, reliability of supply or even additional real-world data collection, which ultimately may result in more health gain for the society.

Barriers for Access to New Medicines: Searching for the Balance Between Rising Costs and Limited Budgets

Introduction: There is continued unmet medical need for new medicines across countries especially for cancer, immunological diseases, and orphan diseases. However, there are growing challenges with funding new medicines at ever increasing prices along with funding increased medicine volumes with the growth in both infectious diseases and non-communicable diseases across countries. This has resulted in the development of new models to better manage the entry of new medicines, new financial models being postulated to finance new medicines as well as strategies to improve prescribing efficiency. However, more needs to be done. Consequently, the primary aim of this paper is to consider potential ways to optimize the use of new medicines balancing rising costs with increasing budgetary pressures to stimulate debate especially from a payer perspective.

Methods: A narrative review of pharmaceutical policies and implications, as well as possible developments, based on key publications and initiatives known to the co-authors principally from a health authority perspective.

Results: A number of initiatives and approaches have been identified including new models to better manage the entry of new medicines based on three pillars (pre-, peri-, and post-launch activities). Within this, we see the growing role of horizon scanning activities starting up to 36 months before launch, managed entry agreements and post launch follow-up. It is also likely there will be greater scrutiny over the effectiveness and value of new cancer medicines given ever increasing prices. This could include establishing minimum effectiveness targets for premium pricing along with re-evaluating prices as more medicines for cancer lose their patent. There will also be a greater involvement of patients especially with orphan diseases. New initiatives could include a greater role of multicriteria decision analysis, as well as looking at the potential for de-linking research and development from commercial activities to enhance affordability.

Conclusion: There are a number of ongoing activities across countries to try and fund new valued medicines whilst attaining or maintaining universal healthcare. Such activities will grow with increasing resource pressures and continued unmet need.

Assessing Lung Cancer Screening Programs under Uncertainty in a Heterogeneous Population

BACKGROUND

Lung cancer screening can reduce cancer mortality. Most implementation studies focus only on low-dose computed tomography (LDCT) and clinical attributes of screening and do not include preferences of potential participants. In this study we evaluated the perceived value of screening programs based on LDCT, breath analysis (BA), or blood biomarkers (BB) according to the perspective of the target population.

METHODS

A multi-criteria decision analysis framework was adopted. The weights of seven attributes of screening (sensitivity, specificity, radiation burden, duration of screening process, waiting time until results are communicated, location of screening, and mode of screening) were obtained from an earlier study that included a broad sample from the Netherlands. Performance data for the screening modalities was obtained from clinical trials and expert opinion. Parameter uncertainty about clinical performances was incorporated probabilistically, while heterogeneity in preferences was analyzed through subgroup analyses.

RESULTS

The mean overall values were 0.58 (CI: 0.57 to 0.59), 0.57 (CI: 0.56 to 0.59), and 0.44 (CI: 0.43 to 0.45) for BB, BA, and LDCT, respectively. Seventy-seven per cent of respondents preferred BB or BA. For most subgroups, the overall values were similar to those of the entire sample. BA had the highest value for respondents who would have been eligible for earlier screening trials.

DISCUSSION

BB and BA seem valuable to participants because they can be applied in a primary care setting. Although LDCT still seems preferable given its strong and positive evidence base, it is important to take non-clinical attributes into account to maximize attendance.

Rapid methods including network meta-analysis to produce evidence in clinical decision support: a decision analysis

BACKGROUND

Conducting systematic reviews is time-consuming but crucial to construct evidence-based patient decision aids, clinical practice guidelines and decision analyses. New methods might enable developers to produce a knowledge base more rapidly. However, trading off scientific rigour for speed when creating a knowledge base is controversial, and the consequences are insufficiently known. We developed and applied faster methods including systematic reviews and network meta-analyses, assessed their feasibility and compared them to a gold standard approach. We also assessed the feasibility of using decision analysis to perform this comparison.

METHODS

Long-term treatment in bipolar disorder was our testing field. We developed two new methods: an empirically based, rapid network meta-analysis (NMA) and an expert NMA, and conducted a patient survey. We applied these methods to collect effect estimates for evidence-based treatments on outcomes important to patients. The relative importance of outcomes was obtained from patients using a stated preference method. We used multi-criteria decision analysis to compare a gold standard NMA with the rapid NMA in terms of the ability of the gold standard NMA to change the ranking and expected values of treatments for individual patients.

RESULTS

Using rapid methods, it was feasible to identify evidence addressing outcomes important to patients. We found that replacing effect estimates from our rapid NMA with estimates from the gold standard NMA resulted in relatively small changes in the ranking and expected value of treatments. The rapid method sufficed to estimate the effects of nine out of ten options. To produce a ranking of treatments accurate for more than 95% of patients, it was necessary to supplement systematic with rapid methods and to use relative importance weights in the analysis. Integrating estimates of the outcome "treatment burden" had a larger impact on rankings than replacing rapid with gold standard methods. Using patients' importance weights only modestly affected results.

CONCLUSIONS

The transfer of knowledge to practice could benefit from faster systematic reviewing methods. The results in this preliminary assessment suggest that an improved rapid NMA approach might replace gold standard NMAs. Decision analysis could be used to compare evidence summarisation methods.

Multi-Criteria Decision Analysis (MCDA) Models in Health Technology Assessment of Orphan Drugs-a Systematic Literature Review. Next Steps in Methodology Development?

Background: Multi-criteria decision analysis (MCDA) is a decision-making tool that can take into account multidimensional factors and enables comparison of (medical) technologies by combining individual criteria into one overall appraisal. The MCDA approach has slowly gained traction within Health Technology Assessment (HTA) and its elements are gradually being incorporated into HTA across Europe. Several groups of scientists have proposed MCDA approaches targeted toward orphan drugs and rare diseases by including criteria specific to rare diseases. The goal of this article is to provide an overview of the current state of knowledge and latest developments in the field of MCDA in HTA for orphan drugs, to review existing models, their design characteristics, as well as to identify opportunities for further model improvement. Methods: A systematic literature search was conducted in January 2018 using four databases: MEDLINE (Pubmed), EBSCO HOST, EMBASE, and Web of science to find publications related to use of MCDA in the rare disease field (keywords: MCDA/orphan drug/rare disease and synonyms). Identified MCDA models were analyzed, e.g., structure, criteria, scoring, and weighting methodology. Results: Two hundred and eleven publications were identified, of which 29 were included after removal of duplicates. 9 authors developed own MCDA models, 7 of which based on literature reviews intended to identify the most important and relevant decision criteria in the model. In 13 publications (8 models) weights were assigned to criteria based on stakeholder input. The most commonly chosen criteria for creation of the MCDA models were: comparative effectiveness/efficacy, the need for intervention, and disease severity. Some models have overlapping criteria, especially in the treatment cost and effectiveness areas. Conclusions: A range of MCDA models for HTA have been developed, each with a slightly different approach, focus, and complexity, including several that specifically target rare diseases and orphan drug appraisal. Models have slowly progressed over the years based on pilots, stakeholder input, sharing experiences and scientific publications. However, full consensus on model structure, criteria selection and weighting is still lacking. A simplification of the MCDA model approach may increase its acceptance. A multi-stakeholder discussion on fundamental design and implementation strategies for MCDA models would be beneficial to this end.

Multicriteria decision analysis (MCDA) for health technology assessment: the Queensland Health experience

Objectives In determining whether new health technologies should be funded, health technology assessment (HTA) committees prefer explicit to implicit methods of analysis in enhancing transparency and consistency of decision making. The aim of this study was to develop and pilot a multicriteria decision analysis (MCDA) framework for the Queensland Department of Health HTA program committee, which weighted decision making criteria according to their perceived importance as determined by group consensus. Methods The criteria used in the MCDA framework were identified by reviewing the five unweighted criteria used in the existing process, consultation with committee members and literature review. Criteria were clearly defined and ordinal categories of lowest to highest preferred were assigned against which technology submissions would be rated. Criteria weights were determined through a discrete choice experiment (DCE) survey of committee members using validated software. Mean weighted technology scores were then used to guide deliberative discussions in determining final funding decisions. Results The MCDA framework created one additional criterion to the previous five. The criteria and their mean weights identified through the DCE survey were clinical benefit and safety (27.2%), quality of evidence (19.2%), implementation capacity (16.9%), innovation (15.4%), burden of disease and clinical need (13.3%) and societal and ethical values (8.0%). Criterion weights varied considerably between individual committee members, with one criterion having a difference of 36.9% between the highest and lowest preference weights. Following deliberative discussions, all but one of 10 submissions were awarded funding. The submission not supported received the third lowest score through the MCDA model. Conclusions This pilot application of an MCDA framework, as a complement to committee deliberation, conferred greater transparency and objectivity on HTA assessment of technologies. The framework converted an implicit, unweighted review process to one that is more explicit, flexible in weighting importance and pragmatic. What is known about the topic? HTA programs involve complex decision-making processes requiring the consideration of multiple criteria. Explicit methods of analysis that use weighted criteria according to their relative importance enhance transparency and consistency of decision making by HTA committees, and are preferred to implicit reviews using unweighted criteria. What does this paper add? This article describes the development and piloting of an MCDA framework that aims to improve transparency, objectivity and consistency of funding decisions of the Queensland HTA committee. Criteria were identified through a review of current processes, committee discussions and a literature review, and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) quality of evidence system. Criteria were weighted using a discrete choice experiment involving committee members. Using weighted criteria, mean technology scores were calculated and incorporated into deliberative discussions to determine funding decisions. What are the implications for practitioners? The MCDA framework described here converted a more implicit, unweighted process to one that was more pragmatic, explicit and flexible in scoring HTA submissions. This framework may be useful to other HTA programs and could be expanded to resource allocation decision making in many other healthcare settings.

A multicriteria resource allocation model for the redesign of services following birth

BACKGROUND

Many healthcare services are under considerable pressure to reduce costs while improving quality. This is particularly true in the United Kingdom's National Health Service where postnatal care is sometimes viewed as having a low priority. There is much debate about the service's redesign and the reallocation of resources, both along care pathways and between groups of mothers and babies with different needs. The aim of this study was to develop a decision support tool that would encourage a systemic approach to service redesign and that could assess the various quality and financial implications of service change options making the consequent trade-offs explicit. The paper describes the development process and an initial implementation as a preliminary exploration of the possible merits of this approach.

METHODS

Other studies have suggested that combining multicriteria decision analysis with programme budgeting and marginal analysis might offer a suitable basis for resource allocation decisions in healthcare systems. The Postnatal care Resource Allocation Model incorporated this approach in a decision support tool to analyse the consequences of varying design parameters, notably staff contacts and time, on the various quality domains and costs. The initial phase of the study focussed on mapping postnatal care, involving interviews and workshops with a variety of stakeholders. This was supplemented with a literature review and the resultant knowledge base was encoded in the decision support tool. The model was then tested with various stakeholders before being used in an NHS Trust in England.

RESULTS

The model provides practical support, helping staff explore options and articulate their proposals for the redesign of postnatal care. The integration of cost and quality domains facilitates trade-offs, allowing staff to explore the benefits of reallocating resources between hospital and community-based care, and different patient-categories.

CONCLUSIONS

The main benefits of the model include its structure for assembling the key data, sharing evidence amongst multi-professional teams and encouraging constructive, systemic debate. Although the model was developed in the context of the routine maternity services for mothers and babies in the days following birth it could be adapted for use in other health care services.

Strengthening the evidence-base of integrated care for people with multi-morbidity in Europe using Multi-Criteria Decision Analysis (MCDA)

BACKGROUND

Evaluation of integrated care programmes for individuals with multi-morbidity requires a broader evaluation framework and a broader definition of added value than is common in cost-utility analysis. This is possible through the use of Multi-Criteria Decision Analysis (MCDA).

METHODS AND RESULTS

This paper presents the seven steps of an MCDA to evaluate 17 different integrated care programmes for individuals with multi-morbidity in 8 European countries participating in the 4-year, EU-funded SELFIE project. In step one, qualitative research was undertaken to better understand the decision-context of these programmes. The programmes faced decisions related to their sustainability in terms of reimbursement, continuation, extension, and/or wider implementation. In step two, a uniform set of decision criteria was defined in terms of outcomes measured across the 17 programmes: physical functioning, psychological well-being, social relationships and participation, enjoyment of life, resilience, person-centeredness, continuity of care, and total health and social care costs. These were supplemented by programme-type specific outcomes. Step three presents the quasi-experimental studies designed to measure the performance of the programmes on the decision criteria. Step four gives details of the methods (Discrete Choice Experiment, Swing Weighting) to determine the relative importance of the decision criteria among five stakeholder groups per country. An example in step five illustrates the value-based method of MCDA by which the performance of the programmes on each decision criterion is combined with the weight of the respective criterion to derive an overall value score. Step six describes how we deal with uncertainty and introduces the Conditional Multi-Attribute Acceptability Curve. Step seven addresses the interpretation of results in stakeholder workshops.

DISCUSSION

By discussing our solutions to the challenges involved in creating a uniform MCDA approach for the evaluation of different programmes, this paper provides guidance to future evaluations and stimulates debate on how to evaluate integrated care for multi-morbidity.

Evaluating the Benefits of New Drugs in Health Technology Assessment Using Multiple Criteria Decision Analysis: A Case Study on Metastatic Prostate Cancer With the Dental and Pharmaceuticals Benefits Agency (TLV) in Sweden

Background. Multiple criteria decision analysis (MCDA) has been identified as a prospective methodology for assisting decision makers in evaluating the benefits of new medicines in health technology assessment (HTA); however, limited empirical evidence exists from real-world applications. Objective. To test in practice a recently developed MCDA methodological framework for HTA, the Advance Value Framework, in a proof-of-concept case study with decision makers. Methods. A multi-attribute value theory methodology was adopted applying the MACBETH questioning protocol through a facilitated decision-analysis modelling approach as part of a decision conference with four experts. Settings. The remit of the Swedish Dental and Pharmaceutical Benefits Agency (Tandvårds- och läkemedelsförmånsverket [TLV]) was adopted but in addition supplementary value dimensions were considered. Patients. Metastatic castrate-resistant prostate cancer patients were considered having received prior chemotherapy. Interventions. Abiraterone, cabazitaxel, and enzalutamide were evaluated as third-line treatments. Measurements. Participants’ value preferences were elicited involving criteria selection, options scoring, criteria weighting, and their aggregation. Results. Eight criteria attributes were finally included in the model relating to therapeutic impact, safety profile, socioeconomic impact, and innovation level with relative importance weights 44.5%, 33.3%, 14.8%, and 7.4% per cluster, respectively. Enzalutamide scored the highest overall weighted preference value score, followed by abiraterone and cabazitaxel. Dividing treatments’ overall weighted preference value scores by their costs derived “costs per unit of value” for ranking the treatments based on value-for-money grounds. Limitations. Study limitations included lack of comparative clinical effects across treatments and the small sample of participants. Conclusion. The Advance Value Framework has the prospects of facilitating the evaluation process in HTA and health care decision making; additional research is recommended to address technical challenges and optimize the use of MCDA for policy making.

A multi-criteria decision approach for ranking unmet needs in healthcare

Early temporary reimbursement (ETR) schemes for new interventions targeting high unmet needs are increasingly applied in pharmaceutical policy. Crucial for these schemes is the assessment of unmet healthcare needs of patients and society. This study develops and tests a multi-criteria decision approach (MCDA) for assessing therapeutic and societal needs. The Belgian unmet needs commission, responsible for creating a list of unmet needs for the ETR programme, has tested this methodology to assess the needs in eight health conditions. For therapeutic need, three criteria were included (impact of the condition on quality of life and on life expectancy and inconvenience of current treatment); for societal need two criteria (condition-related healthcare expenditures per patient, prevalence). The results show that the proposed MCDA is feasible and acceptable for the unmet needs commission. Clear definitions of the criteria and regular repetition of these is needed to avoid variable interpretation of the criteria by the commission members. Quality assessment of the evidence is desired. Rankings resulting from the application have face validity. Considering therapeutic need separately from societal need is considered appropriate. Policy makers should consider the use of MCDA in assessing healthcare needs. MCDA improves the transparency and accountability of the decision making processes and is practical and feasible.

[Multi-criteria decision analysis for health technology resource allocation and assessment: so far and so near?]

Multi-criteria decision analysis (MCDA) is an emerging tool that allows the integration of relevant factors for health technology assessment (HTA). This study aims to present a summary of the methodological characteristics of MCDA: definitions, approaches, applications, and implementation stages. A case study was conducted in the São Paulo State Cancer Institute (ICESP) in order to understand the perspectives of decision-makers in the process of drafting a recommendation for the incorporation of technology in the Brazilian Unified National Health System (SUS), through a report by the Brazilian National Commission for the Incorporation of Technologies in the SUS (CONITEC). Paraconsistent annotated evidential logic Eτ was the methodological approach adopted in the study, since it can serve as an underlying logic for constructs capable of synthesizing objective information (from the scientific literature) and subjective information (from experts' values and preferences in the area of knowledge). It also allows the incorporation of conflicting information (contradictions), as well as vague and even incomplete information in the valuation process, resulting from imperfection of the available scientific evidence. The method has the advantages of allowing explicit consideration of the criteria that influenced the decision, facilitating follow-up and visualization of process stages, allowing assessment of the contribution of each criterion separately, and in aggregate, to the decision's outcome, facilitating the discussion of diverging perspectives by different stakeholder groups, and increasing the understanding of the resulting recommendations. The use of an explicit MCDA approach should facilitate conflict mediation and optimize participation by different stakeholder groups.

Multiple criteria decision analysis in the context of health technology assessment: a simulation exercise on metastatic colorectal cancer with multiple stakeholders in the English setting

BACKGROUND

Multiple criteria decision analysis (MCDA) has appeared as a methodology to address limitations of economic evaluation in health technology assessment (HTA), however there are limited empirical evidence from real world applications. The aim of this study is to test in practice a recently developed MCDA methodological framework known as Advance Value Framework (AVF) through a proof-of-concept case study engaging multiple stakeholders.

METHODS

A multi-attribute value theory methodological process was adopted involving problem structuring, model building, model assessment and model appraisal phases. A facilitated decision analysis modelling approach was used as part of a decision conference with thirteen participants. An expanded scope of the National Institute for Health and Care Excellence (NICE) remit acted as the study setting with the use of supplementary value concerns. Second-line biological treatments were evaluated for metastatic colorectal cancer (mCRC) patients having received prior chemotherapy, including cetuximab monotherapy, panitumumab monotherapy and aflibercept in combination with FOLFIRI chemotherapy. Initially 18 criteria attributes were considered spanning four value domains relating to therapeutic impact, safety profile, innovation level and socioeconomic impact.

RESULTS

Nine criteria attributes were finally included. Cetuximab scored the highest overall weighted preference value score of 45.7 out of 100, followed by panitumumab with 42.3, and aflibercept plus FOLFIRI with 14.4. The relative weights of the two most important criteria (overall survival and Grade 4 adverse events) added up to more than the relative weight of all other criteria together (52.1%). Main methodological limitation was the lack of comparative clinical effects across treatments and challenges included the selection of "lower" and "higher" reference levels on criteria attributes, eliciting preferences across attributes where participants had less experience, and ensuring that all attributes possess the right decision theory properties.

CONCLUSIONS

This first application of AVF produced transparent rankings for three mCRC treatments based on their value, by assessing an explicit set of evaluation criteria while allowing for the elicitation and construction of participants' value preferences and their trade-offs. It proved it can aid the evaluation process and value communication of the alternative treatments for the group participants. Further research is needed to optimise its use as part of policy-making.

Sensibility analysis of MCDA using prospective in Brazilian energy sector

Purpose

The purpose of this paper is to present a hybrid modelling that combines concepts and techniques for scenario building together with a Multi-criteria Decision Aid (MCDA) outranking approach. The paper presents a case to illustrate the proposed methodology.

Design/methodology/approach

The research method is a qualitative and quantitative mixture and it is presented as a study case. Bibliographic research is used to construct the theoretical framework. There are a number of studies that develop a sensibility analysis in MCDA modelling; however, none of them explore the robustness of the MCDA solution with use of scenarios variation.

Findings

The methodology allows the criteria that must be taken into account, according to the decision makers’ values and preferences. It is interesting to note that, depending on the scenario, different weights were applied for each criterion, and the performances of alternatives under each criterion has changed as well.

Practical implications

This need arises in decision problems that are susceptible to the influence of scenario variation.

Originality/value

This proposal was applied to a real case that has taken into account six alternatives, with a prospective analysis of three scenarios, evaluated by four criteria. The authors use prospective scenarios to choose the criterion weights and alternatives evaluation.

Using the Analytic Hierarchy Process for Prioritizing Imaging Tests in Diagnosis of Suspected Appendicitis

RATIONALE AND OBJECTIVES

In clinical guideline or criteria development processes, such as those used in developing American College of Radiology Appropriateness Criteria (ACR AC), experts subjectively evaluate benefits and risks associated with imaging tests and make complex decisions about imaging recommendations. The analytic hierarchy process (AHP) decomposes complex decisions into structured smaller decisions, incorporates quantitative evidence and qualitative expert opinion, and promotes structured consensus building. AHP may supplement and/or improve the transparency of expert opinion contributions to developing guidelines or criteria.

MATERIALS AND METHODS

To conduct an empirical test using health services research tools, we convened a mock ACR AC panel of emergency department radiology and nonradiology physicians to evaluate by multicriteria decision analysis, the relative appropriateness of imaging tests for diagnosing suspected appendicitis. Panel members selected benefit-risk criteria via an online survey and assessed contrast-enhanced computed tomography, magnetic resonance imaging, and ultrasound using an AHP-based software. Participants were asked whether the process was manageable, transparent, and improved shared understanding. Priority scores were converted to rankings and compared to the rank order of ACR AC ratings.

RESULTS

When compared to magnetic resonance and ultrasound imaging, participants agreed with the ACR AC that contrast-enhanced computed tomography is the most appropriate test. Contrary to the ACR AC ratings, study results suggest that magnetic resonance is preferable to ultrasound. When compared to nonradiologists, radiologists' priority scores reflect a stronger preference for computed tomography.

CONCLUSIONS

Study participants addressed decision-making challenges using a relatively efficient data collection mechanism, suggesting that AHP may benefit the ACR AC guideline development process in identifying the relative appropriateness of imaging tests. With additional development, AHP may improve transparency when expert opinion is used in clinical guideline or appropriateness criteria development.

Weighing Clinical Evidence Using Patient Preferences: An Application of Probabilistic Multi-Criteria Decision Analysis

The need for patient engagement has been recognized by regulatory agencies, but there is no consensus about how to operationalize this. One approach is the formal elicitation and use of patient preferences for weighing clinical outcomes. The aim of this study was to demonstrate how patient preferences can be used to weigh clinical outcomes when both preferences and clinical outcomes are uncertain by applying a probabilistic value-based multi-criteria decision analysis (MCDA) method. Probability distributions were used to model random variation and parameter uncertainty in preferences, and parameter uncertainty in clinical outcomes. The posterior value distributions and rank probabilities for each treatment were obtained using Monte-Carlo simulations. The probability of achieving the first rank is the probability that a treatment represents the highest value to patients. We illustrated our methodology for a simplified case on six HIV treatments. Preferences were modeled with normal distributions and clinical outcomes were modeled with beta distributions. The treatment value distributions showed the rank order of treatments according to patients and illustrate the remaining decision uncertainty. This study demonstrated how patient preference data can be used to weigh clinical evidence using MCDA. The model takes into account uncertainty in preferences and clinical outcomes. The model can support decision makers during the aggregation step of the MCDA process and provides a first step toward preference-based personalized medicine, yet requires further testing regarding its appropriate use in real-world settings.

The Value of Medicines: A Crucial but Vague Concept

Health Technology Assessment is increasingly used to evaluate the value of healthcare products and to prioritize resources; however, defining exactly what value is and how it should be measured remains a challenge. In this article, we report the results of a literature review, focusing on nine European countries, with the aim of investigating how value is defined from the perspective of different stakeholders, how definitions of value are used, and how value is incorporated into decision making. Only three articles were identified that presented definitions of value, and there was no single shared definition of value in healthcare, which appears to be a highly subjective concept. The majority of the countries investigated combine clinical assessment with economic evaluation to make reimbursement recommendations; the quality-adjusted life-year is the most commonly used measure of value but does not capture broader aspects of value that may be important to patients and healthcare systems. We describe the use of value-based pricing and multi-criteria decision analysis, two approaches to the incorporation of broader aspects of value into decision making. Overall, we have identified considerable variation in how a product's value is defined by different stakeholders. Although a universal understanding of value in healthcare is important, it is clear that current definitions are insufficient, potentially leading to inconsistent reimbursement decisions. Ultimately, the establishment of clearer policies for defining and measuring value in healthcare is needed, and is likely to lead to improvements in the consistency of decision making.

A Note on the Validity and Reliability of Multi-Criteria Decision Analysis for the Benefit-Risk Assessment of Medicines

The comparative evaluation of benefits and risks is one of the most important tasks during the development, market authorization and post-approval pharmacovigilance of medicinal products. Multi-criteria decision analysis (MCDA) has been recommended to support decision making in the benefit-risk assessment (BRA) of medicines. This paper identifies challenges associated with bias or variability that practitioners may encounter in this field and presents solutions to overcome them. The inclusion of overlapping or preference-complementary criteria, which are frequent violations to the assumptions of this model, should be avoided. For each criterion, a value function translates the original outcomes into preference-related scores. Applying non-linear value functions to criteria defined as the risk of suffering a certain event during the study introduces specific risk behaviours in this prescriptive, rather than descriptive, model and is therefore a questionable practice. MCDA uses weights to compare the importance of the model criteria with each other; during their elicitation a frequent situation where (generally favourable) mild effects are directly traded off against low probabilities of suffering (generally unfavourable) severe effects during the study is known to lead to biased and variable weights and ought to be prevented. The way the outcomes are framed during the elicitation process, positively versus negatively for instance, may also lead to differences in the preference weights, warranting an appropriate justification during each implementation. Finally, extending the weighted-sum MCDA model into a fully inferential tool through a probabilistic sensitivity analysis is desirable. However, this task is troublesome and should not ignore that clinical trial endpoints generally are positively correlated.

Value-Based Assessment of New Medical Technologies: Towards a Robust Methodological Framework for the Application of Multiple Criteria Decision Analysis in the Context of Health Technology Assessment

In recent years, multiple criteria decision analysis (MCDA) has emerged as a likely alternative to address shortcomings in health technology assessment (HTA) by offering a more holistic perspective to value assessment and acting as an alternative priority setting tool. In this paper, we argue that MCDA needs to subscribe to robust methodological processes related to the selection of objectives, criteria and attributes in order to be meaningful in the context of healthcare decision making and fulfil its role in value-based assessment (VBA). We propose a methodological process, based on multi-attribute value theory (MAVT) methods comprising five distinct phases, outline the stages involved in each phase and discuss their relevance in the HTA process. Importantly, criteria and attributes need to satisfy a set of desired properties, otherwise the outcome of the analysis can produce spurious results and misleading recommendations. Assuming the methodological process we propose is adhered to, the application of MCDA presents three very distinct advantages to decision makers in the context of HTA and VBA: first, it acts as an instrument for eliciting preferences on the performance of alternative options across a wider set of explicit criteria, leading to a more complete assessment of value; second, it allows the elicitation of preferences across the criteria themselves to reflect differences in their relative importance; and, third, the entire process of preference elicitation can be informed by direct stakeholder engagement, and can therefore reflect their own preferences. All features are fully transparent and facilitate decision making.

Use of Value of Information in Healthcare Decision Making: Exploring Multiple Perspectives

BACKGROUND

Value of information (VOI) is a tool that can be used to inform decisions concerning additional research in healthcare. VOI estimates the value of obtaining additional information and indicates the optimal design for additional research. Although it is recognized as good practice in handling uncertainty, it is still hardly used in decision making in the Netherlands.

OBJECTIVE

This paper aims to examine the potential value of VOI, barriers and facilitators and the way forward with the use of VOI in the decision-making process for reimbursement of pharmaceuticals in the Netherlands.

METHODS

Three focus group interviews were conducted with researchers, policy makers, and representatives of pharmaceutical companies.

RESULTS

The results revealed that although all stakeholders recognize the relevance of VOI, it is hardly used and many barriers to the performance and use of VOI were identified. One of these barriers is that not all uncertainties are easily incorporated in VOI, and the results may be biased if structural uncertainties are ignored. Furthermore, not all research designs indicated by VOI may be feasible in practice.

CONCLUSIONS

To fully embed VOI into current decision-making processes, a threshold incremental cost-effectiveness ratio and guidelines that clarify when and how VOI should be performed are needed. In addition, it should be clear to all stakeholders how the results of VOI are used in decision making.

Multiple Criteria Decision Analysis for Health Care Decision Making--Emerging Good Practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force

Health care decisions are complex and involve confronting trade-offs between multiple, often conflicting objectives. Using structured, explicit approaches to decisions involving multiple criteria can improve the quality of decision making. A set of techniques, known under the collective heading, multiple criteria decision analysis (MCDA), are useful for this purpose. In 2014, ISPOR established an Emerging Good Practices Task Force. The task force's first report defined MCDA, provided examples of its use in health care, described the key steps, and provided an overview of the principal methods of MCDA. This second task force report provides emerging good-practice guidance on the implementation of MCDA to support health care decisions. The report includes: a checklist to support the design, implementation and review of an MCDA; guidance to support the implementation of the checklist; the order in which the steps should be implemented; illustrates how to incorporate budget constraints into an MCDA; provides an overview of the skills and resources, including available software, required to implement MCDA; and future research directions.

Making Good Decisions in Healthcare with Multi-Criteria Decision Analysis: The Use, Current Research and Future Development of MCDA

Healthcare decision making is usually characterized by a low degree of transparency. The demand for transparent decision processes can be fulfilled only when assessment, appraisal and decisions about health technologies are performed under a systematic construct of benefit assessment. The benefit of an intervention is often multidimensional and, thus, must be represented by several decision criteria. Complex decision problems require an assessment and appraisal of various criteria; therefore, a decision process that systematically identifies the best available alternative and enables an optimal and transparent decision is needed. For that reason, decision criteria must be weighted and goal achievement must be scored for all alternatives. Methods of multi-criteria decision analysis (MCDA) are available to analyse and appraise multiple clinical endpoints and structure complex decision problems in healthcare decision making. By means of MCDA, value judgments, priorities and preferences of patients, insurees and experts can be integrated systematically and transparently into the decision-making process. This article describes the MCDA framework and identifies potential areas where MCDA can be of use (e.g. approval, guidelines and reimbursement/pricing of health technologies). A literature search was performed to identify current research in healthcare. The results showed that healthcare decision making is addressing the problem of multiple decision criteria and is focusing on the future development and use of techniques to weight and score different decision criteria. This article emphasizes the use and future benefit of MCDA.

Multiple Criteria Decision Analysis for Health Care Decision Making--An Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force

Health care decisions are complex and involve confronting trade-offs between multiple, often conflicting, objectives. Using structured, explicit approaches to decisions involving multiple criteria can improve the quality of decision making and a set of techniques, known under the collective heading multiple criteria decision analysis (MCDA), are useful for this purpose. MCDA methods are widely used in other sectors, and recently there has been an increase in health care applications. In 2014, ISPOR established an MCDA Emerging Good Practices Task Force. It was charged with establishing a common definition for MCDA in health care decision making and developing good practice guidelines for conducting MCDA to aid health care decision making. This initial ISPOR MCDA task force report provides an introduction to MCDA - it defines MCDA; provides examples of its use in different kinds of decision making in health care (including benefit risk analysis, health technology assessment, resource allocation, portfolio decision analysis, shared patient clinician decision making and prioritizing patients' access to services); provides an overview of the principal methods of MCDA; and describes the key steps involved. Upon reviewing this report, readers should have a solid overview of MCDA methods and their potential for supporting health care decision making.

Applying the Analytic Hierarchy Process in healthcare research: A systematic literature review and evaluation of reporting

Background

The Analytic Hierarchy Process (AHP), developed by Saaty in the late 1970s, is one of the methods for multi-criteria decision making. The AHP disaggregates a complex decision problem into different hierarchical levels. The weight for each criterion and alternative are judged in pairwise comparisons and priorities are calculated by the Eigenvector method. The slowly increasing application of the AHP was the motivation for this study to explore the current state of its methodology in the healthcare context.

Methods

A systematic literature review was conducted by searching the Pubmed and Web of Science databases for articles with the following keywords in their titles or abstracts: “Analytic Hierarchy Process,” “Analytical Hierarchy Process,” “multi-criteria decision analysis,” “multiple criteria decision,” “stated preference,” and “pairwise comparison.” In addition, we developed reporting criteria to indicate whether the authors reported important aspects and evaluated the resulting studies’ reporting.

Results

The systematic review resulted in 121 articles. The number of studies applying AHP has increased since 2005. Most studies were from Asia (almost 30 %), followed by the US (25.6 %). On average, the studies used 19.64 criteria throughout their hierarchical levels. Furthermore, we restricted a detailed analysis to those articles published within the last 5 years (n = 69). The mean of participants in these studies were 109, whereas we identified major differences in how the surveys were conducted. The evaluation of reporting showed that the mean of reported elements was about 6.75 out of 10. Thus, 12 out of 69 studies reported less than half of the criteria.

Conclusion

The AHP has been applied inconsistently in healthcare research. A minority of studies described all the relevant aspects. Thus, the statements in this review may be biased, as they are restricted to the information available in the papers. Hence, further research is required to discover who should be interviewed and how, how inconsistent answers should be dealt with, and how the outcome and stability of the results should be presented. In addition, we need new insights to determine which target group can best handle the challenges of the AHP.

Estimating the value of medical treatments to patients using probabilistic multi criteria decision analysis

BACKGROUND

Estimating the value of medical treatments to patients is an essential part of healthcare decision making, but is mostly done implicitly and without consulting patients. Multi criteria decision analysis (MCDA) has been proposed for the valuation task, while stated preference studies are increasingly used to measure patient preferences. In this study we propose a methodology for using stated preferences to weigh clinical evidence in an MCDA model that includes uncertainty in both patient preferences and clinical evidence explicitly.

METHODS

A probabilistic MCDA model with an additive value function was developed and illustrated using a case on hypothetical treatments for depression. The patient-weighted values were approximated with Monte Carlo simulations and compared to expert-weighted results. Decision uncertainty was calculated as the probability of rank reversal for the first rank. Furthermore, scenario analyses were done to assess the relative impact of uncertainty in preferences and clinical evidence, and of assuming uniform preference distributions.

RESULTS

The patient-weighted values for drug A, drug B, drug C, and placebo were 0.51 (95% CI: 0.48 to 0.54), 0.51 (95% CI: 0.48 to 0.54), 0.54 (0.49 to 0.58), and 0.15 (95% CI: 0.13 to 0.17), respectively. Drug C was the most preferred treatment and the rank reversal probability for first rank was 27%. This probability decreased to 18% when uncertainty in performances was not included and increased to 41% when uncertainty in criterion weights was not included. With uniform preference distributions, the first rank reversal probability increased to 61%. The expert-weighted values for drug A, drug B, drug C, and placebo were 0.67 (95% CI: 0.65 to 0.68), 0.57 (95% CI: 0.56 to 0.59), 0.67 (95% CI: 0.61 to 0.71), and 0.19 (95% CI: 0.17 to 0.21). The rank reversal probability for the first rank according to experts was 49%.

CONCLUSIONS

Preferences elicited from patients can be used to weigh clinical evidence in a probabilistic MCDA model. The resulting treatment values can be contrasted to results from experts, and the impact of uncertainty can be quantified using rank probabilities. Future research should focus on integrating the model with regulatory decision frameworks and on including other types of uncertainty.