Methods to elicit experts' beliefs over uncertain quantities: application to a cost effectiveness transition model of negative pressure wound therapy for severe pressure ulceration

Potential costs and consequences associated with medication error at hospital discharge: an expert judgement study

OBJECTIVES

Assessing the cost-effectiveness of complex pharmaceutical care interventions and medication error outcomes is hindered by lack of available data on actual outcomes consequent to errors that were intercepted for patient safety reasons. Expert judgement is an approach to acquire data regarding unknown parameters in an economic model which are otherwise insufficient or not possible to obtain. The aim of this paper is to describe a method to approach this problem using findings from a single intervention study and to calculate the potential costs and consequences associated with discharge medication error.

METHODS

Using data from a previous intervention study, the hypothetical consequences of medication error(s) at hospital discharge, in terms of diagnosis, healthcare resource utilisation and impact on health-related quality of life, were identified by expert judgement of anonymised cases. Primary healthcare utilisation costs were derived from published tariffs, inpatient costs were derived by simulation in the hospital discharge activity database test environment and the difference between adjudicated baseline and posterror health state was expressed as quality-adjusted life year (QALY) decrement.

RESULTS

Four experts provided judgement on 81 cases. Of these, 75 were judged to have potential clinical consequences. Between 56 and 69 of the 81 cases were variably judged to require remedial healthcare utilisation. The mean calculated cost per case (representing an individual patient), based on all 81 cases, was €1009.58, 95% CI 726.64 to 1585.67. The mean QALY loss was 0.03 (95% CI 0.01 to 0.05).

CONCLUSION

An expert judgement process proved feasible and useful to estimate financial cost and QALY loss associated with hospital discharge medication error. These estimates will be employed in model-based economic evaluation. This method could be transferred to other prospective observational patient safety research which seeks to assess value for money of complex interventions.

Structured expert elicitation to inform long-term survival extrapolations using alternative parametric distributions: a case study of CAR T therapy for relapsed/ refractory multiple myeloma

Background

Our aim was to extend traditional parametric models used to extrapolate survival in cost-effectiveness analyses (CEAs) by integrating individual-level patient data (IPD) from a clinical trial with estimates from experts regarding long-term survival. This was illustrated using a case study evaluating survival of patients with triple-class exposed relapsed/refractory multiple myeloma treated with the chimeric antigen receptor (CAR) T cell therapy idecabtagene vicleucel (ide-cel, bb2121) in KarMMa (a phase 2, single-arm trial).

Methods

The distribution of patients expected to be alive at 3, 5, and 10 years given the observed survival from KarMMa (13.3 months of follow-up) was elicited from 6 experts using the SHeffield ELicitation Framework. Quantities of interest were elicited from each expert individually, which informed the consensus elicitation including all experts. Estimates for each time point were assumed to follow a truncated normal distribution. These distributions were incorporated into survival models, which constrained the expected survival based on standard survival distributions informed by IPD from KarMMa.

Results

Models for ide-cel that combined KarMMa data with expert opinion were more consistent in terms of survival as well as mean survival at 10 years (survival point estimates under different parametric models were 29–33% at 3 years, 5–17% at 5 years, and 0–6% at 10 years) versus models with KarMMa data alone (11–39% at 3 years, 0–25% at 5 years, and 0–11% at 10 years).

Conclusion

This case study demonstrates a transparent approach to integrate IPD from trials with expert opinion using traditional parametric distributions to ensure long-term survival extrapolations are clinically plausible.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01745-z.

Understanding current UK practice for the incidental identification of vertebral fragility fractures from CT scans: an expert elicitation study

Background

There is an emerging interest in using automated approaches to enable the incidental identification of vertebral fragility fractures (VFFs) on existing medical images visualising the spine.

Aim

To quantify values, and the degree of uncertainty associated with them, for the incidental identification of VFFs from computed tomography (CT) scans in current practice.

Methods

An expert elicitation exercise was conducted to generate point estimates and measures of uncertainty for four values representing the probability of: VFF being correctly reported by the radiologist; the absence of VFF being correctly assessed by the radiologist; being referred for management when a VFF is identified; having a dual-energy X-ray absorptiometry (DXA) scan after general practitioner (GP) referral. Data from a sample of seven experts in the diagnosis and management of people with VFFs were pooled using mathematical aggregation.

Results

The estimated mean values for each probability parameter were: VFF being correctly reported by the radiologist = 0.25 (standard deviation (SD): 0.21); absence of VFF being correctly assessed by the radiologist = 0.89 (0.10); being referred for management when a VFF is identified by the radiologist = 0.15 (0.12); having a DXA scan after GP referral = 0.66 (0.28).

Discussion

These estimates could be used to facilitate the subsequent early economic evaluation of potential new approaches to improve the health outcomes of people with VFFs.

Conclusion

In the absence of epidemiological studies, this study produced point estimates and measures of uncertainty for key parameters needed to describe current pathways for the incidental diagnosis of VFFs.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40520-022-02124-w.

Systematic review of the evidence sources applied to cost-effectiveness analyses for older women with primary breast cancer

OBJECTIVE

To appraise the sources of evidence and methods to estimate input parameter values in decision-analytic model-based cost-effectiveness analyses of treatments for primary breast cancer (PBC) in older patients (≥ 70 years old).

METHODS

Two electronic databases (Ovid Medline, Ovid EMBASE) were searched (inception until 5 September-2021) to identify model-based full economic evaluations of treatments for older women with PBC as part of their base-case target population or age-subgroup analysis. Data sources and methods to estimate four types of input parameters including health-related quality of life (HRQoL); natural history; treatment effect; resource use were extracted and appraised. Quality assessment was completed by reference to the Consolidated Health Economic Evaluation Reporting Standards.

RESULTS

Seven model-based economic evaluations were included (older patients as part of their base-case (n = 3) or subgroup (n = 4) analysis). Data from younger patients (< 70 years) were used frequently to estimate input parameters. Different methods were adopted to adjust these estimates for an older population (HRQoL: disutility multipliers, additive utility decrements; Natural history: calibration of absolute values, one-way sensitivity analyses; Treatment effect: observational data analysis, age-specific behavioural parameters, plausible scenario analyses; Resource use: matched control observational data analysis, age-dependent follow-up costs).

CONCLUSION

Improving estimated input parameters for older PBC patients will improve estimates of cost-effectiveness, decision uncertainty, and the value of further research. The methods reported in this review can inform future cost-effectiveness analyses to overcome data challenges for this population. A better understanding of the value of treatments for these patients will improve population health outcomes, clinical decision-making, and resource allocation decisions.

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

BACKGROUND

Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making.

OBJECTIVES

The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts' judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed.

METHODS

A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the 'choices') and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken.

RESULTS

For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution.

LIMITATIONS

The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are.

CONCLUSIONS

The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required.

FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).

Cost-Benefit Analysis of Single versus Repeated Use of Single-Use Devices in Cataract Surgery

PURPOSE

To estimate the net cost effect associated with the real-world practice of repeated use of designated single-use medical devices (SUDs) versus their proper single use in cataract surgery in Greece.

DESIGN

A cost-benefit analysis model was constructed in the form of a decision tree.

METHODS

A digital expert panel was assembled in order to estimate the probabilities of intraoperative and postoperative complications associated with single and repeated use of SUDs. Unit costs for the management of each complication were obtained from the official Greek bulletins. A Monte Carlo-type sensitivity analysis was performed to assess the robustness of the results.

RESULTS

Based on the probabilities of complications attained from the expert panel, repeated use of SUDs is associated with a higher chance of complications compared to single use, which results in higher cost of complication management. Under the healthcare sector perspective, the total expected cost per cataract surgery is 1,403.98€ (1,244.20€ the initial cost of cataract surgery plus 159.78€ the cost of adverse events) in the case of single use, while for repeated use the total cost is 1,486.29€ (1,146.86€ + 339.43€, respectively) and, thus, repeated use of SUDs in cataract surgery results in 82.31€ higher expected cost per patient compared to their single use. Moreover, the societal perspective analysis indicated even higher additional costs in the case of SUD reuse (108.24€).

CONCLUSION

Repeated use of SUDs in cataract surgery is not appropriate, it jeopardizes patient safety and carries a legal liability for the reuser. The present study, which is the first to attach a monetary value to the common yet questionable practice of SUD reuse, shows that it is not cost beneficial. Therefore, it is expected that the results will have implications in policy formulations to improve the delivery of cataract healthcare.

Methodological issues in economic evaluations of emergency transport systems in low-income and middle-income countries

A recent systematic review identified few papers on the economic evaluation of systems for emergency transport of acutely ill or injured patients. In addition, we found no articles dealing with the methodological challenges posed by such studies in low-income or middle-income countries. We therefore carried out an analysis of issues that are of particular salience to this important topic. This is an intellectual study in which we develop models, identify their limitations, suggest potential extensions to the models and discuss priorities for empirical studies to populate models. First, we develop a general model to calculate changes in survival contingent on the reduced time to treatment that an emergency transport system is designed to achieve. Second, we develop a model to estimate transfer times over an area that will be served by a proposed transfer system. Third, we discuss difficulties in obtaining parameters with which to populate the models. Fourth, we discuss costs, both direct and indirect, of an emergency transfer service. Fifth, we discuss the issue that outcomes other than survival should be considered and that the effects of a service are a weighted sum over all the conditions and severities for which the service caters. Lastly, based on the above work, we identify priorities for research. To our knowledge, this is the first study to identify and frame issues in the health economics of acute transfer systems and to develop models to calculate survival rates from basic parameters, such as time delay/survival relationships, that vary by intervention type and context.

Health economic evaluation of gene replacement therapies: methodological issues and recommendations

Objective: To provide recommendations for addressing previously identified key challenges in health economic evaluations of Gene Replacement Therapies (GRTs), including: 1) the assessment of clinical effectiveness; 2) the valuation of health outcomes; 3) the time horizon and extrapolation of effects beyond trial duration; 4) the estimation of costs; 5) the selection of appropriate discount rates; 6) the incorporation of broader elements of value; and 7) affordability. Methods: A literature review on economic evaluations of GRT was performed. Interviews were conducted with 8 European and US health economic experts with experience in evaluations of GRT. Targeted literature reviews were conducted to investigate further potential solutions to specific challenges. Recommendations: Experts agreed on factors to be considered to ensure the acceptability of historical cohorts by HTA bodies. Existing prospective registries or, if not available, retrospective registries, may be used to analyse different disease trajectories and inform extrapolations. The importance of expert opinion due to limited data was acknowledged. Expert opinion should be obtained using structured elicitation techniques. Broader elements of value, beyond health gains directly related to treatment, can be considered through the application of a factor to inflate the quality-adjusted life years (QALYs) or a higher cost-effectiveness threshold. Additionally, the use of cost-benefit analysis and saved young life equivalents (SAVE) were proposed as alternatives to QALYs for the valuations of outcomes of GRT as they can incorporate broader elements of value and avoid problems of eliciting utilities for paediatric diseases. Conclusions: While some of the limitations of economic evaluations of GRT are inherent to limited clinical data and lack of experience with these treatments, others may be addressed by methodological research to be conducted by health economists.

Cost-effectiveness and value of information analysis of a low-friction environment following skin graft in patients with burn injury

Background

Patients with burn injuries may receive a skin graft to achieve healing in a timely manner. However, in around 7% of cases, the skin graft is lost (fails to attach to the wound site) and a re-grafting procedure is necessary. It has been hypothesised that low-friction (smooth, more slippery) bedding may reduce the risk of skin-graft loss. A before and after feasibility study comparing low-friction with standard bedding in skin-grafted patients was conducted in order to collect proof of concept data. The resulting relative risk on the primary outcome (number of patients with skin graft failure) for the non-randomised study provided no evidence of effect but had a large standard error. The aim of this study is to see if an appropriately powered randomised control trial would be worthwhile.

Methods

A probabilistic decision-analytic model was constructed to compare low-friction bedding to standard care in a population of burn patients who have undergone skin grafting. Results from the before and after study were used as model inputs. The sensitivity of results to bias in the relative risk of graft loss was conducted. Low-friction bedding is considered optimal if expected incremental net benefit (INB) is positive. Uncertainty is assessed using cost-effectiveness acceptability curves. Expected Value of Perfect Partial Information (EVPPI) provides an upper bound for the potential net health benefits of new research for given model input.

Results

At a willingness to pay threshold of £20,000 per QALY, INB = £151 (95% Credible Interval (CrI) −142 to 814), marginally favouring low-friction bedding but with high uncertainty (probability of being cost-effective 70.5%). Expected value of perfect information (EVPI) per patient was £20.29, which results in a population EVPI of £174,765 over a 10-year lifetime for the technology (based on 1000 patients per year who would benefit from the intervention). The parameter contributing most to the uncertainty was the inpatient care cost, i.e. information that could be obtained from the audit of practice and without an expensive trial. These findings were robust to a wide-range of assumptions about the potential bias due to the observational nature of the comparative evidence.

Conclusions

Our study results suggest that an RCT (randomised controlled trial) is unlikely to be worthwhile, but there may be value in a study to estimate the re-graft rates and associated costs in this population.

Expert Elicitation to Inform a Cost-Effectiveness Analysis of Screening for Renal Cancer

BACKGROUND

Population screening for renal cell carcinoma (RCC) using ultrasound has the potential to improve survival outcomes; however, a cost-effectiveness analysis (CEA) has yet to be performed. Owing to the lack of existing evidence, we performed structured expert elicitation to derive unknown quantities to inform the CEA.

OBJECTIVE

To elicit the cancer stage distribution (proportion of individuals with each stage of cancer) for different RCC screening scenarios and the annual transition probabilities for undiagnosed disease becoming diagnosed in the National Health Service.

METHODS

The study design and reporting adhered to the Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations. The elicitation was conducted face-to-face or via telephone between each individual expert and the facilitator, aided by online material. For multinomial data, Connor-Mosimann and modified Connor-Mosimann distributions were fitted for each expert and for all experts combined using mathematical linear pooling.

RESULTS

A total of 24 clinical experts were invited, and 71% participated (7 urologists, 6 oncologists, 4 radiologists). The modified Connor-Mosimann distribution provided the best fit for most elicited quantities. Greater uncertainty was noted for the elicited transition probabilities compared with the elicited stage distributions.

CONCLUSION

We performed the first expert elicitation of RCC screening parameters, crucial information that will inform the CEA of screening. In addition, the elicited quantities may enable future health economic evaluations assessing the value of diagnostic tools and pathways in RCC.

EWMA Document: Negative Pressure Wound Therapy

1. Introduction Since its introduction in clinical practice in the early 1990's negative pressure wounds therapy (NPWT) has become widely used in the management of complex wounds in both inpatient and outpatient care.¹ NPWT has been described as a effective treatment for wounds of many different aetiologies^2,3 and suggested as a gold standard for treatment of wounds such as open abdominal wounds,^4-6 dehisced sternal wounds following cardiac surgery^7,8 and as a valuable agent in complex non-healing wounds.^9,10 Increasingly, NPWT is being applied in the primary and home-care setting, where it is described as having the potential to improve the efficacy of wound management and help reduce the reliance on hospital-based care.¹¹ While the potential of NPWT is promising and the clinical use of the treatment is widespread, highlevel evidence of its effectiveness and economic benefits remain sparse.^12-14 The ongoing controversy regarding high-level evidence in wound care in general is well known. There is a consensus that clinical practice should be evidence-based, which can be difficult to achieve due to confusion about the value of the various approaches to wound management; however, we have to rely on the best available evidence. The need to review wound strategies and treatments in order to reduce the burden of care in an efficient way is urgent. If patients at risk of delayed wound healing are identified earlier and aggressive interventions are taken before the wound deteriorates and complications occur, both patient morbidity and health-care costs can be significantly reduced. There is further a fundamental confusion over the best way to evaluate the effectiveness of interventions in this complex patient population. This is illustrated by reviews of the value of various treatment strategies for non-healing wounds, which have highlighted methodological inconsistencies in primary research. This situation is confounded by differences in the advice given by regulatory and reimbursement bodies in various countries regarding both study design and the ways in which results are interpreted. In response to this confusion, the European Wound Management Association (EWMA) has been publishing a number of interdisciplinary documents^15-19 with the intention of highlighting: The nature and extent of the problem for wound management: from the clinical perspective as well as that of care givers and the patients Evidence-based practice as an integration of clinical expertise with the best available clinical evidence from systematic research The nature and extent of the problem for wound management: from the policy maker and healthcare system perspectives The controversy regarding the value of various approaches to wound management and care is illustrated by the case of NPWT, synonymous with topical negative pressure or vacuum therapy and cited as branded VAC (vacuum-assisted closure) therapy. This is a mode of therapy used to encourage wound healing. It is used as a primary treatment of chronic wounds, in complex acute wounds and as an adjunct for temporary closure and wound bed preparation preceding surgical procedures such as skin grafts and flap surgery. Aim An increasing number of papers on the effect of NPWT are being published. However, due to the low evidence level the treatment remains controversial from the policy maker and health-care system's points of view-particularly with regard to evidence-based medicine. In response EWMA has established an interdisciplinary working group to describe the present knowledge with regard to NPWT and provide overview of its implications for organisation of care, documentation, communication, patient safety, and health economic aspects. These goals will be achieved by the following: Present the rational and scientific support for each delivered statement Uncover controversies and issues related to the use of NPWT in wound management Implications of implementing NPWT as a treatment strategy in the health-care system Provide information and offer perspectives of NPWT from the viewpoints of health-care staff, policy makers, politicians, industry, patients and hospital administrators who are indirectly or directly involved in wound management.

A Bayesian framework for health economic evaluation in studies with missing data

Health economics studies with missing data are increasingly using approaches such as multiple imputation that assume that the data are "missing at random." This assumption is often questionable, as-even given the observed data-the probability that data are missing may reflect the true, unobserved outcomes, such as the patients' true health status. In these cases, methodological guidelines recommend sensitivity analyses to recognise data may be "missing not at random" (MNAR), and call for the development of practical, accessible approaches for exploring the robustness of conclusions to MNAR assumptions. Little attention has been paid to the problem that data may be MNAR in health economics in general and in cost-effectiveness analyses (CEA) in particular. In this paper, we propose a Bayesian framework for CEA where outcome or cost data are missing. Our framework includes a practical, accessible approach to sensitivity analysis that allows the analyst to draw on expert opinion. We illustrate the framework in a CEA comparing an endovascular strategy with open repair for patients with ruptured abdominal aortic aneurysm, and provide software tools to implement this approach.

Costs and consequences of chronic pain due to musculoskeletal disorders from a health system perspective in Chile

Background:

Chronic pain is a prevalent and distressing condition caused by an unceasing pain lasting more than 3 months or a pain that persists beyond the normal healing time. There is evidence of inadequate management partly explained by the unawareness regarding the magnitude of the problem.

Objectives:

To estimate the annual expected costs and consequences of chronic pain caused by musculoskeletal diseases from the health system perspective in Chile.

Methods:

A Markov cohort model was built to represent chronic pain and estimate expected costs and consequences over 1-year time horizon. Transition probabilities were obtained through expert elicitation. Consequences examined were: years lost to disability (YLD), depression, anxiety, and productivity losses. Direct health care costs were estimated using local sources. Probabilistic sensitivity analysis was performed to characterize second-order uncertainty.

Results:

The annual expected cost due to musculoskeletal chronic pain was estimated in USD $1387.2 million, equivalent to 0.417% of the national GDP. Lower back pain and osteoarthritis of the knee explained the larger proportion of the total cost, 31.8% and 27.1%, respectively. Depression attributed to chronic pain is another important consequence accounting for USD $94 million (Bayesian credibility interval 95% $49.1–$156.26). Productivity losses were also important cost, although early retirement and presenteeism were not measured. Chronic pain causes 137,037 YLDs.

Conclusion:

Chronic pain is not only an important cause of disability but also responsible for high social and financial burden in Chile. Public health programs focused on managing chronic pain may decrease burden of disease and possibly reduce costs.

Estimating Future Health Technology Diffusion Using Expert Beliefs Calibrated to an Established Diffusion Model

OBJECTIVES

Estimates of future health technology diffusion, or future uptake over time, are a requirement for different analyses performed within health technology assessments. Methods for obtaining such estimates include constant uptake estimates based on expert opinion or analogous technologies and on extrapolation from initial data points using parametric curves-but remain divorced from established diffusion theory and modeling. We propose an approach to obtaining diffusion estimates using experts' beliefs calibrated to an established diffusion model to address this methodologic gap.

METHODS

We performed an elicitation of experts' beliefs on future diffusion of a new preterm birth screening illustrative case study technology. The elicited quantities were chosen such that they could be calibrated to yield the parameters of the Bass model of new product growth, which was chosen based on a review of the diffusion literature.

RESULTS

With the elicitation of only three quantities per diffusion curve, our approach enabled us to quantify uncertainty about diffusion of the new technology in different scenarios. Pooled results showed that the attainable number of adoptions was predicted to be relatively low compared with what was thought possible. Further research evidence improved the attainable number of adoptions only slightly but resulted in greater speed of diffusion.

CONCLUSIONS

The proposed approach of eliciting experts' beliefs about diffusion and informing the Bass model has the potential to fill the methodologic gap evident in value of implementation and research, as well as budget impact and some cost-effectiveness analyses.

Experiences of Structured Elicitation for Model-Based Cost-Effectiveness Analyses

BACKGROUND

Empirical evidence supporting the cost-effectiveness estimates of particular health care technologies may be limited, or it may even be missing entirely. In these situations, additional information, often in the form of expert judgments, is needed to reach a decision. There are formal methods to quantify experts' beliefs, termed as structured expert elicitation (SEE), but only limited research is available in support of methodological choices. Perhaps as a consequence, the use of SEE in the context of cost-effectiveness modelling is limited.

OBJECTIVES

This article reviews applications of SEE in cost-effectiveness modelling with the aim of summarizing the basis for methodological choices made in each application and recording the difficulties and challenges reported by the authors in the design, conduct, and analyses.

METHODS

The methods used in each application were extracted along with the criteria used to support methodological and practical choices and any issues or challenges discussed in the text. Issues and challenges were extracted using an open field, and then categorised and grouped for reporting.

RESULTS

The review demonstrates considerable heterogeneity in methods used, and authors acknowledge great methodological uncertainty in justifying their choices. Specificities of the context area emerging as potentially important in determining further methodological research in elicitation are between- expert variation and its interpretation, the fact that substantive experts in the area may not be trained in quantitative subjects, that judgments are often needed on various parameter types, the need for some form of assessment of validity, and the need for more integration with behavioural research to devise relevant debiasing strategies.

CONCLUSIONS

This review of experiences of SEE highlights a number of specificities/constraints that can shape the development of guidance and target future research efforts in this area.

Expert Elicitation of Multinomial Probabilities for Decision-Analytic Modeling: An Application to Rates of Disease Progression in Undiagnosed and Untreated Melanoma

BACKGROUND

Expert elicitation is required to inform decision making when relevant "better quality" data either do not exist or cannot be collected. An example of this is to inform decisions as to whether to screen for melanoma. A key input is the counterfactual, in this case the natural history of melanoma in patients who are undiagnosed and hence untreated.

OBJECTIVES

To elicit expert opinion on the probability of disease progression in patients with melanoma that is undetected and hence untreated.

METHODS

A bespoke webinar-based expert elicitation protocol was administered to 14 participants in the United Kingdom, Australia, and New Zealand, comprising 12 multinomial questions on the probability of progression from one disease stage to another in the absence of treatment. A modified Connor-Mosimann distribution was fitted to individual responses to each question. Individual responses were pooled using a Monte-Carlo simulation approach. Participants were asked to provide feedback on the process.

RESULTS

A pooled modified Connor-Mosimann distribution was successfully derived from participants' responses. Feedback from participants was generally positive, with 86% willing to take part in such an exercise again. Nevertheless, only 57% of participants felt that this was a valid approach to determine the risk of disease progression. Qualitative feedback reflected some understanding of the need to rely on expert elicitation in the absence of "hard" data.

CONCLUSIONS

We successfully elicited and pooled the beliefs of experts in melanoma regarding the probability of disease progression in a format suitable for inclusion in a decision-analytic model.

Using expert elicitation to estimate the potential impact of improved diagnostic performance of laboratory tests: a case study on rapid discharge of suspected non-ST elevation myocardial infarction patients

Early health technology assessment can provide insight in the potential cost-effectiveness of new tests to guide further development decisions. This can increase their potential benefit but often requires evidence which is lacking in early test development stages. Then, expert elicitation may be used to generate evidence on the impact of tests on patient management. This is illustrated in a case study on a new triple biomarker test (copeptin, heart-type fatty acid binding protein, and high-sensitivity troponin [HsTn]) at hospital admission. The elicited evidence enables estimation of the impact of using the triple biomarker on time to exclusion of non-ST elevation myocardial infarction compared with current serial HsTn measurement (performed 0, 2, and 6 h after admission). Cardiologists were asked to estimate the effect of the triple biomarker on patient's discharge rates and interventions performed, depending on its diagnostic performance. This elicited evidence was combined with Dutch reimbursement data and published evidence into a decision analytic model. Direct hospital costs and patients' discharge rates were assessed for 3 testing strategies including this triple biomarker (ie, only at admission or combined with HsTn measurements after 2 and 6 h). Direct hospital costs of suspected non-ST elevation myocardial infarction patients using serial HsTn measurements are estimated at €1825 per patient. Combining this triple biomarker with HsTn measurements after 2 and 6 hours is expected to be the most cost-effective strategy. Depending on the diagnostic performance of the triple biomarker, this strategy is estimated to reduce costs with €66 to €205 per patient (ie, 3.6%-11.3% reduction). Expert elicitation can be a valuable tool for early health technology assessment to provide an initial estimate of the cost-effectiveness of new tests prior to their implementation in clinical practice. As demonstrated in our case study, improved diagnostic performance of the triple biomarker may have benefits that should be further explored.

COMPARISON OF METHODS FOR ESTIMATING CONTEMPORARY COSTS: AN APPLICATION TO LIVER TRANSPLANTATION IN THE UNITED KINGDOM

OBJECTIVES

Our study addresses the important issue of estimating treatment costs from historical data. It is a problem frequently faced by health technology assessment analysts. We compared four approaches used to estimate current costs when good quality contemporary data are not available using liver transplantation as an example.

METHODS

First, the total cost estimates extracted for patients from a cohort study, conducted in the 1990s, were inflated using a published inflation multiplier. Second, resource use estimates from the cohort study were extracted for hepatitis C patients and updated using current unit costs. Third, expert elicitation was carried out to identify changes in clinical practice over time and quantify current resource use. Fourth, routine data on resource use were obtained from National Health Service Blood and Transplant (NHSBT).

RESULTS

The first two methods did not account for changes in clinical practice. Also the first was not specific to hepatitis patients. The use of experts confirmed significant changes in clinical practice. However, the quantification of resource use using experts is challenging as clinical specialists may not have a complete overview of clinical pathway. The NHSBT data are the most accurate reflection of transplantation and posttransplantation phase; however, data were not available for the whole pathway of care. The best estimate of total cost, combining NHSBT data and expert elicitation, is £121,211.

DISCUSSION

Observational data from routine care are potentially the most reliable reflection of current resource use. Efforts should be made to make such data readily available and accessible to researchers. Expert elicitation provided reasonable estimates.

Informing Reimbursement Decisions Using Cost-Effectiveness Modelling: A Guide to the Process of Generating Elicited Priors to Capture Model Uncertainties

In informing decisions, utilising health technology assessment (HTA), expert elicitation can provide valuable information, particularly where there is a less-developed evidence-base at the point of market access. In these circumstances, formal methods to elicit expert judgements are preferred to improve the accountability and transparency of the decision-making process, help reduce bias and the use of heuristics, and also provide a structure that allows uncertainty to be expressed. Expert elicitation is the process of transforming the subjective and implicit knowledge of experts into their quantifiable expressions. The use of expert elicitation in HTA is gaining momentum, and there is particular interest in its application to diagnostics, medical devices and complex interventions such as in public health or social care. Compared with the gathering of experimental evidence, elicitation constitutes a reasonably low-cost source of evidence. Given its inherent subject nature, the potential biases in elicited evidence cannot be ignored and, due to its infancy in HTA, there is little guidance to the analyst wishing to conduct a formal elicitation exercise. This article attempts to summarise the stages of designing and conducting an expert elicitation, drawing on key literature and examples, most of which are not in HTA. In addition, we critique their applicability to HTA, given its distinguishing features. There are a number of issues that the analyst should be mindful of, in particular the need to appropriately characterise the uncertainty associated with model inputs and the fact that there are often numerous parameters required, not all of which can be defined using the same quantities. This increases the need for the elicitation task to be as straightforward as possible for the expert to complete.

Expert assessment of the resilience of drinking water and sanitation systems to climate-related hazards

We conducted an expert assessment to obtain expert opinions on the relative global resilience of ten drinking water and five sanitation technologies to the following six climate-related hazards: drought, decreased inter-annual precipitation, flood, superstorm flood, wind damage, and saline intrusion. Resilience scores ranged from 1.7 to 9.9 out of a maximum resilience of 10, with high scores corresponding to high resilience. We find that for some climate-related hazards, such as drought, technologies demonstrated a large range in resilience, indicating that the choice of water and sanitation technologies is important for areas prone to drought. On the other hand, the range of resilience scores for superstorm flooding was much smaller, particularly for sanitation technologies, suggesting that the choice of technology is less of a determinant of functionality for superstorm flooding as compared to other climate-related hazards. For drinking water technologies, only treated piped utility-managed systems that use surface water had resilience scores >6.0 for all hazards, while protected dug wells were found to be one of the least resilient technologies, consistently scoring <5.0 for all hazards except wind damage. In general, sanitation technologies were found to have low to medium resilience, suggesting that sanitation systems need to be adapted to ensure functionality during and after climate-related hazards. The results of the study can be used to help communities decide which technologies are best suited for the climate-related challenges they face and help in future adaptation planning.

Wounds research for patient benefit: a 5-year programme of research

Background

Complex wounds are those that heal by secondary intention and include lower-limb ulcers, pressure ulcers and some surgical wounds. The care of people with complex wounds is costly, with care mainly being delivered by community nurses. There is a lack of current, high-quality data regarding the numbers and types of people affected, care received and outcomes achieved.

Objectives

To (1) assess how high-quality data about complex wounds can be captured effectively for use in both service planning and research while ensuring integration with current clinical data collection systems and minimal impact on staff time; (2) investigate whether or not a clinical register of people with complex wounds could give valid estimates of treatment effects, thus reducing dependence on large-scale randomised controlled trials (RCTs); (3) identify the most important research questions and outcomes for people with complex wounds from the perspectives of patients, carers and health-care professionals; (4) evaluate the potential contributions to decision-making of individual patient data meta-analysis and mixed treatment comparison meta-analysis; and (5) complete and update systematic reviews in topic areas of high priority.

Methods

To meet objectives 1 and 2 we conducted a prevalence survey and developed and piloted a longitudinal disease register. A consultative, deliberative method and in-depth interviews were undertaken to address objective 3. To address objectives 4 and 5 we conducted systematic reviews including mixed treatment comparison meta-analysis.

Results

From the prevalence survey we estimated the point prevalence of all complex wounds to be 1.47 per 1000 people (95% confidence interval 1.38 to 1.56 per 1000 people). Pressure ulcers and venous leg ulcers were the most common type of complex wound. A total of 195 people with a complex wound were recruited to a complex wounds register pilot. We established the feasibility of correctly identifying, extracting and transferring routine NHS data into the register; however, participant recruitment, data collection and tracking individual wounds in people with multiple wounds were challenging. Most patients and health professionals regarded healing of the wound as the primary treatment goal. Patients were greatly troubled by the social consequences of having a complex wound. Complex wounds are frequently a consequence of, and are themselves, a long-term condition but treatment is usually focused on healing the wound. Consultative, deliberative research agenda setting on pressure ulcer prevention and treatment with patients, carers and clinicians yielded 960 treatment uncertainties and a top 12 list of research priorities. Of 167 RCTs of complex wound treatments in a systematic review of study quality, 41% did not specify a primary outcome and the overall quality of the conduct and reporting of the research was poor. Mixed-treatment comparison meta-analysis in areas of high priority identified that matrix hydrocolloid dressings had the highest probability (70%) of being the most effective dressing for diabetic foot ulcers, whereas a hyaluronan fleece dressing had the highest probability (35%) of being the most effective dressing for venous ulcers; however, the quality of this evidence was low and uncertainty is high.

Conclusions

Complex wounds are common and costly with a poor evidence base for many frequent clinical decisions. There is little routine clinical data collection in community nursing. A prospective complex wounds register has the potential to both assist clinical decision-making and provide important research evidence but would be challenging to implement without investment in information technology in NHS community services. Future work should focus on developing insights into typical wound healing trajectories, identifying factors that are prognostic for healing and assessing the cost-effectiveness of selected wound treatments.

Funding

The National Institute for Health Research Programme Grants for Applied Research programme.

A comparison of two methods for expert elicitation in health technology assessments

BACKGROUND

When data needed to inform parameters in decision models are lacking, formal elicitation of expert judgement can be used to characterise parameter uncertainty. Although numerous methods for eliciting expert opinion as probability distributions exist, there is little research to suggest whether one method is more useful than any other method. This study had three objectives: (i) to obtain subjective probability distributions characterising parameter uncertainty in the context of a health technology assessment; (ii) to compare two elicitation methods by eliciting the same parameters in different ways; (iii) to collect subjective preferences of the experts for the different elicitation methods used.

METHODS

Twenty-seven clinical experts were invited to participate in an elicitation exercise to inform a published model-based cost-effectiveness analysis of alternative treatments for prostate cancer. Participants were individually asked to express their judgements as probability distributions using two different methods - the histogram and hybrid elicitation methods - presented in a random order. Individual distributions were mathematically aggregated across experts with and without weighting. The resulting combined distributions were used in the probabilistic analysis of the decision model and mean incremental cost-effectiveness ratios and the expected values of perfect information (EVPI) were calculated for each method, and compared with the original cost-effectiveness analysis. Scores on the ease of use of the two methods and the extent to which the probability distributions obtained from each method accurately reflected the expert's opinion were also recorded.

RESULTS

Six experts completed the task. Mean ICERs from the probabilistic analysis ranged between £162,600-£175,500 per quality-adjusted life year (QALY) depending on the elicitation and weighting methods used. Compared to having no information, use of expert opinion decreased decision uncertainty: the EVPI value at the £30,000 per QALY threshold decreased by 74-86 % from the original cost-effectiveness analysis. Experts indicated that the histogram method was easier to use, but attributed a perception of more accuracy to the hybrid method.

CONCLUSIONS

Inclusion of expert elicitation can decrease decision uncertainty. Here, choice of method did not affect the overall cost-effectiveness conclusions, but researchers intending to use expert elicitation need to be aware of the impact different methods could have.

Extrapolating Survival from Randomized Trials Using External Data: A Review of Methods

This article describes methods used to estimate parameters governing long-term survival, or times to other events, for health economic models. Specifically, the focus is on methods that combine shorter-term individual-level survival data from randomized trials with longer-term external data, thus using the longer-term data to aid extrapolation of the short-term data. This requires assumptions about how trends in survival for each treatment arm will continue after the follow-up period of the trial. Furthermore, using external data requires assumptions about how survival differs between the populations represented by the trial and external data. Study reports from a national health technology assessment program in the United Kingdom were searched, and the findings were combined with “pearl-growing” searches of the academic literature. We categorized the methods that have been used according to the assumptions they made about how the hazards of death vary between the external and internal data and through time, and we discuss the appropriateness of the assumptions in different circumstances. Modeling choices, parameter estimation, and characterization of uncertainty are discussed, and some suggestions for future research priorities in this area are given.

Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold

BACKGROUND

Cost-effectiveness analysis involves the comparison of the incremental cost-effectiveness ratio of a new technology, which is more costly than existing alternatives, with the cost-effectiveness threshold. This indicates whether or not the health expected to be gained from its use exceeds the health expected to be lost elsewhere as other health-care activities are displaced. The threshold therefore represents the additional cost that has to be imposed on the system to forgo 1 quality-adjusted life-year (QALY) of health through displacement. There are no empirical estimates of the cost-effectiveness threshold used by the National Institute for Health and Care Excellence.

OBJECTIVES

(1) To provide a conceptual framework to define the cost-effectiveness threshold and to provide the basis for its empirical estimation. (2) Using programme budgeting data for the English NHS, to estimate the relationship between changes in overall NHS expenditure and changes in mortality. (3) To extend this mortality measure of the health effects of a change in expenditure to life-years and to QALYs by estimating the quality-of-life (QoL) associated with effects on years of life and the additional direct impact on QoL itself. (4) To present the best estimate of the cost-effectiveness threshold for policy purposes.

METHODS

Earlier econometric analysis estimated the relationship between differences in primary care trust (PCT) spending, across programme budget categories (PBCs), and associated disease-specific mortality. This research is extended in several ways including estimating the impact of marginal increases or decreases in overall NHS expenditure on spending in each of the 23 PBCs. Further stages of work link the econometrics to broader health effects in terms of QALYs.

RESULTS

The most relevant 'central' threshold is estimated to be £12,936 per QALY (2008 expenditure, 2008-10 mortality). Uncertainty analysis indicates that the probability that the threshold is < £20,000 per QALY is 0.89 and the probability that it is < £30,000 per QALY is 0.97. Additional 'structural' uncertainty suggests, on balance, that the central or best estimate is, if anything, likely to be an overestimate. The health effects of changes in expenditure are greater when PCTs are under more financial pressure and are more likely to be disinvesting than investing. This indicates that the central estimate of the threshold is likely to be an overestimate for all technologies which impose net costs on the NHS and the appropriate threshold to apply should be lower for technologies which have a greater impact on NHS costs.

LIMITATIONS

The central estimate is based on identifying a preferred analysis at each stage based on the analysis that made the best use of available information, whether or not the assumptions required appeared more reasonable than the other alternatives available, and which provided a more complete picture of the likely health effects of a change in expenditure. However, the limitation of currently available data means that there is substantial uncertainty associated with the estimate of the overall threshold.

CONCLUSIONS

The methods go some way to providing an empirical estimate of the scale of opportunity costs the NHS faces when considering whether or not the health benefits associated with new technologies are greater than the health that is likely to be lost elsewhere in the NHS. Priorities for future research include estimating the threshold for subsequent waves of expenditure and outcome data, for example by utilising expenditure and outcomes available at the level of Clinical Commissioning Groups as well as additional data collected on QoL and updated estimates of incidence (by age and gender) and duration of disease. Nonetheless, the study also starts to make the other NHS patients, who ultimately bear the opportunity costs of such decisions, less abstract and more 'known' in social decisions.

FUNDING

The National Institute for Health Research-Medical Research Council Methodology Research Programme.

Economics of stratified medicine in rheumatoid arthritis

Clinically relevant examples of stratified medicine are available for patients with rheumatoid arthritis (RA). The aim of this study was to understand the current economic evidence for stratified medicine in RA. Two systematic reviews were conducted to identify: (1) all economic evaluations of stratified treatments for rheumatoid arthritis, or those which have used a subgroup analysis, and (2) all stated preference studies of treatments for rheumatoid arthritis. Ten economic evaluations of stratified treatments for RA, 38 economic evaluations including with a subgroup analysis and eight stated preference studies were identified. There was some evidence to support that stratified approaches to treating a patient with RA may be cost-effective. However, there remain key gaps in the economic evidence base needed to support the introduction of stratified medicine in RA into healthcare systems and considerable uncertainty about how proposed stratified approaches will impact future patient preferences, outcomes and costs when used in routine practice.

An internet-based method to elicit experts' beliefs for Bayesian priors: a case study in intracranial stent evaluation

RATIONALE

Bayesian methods provide an interesting approach to assessing an implantable medical device (IMD) that has evolved through successive versions because they allow for explicit incorporation of prior knowledge into the analysis. However, the literature is sparse on the feasibility and reliability of elicitation in cases where expert beliefs are used to form priors.

OBJECTIVES

To develop an Internet-based method for eliciting experts' beliefs about the success rate of an intracranial stenting procedure and to assess their impact on the estimated benefit of the latest version.

STUDY DESIGN AND SETTING

The elicitation questionnaire was administered to a group of nineteen experts. Elicited experts' beliefs were used to inform the prior distributions of a Bayesian hierarchical meta-analysis model, allowing for the estimation of the success rate of each version.

RESULTS

Experts believed that the success rate of the latest version was slightly higher than that of the previous one (median: 80.8 percent versus 75.9 percent). When using noninformative priors in the model, the latest version was found to have a lower success rate (median: 83.1 percent versus 86.0 percent), while no difference between the two versions was detected with informative priors (median: 85.3 percent versus 85.6 percent).

CONCLUSIONS

We proposed a practical method to elicit experts' beliefs on the success rates of successive IMD versions and to explicitly combine all available evidence in the evaluation of the latest one. Our results suggest that the experts were overoptimistic about this last version. Nevertheless, the proposed method should be simplified and assessed in larger, representative samples.

Belief elicitation to populate health economic models of medical diagnostic devices in development

BACKGROUND AND OBJECTIVE

Bayesian methods can be used to elicit experts' beliefs about the clinical value of healthcare technologies. This study investigates a belief-elicitation method for estimating diagnostic performance in an early stage of development of photoacoustic mammography (PAM) imaging versus magnetic resonance imaging (MRI) for detecting breast cancer.

RESEARCH DESIGN

Eighteen experienced radiologists ranked tumor characteristics regarding their importance to detect malignancies. With reference to MRI, radiologists estimated the true positives and negatives of PAM using the variable interval method. An overall probability density function was determined using linear opinion pooling, weighted for individual experts' experience.

RESULT

The most important tumor characteristics are mass margins and mass shape. Respondents considered MRI the better technology to visualize these characteristics. Belief elicitation confirmed this by providing an overall sensitivity of PAM ranging from 58.9 to 85.1% (mode 75.6%) and specificity ranging from 52.2 to 77.6% (mode 66.5%).

CONCLUSION

Belief elicitation allowed estimates to be obtained for the expected diagnostic performance of PAM, although radiologists expressed difficulties in doing so. Heterogeneity within and between experts reflects this uncertainty and the infancy of PAM. Further clinical trials are required to validate the extent to which this belief-elicitation method is predictive for observed test performance.

The economic considerations and implications of the stratification of future oncology therapeutics

Cancer accounts for approximately 13 % of all deaths worldwide. Development of stratification biomarkers, for cancer screening, diagnosis, monitoring, and treatment optimization, is a vital concept to facilitate disease prevention and drug development. The advent of stratified medicine should result in the safer, more effective use of therapeutic drugs to treat cancer, and in reducing the cost associated with inappropriate therapeutic regimens; however, many barriers delay the use of biomarkers in drug development and clinical practice. Since the incorporation of biomarkers in clinical practice might have additional initial costs, the question arises regarding whether the improvement in outcomes is reached at a realistic additional cost. This review presents an overview of economic issues surrounding biomarkers in cancer treatment optimization.

Improving early cycle economic evaluation of diagnostic technologies

The rapidly increasing range and expense of new diagnostics, compels consideration of a different, more proactive approach to health economic evaluation of diagnostic technologies. Early cycle economic evaluation is a decision analytic approach to evaluate technologies in development so as to increase the return on investment as well as patient and societal impact. This paper describes examples of 'early cycle economic evaluations' as applied to diagnostic technologies and highlights challenges in its real-time application. It shows that especially in the field of diagnostics, with rapid technological developments and a changing regulatory climate, early cycle economic evaluation can have a guiding role to improve the efficiency of the diagnostics innovation process. In the next five years the attention will move beyond the methodological and analytic challenges of early cycle economic evaluation towards the challenge of effectively applying it to improve diagnostic research and development and patient value. Future work in this area should therefore be 'strong on principles and soft on metrics', that is, the metrics that resonate most clearly with the various decision makers in this field.

Is expert opinion reliable when estimating transition probabilities? The case of HCV-related cirrhosis in Egypt

BACKGROUND

Data on HCV-related cirrhosis progression are scarce in developing countries in general, and in Egypt in particular. The objective of this study was to estimate the probability of death and transition between different health stages of HCV (compensated cirrhosis, decompensated cirrhosis and hepatocellular carcinoma) for an Egyptian population of patients with HCV-related cirrhosis.

METHODS

We used the "elicitation of expert opinions" method to obtain collective knowledge from a panel of 23 Egyptian experts (among whom 17 were hepatologists or gastroenterologists and 2 were infectiologists). The questionnaire was based on virtual medical cases and asked the experts to assess probability of death or probability of various cirrhosis complications. The design was a Delphi study: we attempted to obtain a consensus between experts via a series of questionnaires interspersed with group response feedback.

RESULTS

We found substantial disparity between experts' answers, and no consensus was reached at the end of the process. Moreover, we obtained high death probability and high risk of hepatocellular carcinoma. The annual transition probability to death was estimated at between 10.1% and 61.5% and the annual probability of occurrence of hepatocellular carcinoma was estimated at between 16.8% and 58.9% (depending on age, gender, time spent in cirrhosis and cirrhosis severity).

CONCLUSIONS

Our results show that eliciting expert opinions is not suited for determining the natural history of diseases due to practitioners' difficulties in evaluating quantities. Cognitive bias occurring during this type of study might explain our results.

Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment

BACKGROUND

Elicitation is a technique that can be used to obtain probability distribution from experts about unknown quantities. We conducted a methodology review of reports where probability distributions had been elicited from experts to be used in model-based health technology assessments.

METHODS

Databases including MEDLINE, EMBASE and the CRD database were searched from inception to April 2013. Reference lists were checked and citation mapping was also used. Studies describing their approach to the elicitation of probability distributions were included. Data was abstracted on pre-defined aspects of the elicitation technique. Reports were critically appraised on their consideration of the validity, reliability and feasibility of the elicitation exercise.

RESULTS

Fourteen articles were included. Across these studies, the most marked features were heterogeneity in elicitation approach and failure to report key aspects of the elicitation method. The most frequently used approaches to elicitation were the histogram technique and the bisection method. Only three papers explicitly considered the validity, reliability and feasibility of the elicitation exercises.

CONCLUSION

Judged by the studies identified in the review, reports of expert elicitation are insufficient in detail and this impacts on the perceived usability of expert-elicited probability distributions. In this context, the wider credibility of elicitation will only be improved by better reporting and greater standardisation of approach. Until then, the advantage of eliciting probability distributions from experts may be lost.

Current methodological issues in the economic assessment of personalized medicine

There is a need for methodological scrutiny in the economic assessment of personalized medicine. In this article, we present a list of 10 specific issues that we argue pose specific methodological challenges that require careful consideration when designing and conducting robust model-based economic evaluations in the context of personalized medicine. Key issues are related to the correct framing of the research question, interpretation of test results, data collection of medical management options after obtaining test results, and expressing the value of tests. The need to formulate the research question clearly and be explicit and specific about the technology being evaluated is essential because various test kits can have the same purpose and yet differ in predictive value, costs, and relevance to practice and patient populations. The correct reporting of sensitivity/specificity, and especially the false negatives and false positives (which are population dependent), of the investigated tests is also considered as a key element. This requires additional structural complexity to establish the relationship between the test result and the consecutive treatment changes and outcomes. This process involves translating the test characteristics into clinical utility, and therefore outlining the clinical and economic consequences of true and false positives and true and false negatives. Information on treatment patterns and on their costs and outcomes, however, is often lacking, especially for false-positive and false-negative test results. The analysis can even become very complex if different tests are combined or sequentially used. This potential complexity can be handled by explicitly showing how these tests are going to be used in practice and then working with the combined sensitivities and specificities of the tests. Each of these issues leads to a higher degree of uncertainty in economic models designed to assess the added value of personalized medicine compared with their simple pharmaceutical counterparts. To some extent, these problems can be overcome by performing early population-level simulations, which can lead to the identification and collection of data on critical input parameters. Finally, it is important to understand that a test strategy does not necessarily lead to more quality-adjusted life-years (QALYs). It is possible that the test will lead to not only fewer QALYs but also fewer costs, which can be defined as "decremental" cost per QALYs. Different decision criteria are needed to interpret such results.

Probability elicitation to inform early health economic evaluations of new medical technologies: a case study in heart failure disease management

OBJECTIVES

Early estimates of the commercial headroom available to a new medical device can assist producers of health technology in making appropriate product investment decisions. The purpose of this study was to illustrate how this quantity can be captured probabilistically by combining probability elicitation with early health economic modeling. The technology considered was a novel point-of-care testing device in heart failure disease management.

METHODS

First, we developed a continuous-time Markov model to represent the patients' disease progression under the current care setting. Next, we identified the model parameters that are likely to change after the introduction of the new device and interviewed three cardiologists to capture the probability distributions of these parameters. Finally, we obtained the probability distribution of the commercial headroom available per measurement by propagating the uncertainty in the model inputs to uncertainty in modeled outcomes.

RESULTS

For a willingness-to-pay value of €10,000 per life-year, the median headroom available per measurement was €1.64 (interquartile range €0.05-€3.16) when the measurement frequency was assumed to be daily. In the subsequently conducted sensitivity analysis, this median value increased to a maximum of €57.70 for different combinations of the willingness-to-pay threshold and the measurement frequency.

CONCLUSIONS

Probability elicitation can successfully be combined with early health economic modeling to obtain the probability distribution of the headroom available to a new medical technology. Subsequently feeding this distribution into a product investment evaluation method enables stakeholders to make more informed decisions regarding to which markets a currently available product prototype should be targeted.

An expert on every street corner? Methods for eliciting distributions in geographically dispersed opinion pools

Recent publications outline developments in eliciting probabilistic opinions from clinical experts with which to inform structural assumptions and parameter estimates in health economic models. We outline approaches taken to date to elicit probabilistic distributions from experts within the health economic literature and outline the appropriate considerations and the resulting process in developing a new elicitation program with the aim of allowing low-cost elicitation of expert opinion from a heterogeneous and geographically dispersed opinion pool while preserving the essential features of good practice elicitation methods.

Methods to assess cost-effectiveness and value of further research when data are sparse: negative-pressure wound therapy for severe pressure ulcers

Health care resources are scarce, and decisions have to be made about how to allocate funds. Often, these decisions are based on sparse or imperfect evidence. One such example is negative-pressure wound therapy (NPWT), which is a widely used treatment for severe pressure ulcers; however, there is currently no robust evidence that it is effective or cost-effective. This work considers the decision to adopt NPWT given a range of alternative treatments, using a decision analytic modeling approach. Literature searches were conducted to identify existing evidence on model parameters. Given the limited evidence base, a second source of evidence, beliefs elicited from experts, was used. Judgments from experts on relevant (uncertain) quantities were obtained through a formal elicitation exercise. Additionally, data derived from a pilot trial were also used to inform the model. The 3 sources of evidence were collated, and the impact of each on cost-effectiveness was evaluated. An analysis of the value of further information indicated that a randomized controlled trial may be worthwhile in reducing decision uncertainty, where from a set of alternative designs, a 3-arm trial with longer follow-up was estimated to be the most efficient. The analyses presented demonstrate how allocation decisions about medical technologies can be explicitly informed when data are sparse and how this kind of analyses can be used to guide future research prioritization, not only indicating whether further research is worthwhile but what type of research is needed and how it should be designed.

A pilot randomised controlled trial of negative pressure wound therapy to treat grade III/IV pressure ulcers [ISRCTN69032034]

BACKGROUND

Negative pressure wound therapy (NPWT) is widely promoted as a treatment for full thickness wounds; however, there is a lack of high-quality research evidence regarding its clinical and cost effectiveness. A trial of NPWT for the treatment of grade III/IV pressure ulcers would be worthwhile but premature without assessing whether such a trial is feasible. The aim of this pilot randomised controlled trial was to assess the feasibility of conducting a future full trial of NPWT for the treatment of grade III and IV pressure ulcers and to pilot all aspects of the trial.

METHODS

This was a two-centre (acute and community), pilot randomised controlled trial. Eligible participants were randomised to receive either NPWT or standard care (SC) (spun hydrocolloid, alginate or foam dressings). Outcome measures were time to healing of the reference pressure ulcer, recruitment rates, frequency of treatment visits, resources used and duration of follow-up.

RESULTS

Three hundred and twelve patients were screened for eligibility into this trial over a 12-month recruitment period and 12/312 participants (3.8%) were randomised: 6 to NPWT and 6 to SC. Only one reference pressure ulcer healed (NPWT group) during follow-up (time to healing 79 days). The mean number of treatment visits per week was 3.1 (NPWT) and 5.7 (SC); 6/6 NPWT and 1/6 SC participants withdrew from their allocated trial treatment. The mean duration of follow-up was 3.8 (NPWT) and 5.0 (SC) months.

CONCLUSIONS

This pilot trial yielded vital information for the planning of a future full study including projected recruitment rate, required duration of follow-up and extent of research nurse support required. Data were also used to inform the cost-effectiveness and value of information analyses, which were conducted alongside the pilot trial.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN69032034.