An expert on every street corner? Methods for eliciting distributions in geographically dispersed opinion pools

Potential costs and consequences associated with medication error at hospital discharge: an expert judgement study

OBJECTIVES

Assessing the cost-effectiveness of complex pharmaceutical care interventions and medication error outcomes is hindered by lack of available data on actual outcomes consequent to errors that were intercepted for patient safety reasons. Expert judgement is an approach to acquire data regarding unknown parameters in an economic model which are otherwise insufficient or not possible to obtain. The aim of this paper is to describe a method to approach this problem using findings from a single intervention study and to calculate the potential costs and consequences associated with discharge medication error.

METHODS

Using data from a previous intervention study, the hypothetical consequences of medication error(s) at hospital discharge, in terms of diagnosis, healthcare resource utilisation and impact on health-related quality of life, were identified by expert judgement of anonymised cases. Primary healthcare utilisation costs were derived from published tariffs, inpatient costs were derived by simulation in the hospital discharge activity database test environment and the difference between adjudicated baseline and posterror health state was expressed as quality-adjusted life year (QALY) decrement.

RESULTS

Four experts provided judgement on 81 cases. Of these, 75 were judged to have potential clinical consequences. Between 56 and 69 of the 81 cases were variably judged to require remedial healthcare utilisation. The mean calculated cost per case (representing an individual patient), based on all 81 cases, was €1009.58, 95% CI 726.64 to 1585.67. The mean QALY loss was 0.03 (95% CI 0.01 to 0.05).

CONCLUSION

An expert judgement process proved feasible and useful to estimate financial cost and QALY loss associated with hospital discharge medication error. These estimates will be employed in model-based economic evaluation. This method could be transferred to other prospective observational patient safety research which seeks to assess value for money of complex interventions.

Protocol for a MULTI-centre feasibility study to assess the use of 99mTc-sestaMIBI SPECT/CT in the diagnosis of kidney tumours (MULTI-MIBI study)

INTRODUCTION

The incidence of renal tumours is increasing and anatomic imaging cannot reliably distinguish benign tumours from renal cell carcinoma. Up to 30% of renal tumours are benign, with oncocytomas the most common type. Biopsy has not been routinely adopted in many centres due to concerns surrounding non-diagnostic rate, bleeding and tumour seeding. As a result, benign masses are often unnecessarily surgically resected. ^99mTc-sestamibi SPECT/CT has shown high diagnostic accuracy for benign renal oncocytomas and other oncocytic renal neoplasms of low malignant potential in single-centre studies. The primary aim of MULTI-MIBI is to assess feasibility of a multicentre study of ^99mTc-sestamibi SPECT/CT against a reference standard of histopathology from surgical resection or biopsy. Secondary aims of the study include obtaining estimates of ^99mTc-sestamibi SPECT/CT sensitivity and specificity and to inform the design and conduct of a future definitive trial.

METHODS AND ANALYSIS

A feasibility prospective multicentre study of participants with indeterminate, clinical T1 renal tumours to undergo ^99mTc-sestamibi SPECT/CT (index test) compared with histopathology from biopsy or surgical resection (reference test). Interpretation of the index and reference tests will be blinded to the results of the other. Recruitment rate as well as estimates of sensitivity, specificity, positive and negative predictive value will be reported. Semistructured interviews with patients and clinicians will provide qualitative data to inform onward trial design and delivery. Training materials for ^99mTc-sestamibi SPECT/CT interpretation will be developed, assessed and optimised. Early health economic modelling using a decision analytic approach for different diagnostic strategies will be performed to understand the potential cost-effectiveness of ^99mTc-sestamibi SPECT/CT.

ETHICS AND DISSEMINATION

Ethical approval has been granted (UK HRA REC 20/YH/0279) protocol V.5.0 dated 21/6/2022. Study outputs will be presented and published nationally and internationally.

TRIAL REGISTRATION NUMBER

ISRCTN12572202.

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

BACKGROUND

Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making.

OBJECTIVES

The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts' judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed.

METHODS

A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the 'choices') and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken.

RESULTS

For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution.

LIMITATIONS

The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are.

CONCLUSIONS

The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required.

FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).

Quantifying expert opinion with discrete choice models: Invasive elodea's influence on Alaska salmonids

Scientific evidence should inform environmental policy, but rapid environmental change brings high ecological uncertainty and associated barriers to the science-management dialogue. Biological invasions of aquatic plants are a worldwide problem with uncertain ecological and economic consequences. We demonstrate that the discrete choice method (DCM) can serve as a structured expert elicitation alternative to quantify expert opinion across a range of possible but uncertain environmental outcomes. DCM is widely applied in the social sciences to better understand and predict human preferences and trade-offs. Here we apply it to Alaska's first submersed invasive aquatic freshwater plant, Elodea spp. (elodea), and its unknown effects on salmonids. While little is known about interactions between elodea and salmonids, ecological research suggests that aquatic plant invasions can have positive and negative, as well as direct and indirect, effects on fish. We use DCM to design hypothetical salmonid habitat scenarios describing elodea's possible effect on critical environmental conditions for salmonids: prey abundance, dissolved oxygen, and vegetation cover. We then observe how experts choose between scenarios that they believe could support persistent salmonid populations in elodea-invaded salmonid habitat. We quantify the relative importance of habitat characteristics that influence expert choice and investigate how experts trade off between habitat characteristics. We take advantage of Bayesian techniques to estimate discrete choice models for individual experts and to simulate expert opinion for specific environmental management situations. We discuss possible applications and advantages of the DCM approach for expert elicitation in the ecological context. We end with methodological questions for future research.

Understanding minimum and ideal factor levels for participation in physical activities by people with haemophilia: An expert elicitation exercise

INTRODUCTION

The benefits of physical activity (PA) for people with haemophilia (PWH) may include improvements in joint, bone and muscle health. However, the factor VIII activity level required to avoid a bleeding episode associated with PA is unknown.

AIM

To elicit the opinion of clinical experts on the minimum level and ideal factor VIII activity ('level') required to avoid a bleeding episode during participation in different types of PA for PWH.

METHODS

Based on the 2017 National Hemophilia Foundation PA descriptions, clinical experts estimated a minimally acceptable and an ideal factor level at which a bleed could be avoided. The uncertainty around estimates was quantified using an approach to construct a probability distribution to represent expert opinion.

RESULTS

Minimum and ideal factor level increased with higher risk PA, whether or not joint morbidity was present, as did the experts' uncertainty in their estimates (ie the range between lowest and highest estimates for minimum and ideal levels). Mean minimum levels ranged from 4% to 48% for low to high risk for people without joint morbidity, and from 7% to 47% for those with joint morbidity. For ideal factor levels, corresponding figures were 9%-52% and 12%-64%, respectively.

CONCLUSION

To support a patient-centric outcome, expert opinion indicates that the clinical norm of 0.01 IU/mL (1%) trough level is insufficient. It is anticipated that introducing a more targeted approach to meet the needs of patients who are increasingly physically active will benefit patients further in addition to recent treatment advances.

Use of Expert Judgement Across NICE Guidance-Making Programmes: A Review of Current Processes and Suitability of Existing Tools to Support the Use of Expert Elicitation

OBJECTIVES

This study aimed to review current use of experts within National Institute for Health and Care Excellence (NICE) guidance-making programmes, identify improvements in use of expert judgement, and to assess tools and protocols to support the elicitation of information from experts for use by NICE.

METHODS

The study comprised a review of NICE process guides; semi-structured interviews with individuals representing each NICE guidance-making programme and a comparison of the suitability of currently available tools and protocols for expert elicitation to the requirements of NICE programmes.

RESULTS

Information elicited from experts and the way in which it is used varies across NICE guidance-making programmes. Experts' involvement can be as intensive as being a member of a committee and thus having direct influence on recommendations or limited one-off consultations on specific parameters. We identified 16 tools for expert elicitation that were potentially relevant. None fully met the requirements of NICE, although an existing tool could be potentially adapted. Ongoing research to develop a reference protocol for expert elicitation in healthcare decision making may be of use in future.

CONCLUSIONS

NICE uses expert judgement across all its guidance-making programmes, but its uses vary considerably. There is no currently available tool for expert elicitation suitable for use by NICE. However, adaptation of an existing tool or ongoing research in the area could address this in the future.

Experiences of Structured Elicitation for Model-Based Cost-Effectiveness Analyses

BACKGROUND

Empirical evidence supporting the cost-effectiveness estimates of particular health care technologies may be limited, or it may even be missing entirely. In these situations, additional information, often in the form of expert judgments, is needed to reach a decision. There are formal methods to quantify experts' beliefs, termed as structured expert elicitation (SEE), but only limited research is available in support of methodological choices. Perhaps as a consequence, the use of SEE in the context of cost-effectiveness modelling is limited.

OBJECTIVES

This article reviews applications of SEE in cost-effectiveness modelling with the aim of summarizing the basis for methodological choices made in each application and recording the difficulties and challenges reported by the authors in the design, conduct, and analyses.

METHODS

The methods used in each application were extracted along with the criteria used to support methodological and practical choices and any issues or challenges discussed in the text. Issues and challenges were extracted using an open field, and then categorised and grouped for reporting.

RESULTS

The review demonstrates considerable heterogeneity in methods used, and authors acknowledge great methodological uncertainty in justifying their choices. Specificities of the context area emerging as potentially important in determining further methodological research in elicitation are between- expert variation and its interpretation, the fact that substantive experts in the area may not be trained in quantitative subjects, that judgments are often needed on various parameter types, the need for some form of assessment of validity, and the need for more integration with behavioural research to devise relevant debiasing strategies.

CONCLUSIONS

This review of experiences of SEE highlights a number of specificities/constraints that can shape the development of guidance and target future research efforts in this area.

EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment

BACKGROUND

Expert opinion is often sought to complement available information needed to inform model-based economic evaluations in health technology assessments. In this context, we define expert elicitation as the process of encoding expert opinion on a quantity of interest, together with associated uncertainty, as a probability distribution. When availability for face-to-face expert elicitation with a facilitator is limited, elicitation can be conducted remotely, overcoming challenges of finding an appropriate time to meet the expert and allowing access to experts situated too far away for practical face-to-face sessions. However, distance elicitation is associated with reduced response rates and limited assistance for the expert during the elicitation session. The aim of this study was to inform the development of a remote elicitation tool by exploring the influence of mode of elicitation on elicited beliefs.

METHODS

An Excel-based tool (EXPLICIT) was developed to assist the elicitation session, including the preparation of the expert and recording of their responses. General practitioners (GPs) were invited to provide expert opinion about population alcohol consumption behaviours. They were randomised to complete the elicitation by either a face-to-face meeting or email. EXPLICIT was used in the elicitation sessions for both arms.

RESULTS

Fifteen GPs completed the elicitation session. Those conducted by email were longer than the face-to-face sessions (13 min 30 s vs 10 min 26 s, p = 0.1) and the email-elicited estimates contained less uncertainty. However, the resulting aggregated distributions were comparable.

CONCLUSIONS

EXPLICIT was useful in both facilitating the elicitation task and in obtaining expert opinion from experts via email. The findings support the opinion that remote, self-administered elicitation is a viable approach within the constraints of HTA to inform policy making, although poor response rates may be observed and additional time for individual sessions may be required.

An internet-based method to elicit experts' beliefs for Bayesian priors: a case study in intracranial stent evaluation

RATIONALE

Bayesian methods provide an interesting approach to assessing an implantable medical device (IMD) that has evolved through successive versions because they allow for explicit incorporation of prior knowledge into the analysis. However, the literature is sparse on the feasibility and reliability of elicitation in cases where expert beliefs are used to form priors.

OBJECTIVES

To develop an Internet-based method for eliciting experts' beliefs about the success rate of an intracranial stenting procedure and to assess their impact on the estimated benefit of the latest version.

STUDY DESIGN AND SETTING

The elicitation questionnaire was administered to a group of nineteen experts. Elicited experts' beliefs were used to inform the prior distributions of a Bayesian hierarchical meta-analysis model, allowing for the estimation of the success rate of each version.

RESULTS

Experts believed that the success rate of the latest version was slightly higher than that of the previous one (median: 80.8 percent versus 75.9 percent). When using noninformative priors in the model, the latest version was found to have a lower success rate (median: 83.1 percent versus 86.0 percent), while no difference between the two versions was detected with informative priors (median: 85.3 percent versus 85.6 percent).

CONCLUSIONS

We proposed a practical method to elicit experts' beliefs on the success rates of successive IMD versions and to explicitly combine all available evidence in the evaluation of the latest one. Our results suggest that the experts were overoptimistic about this last version. Nevertheless, the proposed method should be simplified and assessed in larger, representative samples.

Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment

BACKGROUND

Elicitation is a technique that can be used to obtain probability distribution from experts about unknown quantities. We conducted a methodology review of reports where probability distributions had been elicited from experts to be used in model-based health technology assessments.

METHODS

Databases including MEDLINE, EMBASE and the CRD database were searched from inception to April 2013. Reference lists were checked and citation mapping was also used. Studies describing their approach to the elicitation of probability distributions were included. Data was abstracted on pre-defined aspects of the elicitation technique. Reports were critically appraised on their consideration of the validity, reliability and feasibility of the elicitation exercise.

RESULTS

Fourteen articles were included. Across these studies, the most marked features were heterogeneity in elicitation approach and failure to report key aspects of the elicitation method. The most frequently used approaches to elicitation were the histogram technique and the bisection method. Only three papers explicitly considered the validity, reliability and feasibility of the elicitation exercises.

CONCLUSION

Judged by the studies identified in the review, reports of expert elicitation are insufficient in detail and this impacts on the perceived usability of expert-elicited probability distributions. In this context, the wider credibility of elicitation will only be improved by better reporting and greater standardisation of approach. Until then, the advantage of eliciting probability distributions from experts may be lost.