Use of Expert Judgement Across NICE Guidance-Making Programmes: A Review of Current Processes and Suitability of Existing Tools to Support the Use of Expert Elicitation

Methods to Quantify the Importance of Parameters for Model Updating and Distributional Adaptation

PURPOSE

Decision models are time-consuming to develop; therefore, adapting previously developed models for new purposes may be advantageous. We provide methods to prioritize efforts to 1) update parameter values in existing models and 2) adapt existing models for distributional cost-effectiveness analysis (DCEA).

METHODS

Methods exist to assess the influence of different input parameters on the results of a decision models, including value of information (VOI) and 1-way sensitivity analysis (OWSA). We apply 1) VOI to prioritize searches for additional information to update parameter values and 2) OWSA to prioritize searches for parameters that may vary by socioeconomic characteristics. We highlight the assumptions required and propose metrics that quantify the extent to which parameters in a model have been updated or adapted. We provide R code to quickly carry out the analysis given inputs from a probabilistic sensitivity analysis (PSA) and demonstrate our methods using an oncology case study.

RESULTS

In our case study, updating 2 of 21 probabilistic model parameters addressed 71.5% of the total VOI and updating 3 addressed approximately 100% of the uncertainty. Our proposed approach suggests that these are the 3 parameters that should be prioritized. For model adaptation for DCEA, 46.3% of the total OWSA variation came from a single parameter, while the top 10 input parameters were found to account for more than 95% of the total variation, suggesting efforts should be aimed toward these.

CONCLUSIONS

These methods offer a systematic approach to guide research efforts in updating models with new data or adapting models to undertake DCEA. The case study demonstrated only very small gains from updating more than 3 parameters or adapting more than 10 parameters.

HIGHLIGHTS

It can require considerable analyst time to search for evidence to update a model or to adapt a model to take account of equity concerns.In this article, we provide a quantitative method to prioritze parameters to 1) update existing models to reflect potential new evidence and 2) adapt existing models to estimate distributional outcomes.We define metrics that quantify the extent to which the parameters in a model have been updated or adapted.We provide R code that can quickly rank parameter importance and calculate quality metrics using only the results of a standard probabilistic sensitivity analysis.

Increasing transparency in indirect treatment comparisons: is selecting effect modifiers the missing part of the puzzle? A review of methodological approaches and critical considerations

Failure to adjust for effect modifiers (EMs) in indirect treatment comparisons (ITCs) can produce biased and uncertain effect estimates. This is particularly important for health technology assessments (HTAs), where the availability of new treatments is based on comparative effectiveness results. Much emphasis has been placed on advancing ITC methods to adjust for EMs, yet whether EMs are appropriately identified for the conduct of ITCs in the first place is unclear. To understand the extent of guidance and requirements for the selection of EMs for ITCs currently available and if and how this guidance is applied in practice, a series of pragmatic reviews of guidance documents from HTA and non-payer organizations, primary published ITC analyses, and prior HTA submissions in two indications (non-small cell lung cancer and psoriasis) was conducted. The reviews showed that current ITC guidance mainly focused on developing analytical methods to adjust for EMs. Some organizations, such as HTA bodies in the UK, France and Germany, recommended the use of literature reviews, expert opinion and statistical methods to identify EMs. No detailed guidance on the selection process or the appropriate literature review approach was found. Similar trends were identified through the database search and review of prior HTA submissions; only few published ITCs and submissions included information on the EM selection process which was either based on findings from the literature, trial subgroup analyses, or clinical input. No reference to a systematic selection approach was found. There is an urgent need to fill the guidance gap identified across the reviews by including a step in ITC guidelines on how EMs should be identified through systematic reviews, formal expert elicitation, and a quantitative assessment of the EM distribution. Researchers and manufacturers are also encouraged to improve transparent reporting and justification of their selection of EMs to allow for an independent review of the set of factors being considered for adjustment. Both will contribute toward reducing bias in the ITC results and ultimately increase confidence in decision-making.

Strategies and checklist for designing and conducting palliative care research with family carers: EAPC international expert elicitation study

BACKGROUND

Palliative care services seek to improve the wellbeing of family carers of people living with serious and life-limiting illness. To help achieve this goal, systematic reviews have recommended priority areas for family carer research and the need to improve the quality of study design. Policy makers have also advocated for enhanced family carer support. However, there are specific methodological considerations and challenges in designing and conducting carer research conducted during the course of the serious illness trajectory and in bereavement.

AIM

To develop strategies to improve the design and conduct of research with family carers.

DESIGN

Expert elicitation study using an adapted version of the 'Identify, Discuss, Estimate and Aggregate' elicitation protocol, supplemented with strategies from peer-reviewed literature.

SETTING/PARTICIPANTS

Nine members of the management committee of the European Association for Palliative Care's Reference group on family carer research, comprising international senior research academics in family caregiving.

RESULTS

A compilation of recommended strategies and checklist was created to: (a) help researchers plan research involving family carers focussing on: preparation, conduct and dissemination and (b) assist ethics committees and funding bodies to evaluate proposals.

CONCLUSIONS

The strategies and checklist for conducting research with family carers may enhance methodologically rigorous research. Consequently, researchers, practitioners and policy makers will not only gain a more comprehensive understanding of the unmet needs of family carers but also promote the development of empirically sound interventions.

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review

BACKGROUND

Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual.

OBJECTIVES

We conducted a systematic review to compare EE methods used to generate model input parameters in health research.

DATA SOURCES

PubMed and Web of Science.

STUDY ELIGIBILITY

Modeling studies that reported the use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes.

DATA ABSTRACTION AND SYNTHESIS

Studies were classified as "formal" EE methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion but provided limited information were classified as "indeterminate" methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods.

STUDY APPRAISAL

Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods.

RESULTS

The search identified 1520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16.

LIMITATIONS

Elicitations on nonhealth topics and those reported in the gray literature were not included.

CONCLUSIONS

We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models.

HIGHLIGHTS

We find extensive use of expert elicitation for the development of model input parameters, but most studies do not provide adequate details of their elicitation methods.Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.There is a need to establish expert elicitation best practices and reporting guidelines.

Designing and Implementing Deliberative Processes for Health Technology Assessment: A Good Practices Report of a Joint HTAi/ISPOR Task Force

Objectives

Deliberative processes for health technology assessment (HTA) are intended to facilitate participatory decision making, using discussion and open dialogue between stake-holders. Increasing attention is being given to deliberative processes, but guidance is lacking for those who wish to design or use them. Health Technology Assessment International (HTAi) and ISPOR—The Professional Society for Health Economics and Outcomes Research initiated a joint Task Force to address this gap.

Methods

The joint Task Force consisted of fifteen members with different backgrounds, perspectives, and expertise relevant to the field. It developed guidance and a checklist for deliberative processes for HTA. The guidance builds upon the few, existing initiatives in the field, as well as input from the HTA community following an established consultation plan. In addition, the guidance was subject to two rounds of peer review.

Results

A deliberative process for HTA consists of procedures, activities, and events that support the informed and critical examination of an issue and the weighing of arguments and evidence to guide a subsequent decision. Guidance and an accompanying checklist are provided for (i) developing the governance and structure of an HTA program and (ii) informing how the various stages of an HTA process might be managed using deliberation.

Conclusions

The guidance and the checklist contain a series of questions, grouped by six phases of a model deliberative process. They are offered as practical tools for those wishing to establish or improve deliberative processes for HTA that are fit for local contexts. The tools can also be used for independent scrutiny of deliberative processes.

Designing and Implementing Deliberative Processes for Health Technology Assessment: A Good Practices Report of a Joint HTAi/ISPOR Task Force

OBJECTIVES

Deliberative processes for health technology assessment (HTA) are intended to facilitate participatory decision making, using discussion and open dialogue between stakeholders. Increasing attention is being given to deliberative processes, but guidance is lacking for those who wish to design or use them. Health Technology Assessment International (HTAi) and ISPOR-The Professional Society for Health Economics and Outcomes Research initiated a joint Task Force to address this gap.

METHODS

The joint Task Force consisted of 15 members with different backgrounds, perspectives, and expertise relevant to the field. It developed guidance and a checklist for deliberative processes for HTA. The guidance builds upon the few, existing initiatives in the field, as well as input from the HTA community following an established consultation plan. In addition, the guidance was subject to 2 rounds of peer review.

RESULTS

A deliberative process for HTA consists of procedures, activities, and events that support the informed and critical examination of an issue and the weighing of arguments and evidence to guide a subsequent decision. Guidance and an accompanying checklist are provided for (i) developing the governance and structure of an HTA program and (ii) informing how the various stages of an HTA process might be managed using deliberation.

CONCLUSIONS

The guidance and the checklist contain a series of questions, grouped by 6 phases of a model deliberative process. They are offered as practical tools for those wishing to establish or improve deliberative processes for HTA that are fit for local contexts. The tools can also be used for independent scrutiny of deliberative processes.

Methods for Extrapolating Survival Analyses for the Economic Evaluation of Advanced Therapy Medicinal Products

There are two significant challenges for analysts conducting economic evaluations of advanced therapy medicinal products (ATMPs): (i) estimating long-term treatment effects in the absence of mature clinical data, and (ii) capturing potentially complex hazard functions. This review identifies and critiques a variety of methods that can be used to overcome these challenges. The narrative review is informed by a rapid literature review of methods used for the extrapolation of survival analyses in the economic evaluation of ATMPs. There are several methods that are more suitable than traditional parametric survival modelling approaches for capturing complex hazard functions, including, cure-mixture models and restricted cubic spline models. In the absence of mature clinical data, analysts may augment clinical trial data with data from other sources to aid extrapolation, however, the relative merits of employing methods for including data from different sources is not well understood. Given the high and potentially irrecoverable costs of making incorrect decisions concerning the reimbursement or commissioning of ATMPs, it is important that economic evaluations are correctly specified, and that both parameter and structural uncertainty associated with survival extrapolations are considered. Value of information analyses allow for this uncertainty to be expressed explicitly, and in monetary terms.

Hemophilia Gene Therapy Value Assessment: Methodological Challenges and Recommendations

Gene therapy for hemophilia is designed to produce health gains for patients over many years. Rewarding that value creation on the basis of a one-time treatment implies a large upfront cost. This cost can only be justified by long-term health benefits and being cost-effective compared with conventional treatments. Yet, uncertainties about the long-term benefits make it challenging to assess clinical and economic value of gene therapies at launch. We identify and discuss key methodological challenges in assessing the value of gene therapy for hemophilia, including the immaturity of evidence on the durability of benefits, lack of definition and valuation of cure for chronic diseases, absence of randomized controlled trials, limitations of traditional quality of life measures in hemophilia, approach for qualifying cost-savings compared with current treatments, and choice of perspective. The Institute for Clinical and Economic Review has developed a framework for assessing single or short-term therapies (ICER-SST) and has applied it in hemophilia. After reviewing this framework and its application, we recommend the following when assessing the value of hemophilia gene therapies: (1) leveraging expert clinical opinion to justify assumptions on the durability of benefits; (2) using external synthetic controls and lead-in, self-controlled trials to assess comparative effectiveness; (3) addressing limitations of traditional quality of life measures through the use of modified utility collection approaches; (4) adjusting cost offsets from gene therapies with caution; (5) considering outcome-based contracting to address uncertainties about prices and long-term outcomes; and (6) presenting societal and healthcare system perspectives in parallel.

A toolkit of methods of development-focused health technology assessment

Health technology assessment conducted to inform decisions during technology development (development-focused or DF-HTA) has a number of distinct features compared with HTA conducted to inform reimbursement and usage decisions. In particular, there are a broad range of decisions to be informed related to the development of a technology; multiple markets and decision makers to be considered; a limited (and developing) evidence base; and constrained resources for analysis. These features impact upon methods adopted by analysts. In this paper, we (i) set out methods of DF-HTA against a timeline of technology development; (ii) provide examples of the methods’ use; and (iii) explain how they have been adapted as a result of the features of DF-HTA. We present a toolkit of methods for analysts working with developers of medical technologies. Three categories of methods are described: literature review, stakeholder consultation, and decision analytic modeling. Literature review and stakeholder consultation are often used to fill evidence gaps. Decision analytic modeling is used to synthesize available evidence alongside plausible assumptions to inform developers about price or performance requirements. Methods increase in formality and complexity as the development and evidence base progresses and more resources are available for assessment. We hope this toolkit will be used in conjunction with the framework of features of DF-HTA presented in our earlier article in order to improve the clarity and appropriateness of methods of HTA used in DF-HTA. We also seek to contribute to a continuing dialogue about the nature of, and the best approach to, DF-HTA.

Bibliometrics and Co-Citation Network Analysis of Systematic Reviews of Evidence-Based Nursing Guidelines for Preventing Inpatient Falls

Advances in bibliometrics and co-citation analysis provide the opportunity to analyze quantitatively the large amount of nursing research used in evidence-based nursing. Numerous nursing researchers have attempted to obtain evidence that using evidence-based nursing improves the quality of nursing practices. However, little is known about how these efforts comply with the rigorous methods required for a systematic review in a subject area. This study explored the comprehensiveness of systematic reviews in four guidelines for preventing inpatient falls using bibliometrics and a co-citation network technique. Citations (n = 659) and 9417 unique bibliographic records written by 6537 authors were collected from the guidelines in January 2020. The results showed none of the references spanned all four authoring bodies, whereas only 0.1% and 4.7% of the references spanned three and two of the authoring bodies, respectively. The co-citation analysis revealed differences in the scope and primary concerns among the development groups, even in the same setting, with differences in the highly influential articles and authors. These findings imply that although the systematic reviews of guidelines on fall prevention were of good quality, there remain areas for improvement in terms of harmonizing the selection of bibliographic citations comprehensively.

An Expert Elicitation on the Effects of a Ban on Menthol Cigarettes and Cigars in the United States

INTRODUCTION

The US FDA announced its intention to ban menthol in cigarettes. However, information is needed on how a federal ban would affect population health. We conducted an expert elicitation to gauge the impact of a menthol cigarette and cigar ban in the US.

METHODS

We developed and pilot tested a questionnaire that focused on tobacco use transitions of current smokers (age 18-24 menthol, age 35-54 menthol, and age 35-54 non-menthol) and potential menthol smokers (age 12-24). Using a structured expert elicitation, we estimated mean net transitions under a ban from cigarette use to combustible tobacco product, smokeless tobacco, novel nicotine delivery product (NNDPs, such as e-cigarettes) use, or no tobacco use.

RESULTS

Eleven experts provided responses. Of those ages 12-24 who would have initiated menthol cigarette use in the absence of a ban, the experts estimated that 41% would still initiate combustible products under a ban, while 18% would initiate with NNDPs and 39% would not initiate regular tobacco use. Combustible use by menthol smokers ages 35-54 was expected to decline by 20% post-ban relative to pre-ban rates, half switching to NNDPs and half quitting all tobacco use. Menthol smokers ages 18-24 were expected to reduce combustible use by 30%, with 16% switching to NNDPs. Greater reductions in combustible use were estimated for African-Americans across the three age groups. Negligible impacts were expected for current adult non-menthol smokers.

CONCLUSIONS

According to expert opinion, a menthol ban is expected to substantially reduce smoking initiation and combustible tobacco product use among current menthol smokers.

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

BACKGROUND

Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making.

OBJECTIVES

The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts' judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed.

METHODS

A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the 'choices') and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken.

RESULTS

For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution.

LIMITATIONS

The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are.

CONCLUSIONS

The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required.

FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).