The use of expert elicitation in environmental health impact assessment: a seven step procedure

Overcoming challenges in the economic evaluation of interventions to optimise antibiotic use

Bacteria are becoming increasingly resistant to antibiotics, reducing our ability to treat infections and threatening to undermine modern health care. Optimising antibiotic use is a key element in tackling the problem. Traditional economic evaluation methods do not capture many of the benefits from improved antibiotic use and the potential impact on resistance. Not capturing these benefits is a major obstacle to optimising antibiotic use, as it fails to incentivise the development and use of interventions to optimise the use of antibiotics and preserve their effectiveness (stewardship interventions). Estimates of the benefits of improving antibiotic use involve considerable uncertainty as they depend on the evolution of resistance and associated health outcomes and costs. Here we discuss how economic evaluation methods might be adapted, in the face of such uncertainties. We propose a threshold-based approach that estimates the minimum resistance-related costs that would need to be averted by an intervention to make it cost-effective. If it is probable that without the intervention costs will exceed the threshold then the intervention should be deemed cost-effective.

Use of an expert elicitation methodology to compare welfare impacts of two approaches for blood sampling European badgers (Meles meles) in the field

In the UK and Republic of Ireland, the European badger (Meles meles) is considered the most significant wildlife reservoir of the bacterium Mycobacterium bovis, the cause of bovine tuberculosis (bTB). To expand options for bTB surveillance and disease control, the Animal and Plant Health Agency developed a bespoke physical restraint cage to facilitate collection of a small blood sample from a restrained, conscious badger in the field. A key step, prior to pursuing operational deployment of the novel restraint cage, was an assessment of the relative welfare impacts of the approach. We used an established welfare assessment model to elicit expert opinion during two workshops to compare the impacts of the restraint cage approach with the only current alternative for obtaining blood samples from badgers in the field, which involves administration of a general anaesthetic. Eleven panellists participated in the workshops, comprising experts in the fields of wildlife biology, animal welfare science, badger capture and sampling, and veterinary science. Both approaches were assessed to have negative welfare impacts, although in neither case were overall welfare scores higher than intermediate, never exceeding 5-6 out of a possible 8. Based on our assessments, the restraint cage approach is no worse for welfare compared to using general anaesthesia and possibly has a lower overall negative impact on badger welfare. Our results can be used to integrate consideration of badger welfare alongside other factors, including financial cost and efficiency, when selecting a field method for blood sampling free-living badgers.

Rewiring care delivery through Digital Therapeutics (DTx): a machine learning-enhanced assessment and development (M-LEAD) framework

BACKGROUND

Digital transformation has sparked profound change in the healthcare sector through the development of innovative digital technologies. Digital Therapeutics offer an innovative approach to disease management and treatment. Care delivery is increasingly patient-centered, data-driven, and based on real-time information. These technological innovations can lead to better patient outcomes and support for healthcare professionals, also considering resource scarcity. As these digital technologies continue to evolve, the healthcare field must be ready to integrate them into processes to take advantage of their benefits. This study aims to develop a framework for the development and assessment of Digital Therapeutics.

METHODS

The study was conducted relying on a mixed methodology. 338 studies about Digital Therapeutics resulting from a systematic literature review were analyzed using descriptive statistics through RStudio. Machine learning algorithms were applied to analyze variables and find patterns in the data. The results of these analytical analyses were summarized in a framework qualitatively tested and validated through expert opinion elicitation.

RESULTS

The research provides M-LEAD, a Machine Learning-Enhanced Assessment and Development framework that recommends best practices for developing and assessing Digital Therapeutics. The framework takes as input Digital Therapeutics characteristics, regulatory aspects, study purpose, and assessment domains. The framework produces as outputs recommendations to design the Digital Therapeutics study characteristics.

CONCLUSIONS

The framework constitutes the first step toward standardized guidelines for the development and assessment of Digital Therapeutics. The results may support manufacturers and inform decision-makers of the relevant results of the Digital Therapeutics assessment.

Public health impact of a US ban on menthol in cigarettes and cigars: a simulation study

INTRODUCTION

The US Food and Drug Administration most recently announced its intention to ban menthol cigarettes and cigars nationwide in April 2021. Implementation of the ban will require evidence that it would improve public health. This paper simulates the potential public health impact of a ban on menthol in cigarettes and cigars through its impacts on smoking initiation, smoking cessation and switching to nicotine vaping products (NVPs).

METHODS

After calibrating an established US simulation model to reflect recent use trends in cigarette and NVP use, we extended the model to incorporate menthol and non-menthol cigarette use under a status quo scenario. Applying estimates from a recent expert elicitation on the behavioural impacts of a menthol ban, we developed a menthol ban scenario with the ban starting in 2021. We estimated the public health impact as the difference between smoking and vaping-attributable deaths and life-years lost in the status quo scenario and the menthol ban scenario from 2021 to 2060.

RESULTS

As a result of the ban, overall smoking was estimated to decline by 15% as early as 2026 due to menthol smokers quitting both NVP and combustible use or switching to NVPs. These transitions are projected to reduce cumulative smoking and vaping-attributable deaths from 2021 to 2060 by 5% (650 000 in total) and reduce life-years lost by 8.8% (11.3 million). Sensitivity analyses showed appreciable public health benefits across different parameter specifications.

CONCLUSIONS AND RELEVANCE

Our findings strongly support the implementation of a ban on menthol in cigarettes and cigars.

Guidelines for model adaptation: A study of the transferability of a general seagrass ecosystem Dynamic Bayesian Networks model

In general, it is not feasible to collect enough empirical data to capture the entire range of processes that define a complex system, either intrinsically or when viewing the system from a different geographical or temporal perspective. In this context, an alternative approach is to consider model transferability, which is the act of translating a model built for one environment to another less well‐known situation. Model transferability and adaptability may be extremely beneficial—approaches that aid in the reuse and adaption of models, particularly for sites with limited data, would benefit from widespread model uptake. Besides the reduced effort required to develop a model, data collection can be simplified when transferring a model to a different application context. The research presented in this paper focused on a case study to identify and implement guidelines for model adaptation. Our study adapted a general Dynamic Bayesian Networks (DBN) of a seagrass ecosystem to a new location where nodes were similar, but the conditional probability tables varied. We focused on two species of seagrass (Zostera noltei and Zostera marina) located in Arcachon Bay, France. Expert knowledge was used to complement peer‐reviewed literature to identify which components needed adjustment including parameterization and quantification of the model and desired outcomes. We adopted both linguistic labels and scenario‐based elicitation to elicit from experts the conditional probabilities used to quantify the DBN. Following the proposed guidelines, the model structure of the general DBN was retained, but the conditional probability tables were adapted for nodes that characterized the growth dynamics in Zostera spp. population located in Arcachon Bay, as well as the seasonal variation on their reproduction. Particular attention was paid to the light variable as it is a crucial driver of growth and physiology for seagrasses. Our guidelines provide a way to adapt a general DBN to specific ecosystems to maximize model reuse and minimize re‐development effort. Especially important from a transferability perspective are guidelines for ecosystems with limited data, and how simulation and prior predictive approaches can be used in these contexts.

More vaquita porpoises survive than expected

In 2018, it was estimated that fewer than 20 of Mexico’s endemic vaquita porpoise Phocoena sinus remained, and the species was declining by 47% yr-1. Entanglement in gillnets is the sole threat to the species, and since the last population size estimate, gillnetting has increased in the small area where most vaquitas remain—a 12 × 24 km area in the Gulf of California near San Felipe, Mexico. We conducted research efforts in 2019 and 2021 in that area to estimate the minimum numbers of adults and calves and look for any signs that vaquitas are unhealthy. Through expert elicitation, we estimated between 7 and 15 unique individuals were seen in 2019 and 5-13 were seen in 2021. Calves were seen in both years, and all vaquitas appeared healthy. Population projections from the last full survey indicated that more vaquitas have survived than expected. We suggest that these surviving adult vaquitas may have learned to avoid entanglement in gillnets. These vaquitas and their calves provide hope that the species can survive. However, given the high levels of illegal gillnetting and the theft of equipment which hindered our monitoring efforts, and with only around 10 individuals remaining, survival can only be assured if vaquita habitat is made gillnet-free.

Qualitative modelling of social determinants of health using group model building: the case of debt, poverty, and health

Background

Social determinants of health (SDoH) are known to have a large impact on health outcomes, but their effects are difficult to make visible. They are part of complex systems of variables largely indirect effects on multiple levels, constituting so-called wicked problems. This study describes a participatory approach using group model building (GMB) with stakeholders, in order to develop a qualitative causal model of the health effects of SDoH, taking poverty and debt in the Dutch city of Utrecht as a case study.

Methods

With GMB we utilised the perspective of stakeholders who are directly involved in policy and practice regarding poverty, debt, and/or health. This was done using system dynamic modelling, in three interactive sessions lasting three hours each. In these sessions, they constructed a model, resulting in a system of variables with causal relationships and feedback loops. Subsequently, the results of these GMB sessions were compared to scientific literature and reviewed by a panel of researchers with extensive experience in relevant scientific fields.

Results

The resulting model contains 71 causal relationships between 39 variables, 29 of which are present in feedback loops. The variables of participation in society, stress, shame, social contacts and use of services/provisions appear to hold prominent roles in the model’s mechanisms. Most of the relationships in the model are supported by scientific literature. The researchers reviewing the model in the scientific meeting agreed that the vast majority of relationships would concur with scientific knowledge, but that the model constructed by the stakeholders consists mostly of individual-level factors, while important conditions usually relate to systemic variables.

Conclusions

Building a model with GMB helps grasp the complex situation of a wicked problem, for which it is unlikely that its interrelationships result in a fully intuitive understanding with linear mechanisms. Using this approach, effects of SDoH can be made visible and the body of evidence expanded. Importantly, it elicits stakeholders’ perspectives on a complex reality and offers a non-arbitrary way of formulating the model structure. This qualitative model is also well suited to serve as conceptual input for a quantitative model, which can be used to test and estimate the relationships.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12939-022-01676-7.

An Innovative Approach for Subnational Climate Adaptation of Biodiversity and Ecosystems: The Case Study of a Regional Strategy in Italy

Since climate change impacts are already occurring, urgent adaptive actions are necessary to avoid the worst damages. Regional authorities play an important role in adaptation, but they have few binding guidelines to carry out strategies and plans. Sectoral impacts and adaptive measures strongly differ between regions; therefore, specific results for each territory are needed. Impacts are often not exhaustively reported by literature, dataset and models, thus making it impossible to objectively identify specific adaptive measures. Usual expert elicitation helps to fill this gap but shows some issues. For the Piedmont Strategy, an innovative approach has been proposed, involving experts of private and public bodies (regional authorities, academia, research institutes, parks, associations, NGOs, etc.). They collaborated in two work group, first to identify current and future impacts on biodiversity and ecosystems, and secondly to elaborate and prioritize measures. Involving 143 experts of 46 affiliations, it was possible to quickly edit a cross-validated list of impacts (110) and measures (92) with limited costs. Lastly, a public return of results took place. This approach proved to be effective, efficient and influenced the policymakers, overcoming the tendency to enact long-term actions to face climate change. It could be used internationally by subnational authorities also in other sectors.

Qualitative Risk Assessment for Antimicrobial Resistance among Humans from Salmon Fillet Consumption Due to the High Use of Antibiotics against Bacterial Infections in Farmed Salmon

Background: Worldwide, aquaculture is considered as a hotspot environment for antimicrobial resistance (AMR) due to the intense use of antibiotics in its productive systems. Chile is the second largest producer of farmed salmon worldwide, and tons of antibiotics are used to control bacterial diseases, such as Salmon Rickettsial Syndrome (SRS) and Bacterial Kidney Disease (BKD). However, studies determining the risk of consuming salmon fillets that have been treated with antibiotics during the salmon production are limited. Consulting leading experts in the field could provide a knowledge base to identify and address this question and research gaps. Methods: Multisectoral risk perception of AMR through salmon fillet consumption was evaluated by eliciting expert data obtained through discussions during a workshop and from questionnaires given to experts from academia (n = 15, 63%), the public sector (n = 5, 21%), and the salmon industry (n = 4, 17%). Results: The qualitative risk analysis suggested an overall ‘low’ probability of AMR acquisition by consumption of salmon fillet that had been treated during the production cycle. The risk perception varied slightly between production stages in freshwater and seawater. In consensus with all sectors, this overall ‘low’, but existing, risk was probably associated with bacterial infections and the use of antibiotics. Conclusions: As it is essential to reduce the use of antibiotics in the Chilean salmon industry, this intersectoral approach and consensual results could favor effective implementation of targeted initiatives for the control and prevention of major bacterial diseases.

Reference Case Methods for Expert Elicitation in Health Care Decision Making

BACKGROUND

The evidence used to inform health care decision making (HCDM) is typically uncertain. In these situations, the experience of experts is essential to help decision makers reach a decision. Structured expert elicitation (referred to as elicitation) is a quantitative process to capture experts' beliefs. There is heterogeneity in the existing elicitation methodology used in HCDM, and it is not clear if existing guidelines are appropriate for use in this context. In this article, we seek to establish reference case methods for elicitation to inform HCDM.

METHODS

We collated the methods available for elicitation using reviews and critique. In addition, we conducted controlled experiments to test the accuracy of alternative methods. We determined the suitability of the methods choices for use in HCDM according to a predefined set of principles for elicitation in HCDM, which we have also generated. We determined reference case methods for elicitation in HCDM for health technology assessment (HTA).

RESULTS

In almost all methods choices available for elicitation, we found a lack of empirical evidence supporting recommendations. Despite this, it is possible to define reference case methods for HTA. The reference methods include a focus on gathering experts with substantive knowledge of the quantities being elicited as opposed to those trained in probability and statistics, eliciting quantities that the expert might observe directly, and individual elicitation of beliefs, rather than solely consensus methods. It is likely that there are additional considerations for decision makers in health care outside of HTA.

CONCLUSIONS

The reference case developed here allows the use of different methods, depending on the decision-making setting. Further applied examples of elicitation methods would be useful. Experimental evidence comparing methods should be generated.

Disruptive Innovations for Well-Functioning Food Systems: The Data-Driven “Food and Nutrition Security Under Climate Evolution” Framework

The current climate crisis poses new uncertainties, risks, and vulnerabilities, and is leading to losses for millions of people depending on fragile food systems. Food systems are, however, vastly different across landscapes and communities, and their capacities to respond to climate impacts evolves and changes through time. Humanitarian and development organizations are struggling to keep pace with these changes. Monitoring a large number of diverse food systems during an evolving climate crisis can be expensive and time-consuming. This paper introduces a monitoring approach that uses a combination of open-source earth observations along with national data sources to produce highly contextualized metrics for monitoring Food And Nutrition Security under Climate Evolution (FANSCE). Entirely data-driven, the FANSCE approach has been designed to produce policy recommendations to help monitor, assess, and mitigate climatic impacts on food systems. We developed and tested this approach in Vietnam, where climate variability has become a growing threat to food systems. Our results show that predictors of food and nutrition security differ drastically with the intensity of climate variability. More specifically, our analyses suggest that in areas of high climate variability, levels of food and nutrition security can be significantly predicted based on economic activities, ethnicity, education, health of mothers, and the level of readiness and preparedness to climate impacts of villages and communities. On the other hand, in areas of low climate variability, food and nutrition security are mostly predictable based on the ability of households to access essential services (such as education, health) and communal resources (water, storage, etc.). To support the resilience of food systems, policymakers must regularly monitor how these dimensions react to the changing climate. Addition critical actions to increase food system sustainability in Vietnam include 1) enhanced coordination of institutional responses and capacities across governmental and non-governmental agencies, and 2) better integration of scientific knowledge into national and sub-national decision-making processes.

Palliative care in Japanese long-term care facilities

Background:

Internationally, the demand for the provision of high-quality palliative care is increasing exponentially. Japan is a rapidly ageing society and it is therefore important to understand the current context to develop practice and inform future resource allocation in long-term care facilities (LTCFs).

Aim:

To describe the broad context of palliative care services, practice, education and policies in LTCFs in Japan.

Methods:

A process of expert elicitation and documentary review undertaken. The WHO palliative care model for public health was used as an analytical framework.

Findings:

Several deficits were identified regarding palliative care in Japanese LTCFs: a lack of palliative care strategy and standardised training for staff, and limited information on the use of palliative care medicines, quality improvement tools and frameworks.

Conclusion:

Such deficits need to be addressed to inform current nursing practice, future training needs for staff, quality care initiatives and decision making for advance care planning and resource allocation to respond to the ever-increasing demand for palliative care in these settings.

Energy transition pathways amongst low-income urban households: A mixed method clustering approach

Studies on clean energy transition amongst low-income urban households in the Global South use an array of qualitative and quantitative methods. However, attempts to combine qualitative and quantitative methods are rare and there are a lack of systematic approaches to this. This paper demonstrates a two stage approach using clustering methods to analyse a mixed dataset containing quantitative household survey data and qualitative interview data. By clustering the quantitative and qualitative data separately, latent groups with common characteristics and narratives arising from each of the two analyses are identified. A second stage of clustering identifies links between these qualitative and quantitative clusters and enables inference of energy transition pathways followed by low-income urban households defined by both quantitative characteristics and qualitative narratives. This approach can support interdisciplinary collaboration in energy research, providing a systematic approach to comparing and identifying links between quantitative and qualitative findings.•A mixed dataset comprising of quantitative survey data and qualitative interview data on low-income household energy use is analysed using hierarchical clustering to detect communities within each dataset.•Interviewees are matched to quantitative survey clusters and a second stage of clustering is performed using cluster membership as variables.•Second stage clusters identify common pairs of survey and interview clusters which define energy transition pathways based on socio-economic characteristics, energy use patterns, and narratives for decision making and practices.

One Coin, Two Sides: Eliciting Expert Knowledge From Training Participants in a Capacity-Building Program for Veterinary Professionals

Scientific research may include the elicitation of judgment from non-academic subject-matter experts in order to improve the quality and/or impact of research studies. Elicitation of expert knowledge or judgment is used when data are missing, incomplete, or not representative for the specific setting and processes being studied. Rigorous methods are crucial to ensure robust study results, and yet the quality of the elicitation can be affected by a number of practical constraints, including the understanding that subject-matter experts have of the elicitation process itself. In this paper, we present a case of expert elicitation embedded within an extended training course for veterinary professionals as an example of overcoming these constraints. The coupling of the two activities enabled extended opportunities for training and a relationship of mutual respect to be the foundation for the elicitation process. In addition, the participatory research activities reinforced knowledge synthesis objectives of the educational program. Finally, the synergy between the two concurrent objectives may produce benefits which transcend either independent activity: solutions and ideas built by local professionals, evolving collaborative research and training approaches, and a network of diverse academic and practicing professionals. This approach has the versatility to be adapted to many training and research opportunities.

Self-Reporting of Risk Pathways and Parameter Values for Foot-and-Mouth Disease in Slaughter Cattle from Alternative Production Systems by Kenyan and Ugandan Veterinarians

Countries in which foot-and-mouth disease (FMD) is endemic may face bans on the export of FMD-susceptible livestock and products because of the associated risk for transmission of FMD virus. Risk assessment is an essential tool for demonstrating the fitness of one’s goods for the international marketplace and for improving animal health. However, it is difficult to obtain the necessary data for such risk assessments in many countries where FMD is present. This study bridged the gaps of traditional participatory and expert elicitation approaches by partnering with veterinarians from the National Veterinary Services of Kenya (n = 13) and Uganda (n = 10) enrolled in an extended capacity-building program to systematically collect rich, local knowledge in a format appropriate for formal quantitative analysis. Participants mapped risk pathways and quantified variables that determine the risk of infection among cattle at slaughter originating from each of four beef production systems in each country. Findings highlighted that risk processes differ between management systems, that disease and sale are not always independent events, and that events on the risk pathway are influenced by the actions and motivations of value chain actors. The results provide necessary information for evaluating the risk of FMD among cattle pre-harvest in Kenya and Uganda and provide a framework for similar evaluation in other endemic settings.

Exploring the Feasibility of Comprehensive Uncertainty Assessment in Health Economic Modeling: A Case Study

OBJECTIVES

Decision makers adopt health technologies based on health economic models that are subject to uncertainty. In an ideal world, these models parameterize all uncertainties and reflect them in the cost-effectiveness probability and risk associated with the adoption. In practice, uncertainty assessment is often incomplete, potentially leading to suboptimal reimbursement recommendations and risk management. This study examines the feasibility of comprehensive uncertainty assessment in health economic models.

METHODS

A state transition model on peripheral arterial disease treatment was used as a case study. Uncertainties were identified and added to the probabilistic sensitivity analysis if possible. Parameter distributions were obtained by expert elicitation, and structural uncertainties were either parameterized or explored in scenario analyses, which were model averaged.

RESULTS

A truly comprehensive uncertainty assessment, parameterizing all uncertainty, could not be achieved. Expert elicitation informed 8 effectiveness, utility, and cost parameters. Uncertainties were parameterized or explored in scenario analyses and with model averaging. Barriers included time and resource constraints, also of clinical experts, and lacking guidance regarding some aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The team's multidisciplinary expertise and existing literature and tools were facilitators.

CONCLUSIONS

While comprehensive uncertainty assessment may not be attainable, improvements in uncertainty assessment in general are no doubt desirable. This requires the development of detailed guidance and hands-on tutorials for methods of uncertainty assessment, in particular aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The issue of benefits of uncertainty assessment versus time and resources needed remains unclear.

Guidance for the Harmonisation and Improvement of Economic Evaluations of Personalised Medicine

OBJECTIVE

The objective of this study was to develop guidance contributing to improved consistency and quality in economic evaluations of personalised medicine (PM), given current ambiguity about how to measure the value of PM as well as considerable variation in the methodology and reporting in economic evaluations of PM.

METHODS

A targeted literature review of methodological papers was performed for an overview of modelling challenges in PM. Expert interviews were held to discuss best modelling practice. A systematic literature review of economic evaluations of PM was conducted to gain insight into current modelling practice. The findings were synthesised and used to develop a set of draft recommendations. The draft recommendations were discussed at a stakeholder workshop and subsequently finalised.

RESULTS

Twenty-two methodological papers were identified. Some argued that the challenges in modelling PM can be addressed within existing methodological frameworks, others disagreed. Eighteen experts were interviewed. They believed large uncertainty to be a key concern. Out of 195 economic evaluations of PM identified, 56% addressed none of the identified modelling challenges. A set of 23 recommendations was developed. Eight recommendations focus on the modelling of test-treatment pathways. The use of non-randomised controlled trial data is discouraged but several recommendations are provided in case randomised controlled trial data are unavailable. The parameterisation of structural uncertainty is recommended. Other recommendations consider perspective and discounting; premature survival data; additional value elements; patient and clinician compliance; and managed entry agreements.

CONCLUSIONS

This study provides a comprehensive list of recommendations to modellers of PM and to evaluators and reviewers of PM models.

An Expert Elicitation on the Effects of a Ban on Menthol Cigarettes and Cigars in the United States

INTRODUCTION

The US FDA announced its intention to ban menthol in cigarettes. However, information is needed on how a federal ban would affect population health. We conducted an expert elicitation to gauge the impact of a menthol cigarette and cigar ban in the US.

METHODS

We developed and pilot tested a questionnaire that focused on tobacco use transitions of current smokers (age 18-24 menthol, age 35-54 menthol, and age 35-54 non-menthol) and potential menthol smokers (age 12-24). Using a structured expert elicitation, we estimated mean net transitions under a ban from cigarette use to combustible tobacco product, smokeless tobacco, novel nicotine delivery product (NNDPs, such as e-cigarettes) use, or no tobacco use.

RESULTS

Eleven experts provided responses. Of those ages 12-24 who would have initiated menthol cigarette use in the absence of a ban, the experts estimated that 41% would still initiate combustible products under a ban, while 18% would initiate with NNDPs and 39% would not initiate regular tobacco use. Combustible use by menthol smokers ages 35-54 was expected to decline by 20% post-ban relative to pre-ban rates, half switching to NNDPs and half quitting all tobacco use. Menthol smokers ages 18-24 were expected to reduce combustible use by 30%, with 16% switching to NNDPs. Greater reductions in combustible use were estimated for African-Americans across the three age groups. Negligible impacts were expected for current adult non-menthol smokers.

CONCLUSIONS

According to expert opinion, a menthol ban is expected to substantially reduce smoking initiation and combustible tobacco product use among current menthol smokers.

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

BACKGROUND

Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making.

OBJECTIVES

The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts' judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed.

METHODS

A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the 'choices') and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken.

RESULTS

For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution.

LIMITATIONS

The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are.

CONCLUSIONS

The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required.

FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).

Validation of 4D Components for Measuring Quality of the Public Health Data Collection Process: Elicitation Study

Background

Identification of the essential components of the quality of the data collection process is the starting point for designing effective data quality management strategies for public health information systems. An inductive analysis of the global literature on the quality of the public health data collection process has led to the formation of a preliminary 4D component framework, that is, data collection management, data collection personnel, data collection system, and data collection environment. It is necessary to empirically validate the framework for its use in future research and practice.

Objective

This study aims to obtain empirical evidence to confirm the components of the framework and, if needed, to further develop this framework.

Methods

Expert elicitation was used to evaluate the preliminary framework in the context of the Chinese National HIV/AIDS Comprehensive Response Information Management System. The research processes included the development of an interview guide and data collection form, data collection, and analysis. A total of 3 public health administrators, 15 public health workers, and 10 health care practitioners participated in the elicitation session. A framework qualitative data analysis approach and a quantitative comparative analysis were followed to elicit themes from the interview transcripts and to map them to the elements of the preliminary 4D framework.

Results

A total of 302 codes were extracted from interview transcripts. After iterative and recursive comparison, classification, and mapping, 46 new indicators emerged; 24.8% (37/149) of the original indicators were deleted because of a lack of evidence support and another 28.2% (42/149) were merged. The validated 4D component framework consists of 116 indicators (82 facilitators and 34 barriers). The first component, data collection management, includes data collection protocols and quality assurance. It was measured by 41 indicators, decreased from the original 49% (73/149) to 35.3% (41/116). The second component, data collection environment, was measured by 37 indicators, increased from the original 13.4% (20/149) to 31.9% (37/116). It comprised leadership, training, funding, organizational policy, high-level management support, and collaboration among parallel organizations. The third component, data collection personnel, includes the perception of data collection, skills and competence, communication, and staffing patterns. There was no change in the proportion for data collection personnel (19.5% vs 19.0%), although the number of its indicators was reduced from 29 to 22. The fourth component, the data collection system, was measured using 16 indicators, with a slight decrease in percentage points from 18.1% (27/149) to 13.8% (16/116). It comprised functions, system integration, technical support, and data collection devices.

Conclusions

This expert elicitation study validated and improved the 4D framework. The framework can be useful in developing a questionnaire survey instrument for measuring the quality of the public health data collection process after validation of psychometric properties and item reduction.

Expert Elicitation To Estimate the Feed Safety Impact of Criteria Included in the Canadian Food Inspection Agency Risk Assessment Model for Feed Mills

ABSTRACT

The Canadian Food Inspection Agency is developing an Establishment-based Risk Assessment (ERA) model for commercial and on-farm mills involved in the manufacture, storage, packaging, labeling, or distribution of livestock feed (ERA-Feed Mill model). This model will help inform the allocation of inspection resources on the basis of feed safety risk, including animal health and food safety risk. In a previous study, 34 risk factors, grouped into inherent, mitigation, and compliance clusters, along with assessment criteria were selected. The objective of this current study was to estimate the relative risk (RR) of the 203 assessment criteria on the basis of the impact on feed safety to design an ERA-Feed Mill model algorithm. Furthermore, the intent of this study was to assess the maximum increase or decrease of risk obtained when multiple criteria belonging to a same cluster were identified in a specific feed mill. To do so, a two-round face-to-face expert elicitation was conducted with 28 Canadian feed experts. Results showed no significant association between respondent profiles (years of experience and work sector) and estimated RR. Uniformity of answers between experts improved between rounds. Criteria having the highest increase in risk (median RR ≥ 4) included the presence of materials prohibited to be fed to ruminants in a facility that produces ruminant feed, the presence of multiple livestock species on-site, and historical noncompliances related to the inspection of the feed mill's process control and end-product control programs. Risk mitigation criteria having the highest impact on decreasing the risk were the implementation of feed safety certifications, the use of dedicated manufacturing lines (prohibited materials or medications), and having a hazard sampling plan in place for finished feed. The median RR assigned to each criterion and cluster will be used to build an algorithm of the Canadian Food Inspection Agency's ERA-Feed Mill model.

HIGHLIGHTS

An analytical framework to assess SDG targets within the context of WEFE nexus in the Mediterranean region

Understanding the complex relationships amongst Water, Energy, Food and Ecosystems (WEFE nexus) together with the achievement of Sustainable Development Goals (SDGs) is critical for the development of a sustainable and secure future in the Mediterranean area. In this study, we analysed 29 case studies across the Mediterranean region which describe potential success stories for the implementation of good nexus practices. We developed an analytical framework for investigating the impacts on 15 SDG targets and we also explicitly quantified the magnitude of interconnection of nexus pillars with SDGs. Our findings showed that renewable energies have a predominant role on sustainability. Moreover, to achieve the highest positive impacts on economy, environment and society, it is necessary to ensure that both people and ecosystems benefit from a minimum amount of goods/qualities as expected by specific targets like SDG 6.1-4 (clean water and sanitation) and 15.1-3 (life on land), as well as 7.2-3 (affordable and clean energy) that are strongly linked with 13.1 (climate action). We showed also that the strongest interconections between SDG and WEFE are present for the categories of renewable energy system (RED and REW). However, the analysis showed that there is a tendency to focus on a specific sector (e.g. agriculture) and that the good nexus practices implementation is not enough to understand the achievement and progress towards the SDGs. For that reason, we recommended that a more holistic nexus approach including end of supply chain options should be systematically integrated into the project design or evaluation.

The effects of climate change on Baltic salmon: Framing the problem in collaboration with expert stakeholders

In the Baltic Sea region, salmon are valued for the ecological, economic, and cultural benefits they provide. However, these fish are threatened due to historical overfishing, disease, and reduced access to spawning rivers. Climate change may pose another challenge for salmon management. Therefore, we conducted a problem-framing study to explore the effects climate change may have on salmon and the socio-ecological system they are embedded within. Addressing this emerging issue will require the cooperation of diverse stakeholders and the integration of their knowledge and values in a contentious management context. Therefore, we conducted this problem framing as a participatory process with stakeholders, whose mental models and questionnaire responses form the basis of this study. By framing the climate change problem in this way, we aim to provide a holistic understanding of the problem and incorporate stakeholder perspectives into the management process from an early stage to better address their concerns and establish common ground. We conclude that considering climate change is relevant for Baltic salmon management, although it may not be the most pressing threat facing these fish. Stakeholders disagree about whether climate change will harm or benefit salmon, when it will become a relevant issue in the Baltic context, and whether or not management efforts can mitigate any negative impacts climate change may have on salmon and their fishery. Nevertheless, by synthesizing the stakeholders' influence diagrams, we found 15 themes exemplifying: (1) how climate change may affect salmon, (2) goals for salmon management considering climate change, and (3) strategies for achieving those goals. Further, the stakeholders tended to focus on the riverine environment and the salmon life stages occurring therein, potentially indicating the perceived vulnerability of these life stages to climate change. Interestingly, however, the stakeholders tended to focus on traditional fishery management measures, like catch quotas, to meet their goals for these fish considering climate change. Further, social variables, like "politics," "international cooperation," and "employment" comprised a large proportion of the stakeholders' diagrams, demonstrating the importance of these factors for salmon management.

Developing successful environmental decision support systems: Challenges and best practices

Environmental decision support systems (EDSSs), or DSS applied in the environmental field, have been developed for over 40 years now. However, most of these tools fail to find use or fall out of use extremely quickly. In the aim of aiding in the conception and development of practical and successful decision support systems, i.e. systems that can lead to positive outcomes, this review looks over the existing literature, both EDSS-centric and from broader decision-related fields, to highlight some of the most important challenges influencing the success and usability of these systems. In all, 13 major challenges facing EDSS development were identified and over 60 recommendations and best practices were provided to address these challenges. Though this paper is mainly focused on environmental decision support systems, most of the highlighted information and conclusions are applicable to the development of decision support systems in any field.

Risk Mapping of Influenza D Virus Occurrence in Ruminants and Swine in Togo Using a Spatial Multicriteria Decision Analysis Approach

Influenza D virus (IDV) has been identified in several continents, with serological evidence for the virus in Africa. In order to improve the sensitivity and cost–benefit of IDV surveillance in Togo, risk maps were drawn using a spatial multicriteria decision analysis (MCDA) and experts’ opinion to evaluate the relevance of sampling areas used so far. Areas at highest risk of IDV occurrence were the main cattle markets. The maps were evaluated with previous field surveillance data collected in Togo between 2017 and 2019: 1216 sera from cattle, small ruminants, and swine were screened for antibodies to IDV by hemagglutination inhibition (HI) assays. While further samples collections are needed to validate the maps, the risk maps resulting from the spatial MCDA approach generated here highlight several priority areas for IDV circulation assessment.

Characterizing the role of socioeconomic pathways in shaping future urban heat-related challenges

Urban dwellers worldwide are increasingly affected by more frequent and intense extreme temperature events, ongoing urbanization, and changes in socioeconomic conditions. Decades of research have shown that vulnerability is a crucial determinant of heat-related risk and mortality in cities, yet assessments of future urban heat-related challenges have largely overlooked the contribution of changes in socioeconomic conditions to future heat-related risk and mortality. The scenario framework for climate change research, made up of socioeconomic scenarios (Shared Socioeconomic Pathways - SSPs) combined with climate scenarios (Representative Concentration Pathways - RCPs), facilitates the integration of socioeconomic scenarios into climate risks assessments. In this study, we used Greater Houston (Texas) as a case study to implement the scenario framework at the intra-urban scale. Integrating locally extended SSPs along with a range of sectoral modelling approaches, we combined projections of urban extreme heat - which account for SSP-specific urban heat islands - with projections of future population and vulnerability. We then produced estimates of future heat-related risk and mortality for 2041-2060 (2050s) summers at Census tract level, for multiple combinations of climate and socioeconomic scenarios. Using a scenario matrix, we showed that the projected ~15,738-24,521 future summer excess mortalities compared to 1991-2010 are essentially driven by population growth and changes in vulnerability, with changes in climatic conditions alone being of little influence. We outline methods to apply the new scenario framework at intra-urban scale and to better characterize the contribution of socioeconomic pathways to future urban climate risks. This socio-climatic approach provides comprehensive estimates of future climate risks in urban areas, which are essential for adaptation planning under climatic and socioeconomic uncertainty.

Social-ecological change in the Omo-Turkana basin: A synthesis of current developments

This paper synthesizes current knowledge on the impacts of the Gibe III dam and associated large-scale commercial farming in the Omo-Turkana Basin, based on an expert elicitation coupled with a scoping review and the collective knowledge of an multidisciplinary network of researchers with active data-collection programs in the Basin. We use social-ecological systems and political ecology frameworks to assess the impacts of these interventions on hydrology and ecosystem services in the Basin, and cascading effects on livelihoods, patterns of migration, and conflict dynamics for the people of the region. A landscape-scale transformation is occurring in which commodities, rather than staple foods for local consumption, are becoming the main output of the region. Mitigation measures initiated by the Ethiopian government-notably resettlement schemes-are not adequately buffering affected communities from food insecurity following disruption to indigenous livelihood systems. Therefore, while benefits are accruing to labor migrants, the costs of development are currently borne primarily by the agro-pastoralist indigenous people of the region. We consider measures that might maximize benefits from the changes underway and mitigate their negative outcomes, such as controlled floods, irrigating fodder crops, food aid, and benefit sharing.

Bayesian Network Resource for Meta-Analysis: Cellular Toxicity of Quantum Dots

A web-based resource for meta-analysis of nanomaterials toxicity is developed whereby the utility of Bayesian networks (BNs) is illustrated for exploring the cellular toxicity of Cd-containing quantum dots (QDs). BN models are developed based on a dataset compiled from 517 publications comprising 3028 cell viability data samples and 837 IC₅₀ values. BN QD toxicity (BN-QDTox) models are developed using both continuous (i.e., numerical) and categorical attributes. Using these models, the most relevant attributes identified for correlating IC₅₀ are: QD diameter, exposure time, surface ligand, shell, assay type, surface modification, and surface charge, with the addition of QD concentration for the cell viability analysis. Data exploration via BN models further enables identification of possible association rules for QDs cellular toxicity. The BN models as web-based applications can be used for rapid intelligent query of the available body of evidence for a given nanomaterial and can be readily updated as the body of knowledge expands.

Expert Views on Their Role as Policy Advisor: Pilot Study for the Cases of Electromagnetic Fields, Particulate Matter, and Antimicrobial Resistance

This perspective presents empirical data to demonstrate the existence of different expert views on scientific policy advice on complex environmental health issues. These views are partly research-field specific. According to scientific literature, experts differ in the way they provide policy advice on complex issues such as electromagnetic fields (EMF), particulate matter (PM), and antimicrobial resistance (AMR). Where some experts feel their primary task is to carry out fundamental research, others actively engage in the policy dialogue. Although the literature provides ideas about expert roles, there exists little empirical underpinning. Our aim is to gather empirical evidence about expert roles. The results of an international study indicated that experts on EMF, PM, and AMR differ in the way they view their role in the policy dialogue. For example, experts differed in their views on the need for precaution and their motivation to initiate stakeholder cooperation. Besides, most experts thought that their views on the risks of EMF/PM/AMR did not differ from those of colleagues. Great dissensus was found in views on the best ways of managing risks and uncertainties. In conclusion, the theoretical ideal-typical roles from the literature can be identified to a certain extent.

Defining the appropriateness and inappropriateness of antibiotic prescribing in primary care

Objectives

To assess the appropriateness of prescribing systemic antibiotics for different clinical conditions in primary care, and to quantify 'ideal' antibiotic prescribing proportions in conditions for which antibiotic treatment is sometimes but not always indicated.

Methods

Prescribing guidelines were consulted to define the appropriateness of antibiotic therapy for the conditions that resulted in antibiotic prescriptions between 2013 and 2015 in The Health Improvement Network (THIN) primary care database. The opinions of subject experts were then formally elicited to quantify ideal antibiotic prescribing proportions for 10 common conditions.

Results

Of the antibiotic prescriptions in THIN, 52.5% were for conditions that could be assessed using prescribing guidelines. Among these, the vast majority of prescriptions (91.4%) were for conditions where antibiotic appropriateness is conditional on patient-specific indicators. Experts estimated low ideal prescribing proportions in acute, non-comorbid presentations of many of these conditions, such as cough (10% of patients), rhinosinusitis (11%), bronchitis (13%) and sore throat (13%). Conversely, antibiotics were believed to be appropriate in 75% of non-pregnant women with non-recurrent urinary tract infection. In impetigo and acute exacerbation of chronic obstructive pulmonary disease, experts clustered into distinct groups that believed in either high or low prescribing.

Conclusions

In English primary care, most antibiotics are prescribed for conditions that only sometimes require antibiotic treatment, depending on patient-specific indicators. Experts estimated low ideal prescribing proportions in many of these conditions. Incomplete prescribing guidelines and disagreement about prescribing in some conditions highlight further research needs.

Spatial modelling for low pathogenicity avian influenza virus at the interface of wild birds and backyard poultry

Low pathogenicity avian influenza virus (LPAIV) is endemic in wild birds and poultry in Argentina, and active surveillance has been in place to prevent any eventual virus mutation into a highly pathogenic avian influenza virus (HPAIV), which is exotic in this country. Risk mapping can contribute effectively to disease surveillance and control systems, but it has proven a very challenging task in the absence of disease data. We used a combination of expert opinion elicitation, multicriteria decision analysis (MCDA) and ecological niche modelling (ENM) to identify the most suitable areas for the occurrence of LPAIV at the interface between backyard domestic poultry and wild birds in Argentina. This was achieved by calculating a spatially explicit risk index. As evidenced by the validation and sensitivity analyses, our model was successful in identifying high-risk areas for LPAIV occurrence. Also, we show that the risk for virus occurrence is significantly higher in areas closer to commercial poultry farms. Although the active surveillance systems have been successful in detecting LPAIV-positive backyard farms and wild birds in Argentina, our predictions suggest that surveillance efforts in those compartments could be improved by including high-risk areas identified by our model. Our research provides a tool to guide surveillance activities in the future, and presents a mixed methodological approach which could be implemented in areas where the disease is exotic or rare and a knowledge-driven modelling method is necessary.

Analysis of building plumbing system flushing practices and communications

Drinking water distribution system contamination incidents can prompt public agencies and drinking water utilities to issue do-not-drink and do-not-use advisories. After the contaminant is cleared from distribution mains, consumers are often directed to flush their plumbing. However, little validated guidance and few evaluated communications strategies are available on using flushing to decontaminate building water systems. Additionally, limited data support the effectiveness of current practices and recommendations. In this study, expert elicitation was used to assess existing flushing guidance and develop validated flushing guidance and communications for single-family residences. The resulting guidance recommends progressively opening all cold-water taps from the closest to point of entry to the furthest and allowing the water to run for at least 20 minutes. Hot-water taps should be opened progressively and run for at least 75 minutes. The guidance language and format conformed to grade-level and readability scores within recommended health communication ranges. The readability of eight other flushing guidance documents was also evaluated for contamination incidents from 2008-2015. Seven were written at a 10th-12th grade level, above the 6th-7th grade level recommended for health communications.

A framework for reviewing livestock disease reporting systems in high-risk areas: assessing performance and perceptions towards foot and mouth disease reporting in the Thrace region of Greece, Bulgaria and Turkey

Disease reporting is an essential frontline component of surveillance systems, particularly for detecting incursions of new and emerging diseases. It has the advantages of being comprehensive and continuous, with the potential to reduce the time of disease detection and the extent of consequent spread. A number of exotic diseases, including sheep and goat pox, lumpy skin disease, peste des petits ruminants and foot and mouth disease have historically entered into south-eastern Europe through the Thrace region, which extends across neighbouring areas of Greece, Bulgaria and Turkey. In this high-risk area, multiple factors can reduce the sensitivity of disease reporting across the diverse production systems and animal health services need robust and effective disease reporting systems. While describing a training exercise designed to provide animal health services of the three countries with the knowledge and skills for conducting comprehensive in-country assessments, we provide an initial evaluation of the sensitivity of foot and mouth disease reporting and identify gaps and constraints in the Thrace region. An expert elicitation approach was used to consult official veterinarians from central and local animal health authorities of the three countries, and scenario trees modelling was applied to analyse the collected data. The reported sensitivity of disease reporting often varied between the central and local veterinary authorities within the three countries. Awareness of clinical disease, of reporting procedures and of biosecurity measures affected the early stages of disease reporting, particularly in the production systems identified at lower reporting sensitivity such as small ruminant's herds, mixed bovine herds and backyard herds. Despite its limitations this training exercise provided an effective framework (a) to develop capacities of the veterinary services in the region and (b) to supply initial evidence for guiding further interventions targeting those sectors and stakeholders at lower reporting sensitivity to reduce risks of disease introduction.

Expert assessments of the cost and expected future performance of proton exchange membrane fuel cells for vehicles

Despite decades of development, proton exchange membrane fuel cells (PEMFCs) still lack wide market acceptance in vehicles. To understand the expected trajectories of PEMFC attributes that influence adoption, we conducted an expert elicitation assessment of the current and expected future cost and performance of automotive PEMFCs. We elicited 39 experts' assessments of PEMFC system cost, stack durability, and stack power density under a hypothetical, large-scale production scenario. Experts assessed the median 2017 automotive cost to be $75/kW, stack durability to be 4,000 hours, and stack power density to be 2.5 kW/L. However, experts ranged widely in their assessments. Experts' 2017 best cost assessments ranged from $40 to $500/kW, durability assessments ranged from 1,200 to 12,000 hours, and power density assessments ranged from 0.5 to 4 kW/L. Most respondents expected the 2020 cost to fall short of the 2020 target of the US Department of Energy (DOE). However, most respondents anticipated that the DOE's ultimate target of $30/kW would be met by 2050 and a power density of 3 kW/L would be achieved by 2035. Fifteen experts thought that the DOE's ultimate durability target of 8,000 hours would be met by 2050. In general, experts identified high Pt group metal loading as the most significant barrier to reducing cost. Recommended research and development (R&D) funding was allocated to "catalysts and electrodes," followed in decreasing amount by "fuel cell performance and durability," "membranes and electrolytes," and "testing and technical assessment." Our results could be used to inform public and private R&D decisions and technology roadmaps.

An Expert Survey on the Material Types used to Start Cookstoves

Household air pollution generated using solid-fuel cookstoves is a leading risk factor for morbidity and mortality worldwide. Many studies have quantified cookstove emissions with respect to the main fuels used (e.g., wood, charcoal and other biomass fuels). Anecdotal evidence suggests a variety of fuels are used to ignite cookstoves, however quantitative information on startup fuel types is lacking. Emissions from startup materials contribute to household air pollution. As such, understanding startup practices and the choices of startup materials is integral to understanding the full burden of household air pollution from cookstoves on health and the environment. We conducted an expert elicitation style survey to gather information about startup practices worldwide. Twenty-three respondents from academic and private sectors responded to a survey instrument on cookstove startup, providing information that covered 48 geographic locations across 22 countries. Responses to the open-ended questions were analyzed to quantify how startup materials vary according to factors such as location, seasonality, and the main cooking fuel/stove type. A wide variety of materials were reportedly used to ignite cookstoves, including many non-biomass materials that may have toxic combustion byproducts. Paper, plastic, agricultural wastes, waxes and other petroleum fuels (e.g., kerosene), and rubber-like materials (e.g., tires, footwear) were the most commonly indicated startup materials. Additional materials mentioned included fabrics, plastic packaging, soda bottles, snack food wrappers, and trash. Results from this survey can be used to direct future research on the impacts of startup materials on health and the environment, such as justifying the choice of materials to test in the laboratory.

Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management

Introduction

Natural resource management uses expert judgement to estimate facts that inform important decisions. Unfortunately, expert judgement is often derived by informal and largely untested protocols, despite evidence that the quality of judgements can be improved with structured approaches. We attribute the lack of uptake of structured protocols to the dearth of illustrative examples that demonstrate how they can be applied within pressing time and resource constraints, while also improving judgements.

Aims and methods

In this paper, we demonstrate how the IDEA protocol for structured expert elicitation may be deployed to overcome operational challenges while improving the quality of judgements. The protocol was applied to the estimation of 14 future abiotic and biotic events on the Great Barrier Reef, Australia. Seventy-six participants with varying levels of expertise related to the Great Barrier Reef were recruited and allocated randomly to eight groups. Each participant provided their judgements using the four-step question format of the IDEA protocol (‘Investigate’, ‘Discuss’, ‘Estimate’, ‘Aggregate’) through remote elicitation. When the events were realised, the participant judgements were scored in terms of accuracy, calibration and informativeness.

Results and conclusions

The results demonstrate that the IDEA protocol provides a practical, cost-effective, and repeatable approach to the elicitation of quantitative estimates and uncertainty via remote elicitation. We emphasise that i) the aggregation of diverse individual judgements into pooled group judgments almost always outperformed individuals, and ii) use of a modified Delphi approach helped to remove linguistic ambiguity, and further improved individual and group judgements. Importantly, the protocol encourages review, critical appraisal and replication, each of which is required if judgements are to be used in place of data in a scientific context. The results add to the growing body of literature that demonstrates the merit of using structured elicitation protocols. We urge decision-makers and analysts to use insights and examples to improve the evidence base of expert judgement in natural resource management.

Calculation of the disease burden associated with environmental chemical exposures: application of toxicological information in health economic estimation

Calculation of costs and the Burden of Disease (BoD) is useful in developing resource allocation and prioritization strategies in public and environmental health. While useful, the Disability-Adjusted Life Year (DALY) metric disregards subclinical dysfunctions, adheres to stringent causal criteria, and is hampered by gaps in environmental exposure data, especially from industrializing countries. For these reasons, a recently calculated environmental BoD of 5.18% of the total DALYs is likely underestimated. We combined and extended cost calculations for exposures to environmental chemicals, including neurotoxicants, air pollution, and endocrine disrupting chemicals, where sufficient data were available to determine dose-dependent adverse effects. Environmental exposure information allowed cost estimates for the U.S. and the EU, for OECD countries, though less comprehensive for industrializing countries. As a complement to these health economic estimations, we used attributable risk valuations from expert elicitations to as a third approach to assessing the environmental BoD. For comparison of the different estimates, we used country-specific monetary values of each DALY. The main limitation of DALY calculations is that they are available for few environmental chemicals and primarily based on mortality and impact and duration of clinical morbidity, while less serious conditions are mostly disregarded. Our economic estimates based on available exposure information and dose-response data on environmental risk factors need to be seen in conjunction with other assessments of the total cost for these environmental risk factors, as our estimate overlaps only slightly with the previously estimated environmental DALY costs and crude calculations relying on attributable risks for environmental risk factors. The three approaches complement one another and suggest that environmental chemical exposures contribute costs that may exceed 10% of the global domestic product and that current DALY calculations substantially underestimate the economic costs associated with preventable environmental risk factors. By including toxicological and epidemiological information and data on exposure distributions, more representative results can be obtained from utilizing health economic analyses of the adverse effects associated with environmental chemicals.

Direct costs of transfusion reactions - an expert judgement approach

BACKGROUND AND OBJECTIVES

Despite increasingly meticulous haemovigilance reporting throughout the world, a systematic assessment of the cost of transfusion reactions is still lacking. This is partly caused by the fact that such an assessment requires a subjective expert assessment of the additional costs linked to the adverse reaction. Data on the cost of transfusion reactions could support decision-making regarding blood transfusion safety measures.

MATERIALS AND METHODS

Thirteen experts from nine hospitals were asked to estimate the additional care required following various types of transfusion reactions. Additional care was quantified as the proportion of reactions requiring care, and the amount of care required (e.g. hospitalization days, additional physician's time). Experts were also asked to provide, per type of transfusion reaction, an estimate of the proportion of transfusion reactions preventable. Structured quantitative expert elicitation methods were applied to obtain and combine expert estimates.

RESULTS

The estimated annual in-hospital cost of transfusion reactions in the Netherlands is €933 356 per year (€1.52 per transfusion). Two-thirds (64%) of these are incurred by non-serious transfusion reactions. Circulatory overload, TRALI and anaphylaxis clearly dominate the costs of serious adverse transfusion reactions (66% in total); non-haemolytic transfusion reactions incur 46% of the cost of non-serious transfusion reactions. Additional safety measures targeting circulatory overload and new antibody formation potentially offer the highest cost reduction.

CONCLUSION

In-hospital costs of transfusion reactions are substantial but contribute to less than 1% of the total cost of transfusion in the Netherlands. A considerable part of these costs (24%) might be preventable.

Informing Public Perceptions About Climate Change: A 'Mental Models' Approach

As the specter of climate change looms on the horizon, people will face complex decisions about whether to support climate change policies and how to cope with climate change impacts on their lives. Without some grasp of the relevant science, they may find it hard to make informed decisions. Climate experts therefore face the ethical need to effectively communicate to non-expert audiences. Unfortunately, climate experts may inadvertently violate the maxims of effective communication, which require sharing communications that are truthful, brief, relevant, clear, and tested for effectiveness. Here, we discuss the 'mental models' approach towards developing communications, which aims to help experts to meet the maxims of effective communications, and to better inform the judgments and decisions of non-expert audiences.

A Risk Assessment Framework for Seed Degeneration: Informing an Integrated Seed Health Strategy for Vegetatively Propagated Crops

Pathogen buildup in vegetative planting material, termed seed degeneration, is a major problem in many low-income countries. When smallholder farmers use seed produced on-farm or acquired outside certified programs, it is often infected. We introduce a risk assessment framework for seed degeneration, evaluating the relative performance of individual and combined components of an integrated seed health strategy. The frequency distribution of management performance outcomes was evaluated for models incorporating biological and environmental heterogeneity, with the following results. (1) On-farm seed selection can perform as well as certified seed, if the rate of success in selecting healthy plants for seed production is high; (2) when choosing among within-season management strategies, external inoculum can determine the relative usefulness of 'incidence-altering management' (affecting the proportion of diseased plants/seeds) and 'rate-altering management' (affecting the rate of disease transmission in the field); (3) under severe disease scenarios, where it is difficult to implement management components at high levels of effectiveness, combining management components can be synergistic and keep seed degeneration below a threshold; (4) combining management components can also close the yield gap between average and worst-case scenarios. We also illustrate the potential for expert elicitation to provide parameter estimates when empirical data are unavailable. [Formula: see text] Copyright © 2017 The Author(s). This is an open access article distributed under the CC BY-NC-ND 4.0 International license .

Strategic risk appraisal. Comparing expert- and literature-informed consequence assessments for environmental policy risks receiving national attention

Strategic risk appraisal (SRA) has been applied to compare diverse policy level risks to and from the environment in England and Wales. Its application has relied on expert-informed assessments of the potential consequences from residual risks that attract policy attention at the national scale. Here we compare consequence assessments, across environmental, economic and social impact categories that draw on 'expert'- and 'literature-based' analyses of the evidence for 12 public risks appraised by Government. For environmental consequences there is reasonable agreement between the two sources of assessment, with expert-informed assessments providing a narrower dispersion of impact severity and with median values similar in scale to those produced by an analysis of the literature. The situation is more complex for economic consequences, with a greater spread in the median values, less consistency between the two assessment types and a shift toward higher severity values across the risk portfolio. For social consequences, the spread of severity values is greater still, with no consistent trend between the severities of impact expressed by the two types of assessment. For the latter, the findings suggest the need for a fuller representation of socioeconomic expertise in SRA and the workshops that inform SRA output.

EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment

BACKGROUND

Expert opinion is often sought to complement available information needed to inform model-based economic evaluations in health technology assessments. In this context, we define expert elicitation as the process of encoding expert opinion on a quantity of interest, together with associated uncertainty, as a probability distribution. When availability for face-to-face expert elicitation with a facilitator is limited, elicitation can be conducted remotely, overcoming challenges of finding an appropriate time to meet the expert and allowing access to experts situated too far away for practical face-to-face sessions. However, distance elicitation is associated with reduced response rates and limited assistance for the expert during the elicitation session. The aim of this study was to inform the development of a remote elicitation tool by exploring the influence of mode of elicitation on elicited beliefs.

METHODS

An Excel-based tool (EXPLICIT) was developed to assist the elicitation session, including the preparation of the expert and recording of their responses. General practitioners (GPs) were invited to provide expert opinion about population alcohol consumption behaviours. They were randomised to complete the elicitation by either a face-to-face meeting or email. EXPLICIT was used in the elicitation sessions for both arms.

RESULTS

Fifteen GPs completed the elicitation session. Those conducted by email were longer than the face-to-face sessions (13 min 30 s vs 10 min 26 s, p = 0.1) and the email-elicited estimates contained less uncertainty. However, the resulting aggregated distributions were comparable.

CONCLUSIONS

EXPLICIT was useful in both facilitating the elicitation task and in obtaining expert opinion from experts via email. The findings support the opinion that remote, self-administered elicitation is a viable approach within the constraints of HTA to inform policy making, although poor response rates may be observed and additional time for individual sessions may be required.

Informing Reimbursement Decisions Using Cost-Effectiveness Modelling: A Guide to the Process of Generating Elicited Priors to Capture Model Uncertainties

In informing decisions, utilising health technology assessment (HTA), expert elicitation can provide valuable information, particularly where there is a less-developed evidence-base at the point of market access. In these circumstances, formal methods to elicit expert judgements are preferred to improve the accountability and transparency of the decision-making process, help reduce bias and the use of heuristics, and also provide a structure that allows uncertainty to be expressed. Expert elicitation is the process of transforming the subjective and implicit knowledge of experts into their quantifiable expressions. The use of expert elicitation in HTA is gaining momentum, and there is particular interest in its application to diagnostics, medical devices and complex interventions such as in public health or social care. Compared with the gathering of experimental evidence, elicitation constitutes a reasonably low-cost source of evidence. Given its inherent subject nature, the potential biases in elicited evidence cannot be ignored and, due to its infancy in HTA, there is little guidance to the analyst wishing to conduct a formal elicitation exercise. This article attempts to summarise the stages of designing and conducting an expert elicitation, drawing on key literature and examples, most of which are not in HTA. In addition, we critique their applicability to HTA, given its distinguishing features. There are a number of issues that the analyst should be mindful of, in particular the need to appropriately characterise the uncertainty associated with model inputs and the fact that there are often numerous parameters required, not all of which can be defined using the same quantities. This increases the need for the elicitation task to be as straightforward as possible for the expert to complete.