Use (and abuse) of expert elicitation in support of decision making for public policy

Evaluating models of expert judgment to inform assessment of ecosystem viability and collapse

Expert judgment underpins assessment of threatened ecosystems. However, experts are often narrowly defined, and variability in their judgments may be substantial. Models built from structured elicitation with large diverse expert panels can contribute to more consistent and transparent decision-making. We conducted a structured elicitation under a broad definition of expertise to examine variation in judgments of ecosystem viability and collapse in a critically endangered ecosystem. We explored whether variation in judgments among 83 experts was related to affiliation and management expertise and assessed performance of an average model based on common ecosystem indicators. There were systematic differences among individuals, much of which were not explained by affiliation or expertise. However, of the individuals affiliated with government, those in conservation and environmental departments were more likely to determine a patch was viable than those in agriculture and rural land management. Classification errors from an average model, in which all individuals were weighted equally, were highest among government agriculture experts (27%) and lowest among government conservation experts (12%). Differences were mostly cases in which the average model predicted a patch was viable but the individual thought it was not. These differences arose primarily for areas that were grazed or cleared of mature trees. These areas are often the target of restoration, but they are also valuable for agriculture. These results highlight the potential for conflicting advice and disagreement about policies and actions for conserving and restoring threatened ecosystems. Although adoption of an average model can improve consistency of ecosystem assessment, it can fail to capture and convey diverse opinions held by experts. Structured elicitation and models of ecosystem viability play an important role in providing data-driven evidence of where differences arise among experts to support engagement and discussion among stakeholders and decision makers and to improve the management of threatened ecosystems.

Estimating public health risks of infectious disease events: A Canadian approach to rapid risk assessment

Background

The COVID-19 pandemic highlighted the need for timely, evidence-based rapid risk assessments (RRA) of infectious disease events to inform public health action during rapidly evolving situations with high uncertainty. In 2022, the Public Health Agency of Canada established a coordinated approach to public health risk assessment, including a methodology for qualitative RRA of infectious disease threats.

Objective

To describe the RRA methodology and illustrate its use with examples from different infectious hazards of public health concern.

Methods

The RRA methodology employs the risk pathway to describe the sequence of events leading from a hazard's source to the adverse event of concern and subsequent impacts; define specific questions to be addressed; and identify relevant knowledge gaps, limitations and recommendations. Qualitative likelihood and impact estimates are derived through integration of evidence review and expert opinion and are communicated together with corresponding levels of uncertainty. The impacts of the event are based on an assessment of the most likely spread scenario within Canada, considering individual-level impact on affected individuals, the impact on the general population and, if relevant, sub-groups at higher risk.

Results

This RRA approach aligns with well-established international methods and provides flexibility to accommodate a broad range of risk questions. It has been implemented to estimate the risk of various threats of concern to Canada, including mpox, avian influenza A(H5N1) and measles.

Conclusion

Given the broad range and complexity of public health hazards, RRAs provide a timely, coordinated and systematic process for characterizing and communicating the risk to inform risk mitigation and decision-making and to guide appropriate public health response.

Quantitative support for the benefits of proactive management for wildlife disease control

Finding effective pathogen mitigation strategies is one of the biggest challenges humans face today. In the context of wildlife, emerging infectious diseases have repeatedly caused widespread host morbidity and population declines of numerous taxa. In areas yet unaffected by a pathogen, a proactive management approach has the potential to minimize or prevent host mortality. However, typically critical information on disease dynamics in a novel host system is lacking, empirical evidence on efficacy of management interventions is limited, and there is a lack of validated predictive models. As such, quantitative support for identifying effective management interventions is largely absent, and the opportunity for proactive management is often missed. We considered the potential invasion of the chytrid fungus, Batrachochytrium salamandrivorans (Bsal), whose expected emergence in North America poses a severe threat to hundreds of salamander species in this global salamander biodiversity hotspot. We developed and parameterized a dynamic multistate occupancy model to forecast host and pathogen occurrence, following expected emergence of the pathogen, and evaluated the response of salamander populations to different management scenarios. Our model forecasted that taking no action is expected to be catastrophic to salamander populations. Proactive action was predicted to maximize host occupancy outcomes relative to wait-and-see reactive management, thus providing quantitative support for proactive management opportunities. The eradication of Bsal was unlikely under all the evaluated management options. Contrary to our expectations, even early pathogen detection had little effect on Bsal or host occupancy outcomes. Our results provide quantitative support that proactive management is the optimal strategy for promoting persistence of disease-threatened salamander populations. Our approach fills a critical gap by defining a framework for evaluating management options prior to pathogen invasion and can thus serve as a template for addressing novel disease threats that jeopardize wildlife and human health.

Weight of evidence evaluation of the metabolism disrupting effects of triphenyl phosphate using an expert knowledge elicitation approach

Identification of Endocrine-Disrupting Chemicals (EDCs) in a regulatory context requires a high level of evidence. However, lines of evidence (e.g. human, in vivo, in vitro or in silico) are heterogeneous and incomplete for quantifying evidence of the adverse effects and mechanisms involved. To date, for the regulatory appraisal of metabolism-disrupting chemicals (MDCs), no harmonised guidance to assess the weight of evidence has been developed at the EU or international level. To explore how to develop this, we applied a formal Expert Knowledge Elicitation (EKE) approach within the European GOLIATH project. EKE captures expert judgment in a quantitative manner and provides an estimate of uncertainty of the final opinion. As a proof of principle, we selected one suspected MDC -triphenyl phosphate (TPP) - based on its related adverse endpoints (obesity/adipogenicity) relevant to metabolic disruption and a putative Molecular Initiating Event (MIE): activation of peroxisome proliferator activated receptor gamma (PPARγ). We conducted a systematic literature review and assessed the quality of the lines of evidence with two independent groups of experts within GOLIATH, with the objective of categorising the metabolic disruption properties of TPP, by applying an EKE approach. Having followed the entire process separately, both groups arrived at the same conclusion, designating TPP as a "suspected MDC" with an overall quantitative agreement exceeding 85%, indicating robust reproducibility. The EKE method provides to be an important way to bring together scientists with diverse expertise and is recommended for future work in this area.

Assessing the risk of takeover catastrophe from large language models

This article presents a risk analysis of large language models (LLMs), a type of "generative" artificial intelligence (AI) system that produces text, commonly in response to textual inputs from human users. The article is specifically focused on the risk of LLMs causing an extreme catastrophe in which they do something akin to taking over the world and killing everyone. The possibility of LLM takeover catastrophe has been a major point of public discussion since the recent release of remarkably capable LLMs such as ChatGPT and GPT-4. This arguably marks the first time when actual AI systems (and not hypothetical future systems) have sparked concern about takeover catastrophe. The article's analysis compares (A) characteristics of AI systems that may be needed for takeover, as identified in prior theoretical literature on AI takeover risk, with (B) characteristics observed in current LLMs. This comparison reveals that the capabilities of current LLMs appear to fall well short of what may be needed for takeover catastrophe. Future LLMs may be similarly incapable due to fundamental limitations of deep learning algorithms. However, divided expert opinion on deep learning and surprise capabilities found in current LLMs suggests some risk of takeover catastrophe from future LLMs. LLM governance should monitor for changes in takeover characteristics and be prepared to proceed more aggressively if warning signs emerge. Unless and until such signs emerge, more aggressive governance measures may be unwarranted.

Adopting electric school buses in the United States: Health and climate benefits

Electric school buses have been proposed as an alternative to reduce the health and climate impacts of the current U.S. school bus fleet, of which a substantial share are highly polluting old diesel vehicles. However, the climate and health benefits of electric school buses are not well known. As they are substantially more costly than diesel buses, assessing their benefits is needed to inform policy decisions. We assess the health benefits of electric school buses in the United States from reduced adult mortality and childhood asthma onset risks due to exposure to ambient fine particulate matter (PM2.5). We also evaluate climate benefits from reduced greenhouse-gas emissions. We find that replacing the average diesel bus in the U.S. fleet in 2017 with an electric bus yields $84,200 in total benefits. Climate benefits amount to $40,400/bus, whereas health benefits amount to $43,800/bus due to 4.42*10-3 fewer PM2.5-attributable deaths ($40,000 of total) and 7.42*10-3 fewer PM2.5-attributable new childhood asthma cases ($3,700 of total). However, health benefits of electric buses vary substantially by driving location and model year (MY) of the diesel buses they replace. Replacing old, MY 2005 diesel buses in large cities yields $207,200/bus in health benefits and is likely cost-beneficial, although other policies that accelerate fleet turnover in these areas deserve consideration. Electric school buses driven in rural areas achieve small health benefits from reduced exposure to ambient PM2.5. Further research assessing benefits of reduced exposure to in-cabin air pollution among children riding buses would be valuable to inform policy decisions.

Overcoming challenges in the economic evaluation of interventions to optimise antibiotic use

Bacteria are becoming increasingly resistant to antibiotics, reducing our ability to treat infections and threatening to undermine modern health care. Optimising antibiotic use is a key element in tackling the problem. Traditional economic evaluation methods do not capture many of the benefits from improved antibiotic use and the potential impact on resistance. Not capturing these benefits is a major obstacle to optimising antibiotic use, as it fails to incentivise the development and use of interventions to optimise the use of antibiotics and preserve their effectiveness (stewardship interventions). Estimates of the benefits of improving antibiotic use involve considerable uncertainty as they depend on the evolution of resistance and associated health outcomes and costs. Here we discuss how economic evaluation methods might be adapted, in the face of such uncertainties. We propose a threshold-based approach that estimates the minimum resistance-related costs that would need to be averted by an intervention to make it cost-effective. If it is probable that without the intervention costs will exceed the threshold then the intervention should be deemed cost-effective.

Modeling direct air carbon capture and storage in a 1.5 °C climate future using historical analogs

Limiting the rise in global temperature to 1.5 °C will rely, in part, on technologies to remove CO2 from the atmosphere. However, many carbon dioxide removal (CDR) technologies are in the early stages of development, and there is limited data to inform predictions of their future adoption. Here, we present an approach to model adoption of early-stage technologies such as CDR and apply it to direct air carbon capture and storage (DACCS). Our approach combines empirical data on historical technology analogs and early adoption indicators to model a range of feasible growth pathways. We use these pathways as inputs to an integrated assessment model (the Global Change Analysis Model, GCAM) and evaluate their effects under an emissions policy to limit end-of-century temperature change to 1.5 °C. Adoption varies widely across analogs, which share different strategic similarities with DACCS. If DACCS growth mirrors high-growth analogs (e.g., solar photovoltaics), it can reach up to 4.9 GtCO2 removal by midcentury, compared to as low as 0.2 GtCO2 for low-growth analogs (e.g., natural gas pipelines). For these slower growing analogs, unabated fossil fuel generation in 2050 is reduced by 44% compared to high-growth analogs, with implications for energy investments and stranded assets. Residual emissions at the end of the century are also substantially lower (by up to 43% and 34% in transportation and industry) under lower DACCS scenarios. The large variation in growth rates observed for different analogs can also point to policy takeaways for enabling DACCS.

Using the R package popharvest to assess the sustainability of offtake in birds

The R package popharvest was designed to help assess the sustainability of offtake in birds when only limited demographic information is available. In this article, we describe some basics of harvest theory and then discuss several considerations when using the different approaches in popharvest to assess whether observed harvests are unsustainable. Throughout, we emphasize the importance of distinguishing between the scientific and policy aspects of managing offtake. The principal product of popharvest is a sustainable harvest index (SHI), which can indicate whether the harvest is unsustainable but not the converse. SHI is estimated based on a simple, scalar model of logistic population growth, whose parameters may be estimated using limited knowledge of demography. Uncertainty in demography leads to a distribution of SHI values and it is the purview of the decision-maker to determine what amounts to an acceptable risk when failing to reject the null hypothesis of sustainability. The attitude toward risk, in turn, will likely depend on the decision-maker's objective(s) in managing offtake. The management objective as specified in popharvest is a social construct, informed by biology, but ultimately it is an expression of social values that usually vary among stakeholders. We therefore suggest that any standardization of criteria for management objectives in popharvest will necessarily be subjective and, thus, hard to defend in diverse decision-making situations. Because of its ease of use, diverse functionalities, and a minimal requirement of demographic information, we expect the use of popharvest to become widespread. Nonetheless, we suggest that while popharvest provides a useful platform for rapid assessments of sustainability, it cannot substitute for sufficient expertise and experience in harvest theory and management.

Developing expert scientific consensus on the environmental and societal effects of marine artificial structures prior to decommissioning

Thousands of artificial ('human-made') structures are present in the marine environment, many at or approaching end-of-life and requiring urgent decisions regarding their decommissioning. No consensus has been reached on which decommissioning option(s) result in optimal environmental and societal outcomes, in part, owing to a paucity of evidence from real-world decommissioning case studies. To address this significant challenge, we asked a worldwide panel of scientists to provide their expert opinion. They were asked to identify and characterise the ecosystem effects of artificial structures in the sea, their causes and consequences, and to identify which, if any, should be retained following decommissioning. Experts considered that most of the pressures driving ecological and societal effects from marine artificial structures (MAS) were of medium severity, occur frequently, and are dependent on spatial scale with local-scale effects of greater magnitude than regional effects. The duration of many effects following decommissioning were considered to be relatively short, in the order of days. Overall, environmental effects of structures were considered marginally undesirable, while societal effects marginally desirable. Experts therefore indicated that any decision to leave MAS in place at end-of-life to be more beneficial to society than the natural environment. However, some individual environmental effects were considered desirable and worthy of retention, especially in certain geographic locations, where structures can support improved trophic linkages, increases in tourism, habitat provision, and population size, and provide stability in population dynamics. The expert analysis consensus that the effects of MAS are both negative and positive for the environment and society, gives no strong support for policy change whether removal or retention is favoured until further empirical evidence is available to justify change to the status quo. The combination of desirable and undesirable effects associated with MAS present a significant challenge for policy- and decision-makers in their justification to implement decommissioning options. Decisions may need to be decided on a case-by-case basis accounting for the trade-off in costs and benefits at a local level.

Expert elicitation of the timing and uncertainty to establish a geologic sequestration well for CO2 in the United States

Many studies anticipate that carbon capture and sequestration (CCS) will be essential to decarbonizing the U.S. economy. However, prior work has not estimated the time required to develop, approve, and implement a geologic sequestration site in the United States. We generate such an estimate by identifying six clearance points that must be passed before a sequestration site can become operational. For each clearance point (CP), we elicit expert judgments of the time required in the form of probability distributions and then use stochastic simulation to combine and sum the results. We find that, on average, there is a 90% chance that the time required lies between 5.5 and 9.6 y, with an upper bound of 12 y. Even using the most optimistic expert judgements, the lower bound on time is 2.7 y, and the upper bound is 8.3 y. Using the most pessimistic judgements, the lower bound is 3.5 y and the upper bound is 19.2 y. These estimates suggest that strategies must be found to safely accelerate the process. We conclude the paper by discussing seven potential strategies.

Vaccine Nationalism Counterintuitively Erodes Public Trust in Leaders

Global access to resources like vaccines is key for containing the spread of infectious diseases. However, wealthy countries often pursue nationalistic policies, stockpiling doses rather than redistributing them globally. One possible motivation behind vaccine nationalism is a belief among policymakers that citizens will mistrust leaders who prioritize global needs over domestic protection. In seven experiments (total N = 4,215 adults), we demonstrate that such concerns are misplaced: Nationally representative samples across multiple countries with large vaccine surpluses (Australia, Canada, United Kingdom, and United States) trusted redistributive leaders more than nationalistic leaders-even the more nationalistic participants. This preference generalized across different diseases and manifested in both self-reported and behavioral measures of trust. Professional civil servants, however, had the opposite intuition and predicted higher trust in nationalistic leaders, and a nonexpert sample also failed to predict higher trust in redistributive leaders. We discuss how policymakers' inaccurate intuitions might originate from overestimating others' self-interest.

Perspectives on informative Bayesian methods in pediatrics

Bayesian methods have been proposed as a natural fit for pediatric extrapolation, as they allow the incorporation of relevant external data to reduce the required sample size and hence trial burden for the pediatric patient population. In this paper we will discuss our experience and perspectives with these methods in pediatric trials. We will present some of the background and thinking underlying pediatric extrapolation and discuss the use of Bayesian methods within this context. We will present two recent case examples illustrating the value of a Bayesian approach in this setting and present perspectives on some of the issues that we have encountered in these and other cases.

Microalgae-microplastics interactions at environmentally relevant concentrations: Implications toward ecology, bioeconomy, and UN SDGs

Microplastics (MPs) are one of the emerging pollutants, causing potential harm to aquatic ecosystems and serious concern in achieving UN Sustainable Development Goals (SDGs). Realizing the occurrence of varying concentrations of MPs in the environment, this investigation presents multi-dimensional insights into the ecological and bioeconomic implications at environmentally relevant concentrations. We pursued a multi-step approach to gain a comprehensive understanding on the effects of microalgae‒MPs interactions and their expansive implications toward SDGs. Baseline data generated using a model microalga, Raphidocelis subcapitata, and three MPs (polyethylene terephthalate, PET; polyvinyl chloride, PVC; and polystyrene, PS) indicated 10‒15 % reduction in microalgal growth rate relative to the control, pointing to a heightened energy demand. The biochemical impacts displayed concentration-dependent variability. Using the baseline data, we developed a linear regression model to dissect the interaction effects around the primary dimensions of Ecology and Bioeconomy. Notably, a correlation matrix for carbon allocation pinpointed PET as having a more pronounced impact compared to PVC and PS, with the model accounting for 33.72 % of the observed variance. Extending our insights from the model, we adopted an evidence-based methodology to outline the broader implications across the Ecology and Bioeconomy domains, and subsequently identified their associations with specific SDGs. Further probing into microalgae‒MPs interaction effects at environmentally relevant concentration, our model revealed that the selected MPs perturbed the ecological variables. Interestingly, when carbon allocation was assessed to study bioeconomic implications, there were contrasting effects on starch synthesis (beneficially) and lipid synthesis (detrimentally). The present combined analysis revealed that MPs, beyond their traditional association with SDG 14 (Life Below Water), directly and indirectly affect five other SDGs through their interactions with microalgae. This study thus underscores the complex and interconnected nature of MPs pollution at environmentally relevant concentrations and their impacts on ecological and bioeconomic aspects of SDGs, thereby highlighting the urgent need for additional research and effective mitigation strategies.

Confidence deficits and reducibility: Toward a coherent conceptualization of uncertainty level

Outside of the field of risk analysis, an important theoretical conversation on the slippery concept of uncertainty has unfolded over the last 40 years within the adjacent field of environmental risk. This literature has become increasingly standardized behind the tripartite distinction between uncertainty location, the nature of uncertainty, and uncertainty level, popularized by the "W&H framework." This article introduces risk theorists and practitioners to the conceptual literature on uncertainty with the goal of catalyzing further development and clarification of the uncertainty concept within the field of risk analysis. It presents two critiques of the W&H framework's dimension of uncertainty level-the dimension that attempts to define the characteristics separating greater uncertainties from lesser uncertainties. First, I argue the framework's conceptualization of uncertainty level lacks a clear and consistent epistemological position and fails to acknowledge or reconcile the tensions between Bayesian and frequentist perspectives present within the framework. This article reinterprets the dimension of uncertainty level from a Bayesian perspective, which understands uncertainty as a mental phenomenon arising from "confidence deficits" as opposed to the ill-defined notion of "knowledge deficits" present in the framework. And second, I elaborate the undertheorized concept of uncertainty "reducibility." These critiques inform a clarified conceptualization of uncertainty level that can be integrated with risk analysis concepts and usefully applied by modelers and decisionmakers engaged in model-based decision support.

Probabilistic projections of granular energy technology diffusion at subnational level

Projections of granular energy technology diffusion can support decision-making on climate mitigation policies and infrastructure investments. However, such projections often do not account for uncertainties and have low spatial resolution. S-curve models of technology diffusion are widely used to project future installations, but the results of the different models can vary significantly. We propose a method to create probabilistic projections of granular energy technology diffusion at subnational level based on historical time series data and testing how various projection models perform in terms of accuracy and uncertainty to inform the choice of models. As a case study, we investigate the growth of solar photovoltaics, heat pumps, and battery electric vehicles at municipality level throughout Switzerland in 2000-2021 (testing) and until 2050 (projections). Consistently for all S-curve models and technologies, we find that the medians of the probabilistic projections anticipate the diffusion of the technologies more accurately than the respective deterministic projections. While accuracy and probabilistic density intervals of the models vary across technologies, municipalities, and years, Bertalanffy and two versions of the generalized Richards model estimate the future diffusion with higher accuracy and sharpness than logistic, Gompertz, and Bass models. The results also highlight that all models come with trade-offs and eventually a combination of models with weights is needed. Based on these weighted probabilistic projections, we show that, given the current dynamics of diffusion in solar photovoltaics, heat pumps, and battery electric vehicles in Switzerland, the net-zero emissions target would be missed by 2050 with high certainty.

Vulnerability to climate change of United States marine mammal stocks in the western North Atlantic, Gulf of Mexico, and Caribbean

Climate change and climate variability are affecting marine mammal species and these impacts are projected to continue in the coming decades. Vulnerability assessments provide a framework for evaluating climate impacts over a broad range of species using currently available information. We conducted a trait-based climate vulnerability assessment using expert elicitation for 108 marine mammal stocks and stock groups in the western North Atlantic, Gulf of Mexico, and Caribbean Sea. Our approach combined the exposure (projected change in environmental conditions) and sensitivity (ability to tolerate and adapt to changing conditions) of marine mammal stocks to estimate vulnerability to climate change, and categorize stocks with a vulnerability index. The climate vulnerability score was very high for 44% (n = 47) of these stocks, high for 29% (n = 31), moderate for 20% (n = 22), and low for 7% (n = 8). The majority of stocks (n = 78; 72%) scored very high exposure, whereas 24% (n = 26) scored high, and 4% (n = 4) scored moderate. The sensitivity score was very high for 33% (n = 36) of these stocks, high for 18% (n = 19), moderate for 34% (n = 37), and low for 15% (n = 16). Vulnerability results were summarized for stocks in five taxonomic groups: pinnipeds (n = 4; 25% high, 75% moderate), mysticetes (n = 7; 29% very high, 57% high, 14% moderate), ziphiids (n = 8; 13% very high, 50% high, 38% moderate), delphinids (n = 84; 52% very high, 23% high, 15% moderate, 10% low), and other odontocetes (n = 5; 60% high, 40% moderate). Factors including temperature, ocean pH, and dissolved oxygen were the primary drivers of high climate exposure, with effects mediated through prey and habitat parameters. We quantified sources of uncertainty by bootstrapping vulnerability scores, conducting leave-one-out analyses of individual attributes and individual scorers, and through scoring data quality for each attribute. These results provide information for researchers, managers, and the public on marine mammal responses to climate change to enhance the development of more effective marine mammal management, restoration, and conservation activities that address current and future environmental variation and biological responses due to climate change.

Underestimated economic and social burdens of non-Typhoidal Salmonella infections: The One Health perspective from Nigeria

Background

The non-typhoidal salmonellosis (NTS) is a pathogenic bacterial zoonosis with substantial but often under-appreciated public health impacts. The NTS is prevalent in poultry and humans in Nigeria, yet its economic and social burden have not been determined through any empirical study. To bridge the gap, we evaluated the impact of NTS in social and economic terms.

Methods

Relevant population, economic and epidemiological data were retrieved from peer-reviewed publications, open sources and relevant authorities. Additional data were obtained through experts' opinions and field surveys. Using a customized and validated Microsoft Excel® tool, economic analysis was conducted.

Results

Using the year 2020 reference point, the burden of NTS was 325,731 cases and a total of 1043 human deaths, at a disability-adjusted life year (DALYs) of 37,321. The cost associated with infection in humans was US$ 473,982,068. A total loss of US$ 456,905,311 was estimated in poultry including the direct value of animal loss, US$ 224,236,769, loss from salvage slaughter and culling, US$ 220,386,556, and value of foregone production, US$ 12,281,987.

Interpretation

The outcomes of this important work provide empirical evidence to support informed decisions and investments in the control and eradication of human and poultry salmonellosis (NTS) in Nigeria.

Public health impact of a US ban on menthol in cigarettes and cigars: a simulation study

INTRODUCTION

The US Food and Drug Administration most recently announced its intention to ban menthol cigarettes and cigars nationwide in April 2021. Implementation of the ban will require evidence that it would improve public health. This paper simulates the potential public health impact of a ban on menthol in cigarettes and cigars through its impacts on smoking initiation, smoking cessation and switching to nicotine vaping products (NVPs).

METHODS

After calibrating an established US simulation model to reflect recent use trends in cigarette and NVP use, we extended the model to incorporate menthol and non-menthol cigarette use under a status quo scenario. Applying estimates from a recent expert elicitation on the behavioural impacts of a menthol ban, we developed a menthol ban scenario with the ban starting in 2021. We estimated the public health impact as the difference between smoking and vaping-attributable deaths and life-years lost in the status quo scenario and the menthol ban scenario from 2021 to 2060.

RESULTS

As a result of the ban, overall smoking was estimated to decline by 15% as early as 2026 due to menthol smokers quitting both NVP and combustible use or switching to NVPs. These transitions are projected to reduce cumulative smoking and vaping-attributable deaths from 2021 to 2060 by 5% (650 000 in total) and reduce life-years lost by 8.8% (11.3 million). Sensitivity analyses showed appreciable public health benefits across different parameter specifications.

CONCLUSIONS AND RELEVANCE

Our findings strongly support the implementation of a ban on menthol in cigarettes and cigars.

First Expert Elicitation of Knowledge on Possible Drivers of Observed Increasing Human Cases of Tick-Borne Encephalitis in Europe

Tick-borne encephalitis (TBE) is a viral disease endemic in Eurasia. The virus is mainly transmitted to humans via ticks and occasionally via the consumption of unpasteurized milk products. The European Centre for Disease Prevention and Control reported an increase in TBE incidence over the past years in Europe as well as the emergence of the disease in new areas. To better understand this phenomenon, we investigated the drivers of TBE emergence and increase in incidence in humans through an expert knowledge elicitation. We listed 59 possible drivers grouped in eight domains and elicited forty European experts to: (i) allocate a score per driver, (ii) weight this score within each domain, and (iii) weight the different domains and attribute an uncertainty level per domain. An overall weighted score per driver was calculated, and drivers with comparable scores were grouped into three terminal nodes using a regression tree analysis. The drivers with the highest scores were: (i) changes in human behavior/activities; (ii) changes in eating habits or consumer demand; (iii) changes in the landscape; (iv) influence of humidity on the survival and transmission of the pathogen; (v) difficulty to control reservoir(s) and/or vector(s); (vi) influence of temperature on virus survival and transmission; (vii) number of wildlife compartments/groups acting as reservoirs or amplifying hosts; (viii) increase of autochthonous wild mammals; and (ix) number of tick species vectors and their distribution. Our results support researchers in prioritizing studies targeting the most relevant drivers of emergence and increasing TBE incidence.

Exploring fishing threat at fleet segment and subregional scale: Least expert knowledge and a resilience versus disturbance-based approach as conservation's tools for cartilaginous fish

Based on an explorative but rigorous elicitation framework, we obtained the bycatch fishing probability at the fishing fleet segment level using expert estimates. Based on the knowledge of three scientific experts, we developed a new and creative structured method for smart and fast fishery-related risk assessments for species of high conservation concern. In order to test the method here propose, we applied it to 76 cartilaginous fish species (included in the IUCN Red Lists) and on five different fishing segments at both Italian and Mediterranean scale. The method produced qualitative results specific to the threat posed by fishing for each species and each segment with information between and within the segments. Based on the interpretation of resilience-disturbance interactions developed for ecological systems, the quantitative results provided reliable cumulative metrics, measuring the extinction risk due to fishing and the response to overfishing for the species considered. Additionally, the results highlight that the method perform best on a small geographic scale. Therefore, the application of this new method on other subregional or local scales where very few data are available (e.g., fishing effort) could be a valuable tool for the preliminary assessment for species of conservation concern. In fact, despite the absence of detailed catch data at local geographic scales, the flexibility of this method could help to highlight potential fishery-related conservation problems and thus redirect conservation strategies for threatened marine species such as many sharks and rays species.

A novel expert-driven methodology to develop thermal response curves and project habitat thermal suitability for cetaceans under a changing climate

Over the last decades, global warming has contributed to changes in marine species composition, abundance and distribution, in response to changes in oceanographic conditions such as temperature, acidification, and deoxygenation. Experimentally derived thermal limits, which are known to be related to observed latitudinal ranges, have been used to assess variations in species distribution patterns. However, such experiments cannot be undertaken on free-swimming large marine predators with wide-range distribution, like cetaceans. An alternative approach is to elicit expert's knowledge to derive species' thermal suitability and assess their thermal responses, something that has never been tested in these taxa. We developed and applied a methodology based on expert-derived thermal suitability curves and projected future responses for several species under different climate scenarios. We tested this approach with ten cetacean species currently present in the biogeographic area of Macaronesia (North Atlantic) under Representative Concentration Pathways 2.6, 4.5 and 8.5, until 2050. Overall, increases in annual thermal suitability were found for Balaenoptera edeni, Globicephala macrorhynchus, Mesoplodon densirostris, Physeter macrocephalus, Stenella frontalis, Tursiops truncatus and Ziphius cavirostris. Conversely, our results indicated a decline in thermal suitability for B. physalus, Delphinus delphis, and Grampus griseus. Our study reveals potential responses in cetaceans' thermal suitability, and potentially in other highly mobile and large predators, and it tests this method's applicability, which is a novel application for this purpose and group of species. It aims to be a cost-efficient tool to support conservation managers and practitioners.

Adaptive Management of Flows in a Regulated River: Flow-ecology Relationships Revealed by a 26-year, Five-treatment Flow Experiment

Adaptive management (AM) is often proposed as a means to resolve uncertainty in the management of socio-ecological systems but successful implementation of AM is rare. We report results from a 26 year, five-treatment, AM experiment designed to inform decision makers about the response of juvenile salmonids (Oncorhynchus spp.) to flow releases from a dam on the regulated Bridge River, British Columbia, Canada. Treatments consisted of a baseline (no dam release) and four different dam release regimes that followed a semi-natural hydrograph but varied in the magnitude of spring-summer freshet flows. We found total salmonid biomass was highest at the lowest flow release, and decreased with increasing flow, consistent with a priori predictions made by an expert solicitation process. Species-specific responses were observed that in some cases could be attributed to interactions between the flow regime and life history. The relationship between juvenile biomass and flow resulting from the experiment can inform decisions on water management for this river. The documentation of successful AM experiments is sorely needed to allow for reflection on the circumstances when AM is likely to deliver desirable outcomes, and to improve other decision processes that require fewer resources and less time to implement.

Green building standards and the United Nations' Sustainable Development Goals

Sustainable practices in the building industry are strongly influenced by published green and sustainable building and real-estate standards (GSBRES). Therefore, it is crucial to assess how these standards contribute to achieving the United Nations' Sustainable Development Goals (SDGs). This paper evaluates the extent to which GSBRESs align with the 2030 Agenda for Sustainable Development, lending a particular focus to the call for transformative change implicit in the SDGs. To this end, we develop a methodology that combines qualitative and quantitative analysis to assess the overlap between the content of three GSBRESs (LEED for design, BOMA BEST for operation, and GRESB for investment) and the SDGs. Despite the overlaps between the attributes of the GSBRESs and the general topics of the SDGs, we find that less than 20% of GSBRES attributes address the specific targets of the 2030 Agenda. Most importantly, the qualitative analysis shows that less than 10% of the standards' scores is attributed to transformative change. We conclude that claims that the GSBRESs are effective in advancing the SDGs are overstated and, without further empirical evidence, caution that they increase the risk of sustainable development greenwashing. We recommend that the standards be repositioned to adopt transformation-focused indicators related to a project's long-term impacts.

Data Hunches: Incorporating Personal Knowledge into Visualizations

The trouble with data is that it frequently provides only an imperfect representation of a phenomenon of interest. Experts who are familiar with their datasets will often make implicit, mental corrections when analyzing a dataset, or will be cautious not to be overly confident about their findings if caveats are present. However, personal knowledge about the caveats of a dataset is typically not incorporated in a structured way, which is problematic if others who lack that knowledge interpret the data. In this work, we define such analysts' knowledge about datasets as data hunches. We differentiate data hunches from uncertainty and discuss types of hunches. We then explore ways of recording data hunches, and, based on a prototypical design, develop recommendations for designing visualizations that support data hunches. We conclude by discussing various challenges associated with data hunches, including the potential for harm and challenges for trust and privacy. We envision that data hunches will empower analysts to externalize their knowledge, facilitate collaboration and communication, and support the ability to learn from others' data hunches.

A framework for supporting health capability-based planning: Identifying and structuring health capabilities

The COVID-19 pandemic has highlighted that health security systems must be redesigned, in a way that they are better prepared and ready to cope with multiple and diverse health threats, from predictable and well-known epidemics to unexpected and challenging pandemics. A powerful way of accomplishing this goal is to focus the planning on health capabilities. This focus may enhance the ability to respond to and recover from health threats and emergencies, while helping to identify the level of resources required to maintain and build up those capabilities that are critical in ensuring the preparedness of health security systems. However, current attempts for defining and organizing health capabilities have some important limitations. First, such attempts were not designed to consider diverse scenarios and multiple health threats. Second, they provide a limited representation of capabilities and lack a systemic perspective. Third, they struggle to identify capability and resource gaps. In this article, we thus propose a new framework for identifying and structuring health capabilities and support health capability planning. The suggested framework has three main potential benefits. First, the framework may help policymakers in planning under high levels of uncertainty, by considering multiple realistic and stressful scenarios. Second, it can provide risk analysts with a more comprehensive representation of health capabilities and their complex relationships. Third, it can support planners in identifying resource and capability gaps. We illustrate the use of the framework in practice considering an outbreak scenario caused by three different health threats (COVID-19, Ebola, and Influenza viruses).

Residual Disease After Primary Surgical Treatment for Advanced Epithelial Ovarian Cancer, Part 2: Network Meta-analysis Incorporating Expert Elicitation to Adjust for Publication Bias

BACKGROUND

Previous work has identified a strong association between the achievements of macroscopic cytoreduction and improved overall survival (OS) after primary surgical treatment of advanced epithelial ovarian cancer. Despite the use of contemporary methodology, resulting in the most comprehensive currently available evidence to date in this area, opponents remain skeptical.

AREAS OF UNCERTAINTY

We aimed to conduct sensitivity analyses to adjust for potential publication bias, to confirm or refute existing conclusions and recommendations, leveraging elicitation to incorporate expert opinion. We recommend our approach as an exemplar that should be adopted in other areas of research.

DATA SOURCES

We conducted random-effects network meta-analyses in frequentist and Bayesian (using Markov Chain Montel Carlo simulation) frameworks comparing OS across residual disease thresholds in women with advanced epithelial ovarian cancer after primary cytoreductive surgery. Elicitation methods among experts in gynecology were used to derive priors for an extension to a previously reported Copas selection model and a novel approach using effect estimates calculated from the elicitation exercise, to attempt to adjust for publication bias and increase confidence in the certainty of the evidence.

THERAPEUTIC ADVANCES

Analyses using data from 25 studies (n = 20,927 women) all showed the prognostic importance of complete cytoreduction (0 cm) in both frameworks. Experts accepted publication bias was likely, but after adjustment for their opinions, published results overpowered the informative priors incorporated into the Bayesian sensitivity analyses. Effect estimates were attenuated but conclusions were robust in all analyses.

CONCLUSIONS

There remains a strong association between the achievement of complete cytoreduction and improved OS even after adjustment for publication bias using strong informative priors formed from an expert elicitation exercise. The concepts of the elicitation survey should be strongly considered for utilization in other meta-analyses.

Assessing natural global catastrophic risks

The risk of global catastrophe from natural sources may be significantly larger than previous analyses have found. In the study of global catastrophic risk (GCR), one line of thinking posits that deep human history renders natural GCRs insignificant. Essentially, the fact that natural hazards did not cause human extinction at any previous time makes it unlikely that they would do so now. This paper finds flaws in this argument, refines the theory of natural GCR, analyzes the space of natural GCRs, and presents implications for decision-making and research. The paper analyzes natural climate change, natural pandemics, near-Earth objects (asteroids, comets, and meteors), space weather (coronal mass ejections, solar flares, and solar particle events), stellar explosions (gamma-ray bursts and supernovae), and volcanic eruptions. Almost all natural GCR scenarios involve important interactions between the natural hazard and human civilization. Several natural GCR scenarios may have high ongoing probability. Deep human history provides little information about the resilience of modern global civilization to natural global catastrophes. The natural GCRs should not be dismissed on grounds of deep human history. Work on natural GCRs should account for their important human dimensions. A case can even be made for abandoning the distinction between natural and anthropogenic GCR.

Predictive validity in drug discovery: what it is, why it matters and how to improve it

Successful drug discovery is like finding oases of safety and efficacy in chemical and biological deserts. Screens in disease models, and other decision tools used in drug research and development (R&D), point towards oases when they score therapeutic candidates in a way that correlates with clinical utility in humans. Otherwise, they probably lead in the wrong direction. This line of thought can be quantified by using decision theory, in which 'predictive validity' is the correlation coefficient between the output of a decision tool and clinical utility across therapeutic candidates. Analyses based on this approach reveal that the detectability of good candidates is extremely sensitive to predictive validity, because the deserts are big and oases small. Both history and decision theory suggest that predictive validity is under-managed in drug R&D, not least because it is so hard to measure before projects succeed or fail later in the process. This article explains the influence of predictive validity on R&D productivity and discusses methods to evaluate and improve it, with the aim of supporting the application of more effective decision tools and catalysing investment in their creation.

Determining our climate policy future: expert opinions about negative emissions and solar radiation management pathways

Negative emissions technologies and solar radiation management techniques could contribute towards climate stability, either by removing carbon dioxide from the atmosphere and storing it permanently or reflecting sunlight away from the atmosphere. Despite concerns about them, such options are increasingly being discussed as crucial complements to traditional climate change mitigation and adaptation. Expectations around negative emissions and solar radiation management and their associated risks and costs shape public and private discussions of how society deals with the climate crisis. In this study, we rely on a large expert survey (N = 74) to critically examine the future potential of both negative emission options (e.g., carbon dioxide removal) and solar radiation management techniques. We designed a survey process that asked a pool of prominent experts questions about (i) the necessity of adopting negative emissions or solar radiation management options, (ii) the desirability of such options when ranked against each other, (iii) estimations of future efficacy in terms of temperature reductions achieved or gigatons of carbon removed, (iv) expectations about future scaling, commercialization, and deployment targets, and (v) potential risks and barriers. Unlike other elicitation processes where experts are more positive or have high expectations about novel options, our results are more critical and cautionary. We find that some options (notably afforestation and reforestation, ecosystem restoration, and soil carbon sequestration) are envisioned frequently as necessary, desirable, feasible, and affordable, with minimal risks and barriers (compared to other options). This contrasts with other options envisaged as unnecessary risky or costly, notably ocean alkalization or fertilization, space-based reflectors, high-altitude sunshades, and albedo management via clouds. Moreover, only the options of afforestation and reforestation and soil carbon sequestration are expected to be widely deployed before 2035, which raise very real concerns about climate and energy policy in the near- to mid-term.

The distribution of initial estimates moderates the effect of social influence on the wisdom of the crowd

Whether, and under what conditions, groups exhibit “crowd wisdom” has been a major focus of research across the social and computational sciences. Much of this work has focused on the role of social influence in promoting the wisdom of the crowd versus leading the crowd astray and has resulted in conflicting conclusions about how social network structure determines the impact of social influence. Here, we demonstrate that it is not enough to consider the network structure in isolation. Using theoretical analysis, numerical simulation, and reanalysis of four experimental datasets (totaling 2885 human subjects), we find that the wisdom of crowds critically depends on the interaction between (i) the centralization of the social influence network and (ii) the distribution of the initial individual estimates. By adopting a framework that integrates both the structure of the social influence and the distribution of the initial estimates, we bring previously conflicting results under one theoretical framework and clarify the effects of social influence on the wisdom of crowds.

A retrospective assessment of COVID-19 model performance in the USA

Coronavirus disease 2019 (COVID-19) forecasts from over 100 models are readily available. However, little published information exists regarding the performance of their uncertainty estimates (i.e. probabilistic performance). To evaluate their probabilistic performance, we employ the classical model (CM), an established method typically used to validate expert opinion. In this analysis, we assess both the predictive and probabilistic performance of COVID-19 forecasting models during 2021. We also compare the performance of aggregated forecasts (i.e. ensembles) based on equal and CM performance-based weights to an established ensemble from the Centers for Disease Control and Prevention (CDC). Our analysis of forecasts of COVID-19 mortality from 22 individual models and three ensembles across 49 states indicates that-(i) good predictive performance does not imply good probabilistic performance, and vice versa; (ii) models often provide tight but inaccurate uncertainty estimates; (iii) most models perform worse than a naive baseline model; (iv) both the CDC and CM performance-weighted ensembles perform well; but (v) while the CDC ensemble was more informative, the CM ensemble was more statistically accurate across states. This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modelling.

A statistical modelling approach for source attribution meta-analysis of sporadic infection with foodborne pathogens

Numerous source attribution studies for foodborne pathogens based on epidemiological and microbiological methods are available. These studies provide empirical data for modelling frameworks that synthetize the quantitative evidence at our disposal and reduce reliance on expert elicitations. Here, we develop a statistical model within a Bayesian estimation framework to integrate attribution estimates from expert elicitations with estimates from microbial subtyping and case-control studies for sporadic infections with four major bacterial zoonotic pathogens in the Netherlands (Campylobacter, Salmonella, Shiga toxin-producing E. coli [STEC] O157 and Listeria). For each pathogen, we pooled the published fractions of human cases attributable to each animal reservoir from the microbial subtyping studies, accounting for the uncertainty arising from the different typing methods, attribution models, and year(s) of data collection. We then combined the population attributable fractions (PAFs) from the case-control studies according to five transmission pathways (domestic food, environment, direct animal contact, human-human transmission and travel) and 11 groups within the foodborne pathway (beef/lamb, pork, poultry meat, eggs, dairy, fish/shellfish, fruit/vegetables, beverages, grains, composite foods and food handlers/vermin). The attribution estimates were biologically plausible, allowing the human cases to be attributed in several ways according to reservoirs, transmission pathways and food groups. All pathogens were predominantly foodborne, with Campylobacter being mostly attributable to the chicken reservoir, Salmonella to pigs (albeit closely followed by layers), and Listeria and STEC O157 to cattle. Food-wise, the attributions reflected those at the reservoir level in terms of ranking. We provided a modelling solution to reach consensus attribution estimates reflecting the empirical evidence in the literature that is particularly useful for policy-making and is extensible to other pathogens and domains.

Research priorities for global food security under extreme events

Extreme events, such as those caused by climate change, economic or geopolitical shocks, and pest or disease epidemics, threaten global food security. The complexity of causation, as well as the myriad ways that an event, or a sequence of events, creates cascading and systemic impacts, poses significant challenges to food systems research and policy alike. To identify priority food security risks and research opportunities, we asked experts from a range of fields and geographies to describe key threats to global food security over the next two decades and to suggest key research questions and gaps on this topic. Here, we present a prioritization of threats to global food security from extreme events, as well as emerging research questions that highlight the conceptual and practical challenges that exist in designing, adopting, and governing resilient food systems. We hope that these findings help in directing research funding and resources toward food system transformations needed to help society tackle major food system risks and food insecurity under extreme events.

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review

BACKGROUND

Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual.

OBJECTIVES

We conducted a systematic review to compare EE methods used to generate model input parameters in health research.

DATA SOURCES

PubMed and Web of Science.

STUDY ELIGIBILITY

Modeling studies that reported the use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes.

DATA ABSTRACTION AND SYNTHESIS

Studies were classified as "formal" EE methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion but provided limited information were classified as "indeterminate" methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods.

STUDY APPRAISAL

Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods.

RESULTS

The search identified 1520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16.

LIMITATIONS

Elicitations on nonhealth topics and those reported in the gray literature were not included.

CONCLUSIONS

We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models.

HIGHLIGHTS

We find extensive use of expert elicitation for the development of model input parameters, but most studies do not provide adequate details of their elicitation methods.Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.There is a need to establish expert elicitation best practices and reporting guidelines.

Mortality Attributable to Long-Term Exposure to Ambient Fine Particulate Matter: Insights from the Epidemiologic Evidence for Understudied Locations

Epidemiologic cohort studies have consistently demonstrated that long-term exposure to ambient fine particles (PM_2.5) is associated with mortality. Nevertheless, extrapolating results to understudied locations may involve considerable uncertainty. To explore this issue, this review discusses the evidence for (i) the associated risk of mortality, (ii) the shape of the concentration-response function, (iii) a causal interpretation, and (iv) how the source mix/composition of PM_2.5 and population characteristics may alter the effect. The accumulated evidence suggests the following: (i) In the United States, the change in all-cause mortality risk per μg/m³ is about 0.8%. (ii) The concentration-response function appears nonlinear. (iii) Causation is overwhelmingly supported. (iv) Fossil fuel combustion-related sources are likely more toxic than others, and age, race, and income may modify the effect. To illustrate the use of our findings in support of a risk assessment in an understudied setting, we consider Kuwait. However, given the complexity of this relationship and the heterogeneity in reported effects, it is unreasonable to think that, in such circumstances, point estimates can be meaningful. Consequently, quantitative probabilistic estimates, which cannot be derived objectively, become essential. Formally elicited expert judgment can provide such estimates, and this review provides the evidence to support an elicitation.

Guidelines, guidelines and more guidelines for haemorrhoid treatment: A review to sort the wheat from the chaff

AIM

Guidelines benefit patients and clinicians by distilling evidence into easy-to-read recommendations. The literature around the management of haemorrhoids is immense and guidelines are invaluable to improve treatment integrity and patient outcomes. We identified current haemorrhoid guidelines and assessed them for quality and consistency.

METHODS

A systematic search of the literature from January 2011 to October 2021 was carried out. Guidelines identified were assessed for quality using the AGREE II instrument and for consistency in terms of tabulated treatment recommendations.

RESULTS

During this period nine guidelines were identified worldwide. The general quality was poor with only one guideline considered of high enough quality for use. In general, expert selection criteria for guideline development groups were vaguely defined. There were inconsistencies in the interpretation of the published evidence leading to variation in treatment recommendations.

DISCUSSION

Fewer, higher quality guidelines, with more consistent results, are needed. Particular attention should be given to defining the selection of experts involved.

An Innovative Approach for Subnational Climate Adaptation of Biodiversity and Ecosystems: The Case Study of a Regional Strategy in Italy

Since climate change impacts are already occurring, urgent adaptive actions are necessary to avoid the worst damages. Regional authorities play an important role in adaptation, but they have few binding guidelines to carry out strategies and plans. Sectoral impacts and adaptive measures strongly differ between regions; therefore, specific results for each territory are needed. Impacts are often not exhaustively reported by literature, dataset and models, thus making it impossible to objectively identify specific adaptive measures. Usual expert elicitation helps to fill this gap but shows some issues. For the Piedmont Strategy, an innovative approach has been proposed, involving experts of private and public bodies (regional authorities, academia, research institutes, parks, associations, NGOs, etc.). They collaborated in two work group, first to identify current and future impacts on biodiversity and ecosystems, and secondly to elaborate and prioritize measures. Involving 143 experts of 46 affiliations, it was possible to quickly edit a cross-validated list of impacts (110) and measures (92) with limited costs. Lastly, a public return of results took place. This approach proved to be effective, efficient and influenced the policymakers, overcoming the tendency to enact long-term actions to face climate change. It could be used internationally by subnational authorities also in other sectors.

Climate Adaptation and Successful Adaptation Definitions: Latin American Perspectives Using the Delphi Method

Across the world, policies and measures are being developed and implemented to reduce the risks of climate change and adapt to its current and projected adverse effects. The Paris Agreement established the global stocktake to evaluate the collective progress made on adaptation. Nevertheless, various challenges still exist when evaluating adaptation progress, among which is the lack of standard definitions to support evaluation efforts. Therefore, we investigated the views of experts regarding the definitions of adaptation given by the Intergovernmental Panel on Climate Change (IPCC) and the definition of successful adaptation by Doria et al., with a focus on Latin America. Using the Delphi method, we obtained relevant knowledge and perspectives. As a result, we identified a high level of consensus (85%) among the experts regarding the IPCC’s definition of climate adaptation. However, there was no consensus on the definition of successful adaptation. For both definitions, we present the elements on which the experts agreed and disagreed, as well as the proposed elements that could improve the definitions to support adaptation evaluation efforts. Additionally, we introduce a list of criteria and indicators that could improve the evaluation of adaptation at different management levels and facilitate the aggregation of information on adaptation progress.

Methods for Extrapolating Survival Analyses for the Economic Evaluation of Advanced Therapy Medicinal Products

There are two significant challenges for analysts conducting economic evaluations of advanced therapy medicinal products (ATMPs): (i) estimating long-term treatment effects in the absence of mature clinical data, and (ii) capturing potentially complex hazard functions. This review identifies and critiques a variety of methods that can be used to overcome these challenges. The narrative review is informed by a rapid literature review of methods used for the extrapolation of survival analyses in the economic evaluation of ATMPs. There are several methods that are more suitable than traditional parametric survival modelling approaches for capturing complex hazard functions, including, cure-mixture models and restricted cubic spline models. In the absence of mature clinical data, analysts may augment clinical trial data with data from other sources to aid extrapolation, however, the relative merits of employing methods for including data from different sources is not well understood. Given the high and potentially irrecoverable costs of making incorrect decisions concerning the reimbursement or commissioning of ATMPs, it is important that economic evaluations are correctly specified, and that both parameter and structural uncertainty associated with survival extrapolations are considered. Value of information analyses allow for this uncertainty to be expressed explicitly, and in monetary terms.

Understanding current UK practice for the incidental identification of vertebral fragility fractures from CT scans: an expert elicitation study

Background

There is an emerging interest in using automated approaches to enable the incidental identification of vertebral fragility fractures (VFFs) on existing medical images visualising the spine.

Aim

To quantify values, and the degree of uncertainty associated with them, for the incidental identification of VFFs from computed tomography (CT) scans in current practice.

Methods

An expert elicitation exercise was conducted to generate point estimates and measures of uncertainty for four values representing the probability of: VFF being correctly reported by the radiologist; the absence of VFF being correctly assessed by the radiologist; being referred for management when a VFF is identified; having a dual-energy X-ray absorptiometry (DXA) scan after general practitioner (GP) referral. Data from a sample of seven experts in the diagnosis and management of people with VFFs were pooled using mathematical aggregation.

Results

The estimated mean values for each probability parameter were: VFF being correctly reported by the radiologist = 0.25 (standard deviation (SD): 0.21); absence of VFF being correctly assessed by the radiologist = 0.89 (0.10); being referred for management when a VFF is identified by the radiologist = 0.15 (0.12); having a DXA scan after GP referral = 0.66 (0.28).

Discussion

These estimates could be used to facilitate the subsequent early economic evaluation of potential new approaches to improve the health outcomes of people with VFFs.

Conclusion

In the absence of epidemiological studies, this study produced point estimates and measures of uncertainty for key parameters needed to describe current pathways for the incidental diagnosis of VFFs.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40520-022-02124-w.

Reimagining peer review as an expert elicitation process

Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers and the wider public. We might assume that editors can identify the 'best' experts and rely on them for peer review. But decades of research on both expert decision-making and peer review suggests they cannot. In the absence of a clear criterion for demarcating reliable, insightful, and accurate expert assessors of research quality, the best safeguard against unwanted biases and uneven power distributions is to introduce greater transparency and structure into the process. This paper argues that peer review would therefore benefit from applying a series of evidence-based recommendations from the empirical literature on structured expert elicitation. We highlight individual and group characteristics that contribute to higher quality judgements, and elements of elicitation protocols that reduce bias, promote constructive discussion, and enable opinions to be objectively and transparently aggregated.

Evaluating the effect of expert elicitation techniques on population status assessment in the face of large uncertainty

Population projection models are important tools for conservation and management. They are often used for population status assessments, for threat analyses, and to predict the consequences of conservation actions. Although conservation decisions should be informed by science, critical decisions are often made with very little information to support decision-making. Conversely, postponing decisions until better information is available may reduce the benefit of a conservation decision. When empirical data are limited or lacking, expert elicitation can be used to supplement existing data and inform model parameter estimates. The use of rigorous techniques for expert elicitation that account for uncertainty can improve the quality of the expert elicited values and therefore the accuracy of the projection models. One recurring challenge for summarizing expert elicited values is how to aggregate them. Here, we illustrate a process for population status assessment using a combination of expert elicitation and data from the ecological literature. We discuss the importance of considering various aggregation techniques, and illustrate this process using matrix population models for the wood turtle (Glyptemys insculpta) to assist U.S. Fish and Wildlife Service decision-makers with their Species Status Assessment. We compare estimates of population growth using data from the ecological literature and four alternative aggregation techniques for the expert-elicited values. The estimate of population growth rate based on estimates from the literature (λmean = 0.952, 95% CI: 0.87-1.01) could not be used to unequivocally reject the hypotheses of a rapidly declining population nor the hypothesis of a stable, or even slightly growing population, whereas our results for the expert-elicited estimates supported the hypothesis that the wood turtle population will decline over time. Our results showed that the aggregation techniques used had an impact on model estimates, suggesting that the choice of techniques should be carefully considered. We discuss the benefits and limitations associated with each method and their relevance to the population status assessment. We note a difference in the temporal scope or inference between the literature-based estimates that provided insights about historical changes, whereas the expert-based estimates were forward looking. Therefore, conducting an expert-elicitation in addition to using parameter estimates from the literature improved our understanding of our species of interest.