Uncertainty Quantification with Experts: Present Status and Research Needs

Elicitation of Rank Correlations with Probabilities of Concordance: Method and Application to Building Management

Constructing Bayesian networks (BN) for practical applications presents significant challenges, especially in domains with limited empirical data available. In such situations, field experts are often consulted to estimate the model's parameters, for instance, rank correlations in Gaussian copula-based Bayesian networks (GCBN). Because there is no consensus on a 'best' approach for eliciting these correlations, this paper proposes a framework that uses probabilities of concordance for assessing dependence, and the dependence calibration score to aggregate experts' judgments. To demonstrate the relevance of our approach, the latter is implemented to populate a GCBN intended to estimate the condition of air handling units' components-a key challenge in building asset management. While the elicitation of concordance probabilities was well received by the questionnaire respondents, the analysis of the results reveals notable disparities in the experts' ability to quantify uncertainty. Moreover, the application of the dependence calibration aggregation method was hindered by the absence of relevant seed variables, thus failing to evaluate the participants' field expertise. All in all, while the authors do not recommend to use the current model in practice, this study suggests that concordance probabilities should be further explored as an alternative approach for the elicitation of dependence.

Robinson A. Co-designing and building an expert-elicited non-parametric Bayesian network model: demonstrating a methodology using a Bonamia Ostreae spread risk case study

The development and use of probabilistic models, particularly Bayesian networks (BN), to support risk-based decision making is well established. Striking an efficient balance between satisfying model complexity and ease of development requires continuous compromise. Codesign, wherein the structural content of the model is developed hand-in-hand with the experts who will be accountable for the parameter estimates, shows promise, as do so-called nonparametric Bayesian networks (NPBNs), which provide a light-touch approach to capturing complex relationships among nodes. We describe and demonstrate the process of codesigning, building, quantifying, and validating an NPBN model for emerging risks and the consequences of potential management decisions using structured expert judgment (SEJ). We develop a case study of the local spread of a marine pathogen, namely, Bonamia ostreae. The BN was developed through a series of semistructured workshops that incorporated extensive feedback from many experts. The model was then quantified with a combination of field and expert-elicited data. The IDEA protocol for SEJ was used in its hybrid (remote and face-to-face) form to elicit information about more than 100 parameters. This article focuses on the modeling and quantification process, the methodological challenges, and the way these were addressed.

Methods for Eliciting Informative Prior Distributions: A Critical Review

Eliciting informative prior distributions for Bayesian inference can often be complex and challenging. Although popular methods rely on asking experts probability-based questions to quantify uncertainty, these methods are not without their drawbacks, and many alternative elicitation methods exist. This paper explores methods for eliciting informative priors categorized by type and briefly discusses their strengths and limitations. Most of the review literature in this field focuses on a particular type of elicitation approach. The primary aim of this work, however, is to provide a more complete yet macro view of the state of the art by highlighting new (and old) approaches in one clear easy-to-read article. Two representative applications are used throughout to explore the suitability, or lack thereof, of the existing methods, one of which highlights a challenge that has not been addressed in the literature yet. We identify some of the gaps in the present work and discuss directions for future research.

One Coin, Two Sides: Eliciting Expert Knowledge From Training Participants in a Capacity-Building Program for Veterinary Professionals

Scientific research may include the elicitation of judgment from non-academic subject-matter experts in order to improve the quality and/or impact of research studies. Elicitation of expert knowledge or judgment is used when data are missing, incomplete, or not representative for the specific setting and processes being studied. Rigorous methods are crucial to ensure robust study results, and yet the quality of the elicitation can be affected by a number of practical constraints, including the understanding that subject-matter experts have of the elicitation process itself. In this paper, we present a case of expert elicitation embedded within an extended training course for veterinary professionals as an example of overcoming these constraints. The coupling of the two activities enabled extended opportunities for training and a relationship of mutual respect to be the foundation for the elicitation process. In addition, the participatory research activities reinforced knowledge synthesis objectives of the educational program. Finally, the synergy between the two concurrent objectives may produce benefits which transcend either independent activity: solutions and ideas built by local professionals, evolving collaborative research and training approaches, and a network of diverse academic and practicing professionals. This approach has the versatility to be adapted to many training and research opportunities.

Self-Reporting of Risk Pathways and Parameter Values for Foot-and-Mouth Disease in Slaughter Cattle from Alternative Production Systems by Kenyan and Ugandan Veterinarians

Countries in which foot-and-mouth disease (FMD) is endemic may face bans on the export of FMD-susceptible livestock and products because of the associated risk for transmission of FMD virus. Risk assessment is an essential tool for demonstrating the fitness of one’s goods for the international marketplace and for improving animal health. However, it is difficult to obtain the necessary data for such risk assessments in many countries where FMD is present. This study bridged the gaps of traditional participatory and expert elicitation approaches by partnering with veterinarians from the National Veterinary Services of Kenya (n = 13) and Uganda (n = 10) enrolled in an extended capacity-building program to systematically collect rich, local knowledge in a format appropriate for formal quantitative analysis. Participants mapped risk pathways and quantified variables that determine the risk of infection among cattle at slaughter originating from each of four beef production systems in each country. Findings highlighted that risk processes differ between management systems, that disease and sale are not always independent events, and that events on the risk pathway are influenced by the actions and motivations of value chain actors. The results provide necessary information for evaluating the risk of FMD among cattle pre-harvest in Kenya and Uganda and provide a framework for similar evaluation in other endemic settings.