False confidence, non-additive beliefs, and valid statistical inference

Confidence in confidence distributions!

The recent article ‘Satellite conjunction analysis and the false confidence theorem’ (Balch et al . 2019, Satellite conjunction analysis and the false confidence theorem. Proc. R. Soc. A 475 , 20180565) points to certain difficulties with Bayesian analysis when used for models for satellite conjuntion and ensuing operative decisions. Here, we supplement these previous analyses and findings with further insights, uncovering what we perceive of as being the crucial points, explained in a prototype set-up where exact analysis is attainable. We also show that a different and frequentist method, involving confidence distributions, is free of the false confidence syndrome.

Satellite conjunction analysis and the false confidence theorem

Satellite conjunction analysis is the assessment of collision risk during a close encounter between a satellite and another object in orbit. A counterintuitive phenomenon has emerged in the conjunction analysis literature, namely, probability dilution, in which lower quality data paradoxically appear to reduce the risk of collision. We show that probability dilution is a symptom of a fundamental deficiency in probabilistic representations of statistical inference, in which there are propositions that will consistently be assigned a high degree of belief, regardless of whether or not they are true. We call this deficiency false confidence. In satellite conjunction analysis, it results in a severe and persistent underestimate of collision risk exposure. We introduce the Martin–Liu validity criterion as a benchmark by which to identify statistical methods that are free from false confidence. Such inferences will necessarily be non-probabilistic. In satellite conjunction analysis, we show that uncertainty ellipsoids satisfy the validity criterion. Performing collision avoidance manoeuvres based on ellipsoid overlap will ensure that collision risk is capped at the user-specified level. Furthermore, this investigation into satellite conjunction analysis provides a template for recognizing and resolving false confidence issues as they occur in other problems of statistical inference.