Assessing the Skill of Yes/No Predictions

Should healthy, middle-aged women receive precautionary mammograms? Should trauma surgeons use the popular TRISS score to predict the likelihood of patient survival? These are examples of questions confronting us when we decide whether to use a yes/no prediction. In order to trust a prediction we must show that it is more valuable than would be our best guess of the future in the absence of the prediction. Calculating value means identifying our loss should the prediction err and examining the past performance of the prediction with respect to that loss. A statistical test to do this is developed. Predictions that pass this test are said to have skill. Only skillful predictions should be used. Graphical and numerical methods to identify skill will be demonstrated. The usefulness of mammograms is explored.