Natural Frequencies Do Not Foster Public Understanding of Medical Test Results

The impact of problem domain on Bayesian inferences: A systematic investigation

Sparse (and occasionally contradictory) evidence exists regarding the impact of domain on probabilistic updating, some of which suggests that Bayesian word problems with medical content may be especially challenging. The present research aims to address this gap in knowledge through three pre-registered online studies, which involved a total of 2,238 participants. Bayesian word problems were related to one of three domains: medical, daily-life, and abstract. In the first two cases, problems presented realistic content and plausible numerical information, while in the latter, problems contained explicitly imaginary elements. Problems across domains were matched in terms of all relevant statistical values and, as much as possible, wording. Studies 1 and 2 utilized the same set of problems, but different response elicitation methods (i.e., an open-ended and a multiple-choice question, respectively). Study 3 involved a larger number of participants per condition and a smaller set of problems to more thoroughly investigate the magnitude of differences between the domains. There was a generally low rate of correct responses (17.2%, 17.4%, and 14.3% in Studies 1, 2, and 3, respectively), consistent with accuracy levels commonly observed in the literature for this specific task with online samples. Nonetheless, a small but significant difference between domains was observed: participants' accuracy did not differ between medical and daily-life problems, while it was significantly higher in corresponding abstract problems. These results suggest that medical problems are not inherently more difficult to solve, but rather that performance is improved with abstract problems for which participants cannot draw from their background knowledge.

When intuitive Bayesians need to be good readers: The problem-wording effect on Bayesian reasoning

Are humans intuitive Bayesians? It depends. People seem to be Bayesians when updating probabilities from experience but not when acquiring probabilities from descriptions (i.e., Bayesian textbook problems). Decades of research on textbook problems have focused on how the format of the statistical information (e.g., the natural frequency effect) affects such reasoning. However, it pays much less attention to the wording of these problems. Mathematical problem-solving literature indicates that wording is critical for performance. Wording effects (the wording varied across the problems and manipulations) can also have far-reaching consequences. These may have confounded between-format comparisons and moderated within-format variability in prior research. Therefore, across seven experiments (N = 4909), we investigated the impact of the wording of medical screening problems and statistical formats on Bayesian reasoning in a general adult population. Participants generated more Bayesian answers with natural frequencies than with single-event probabilities, but only with the improved wording. The improved wording of the natural frequencies consistently led to more Bayesian answers than the natural frequencies with standard wording. The improved wording effect occurred mainly due to a more efficient description of the statistical information-cueing required mathematical operations, an unambiguous association of numbers with their reference class and verbal simplification. The wording effect extends the current theoretical explanations of Bayesian reasoning and bears methodological and practical implications. Ultimately, even intuitive Bayesians must be good readers when solving Bayesian textbook problems.

Heterogeneity of rules in Bayesian reasoning: A toolbox analysis

How do people infer the Bayesian posterior probability from stated base rate, hit rate, and false alarm rate? This question is not only of theoretical relevance but also of practical relevance in medical and legal settings. We test two competing theoretical views: single-process theories versus toolbox theories. Single-process theories assume that a single process explains people's inferences and have indeed been observed to fit people's inferences well. Examples are Bayes's rule, the representativeness heuristic, and a weighing-and-adding model. Their assumed process homogeneity implies unimodal response distributions. Toolbox theories, in contrast, assume process heterogeneity, implying multimodal response distributions. After analyzing response distributions in studies with laypeople and professionals, we find little support for the single-process theories tested. Using simulations, we find that a single process, the weighing-and-adding model, nevertheless can best fit the aggregate data and, surprisingly, also achieve the best out-of-sample prediction even though it fails to predict any single respondent's inferences. To identify the potential toolbox of rules, we test how well candidate rules predict a set of over 10,000 inferences (culled from the literature) from 4,188 participants and 106 different Bayesian tasks. A toolbox of five non-Bayesian rules plus Bayes's rule captures 64% of inferences. Finally, we validate the Five-Plus toolbox in three experiments that measure response times, self-reports, and strategy use. The most important conclusion from these analyses is that the fitting of single-process theories to aggregate data risks misidentifying the cognitive process. Antidotes to that risk are careful analyses of process and rule heterogeneity across people.

Tversky and Kahneman's Cognitive Illusions: Who Can Solve Them, and Why?

In the present paper we empirically investigate the psychometric properties of some of the most famous statistical and logical cognitive illusions from the "heuristics and biases" research program by Daniel Kahneman and Amos Tversky, who nearly 50 years ago introduced fascinating brain teasers such as the famous Linda problem, the Wason card selection task, and so-called Bayesian reasoning problems (e.g., the mammography task). In the meantime, a great number of articles has been published that empirically examine single cognitive illusions, theoretically explaining people's faulty thinking, or proposing and experimentally implementing measures to foster insight and to make these problems accessible to the human mind. Yet these problems have thus far usually been empirically analyzed on an individual-item level only (e.g., by experimentally comparing participants' performance on various versions of one of these problems). In this paper, by contrast, we examine these illusions as a group and look at the ability to solve them as a psychological construct. Based on an sample of N = 2,643 Luxembourgian school students of age 16-18 we investigate the internal psychometric structure of these illusions (i.e., Are they substantially correlated? Do they form a reflexive or a formative construct?), their connection to related constructs (e.g., Are they distinguishable from intelligence or mathematical competence in a confirmatory factor analysis?), and the question of which of a person's abilities can predict the correct solution of these brain teasers (by means of a regression analysis).

Public's understanding of swab test results for SARS-CoV-2: an online behavioural experiment during the April 2020 lockdown

OBJECTIVE

Although widespread testing for SARS-CoV-2 is in place, little is known about how well the public understands these results. We aimed to provide a comprehensive overview of the general public's grasp of the accuracy and significance of the results of the swab test.

DESIGN

Web-based behavioural experiment.

SETTING

Italy during the April 2020 lockdown.

PARTICIPANTS

566 Italian residents.

MAIN OUTCOME MEASURES

Participants' estimates of the SARS-CoV-2 prevalence; the predictive and diagnostic accuracy of the test; the behavioural impact of (positive vs negative) test results; the perceived usefulness of a short-term repetition of the test following positive or negative results; and rankings of causes for false positives and false negatives.

RESULTS

Most participants considered the swab test useful (89.6%) and provided predictive values consistent with their estimates of test diagnostic accuracy and infection prevalence (67.0%). Participants acknowledged the effects of symptomatic status and geographical location on prevalence (all p<0.001) but failed to take this information into account when estimating the positive or negative predictive value. Overall, test specificity was underestimated (91.5%, 95% CI 90.2% to 92.8%); test sensitivity was overestimated (89.7%, 95% CI 88.3% to 91.0%). Positive results were evaluated as more informative than negative ones (91.6, 95% CI 90.2 to 93.1 and 41.0, 95% CI 37.9 to 44.0, respectively, p<0.001); a short-term repetition of the test was considered more useful after a positive than a negative result (62.7, 95% CI 59.6 to 65.7 and 47.2, 95% CI 44.4 to 50.0, respectively, p=0.013). Human error and technical characteristics were assessed as more likely to be the causes of false positives (p<0.001); the level of the viral load as the cause of false negatives (p<0.001).

CONCLUSIONS

While some aspects of the swab for SARS-CoV-2 are well grasped, others are not and may have a strong bearing on the general public's health and well-being. The obtained findings provide policymakers with a detailed picture that can guide the design and implementation of interventions for improving efficient communication with the general public as well as adherence to precautionary behaviour.

[Communicating test results in a comprehensible manner: A randomized controlled trial of word usage in doctor-patient communication]

INTRODUCTION

Doctor-patient communication is one of the hallmarks of good medical treatment. Mutual understanding is of foremost importance, in particular when communicating non-numerical test results.

METHODS

Using a two-step approach, this study analyses the influence of wording on the correct understanding of medical test results by people without a medical background. In a first step, a qualitative analysis of physicians' letters helped to identify adjectives that are frequently used to communicate test results. In a second step, a parallel randomised study was conducted to test the comprehensibility of particularly relevant adjectives, combining the written communication of test results with a subsequent survey. 1,131 participants, representing the population of Germany with regard to age, gender, and educational level, were recruited via an online platform. The participants read a scenario involving the communication of the results of a breath test, whereby non-numerical test results were described as being either "positive" vs. "negative" (n=566) or "abnormal" vs. "normal" (n=565). Participant assignment to one of these groups took place in a randomised way. The outcomes measured included the subjective and objective understanding of test results as well as the participants' subjective comprehension of the physician communicating with them.

RESULTS

People without a medical background can understand medical test results more readily when neutral, descriptive adjectives are used rather than adjectives considered as being judgmental in everyday language. 54 % of the participants who read test results using the adjectives "positive" vs. "negative" and 65 % of the participants who read test results using the adjectives "abnormal" vs. "normal", respectively, understood the results correctly. This relative difference of 20.4 % in the number of participants with a correct understanding is statistically significant (Chi square=13.061; p=0.001). There was also a considerable difference in the subjective understanding (means of 5.04 of "positive" vs. "negative" and 5.47 for "abnormal" vs. "normal" on a 7-point Likert scale; absolute mean difference 0.42 [95 % CI: 0.20; 0.64]) as well as in the subjective comprehension of the communicating physician (means of 4.49 for "positive" vs. "negative" and 4.95 for "abnormal" vs. "normal" on a 7-point Likert scale; absolute mean difference 0.45 [95% CI: 0.23; 0.67]). A higher level of comprehension for the words "abnormal" vs. "normal" was consistent across the overall sample. It is mainly people with no school-leaving certificate and a lower educational level who benefit from the changed wording.

CONCLUSIONS

In the process of communicating non-numerical medical test results, people without a medical background understand neutral, descriptive adjectives better than adjectives that are considered judgmental in everyday usage. A corresponding change of written and oral communication can easily be implemented by medical experts in their everyday practice and particularly supports a population group that already suffers disadvantages in the medical system.

A qualitative exploration of clinicians' strategies to communicate risks to patients in the complex reality of clinical practice

BACKGROUND

Risk communication, situated in the model of shared decision making (SDM), is an essential element in daily clinical practice. The scientific literature makes a number of generic recommendations. Yet the application of risk communication remains a challenge in patient-clinician encounters. How clinicians actually communicate risk during consultations is not well understood. We aimed to explore the risk communication strategies used by clinicians and extract narratives and visualizations of those strategies to help inform medical education.

METHODS

In this qualitative descriptive study, we interviewed fifteen purposely sampled clinicians from several medical disciplines, who were familiar with the concept of SDM. Deductive and inductive content analysis was used during an iterative data collection and analyses process.

RESULTS

Our study identified various strategies reported to be used by clinicians to address the complexities of risk communication such as dealing with uncertainty. These included verbal, numerical and visual risk communication and framing. Clinicians were familiar with recommended risk formats such as natural frequencies and population pictograms. However, it became clear that clinicians' expertise and communication goals also play an important role in the risk talk. Clinicians try to lay a foundation for balanced decision-making and to incorporate patient preferences while faced with several challenges such as the dilemma of raising awareness but triggering anxiety or fan fear in patients. Consequently, they also use communication goals such as influencing mindset and reassuring patients. Additionally, clinicians frequently have to account for the illusion of certainty in the risk talk.

CONCLUSION

Risk communication is a multi-faceted construct that cannot be dealt with in isolation from the clinical context. For future research we recommend considering a more practical framework within the clinical setting and to take a goal-directed approach into account when investigating and teaching the topic. The patient perspective should also be addressed in further research.

Fostering the Understanding of Positive Test Results

Background

The majority of health-service users seem unable to properly compute the positive predictive value of medical tests. The research reported in the present study sought to investigate whether, and to what extent, probabilistic inferences about a positive test result can be improved by changing the traditional way in which probability judgments are elicited and medical information is presented.

Methods

Online survey respondents were presented with a positive test result regarding a pregnant woman, and had to estimate the chances that her unborn baby had an anomaly (standard judgment), to apportion the numbers of chances for and against this hypothesis (distributive judgment), and to indicate whether the hypothesis that the baby had an anomaly was more or less likely than its alternative (relative judgment). Test sensitivity and information framing were also manipulated.

Results

Irrespective of education and to some extent of numeracy, the majority of respondents produced correct distributive assessments of chances, which were in line with relative judgments and more accurate than standard ones. When information displayed exclusively positive test results, inferences resulted further improved and unaffected by test sensitivity.

Conclusions

Simple communication strategies that prompt extensional reasoning on the relevant set of number of chances can help individuals to overcome probabilistic inference errors.

Associations between Case Fatality Rates and Self-Reported Fear of Neoplasms and Circulatory Diseases

Background. According to decision by sampling theory, people store relative frequencies of events in memory, and these values constitute subjective representations of events. Because fear is a natural response to the threat of death, we hypothesized that case fatality rate (CFR) statistics, which represent how deadly a disease is, would be positively correlated with self-reported fear ratings of neoplasms and circulatory diseases. Methods. Participants (N = 239) were asked to rate various neoplasms and circulatory diseases (110 diseases in total) on fear, typicality, and disgust scales (e.g., 1 = no fear, 10 = intense fear). They also estimated mortality and morbidity rates for the same set of diseases. Finally, they completed the Berlin Numeracy Test. CFRs were obtained from the World Health Organization (WHO) database. The association between relative CFR and fear ratings was tested using correlation analyses and a multilevel linear model with Bayesian inference techniques. Results. We found that fear ratings were related to relative CFRs (r = 0.42, [0.25, 0.56], BF = 3511). This effect was present on aggregate and, to some extent, on individual levels, even after controlling for other ratings, morbidity rate, participants' estimates of mortality and morbidity statistics, numeracy, sex, age, and knowledge of WHO statistics. Also, women rated neoplasms as more frightening than circulatory diseases, and typicality ratings were related to morbidity rates. Limitations. Limited number of diagnostic entities and categories, lack of control over the technicality of disease names and participants' experience of diseases, and study sample (83% young women). Conclusions. We present initial evidence that implicit acquisition of CFRs of diseases through everyday experience may be related to the intensity of fear reactions to them.

An Eye-Tracking Study of Statistical Reasoning With Tree Diagrams and 2 × 2 Tables

Changing the information format from probabilities into frequencies as well as employing appropriate visualizations such as tree diagrams or 2 × 2 tables are important tools that can facilitate people's statistical reasoning. Previous studies have shown that despite their widespread use in statistical textbooks, both of those visualization types are only of restricted help when they are provided with probabilities, but that they can foster insight when presented with frequencies instead. In the present study, we attempt to replicate this effect and also examine, by the method of eye tracking, why probabilistic 2 × 2 tables and tree diagrams do not facilitate reasoning with regard to Bayesian inferences (i.e., determining what errors occur and whether they can be explained by scan paths), and why the same visualizations are of great help to an individual when they are combined with frequencies. All ten inferences of N = 24 participants were based solely on tree diagrams or 2 × 2 tables that presented either the famous "mammography context" or an "economics context" (without additional textual wording). We first asked participants for marginal, conjoint, and (non-inverted) conditional probabilities (or frequencies), followed by related Bayesian tasks. While solution rates were higher for natural frequency questions as compared to probability versions, eye-tracking analyses indeed yielded noticeable differences regarding eye movements between correct and incorrect solutions. For instance, heat maps (aggregated scan paths) of distinct results differed remarkably, thereby making correct and faulty strategies visible in the line of theoretical classifications. Moreover, the inherent structure of 2 × 2 tables seems to help participants avoid certain Bayesian mistakes (e.g., "Fisherian" error) while tree diagrams seem to help steer them away from others (e.g., "joint occurrence"). We will discuss resulting educational consequences at the end of the paper.

Why Can Only 24% Solve Bayesian Reasoning Problems in Natural Frequencies: Frequency Phobia in Spite of Probability Blindness

For more than 20 years, research has proven the beneficial effect of natural frequencies when it comes to solving Bayesian reasoning tasks (Gigerenzer and Hoffrage, 1995). In a recent meta-analysis, McDowell and Jacobs (2017) showed that presenting a task in natural frequency format increases performance rates to 24% compared to only 4% when the same task is presented in probability format. Nevertheless, on average three quarters of participants in their meta-analysis failed to obtain the correct solution for such a task in frequency format. In this paper, we present an empirical study on what participants typically do wrong when confronted with natural frequencies. We found that many of them did not actually use natural frequencies for their calculations, but translated them back into complicated probabilities instead. This switch from the intuitive presentation format to a less intuitive calculation format will be discussed within the framework of psychological theories (e.g., the Einstellung effect).

From reading numbers to seeing ratios: a benefit of icons for risk comprehension

Promoting a better understanding of statistical data is becoming increasingly important for improving risk comprehension and decision-making. In this regard, previous studies on Bayesian problem solving have shown that iconic representations help infer frequencies in sets and subsets. Nevertheless, the mechanisms by which icons enhance performance remain unclear. Here, we tested the hypothesis that the benefit offered by icon arrays lies in a better alignment between presented and requested relationships, which should facilitate the comprehension of the requested ratio beyond the represented quantities. To this end, we analyzed individual risk estimates based on data presented either in standard verbal presentations (percentages and natural frequency formats) or as icon arrays. Compared to the other formats, icons led to estimates that were more accurate, and importantly, promoted the use of equivalent expressions for the requested probability. Furthermore, whereas the accuracy of the estimates based on verbal formats depended on their alignment with the text, all the estimates based on icons were equally accurate. Therefore, these results support the proposal that icons enhance the comprehension of the ratio and its mapping onto the requested probability and point to relational misalignment as potential interference for text-based Bayesian reasoning. The present findings also argue against an intrinsic difficulty with understanding single-event probabilities.

Visualizing the Bayesian 2-test case: The effect of tree diagrams on medical decision making

In medicine, diagnoses based on medical test results are probabilistic by nature. Unfortunately, cognitive illusions regarding the statistical meaning of test results are well documented among patients, medical students, and even physicians. There are two effective strategies that can foster insight into what is known as Bayesian reasoning situations: (1) translating the statistical information on the prevalence of a disease and the sensitivity and the false-alarm rate of a specific test for that disease from probabilities into natural frequencies, and (2) illustrating the statistical information with tree diagrams, for instance, or with other pictorial representation. So far, such strategies have only been empirically tested in combination for “1-test cases”, where one binary hypothesis (“disease” vs. “no disease”) has to be diagnosed based on one binary test result (“positive” vs. “negative”). However, in reality, often more than one medical test is conducted to derive a diagnosis. In two studies, we examined a total of 388 medical students from the University of Regensburg (Germany) with medical “2-test scenarios”. Each student had to work on two problems: diagnosing breast cancer with mammography and sonography test results, and diagnosing HIV infection with the ELISA and Western Blot tests. In Study 1 (N = 190 participants), we systematically varied the presentation of statistical information (“only textual information” vs. “only tree diagram” vs. “text and tree diagram in combination”), whereas in Study 2 (N = 198 participants), we varied the kinds of tree diagrams (“complete tree” vs. “highlighted tree” vs. “pruned tree”). All versions were implemented in probability format (including probability trees) and in natural frequency format (including frequency trees). We found that natural frequency trees, especially when the question-related branches were highlighted, improved performance, but that none of the corresponding probabilistic visualizations did.

Natural Frequencies Do Foster Public Understanding of Medical Tests: Comment on Pighin, Gonzalez, Savadori, and Girotto (2016)

Patients and doctors often need to make decisions based on the results of medical tests. When these results are presented in the form of conditional probabilities, even doctors find it difficult to interpret them correctly. There is over 20 y of research supporting the finding that people are better able to calculate the correct positive predictive value of a test when given information in natural frequencies, as opposed to conditional probabilities. Natural frequencies are one of a few psychological tools that have made it into evidence-based medicine. Recently, Pighin and others (Med Decis Making 2016;36:686-91) argued that natural frequencies could hinder informed decision making, a critique based on a single task and a crude scoring criterion we refer to as the 50%-Split. Our commentary addresses these criticisms based on three analyses. First, we show how the 50%-Split scoring used by Pighin and others misclassifies known errors, such as solely attending to the hit rate (true-positive rate) of the test, as strategies that support understanding. Second, we reanalyze data from 21 additional problems completed by various participant groups to show that their scoring criterion does not support their results in 19 out of 21 cases. Third, we apply the mean deviation scoring method and show that, when given information in natural frequency formats, participants provide estimates that are closer to the correct Bayesian solution than for conditional probability formats. In each analysis, natural frequencies lead to more correct judgements and therefore promote informed decision making relative to conditional probabilities. We welcome further discussions of performance metrics that can provide insight into how the public and therefore patients understand the implications of medical test results.

Another chance for good reasoning

Disagreement on the "probability status" of chances casts doubt on Girotto and Gonzalez's (2001) conclusion that the human mind can make sound Bayesian inferences involving single-event probabilities. The main objection raised has been that chances are de facto natural frequencies disguised as probabilities. In the present study, we empirically demonstrated that numbers of chances are perceived as being distinct from natural frequencies and that they have a facilitatory effect on Bayesian inference tasks that is completely independent from their (minor) frequentist readings. Overall, therefore, our results strongly disconfirm the hypothesis that natural frequencies are a privileged cognitive representational format for Bayesian inferences and suggest that a significant portion of laypeople adequately handle genuine single-event probability problems once these are rendered computationally more accessible by using numbers of chances.