Visual representation of statistical information improves diagnostic inferences in doctors and their patients

Too Worried to Judge: On the Role of Perceived Severity in Medical Decision-Making

Ideally, decisions regarding one’s health should be made after assessing the objective probabilities of relevant outcomes. Nevertheless, previous beliefs and emotional reactions also have a role in decision-making. Furthermore, the comprehension of probabilities is commonly affected by the presentation format, and by numeracy. This study aimed to assess the extent to which the influence of these factors might vary between different medical conditions. A sample of university students were presented with two health scenarios containing statistical information on the prevalence of breast cancer and hypertension either through icon arrays (N = 71) or natural frequencies (N = 72). They also received information regarding a preventive measure (mammogram/low-sodium diet) and the likelihood of a positive mammogram or a rich-sodium diet either when suffering or not suffering from the disease. Before seeing the data, participants rated the severity of the disease and the inconvenience of the preventive measure. After reading the health scenario, participants had to rate its difficulty, and how worrisome it was. They had also to rate the prior probability of suffering from this medical condition, and the posterior probability of it, provided a positive mammogram or a rich-sodium diet. Finally, they rated the extent to which they would recommend the preventive measures. All the rates used the same 1 (little)-8 (a great deal) scale. Participants’ numeracy was also assessed. The scenarios differed significantly in perceived severity and worry, with the cancer scenario obtaining higher scores. Importantly, regression analyses showed that the recommendations in the two health scenarios depended on different variables. A model taking into consideration severity and worry rates best explained decisions in the cancer scenario; in contrast, in the hypertension scenario the model that best explained the recommendations comprised both the posterior probability estimate and the severity rate. Neither numeracy nor presentation format affected recommendation but both affected difficulty, worrying and probability rates. We conclude that previous perceptions of the severity of a health condition modulate the use of probabilistic information for decision-making. The roles of presentation format and numeracy in enabling patients to understand statistical information are also discussed.

Why Can Only 24% Solve Bayesian Reasoning Problems in Natural Frequencies: Frequency Phobia in Spite of Probability Blindness

For more than 20 years, research has proven the beneficial effect of natural frequencies when it comes to solving Bayesian reasoning tasks (Gigerenzer and Hoffrage, 1995). In a recent meta-analysis, McDowell and Jacobs (2017) showed that presenting a task in natural frequency format increases performance rates to 24% compared to only 4% when the same task is presented in probability format. Nevertheless, on average three quarters of participants in their meta-analysis failed to obtain the correct solution for such a task in frequency format. In this paper, we present an empirical study on what participants typically do wrong when confronted with natural frequencies. We found that many of them did not actually use natural frequencies for their calculations, but translated them back into complicated probabilities instead. This switch from the intuitive presentation format to a less intuitive calculation format will be discussed within the framework of psychological theories (e.g., the Einstellung effect).

From reading numbers to seeing ratios: a benefit of icons for risk comprehension

Promoting a better understanding of statistical data is becoming increasingly important for improving risk comprehension and decision-making. In this regard, previous studies on Bayesian problem solving have shown that iconic representations help infer frequencies in sets and subsets. Nevertheless, the mechanisms by which icons enhance performance remain unclear. Here, we tested the hypothesis that the benefit offered by icon arrays lies in a better alignment between presented and requested relationships, which should facilitate the comprehension of the requested ratio beyond the represented quantities. To this end, we analyzed individual risk estimates based on data presented either in standard verbal presentations (percentages and natural frequency formats) or as icon arrays. Compared to the other formats, icons led to estimates that were more accurate, and importantly, promoted the use of equivalent expressions for the requested probability. Furthermore, whereas the accuracy of the estimates based on verbal formats depended on their alignment with the text, all the estimates based on icons were equally accurate. Therefore, these results support the proposal that icons enhance the comprehension of the ratio and its mapping onto the requested probability and point to relational misalignment as potential interference for text-based Bayesian reasoning. The present findings also argue against an intrinsic difficulty with understanding single-event probabilities.

How to Think Rationally about World Problems

I agree with the target essay that psychology has something to offer in helping to address societal problems. Intelligence has helped meliorate some social problems throughout history, including the period of time that is covered by the Flynn effect, but I agree with Sternberg that other psychological characteristics may be contributing as well, particularly increases in rationality. I also believe that increasing human rationality could have a variety of positive societal affects at levels somewhat smaller in grain size than the societal problems that Sternberg focuses on. Some of the societal problems that Sternberg lists, however, I do not think would be remedied by increases in rationality, intelligence, or wisdom, because remedy might be the wrong word in the context of these issues. Issues such as how much inequality of income to tolerate, how much pollution to tolerate, and how much we should sacrifice economic growth for potential future changes in global temperature represent issues of clashing values, not the inability to process information, nor the lack of information, nor the failure to show wisdom.

Emergency Physician Risk Estimates and Admission Decisions for Chest Pain: A Web-Based Scenario Study

STUDY OBJECTIVE

We conducted this study to better understand how emergency physicians estimate risk and make admission decisions for patients with low-risk chest pain.

METHODS

We created a Web-based survey consisting of 5 chest pain scenarios that included history, physical examination, ECG findings, and basic laboratory studies, including a negative initial troponin-level result. We administered the scenarios in random order to emergency medicine residents and faculty at 11 US emergency medicine residency programs. We randomized respondents to receive questions about 1 of 2 endpoints, acute coronary syndrome or serious complication (death, dysrhythmia, or congestive heart failure within 30 days). For each scenario, the respondent provided a quantitative estimate of the probability of the endpoint, a qualitative estimate of the risk of the endpoint (very low, low, moderate, high, or very high), and an admission decision. Respondents also provided demographic information and completed a 3-item Fear of Malpractice scale.

RESULTS

Two hundred eight (65%) of 320 eligible physicians completed the survey, 73% of whom were residents. Ninety-five percent of respondents were wholly consistent (no admitted patient was assigned a lower probability than a discharged patient). For individual scenarios, probability estimates covered at least 4 orders of magnitude; admission rates for scenarios varied from 16% to 99%. The majority of respondents (>72%) had admission thresholds at or below a 1% probability of acute coronary syndrome. Respondents did not fully differentiate the probability of acute coronary syndrome and serious outcome; for each scenario, estimates for the two were quite similar despite a serious outcome being far less likely. Raters used the terms "very low risk" and "low risk" only when their probability estimates were less than 1%.

CONCLUSION

The majority of respondents considered any probability greater than 1% for acute coronary syndrome or serious outcome to be at least moderate risk and warranting admission. Physicians used qualitative terms in ways fundamentally different from how they are used in ordinary conversation, which may lead to miscommunication during shared decisionmaking processes. These data suggest that probability or utility models are inadequate to describe physician decisionmaking for patients with chest pain.

Improving Understanding of Diagnostic Test Outcomes

BACKGROUND

Understanding diagnostic test outcomes requires determining the positive predictive value (PPV) of the test, which most laypeople and medical professionals struggle to do. Despite advances found with frequency formats and visual aids, less than 40% of people can typically identify this value. This study tests the impact of using congruent reference classes in problem-question pairings, evaluates the role of numeracy, and assesses how diagnostic value estimates affect the reported likelihood to use the test.

METHOD

A 3 × 2, Pairing (congruent test-focus, congruent condition-focus, incongruent) × Response Format (frequency, percentage) mixed design experiment was conducted, in which participants answered diagnostic questions about 7 medical problems presented in a format focusing either on the reference class of those who test positive or those who have the condition. Answer accuracy, numeracy, and ratings of likelihood to use estimates were assessed.

RESULTS

Focusing on the congruent test reference class allowed 87% of participants to consistently identify the PPV, and focusing on the congruent condition reference class led 63% of participants to consistently identify the sensitivity (SEN). Aligning reference classes was especially beneficial for those with lower numeracy, increasing accuracy on problems from 21% for incongruent pairings to 66% for congruent pairings. Ratings of likelihood to use the test were closely tied to participants' estimates of diagnostic values, regardless of the accuracy of those estimates.

CONCLUSIONS

Although often overlooked, a straightforward mapping of reference classes from the relevant diagnostic information to the question of interest reduces confusion and substantially increases accuracy in estimates of diagnostic values. These findings can be used to strengthen training in the assessment of uncertainties associated with medical test results.

Visualizing the Bayesian 2-test case: The effect of tree diagrams on medical decision making

In medicine, diagnoses based on medical test results are probabilistic by nature. Unfortunately, cognitive illusions regarding the statistical meaning of test results are well documented among patients, medical students, and even physicians. There are two effective strategies that can foster insight into what is known as Bayesian reasoning situations: (1) translating the statistical information on the prevalence of a disease and the sensitivity and the false-alarm rate of a specific test for that disease from probabilities into natural frequencies, and (2) illustrating the statistical information with tree diagrams, for instance, or with other pictorial representation. So far, such strategies have only been empirically tested in combination for “1-test cases”, where one binary hypothesis (“disease” vs. “no disease”) has to be diagnosed based on one binary test result (“positive” vs. “negative”). However, in reality, often more than one medical test is conducted to derive a diagnosis. In two studies, we examined a total of 388 medical students from the University of Regensburg (Germany) with medical “2-test scenarios”. Each student had to work on two problems: diagnosing breast cancer with mammography and sonography test results, and diagnosing HIV infection with the ELISA and Western Blot tests. In Study 1 (N = 190 participants), we systematically varied the presentation of statistical information (“only textual information” vs. “only tree diagram” vs. “text and tree diagram in combination”), whereas in Study 2 (N = 198 participants), we varied the kinds of tree diagrams (“complete tree” vs. “highlighted tree” vs. “pruned tree”). All versions were implemented in probability format (including probability trees) and in natural frequency format (including frequency trees). We found that natural frequency trees, especially when the question-related branches were highlighted, improved performance, but that none of the corresponding probabilistic visualizations did.

Natural Frequencies Do Foster Public Understanding of Medical Tests: Comment on Pighin, Gonzalez, Savadori, and Girotto (2016)

Patients and doctors often need to make decisions based on the results of medical tests. When these results are presented in the form of conditional probabilities, even doctors find it difficult to interpret them correctly. There is over 20 y of research supporting the finding that people are better able to calculate the correct positive predictive value of a test when given information in natural frequencies, as opposed to conditional probabilities. Natural frequencies are one of a few psychological tools that have made it into evidence-based medicine. Recently, Pighin and others (Med Decis Making 2016;36:686-91) argued that natural frequencies could hinder informed decision making, a critique based on a single task and a crude scoring criterion we refer to as the 50%-Split. Our commentary addresses these criticisms based on three analyses. First, we show how the 50%-Split scoring used by Pighin and others misclassifies known errors, such as solely attending to the hit rate (true-positive rate) of the test, as strategies that support understanding. Second, we reanalyze data from 21 additional problems completed by various participant groups to show that their scoring criterion does not support their results in 19 out of 21 cases. Third, we apply the mean deviation scoring method and show that, when given information in natural frequency formats, participants provide estimates that are closer to the correct Bayesian solution than for conditional probability formats. In each analysis, natural frequencies lead to more correct judgements and therefore promote informed decision making relative to conditional probabilities. We welcome further discussions of performance metrics that can provide insight into how the public and therefore patients understand the implications of medical test results.

Imagining Replications: Graphical Prediction & Discrete Visualizations Improve Recall & Estimation of Effect Uncertainty

People often have erroneous intuitions about the results of uncertain processes, such as scientific experiments. Many uncertainty visualizations assume considerable statistical knowledge, but have been shown to prompt erroneous conclusions even when users possess this knowledge. Active learning approaches been shown to improve statistical reasoning, but are rarely applied in visualizing uncertainty in scientific reports. We present a controlled study to evaluate the impact of an interactive, graphical uncertainty prediction technique for communicating uncertainty in experiment results. Using our technique, users sketch their prediction of the uncertainty in experimental effects prior to viewing the true sampling distribution from an experiment. We find that having a user graphically predict the possible effects from experiment replications is an effective way to improve one's ability to make predictions about replications of new experiments. Additionally, visualizing uncertainty as a set of discrete outcomes, as opposed to a continuous probability distribution, can improve recall of a sampling distribution from a single experiment. Our work has implications for various applications where it is important to elicit peoples' estimates of probability distributions and to communicate uncertainty effectively.

Aiding risk information learning through simulated experience (ARISE): Using simulated outcomes to improve understanding of conditional probabilities in prenatal Down syndrome screening

OBJECTIVE

To determine whether the use of simulated experiences to communicate statistical information can improve an individual's understanding of conditional probabilities-specifically the positive predictive value (PPV) of prenatal screening tests for Down syndrome.

METHODS

In Experiment 1 (N=64) and Experiment 2 (N=180) participants were asked to estimate the PPV of a prenatal screening test for Down syndrome based on either (1) explicit statistics regarding the prevalence of Down syndrome and the sensitivity and specificity of a prenatal screening test for Down syndrome, or (2) experiencing up to 5000 simulated test results over a short time.

RESULTS

Participants' estimates of the PPV were more accurate when they had learned via simulated experiences (79% accuracy) compared with estimates based on explicitly described statistics (14%). Participants in the simulated experience condition also reported decreased interest in screening and decreased concern with a positive test result.

CONCLUSION

Simulated experiences improve PPV estimates, compared to estimates derived from explicitly provided statistics, while also shifting attitudes away from screening.

PRACTICE IMPLICATIONS

The use of simulated experiences may prove to be simple but powerful tool to communicate complex statistical information to patients in medical decision making situations.

Visual Aids for Improving Patient Decision Making in Severe Symptomatic Carotid Stenosis

BACKGROUND

Because of the large amount of information to process and the limited time of a clinical consult, choosing between carotid endarterectomy (CEA) and carotid angioplasty with stenting (CAS) can be confusing for patients with severe symptomatic internal carotid stenosis (ICA).

GOAL

We aim to develop a visual aid tool to help clinicians and patients in the decision-making process of selecting between CEA and CAS.

MATERIALS AND METHODS

Based on pooled analysis from randomized controlled trials including patients with symptomatic and severe ICA (SSICA), we generated visual plots comparing CEA with CAS for 3 prespecified postprocedural time points: (1) any stroke or death at 4 months, and (2) any stroke or death in the first 30 days and ipsilateral stroke thereafter at 5 years and (3) at 10 years.

RESULTS

A total of 4574 participants (2393 assigned to CAS, and 2361 to CEA) were included in the analyses. For every 100 patients with SSICA, 6 would develop any stroke or death in the CEA group compared with 9 undergoing CAS at 4 months (hazard ratio [HR] 1.53; 95%CI 1.20-1.95). At 5 years, 7 patients in the CEA group would develop any periprocedural stroke or death and ipsilateral stroke thereafter versus 12 undergoing CAS (HR 1.72; 95%CI 1.24-2.39), compared with 10 patients in the CEA and 13 in the CAS groups at 10 years (HR 1.17; 95%CI 0.82-1.66).

CONCLUSION

Visual aids presented in this study could potentially help patients with severe symptomatic internal carotid stenosis to better weigh the risks and benefits of CEA versus CAS as a function of time, allowing for the prioritization of personal preferences, and should be prospectively assessed.

Visualizing risks in cancer communication: A systematic review of computer-supported visual aids

OBJECTIVE

Health websites are becoming important sources for cancer information. Lay users, patients and carers seek support for critical decisions, but they are prone to common biases when quantitative information is presented. Graphical representations of risk data can facilitate comprehension, and interactive visualizations are popular. This review summarizes the evidence on computer-supported graphs that present risk data and their effects on various measures.

METHODS

The systematic literature search was conducted in several databases, including MEDLINE, EMBASE and CINAHL. Only studies with a controlled design were included. Relevant publications were carefully selected and critically appraised by two reviewers.

RESULTS

Thirteen studies were included. Ten studies evaluated static graphs and three dynamic formats. Most decision scenarios were hypothetical. Static graphs could improve accuracy, comprehension, and behavioural intention. But the results were heterogeneous and inconsistent among the studies. Dynamic formats were not superior or even impaired performance compared to static formats.

CONCLUSIONS

Static graphs show promising but inconsistent results, while research on dynamic visualizations is scarce and must be interpreted cautiously due to methodical limitations.

PRACTICE IMPLICATIONS

Well-designed and context-specific static graphs can support web-based cancer risk communication in particular populations. The application of dynamic formats cannot be recommended and needs further research.

Designing Visual Aids That Promote Risk Literacy: A Systematic Review of Health Research and Evidence-Based Design Heuristics

Background Effective risk communication is essential for informed decision making. Unfortunately, many people struggle to understand typical risk communications because they lack essential decision-making skills. Objective The aim of this study was to review the literature on the effect of numeracy on risk literacy, decision making, and health outcomes, and to evaluate the benefits of visual aids in risk communication. Method We present a conceptual framework describing the influence of numeracy on risk literacy, decision making, and health outcomes, followed by a systematic review of the benefits of visual aids in risk communication for people with different levels of numeracy and graph literacy. The systematic review covers scientific research published between January 1995 and April 2016, drawn from the following databases: Web of Science, PubMed, PsycINFO, ERIC, Medline, and Google Scholar. Inclusion criteria were investigation of the effect of numeracy and/or graph literacy, and investigation of the effect of visual aids or comparison of their effect with that of numerical information. Thirty-six publications met the criteria, providing data on 27,885 diverse participants from 60 countries. Results Transparent visual aids robustly improved risk understanding in diverse individuals by encouraging thorough deliberation, enhancing cognitive self-assessment, and reducing conceptual biases in memory. Improvements in risk understanding consistently produced beneficial changes in attitudes, behavioral intentions, trust, and healthy behaviors. Visual aids were found to be particularly beneficial for vulnerable and less skilled individuals. Conclusion Well-designed visual aids tend to be highly effective tools for improving informed decision making among diverse decision makers. We identify five categories of practical, evidence-based guidelines for heuristic evaluation and design of effective visual aids.

Designing Graphs to Communicate Risks: Understanding How the Choice of Graphical Format Influences Decision Making

Previous research suggests that the choice of graphical format for communicating risk information affects both understanding of the risk magnitude and the likelihood of acting to decrease risk. However, the mechanisms through which these effects work are poorly understood. To explore these mechanisms using a real-world scenario, we examined the relative impact of two graphical displays for depicting the risk of exposure to unexploded ammunition during potential land redevelopment. One display depicted only the foreground information graphically (a bar graph of the number of people harmed), and a second depicted the foreground and background graphically (a stacked bar graph representing both the number harmed and at risk). We presented 296 participants with either the foreground-only or the foreground and background graphical display and measured a broad set of outcome variables, examining (1) the graphical display effect on each of the outcome measures and (2) the pathways by which any display effects work to influence decision making. We found that the foreground-only graphical display increased perceived likelihood and experienced fear, which produced greater worry, which in turn increased risk aversion. In addition, a positive evaluation of the communication materials increased support for policies related to land redevelopment, whether those policies were risk taking or risk mitigating. Finally, the foreground-only graphical display decreased understanding of the risk magnitude, showing that approaches to accomplish one risk communication goal (promoting risk-averse decisions) may do so at the expense of another goal (increasing understanding).

The Impact of Visualizing Nested Sets. An Empirical Study on Tree Diagrams and Unit Squares

It is an ongoing debate, what properties of visualizations increase people's performance when solving Bayesian reasoning tasks. In the discussion of the properties of two visualizations, i.e., the tree diagram and the unit square, we emphasize how both visualizations make relevant subset relations transparent. Actually, the unit square with natural frequencies reveals the subset relation that is essential for the Bayes' rule in a numerical and geometrical way whereas the tree diagram with natural frequencies does it only in a numerical way. Accordingly, in a first experiment with 148 university students, the unit square outperformed the tree diagram when referring to the students' ability to quantify the subset relation that must be applied in Bayes' rule. As hypothesized, in a second experiment with 143 students, the unit square was significantly more effective when the students' performance in tasks based on Bayes' rule was regarded. Our results could inform the debate referring to Bayesian reasoning since we found that the graphical transparency of nested sets could explain these visualizations' effect.

Visual elements in direct-to-consumer advertising: Messages communicated to patients with arthritis

Direct-to-consumer (DTC) advertising saturates popular health magazines, communicating persuasive messages to readers that may influence attitudes and behaviors. This research used a two-prong approach to investigate the visual elements used in DTC advertising and their influence on consumers' understanding of a disease and its treatment options. An analysis was conducted of DTC advertisements (N = 62) from a population sample of Arthritis Today magazine, 2000-2010. Three panels of people with arthritis were used to validate the findings and discuss implications for health literacy. Pharmaceutical companies have an opportunity to communicate tailored messages to readers of niche publications and improve disease management.

Improving the Understanding of Publicly Reported Healthcare-Associated Infection (HAI) Data

OBJECTIVE Hospital-acquired infection (HAI) data are reported to the public on the Centers for Medicare and Medicaid Services (CMS) Hospital Compare website. We previously found that public understanding of these data is poor. Our objective was to develop an improved method for presenting HAI data that could be used on the CMS website. DESIGN Randomized controlled trial comparing understanding of data presented using the current CMS presentation strategy versus a new strategy. SETTING A 760-bed tertiary referral hospital. PARTICIPANTS A total of 61 patients were randomly selected within 24 hours of admission. INTERVENTION Participants were shown HAI data as presented on the CMS Hospital Compare website (control arm) or data formatted using a new method (experimental arm). RESULTS No statistically significant demographic differences were identified between study arms. Although 47% percent of participants said a website for comparing hospitals would have been helpful, only 10% had ever used such a website. Participants viewing data using the new presentation strategy compared hospitals correctly 56% of the time, compared with 32% in the control arm (P=.0002). CONCLUSIONS Understanding of HAI data increased significantly with the new data presentation method compared to the method currently used on the CMS Hospital Compare website. Many participants expressed interest in a website for comparing hospitals. Improved methods for presenting CMS HAI data, such as the one assessed here, should be adopted to increase public understanding. Infect Control Hosp Epidemiol 2016;1-6.

Improving risk literacy in surgeons

OBJECTIVE

To effectively practice evidence-based medicine, surgeons need to understand and be able to communicate health-relevant numerical information. We present the first study examining risk literacy in surgeons by assessing numeracy and surgical risk comprehension. Our study also investigated whether visual aids improve risk comprehension in surgeons with limited numeracy.

METHODS

Participants were 292 surgeons from 60 countries who completed an instrument measuring numeracy and evaluated the results of a randomized controlled trial including post-surgical side-effects. Half of the surgeons received this information in numbers. The other half received the information represented visually. Accuracy of risk estimation, reading latency, and estimate latency (i.e., deliberation) were assessed.

RESULTS

Some surgeons have low numeracy and could not correctly interpret surgical risks without additional support. Visual aids made risks transparent and eliminated differences in risk understanding between more and less numerate surgeons, increasing the amount of time that less numerate surgeons spent deliberating about risks.

CONCLUSIONS

Visual aids can be an efficient and inexpensive means of improving risk comprehension and clinical judgement in surgeons with low numerical and statistical skills.

PRACTICE IMPLICATIONS

Programs designed to help professionals represent and communicate health-relevant numerical information in simple transparent graphs may unobtrusively promote informed decision making.

Is patients' numeracy related to physical and mental health?

OBJECTIVE

There is compelling evidence showing that health literacy influences health outcomes. However, there is a dearth of research investigating this issue in the vast literature on numeracy-the ability to accurately interpret numerical information about risk, a skill that is only moderately correlated with health literacy. In a cross-sectional study, we investigated whether objective and subjective numeracy is related to objective and subjective health outcomes. Objective (subjective) numeracy is actual (self-reported) numerical competence. Objective outcomes include prevalence of comorbidity and prescribed medications. Subjective outcomes include perceptions of physical and mental health.

METHODS

A convenience sample of 502 male individuals receiving outpatient care at a Veterans Affairs Medical Center reported their demographics and answered a survey measuring objective and subjective numeracy, trust in physicians, satisfaction with role in medical decision making, perceptions of physical and mental health, and risky habits. We computed patients' body mass index (BMI) and their age-adjusted Charlson index-an extensively studied comorbidity index for predicting mortality in clinical research. We retrieved number of prescribed medications from medical records.

RESULTS

Compared with patients who had high objective numeracy, patients with low objective numeracy showed higher prevalence of comorbidities and took more prescribed medications. Compared with patients who had high subjective numeracy, patients with low subjective numeracy had more negative perceptions of their physical and mental health. These conclusions held after controlling for the effect of demographics, risky habits, BMI, trust in physicians, and satisfaction with role in decision making, suggesting that numeracy has a unique, significant contribution to health outcomes beyond the effect of these factors.

CONCLUSIONS

Our research documents for the first time that self-reported numeracy is related to perceptions of health, whereas objective numeracy is related to actual health, laying the groundwork for future research on the effect of numeracy on health outcomes.

Natural Frequencies Do Not Foster Public Understanding of Medical Test Results

Major organizations recommend presenting medical test results in terms of natural frequencies, rather than single-event probabilities. The evidence, however, is that natural frequency presentations benefit at most one-fifth of samples of health service users and patients. Only one study reported a substantial benefit of these presentations. Here, we replicate that study, testing online survey respondents. Study 1 attributed the previously reported benefit of natural frequencies to a scoring artifact. Study 2 showed that natural frequencies may elicit evaluations that conflict with the normatively correct one, potentially hindering informed decision making. Ironically, these evaluations occurred less often when respondents reasoned about single-event probabilities. These results suggest caution in promoting natural frequencies as the best way to communicate medical test data to health service users and patients.

Empirical relationships between numeracy and treatment decision making: A scoping review of the literature

OBJECTIVES

To determine what is known from the existing literature about empirical relationships between numeracy and the three stages of the treatment decision making process (information exchange, deliberation, and deciding on the treatment to implement). We also examined if, and how, numeracy was mentioned in relation to health literacy.

METHODS

A search of four databases returned 2772 records. After de-duplication and three levels of relevance screening, 38 primary studies were included in this review.

RESULTS

Relationships between numeracy and the information exchange stage have received greater attention than relationships between numeracy and the deliberation and deciding on the treatment to implement stages. The lack of overlap in the empirical relationships examined in studies, the measure(s) of numeracy used, and the characteristics of study populations, made findings difficult to compare. Multiple knowledge gaps and measurement-related problems were identified. Numeracy and health literacy have largely been treated as separate concepts.

CONCLUSION

More research is needed to better understand the importance of numeracy and health literacy to treatment decision making.

PRACTICE IMPLICATIONS

Decision aids designed for patients with different levels of health literacy may not meet the needs of patients with different levels of numeracy. The numeracy skills of health professionals require attention.

Improving Bayesian Reasoning: The Effects of Phrasing, Visualization, and Spatial Ability

Decades of research have repeatedly shown that people perform poorly at estimating and understanding conditional probabilities that are inherent in Bayesian reasoning problems. Yet in the medical domain, both physicians and patients make daily, life-critical judgments based on conditional probability. Although there have been a number of attempts to develop more effective ways to facilitate Bayesian reasoning, reports of these findings tend to be inconsistent and sometimes even contradictory. For instance, the reported accuracies for individuals being able to correctly estimate conditional probability range from 6% to 62%. In this work, we show that problem representation can significantly affect accuracies. By controlling the amount of information presented to the user, we demonstrate how text and visualization designs can increase overall accuracies to as high as 77%. Additionally, we found that for users with high spatial ability, our designs can further improve their accuracies to as high as 100%. By and large, our findings provide explanations for the inconsistent reports on accuracy in Bayesian reasoning tasks and show a significant improvement over existing methods. We believe that these findings can have immediate impact on risk communication in health-related fields.

Natural frequencies improve Bayesian reasoning in simple and complex inference tasks

Representing statistical information in terms of natural frequencies rather than probabilities improves performance in Bayesian inference tasks. This beneficial effect of natural frequencies has been demonstrated in a variety of applied domains such as medicine, law, and education. Yet all the research and applications so far have been limited to situations where one dichotomous cue is used to infer which of two hypotheses is true. Real-life applications, however, often involve situations where cues (e.g., medical tests) have more than one value, where more than two hypotheses (e.g., diseases) are considered, or where more than one cue is available. In Study 1, we show that natural frequencies, compared to information stated in terms of probabilities, consistently increase the proportion of Bayesian inferences made by medical students in four conditions-three cue values, three hypotheses, two cues, or three cues-by an average of 37 percentage points. In Study 2, we show that teaching natural frequencies for simple tasks with one dichotomous cue and two hypotheses leads to a transfer of learning to complex tasks with three cue values and two cues, with a proportion of 40 and 81% correct inferences, respectively. Thus, natural frequencies facilitate Bayesian reasoning in a much broader class of situations than previously thought.

A Novel Method for Real-Time Audio Recording With Intraoperative Video

OBJECTIVE

Although laparoscopic surgery has become widespread, effective and efficient education in laparoscopic surgery is difficult. Instructive laparoscopy videos with appropriate annotations are ideal for initial training in laparoscopic surgery; however, the method we use at our institution for creating laparoscopy videos with audio is not generalized, and there have been no detailed explanations of any such method. Our objectives were to demonstrate the feasibility of low-cost simple methods for recording surgical videos with audio and to perform a preliminary safety evaluation when obtaining these recordings during operations.

DESIGN

We devised a method for the synchronous recording of surgical video with real-time audio in which we connected an amplifier and a wireless microphone to an existing endoscopy system and its equipped video-recording device. We tested this system in 209 cases of laparoscopic surgery in operating rooms between August 2010 and July 2011 and prospectively investigated the results of the audiovisual recording method and examined intraoperative problems.

SETTING

Numazu City Hospital in Numazu city, Japan.

PARTICIPANTS

Surgeons, instrument nurses, and medical engineers.

RESULTS

In all cases, the synchronous input of audio and video was possible. The recording system did not cause any inconvenience to the surgeon, assistants, instrument nurse, sterilized equipment, or electrical medical equipment. Statistically significant differences were not observed between the audiovisual group and control group regarding the operating time, which had been divided into 2 slots-performed by the instructors or by trainees (p > 0.05).

CONCLUSIONS

This recording method is feasible and considerably safe while posing minimal difficulty in terms of technology, time, and expense. We recommend this method for both surgical trainees who wish to acquire surgical skills effectively and medical instructors who wish to teach surgical skills effectively.

Effects of visualizing statistical information - an empirical study on tree diagrams and 2 × 2 tables

In their research articles, scholars often use 2 × 2 tables or tree diagrams including natural frequencies in order to illustrate Bayesian reasoning situations to their peers. Interestingly, the effect of these visualizations on participants’ performance has not been tested empirically so far (apart from explicit training studies). In the present article, we report on an empirical study (3 × 2 × 2 design) in which we systematically vary visualization (no visualization vs. 2 × 2 table vs. tree diagram) and information format (probabilities vs. natural frequencies) for two contexts (medical vs. economical context; not a factor of interest). Each of N = 259 participants (students of age 16–18) had to solve two typical Bayesian reasoning tasks (“mammography problem” and “economics problem”). The hypothesis is that 2 × 2 tables and tree diagrams – especially when natural frequencies are included – can foster insight into the notoriously difficult structure of Bayesian reasoning situations. In contrast to many other visualizations (e.g., icon arrays, Euler diagrams), 2 × 2 tables and tree diagrams have the advantage that they can be constructed easily. The implications of our findings for teaching Bayesian reasoning will be discussed.

Toward an ecological analysis of Bayesian inferences: how task characteristics influence responses

In research on Bayesian inferences, the specific tasks, with their narratives and characteristics, are typically seen as exchangeable vehicles that merely transport the structure of the problem to research participants. In the present paper, we explore whether, and possibly how, task characteristics that are usually ignored influence participants’ responses in these tasks. We focus on both quantitative dimensions of the tasks, such as their base rates, hit rates, and false-alarm rates, as well as qualitative characteristics, such as whether the task involves a norm violation or not, whether the stakes are high or low, and whether the focus is on the individual case or on the numbers. Using a data set of 19 different tasks presented to 500 different participants who provided a total of 1,773 responses, we analyze these responses in two ways: first, on the level of the numerical estimates themselves, and second, on the level of various response strategies, Bayesian and non-Bayesian, that might have produced the estimates. We identified various contingencies, and most of the task characteristics had an influence on participants’ responses. Typically, this influence has been stronger when the numerical information in the tasks was presented in terms of probabilities or percentages, compared to natural frequencies – and this effect cannot be fully explained by a higher proportion of Bayesian responses when natural frequencies were used. One characteristic that did not seem to influence participants’ response strategy was the numerical value of the Bayesian solution itself. Our exploratory study is a first step toward an ecological analysis of Bayesian inferences, and highlights new avenues for future research.

How well do health professionals interpret diagnostic information? A systematic review

OBJECTIVE

To evaluate whether clinicians differ in how they evaluate and interpret diagnostic test information.

DESIGN

Systematic review.

DATA SOURCES

MEDLINE, EMBASE and PsycINFO from inception to September 2013; bibliographies of retrieved studies, experts and citation search of key included studies.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Primary studies that provided information on the accuracy of any diagnostic test (eg, sensitivity, specificity, likelihood ratios) to health professionals and that reported outcomes relating to their understanding of information on or implications of test accuracy.

RESULTS

We included 24 studies. 6 assessed ability to define accuracy metrics: health professionals were less likely to identify the correct definition of likelihood ratios than of sensitivity and specificity. -25 studies assessed Bayesian reasoning. Most assessed the influence of a positive test result on the probability of disease: they generally found health professionals' estimation of post-test probability to be poor, with a tendency to overestimation. 3 studies found that approaches based on likelihood ratios resulted in more accurate estimates of post-test probability than approaches based on estimates of sensitivity and specificity alone, while 3 found less accurate estimates. 5 studies found that presenting natural frequencies rather than probabilities improved post-test probability estimation and speed of calculations.

CONCLUSIONS

Commonly used measures of test accuracy are poorly understood by health professionals. Reporting test accuracy using natural frequencies and visual aids may facilitate improved understanding and better estimation of the post-test probability of disease.

Comprehension and computation in Bayesian problem solving

Humans have long been characterized as poor probabilistic reasoners when presented with explicit numerical information. Bayesian word problems provide a well-known example of this, where even highly educated and cognitively skilled individuals fail to adhere to mathematical norms. It is widely agreed that natural frequencies can facilitate Bayesian inferences relative to normalized formats (e.g., probabilities, percentages), both by clarifying logical set-subset relations and by simplifying numerical calculations. Nevertheless, between-study performance on “transparent” Bayesian problems varies widely, and generally remains rather unimpressive. We suggest there has been an over-focus on this representational facilitator (i.e., transparent problem structures) at the expense of the specific logical and numerical processing requirements and the corresponding individual abilities and skills necessary for providing Bayesian-like output given specific verbal and numerical input. We further suggest that understanding this task-individual pair could benefit from considerations from the literature on mathematical cognition, which emphasizes text comprehension and problem solving, along with contributions of online executive working memory, metacognitive regulation, and relevant stored knowledge and skills. We conclude by offering avenues for future research aimed at identifying the stages in problem solving at which correct vs. incorrect reasoners depart, and how individual differences might influence this time point.

Visual aids improve diagnostic inferences and metacognitive judgment calibration

Visual aids can improve comprehension of risks associated with medical treatments, screenings, and lifestyles. Do visual aids also help decision makers accurately assess their risk comprehension? That is, do visual aids help them become well calibrated? To address these questions, we investigated the benefits of visual aids displaying numerical information and measured accuracy of self-assessment of diagnostic inferences (i.e., metacognitive judgment calibration) controlling for individual differences in numeracy. Participants included 108 patients who made diagnostic inferences about three medical tests on the basis of information about the sensitivity and false-positive rate of the tests and disease prevalence. Half of the patients received the information in numbers without a visual aid, while the other half received numbers along with a grid representing the numerical information. In the numerical condition, many patients–especially those with low numeracy–misinterpreted the predictive value of the tests and profoundly overestimated the accuracy of their inferences. Metacognitive judgment calibration mediated the relationship between numeracy and accuracy of diagnostic inferences. In contrast, in the visual aid condition, patients at all levels of numeracy showed high-levels of inferential accuracy and metacognitive judgment calibration. Results indicate that accurate metacognitive assessment may explain the beneficial effects of visual aids and numeracy–a result that accords with theory suggesting that metacognition is an essential part of risk literacy. We conclude that well-designed risk communications can inform patients about healthrelevant numerical information while helping them assess the quality of their own risk comprehension.

Presentation of Diagnostic Information to Doctors May Change Their Interpretation and Clinical Management: A Web-Based Randomised Controlled Trial

BACKGROUND

There is little evidence on how best to present diagnostic information to doctors and whether this makes any difference to clinical management. We undertook a randomised controlled trial to see if different data presentations altered clinicians' decision to further investigate or treat a patient with a fictitious disorder ("Green syndrome") and their ability to determine post-test probability.

METHODS

We recruited doctors registered with the United Kingdom's largest online network for medical doctors between 10 July and 6" November 2012. Participants were randomised to one of four arms: (a) text summary of sensitivity and specificity, (b) Fagan's nomogram, (c) probability-modifying plot (PMP), (d) natural frequency tree (NFT). The main outcome measure was the decision whether to treat, not treat or undertake a brain biopsy on the hypothetical patient and the correct post-test probability. Secondary outcome measures included knowledge of diagnostic tests.

RESULTS

917 participants attempted the survey and complete data were available from 874 (95.3%). Doctors randomized to the PMP and NFT arms were more likely to treat the patient than those randomized to the text-only arm. (ORs 1.49, 95% CI 1.02, 2.16) and 1.43, 95% CI 0.98, 2.08 respectively). More patients randomized to the PMP (87/218-39.9%) and NFT (73/207-35.3%) arms than the nomogram (50/194-25.8%) or text only (30/255-11.8%) arms reported the correct post-test probability (p <0.001). Younger age, postgraduate training and higher self-rated confidence all predicted better knowledge performance. Doctors with better knowledge were more likely to view an optional learning tutorial (OR per correct answer 1.18, 95% CI 1.06, 1.31).

CONCLUSIONS

Presenting diagnostic data using a probability-modifying plot or natural frequency tree influences the threshold for treatment and improves interpretation of tests results compared to text summary of sensitivity and specificity or Fagan's nomogram.

Natural frequencies facilitate diagnostic inferences of managers

In Bayesian inference tasks, information about base rates as well as hit rate and false-alarm rate needs to be integrated according to Bayes' rule after the result of a diagnostic test became known. Numerous studies have found that presenting information in a Bayesian inference task in terms of natural frequencies leads to better performance compared to variants with information presented in terms of probabilities or percentages. Natural frequencies are the tallies in a natural sample in which hit rate and false-alarm rate are not normalized with respect to base rates. The present research replicates the beneficial effect of natural frequencies with four tasks from the domain of management, and with management students as well as experienced executives as participants. The percentage of Bayesian responses was almost twice as high when information was presented in natural frequencies compared to a presentation in terms of percentages. In contrast to most tasks previously studied, the majority of numerical responses were lower than the Bayesian solutions. Having heard of Bayes' rule prior to the study did not affect Bayesian performance. An implication of our work is that textbooks explaining Bayes' rule should teach how to represent information in terms of natural frequencies instead of how to plug probabilities or percentages into a formula.

Instruction in information structuring improves Bayesian judgment in intelligence analysts

An experiment was conducted to test the effectiveness of brief instruction in information structuring (i.e., representing and integrating information) for improving the coherence of probability judgments and binary choices among intelligence analysts. Forty-three analysts were presented with comparable sets of Bayesian judgment problems before and immediately after instruction. After instruction, analysts' probability judgments were more coherent (i.e., more additive and compliant with Bayes theorem). Instruction also improved the coherence of binary choices regarding category membership: after instruction, subjects were more likely to invariably choose the category to which they assigned the higher probability of a target's membership. The research provides a rare example of evidence-based validation of effectiveness in instruction to improve the statistical assessment skills of intelligence analysts. Such instruction could also be used to improve the assessment quality of other types of experts who are required to integrate statistical information or make probabilistic assessments.

Good fences make for good neighbors but bad science: a review of what improves Bayesian reasoning and why

Bayesian reasoning, defined here as the updating of a posterior probability following new information, has historically been problematic for humans. Classic psychology experiments have tested human Bayesian reasoning through the use of word problems and have evaluated each participant’s performance against the normatively correct answer provided by Bayes’ theorem. The standard finding is of generally poor performance. Over the past two decades, though, progress has been made on how to improve Bayesian reasoning. Most notably, research has demonstrated that the use of frequencies in a natural sampling framework—as opposed to single-event probabilities—can improve participants’ Bayesian estimates. Furthermore, pictorial aids and certain individual difference factors also can play significant roles in Bayesian reasoning success. The mechanics of how to build tasks which show these improvements is not under much debate. The explanations for why naturally sampled frequencies and pictures help Bayesian reasoning remain hotly contested, however, with many researchers falling into ingrained “camps” organized around two dominant theoretical perspectives. The present paper evaluates the merits of these theoretical perspectives, including the weight of empirical evidence, theoretical coherence, and predictive power. By these criteria, the ecological rationality approach is clearly better than the heuristics and biases view. Progress in the study of Bayesian reasoning will depend on continued research that honestly, vigorously, and consistently engages across these different theoretical accounts rather than staying “siloed” within one particular perspective. The process of science requires an understanding of competing points of view, with the ultimate goal being integration.

Counseling women with a previous cesarean birth: toward a shared decision-making partnership

Pregnant women who had a previous cesarean birth must choose whether to have a repeat cesarean or to attempt a vaginal birth. Many of these women are candidates for a trial of labor. Current practice guidelines recommend that women should be thoroughly counseled during prenatal care about the benefits and harms of both a trial of labor after cesarean (TOLAC) and an elective repeat cesarean delivery and be offered the opportunity to make an informed decision about mode of birth in collaboration with their provider. The purpose of this article is to improve the process of counseling, decision making, and informed consent by increasing health care providers' knowledge about the essential elements of shared decision making. Factors that affect the decisions to be made and concepts that are critical for effective counseling are explored, including clinical considerations, women's perspectives, decision-making models, health literacy and numeracy, communicating risk, and the use of decision aids. Issues related to birth sites for TOLAC are also discussed, including access, safety, refusal of surgery, and clinical management.

Communicating Health Risks With Visual Aids

Informed decision making requires that people understand health risks. Unfortunately, many people are not risk literate and are biased by common risk communication practices. In this article, we review a collection of studies investigating the benefits of visual aids for communicating health risks to diverse vulnerable people (e.g., varying in abilities, ages, risk characteristics, and cultural backgrounds). These studies show that appropriately designed visual aids are often highly effective, transparent, and ethically desirable tools for improving decision making, changing attitudes, and reducing risky behavior. Theoretical mechanisms, open questions, and emerging applications are discussed.

Simple but powerful health messages for increasing condom use in young adults

In a large longitudinal study involving young adults, we conducted an eight-hour STD educational intervention and examined the impact of the intervention on the efficacy of a message for promoting condom use. The message was framed in positive or negative terms and was presented visually or in numbers (percentages or frequencies). Results indicated that the numerical positive-framed message increased condom use among young adults who did not receive the intervention, whereas the numerical negative-framed message did not. Attitudes toward condom use along with changes in intentions to use condoms mediated this framing effect. In contrast, the positive-framed and negative-framed messages were equally and highly effective for promoting condom use when the messages were presented visually or when young adults received the STD educational intervention before reading the message, suggesting that the simple brochures featuring visual aids were as effective in changing attitudes and behavioral intentions as the extensive intervention. These findings add to a growing body of evidence detailing the mechanisms that allow well-constructed visual aids to be among the most effective, transparent, memorable, and ethically desirable means of risk communication. Clinical and public health implications are discussed.