Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study

BACKGROUND

Rare disease diagnosis is often delayed by years. A primary factor for this delay is a lack of knowledge and awareness regarding rare diseases. Probabilistic diagnostic decision support systems (DDSSs) have the potential to accelerate rare disease diagnosis by suggesting differential diagnoses for physicians based on case input and incorporated medical knowledge. We examine the DDSS prototype Ada DX and assess its potential to provide accurate rare disease suggestions early in the course of rare disease cases.

RESULTS

Ada DX suggested the correct disease earlier than the time of clinical diagnosis among the top five fit disease suggestions in 53.8% of cases (50 of 93), and as the top fit disease suggestion in 37.6% of cases (35 of 93). The median advantage of correct disease suggestions compared to the time of clinical diagnosis was 3 months or 50% for top five fit and 1 month or 21% for top fit. The correct diagnosis was suggested at the first documented patient visit in 33.3% (top 5 fit), and 16.1% of cases (top fit), respectively. Wilcoxon signed-rank test shows a significant difference between the time to clinical diagnosis and the time to correct disease suggestion for both top five fit and top fit (z-score -6.68, respective -5.71, α=0.05, p-value <0.001).

CONCLUSION

Ada DX provided accurate rare disease suggestions in most rare disease cases. In many cases, Ada DX provided correct rare disease suggestions early in the course of the disease, sometimes at the very beginning of a patient journey. The interpretation of these results indicates that Ada DX has the potential to suggest rare diseases to physicians early in the course of a case. Limitations of this study derive from its retrospective and unblinded design, data input by a single user, and the optimization of the knowledge base during the course of the study. Results pertaining to the system's accuracy should be interpreted cautiously. Whether the use of Ada DX reduces the time to diagnosis in rare diseases in a clinical setting should be validated in prospective studies.

The anchoring bias reflects rational use of cognitive resources

Cognitive biases, such as the anchoring bias, pose a serious challenge to rational accounts of human cognition. We investigate whether rational theories can meet this challenge by taking into account the mind's bounded cognitive resources. We asked what reasoning under uncertainty would look like if people made rational use of their finite time and limited cognitive resources. To answer this question, we applied a mathematical theory of bounded rationality to the problem of numerical estimation. Our analysis led to a rational process model that can be interpreted in terms of anchoring-and-adjustment. This model provided a unifying explanation for ten anchoring phenomena including the differential effect of accuracy motivation on the bias towards provided versus self-generated anchors. Our results illustrate the potential of resource-rational analysis to provide formal theories that can unify a wide range of empirical results and reconcile the impressive capacities of the human mind with its apparently irrational cognitive biases.

Reduced activation in ventral striatum and ventral tegmental area during probabilistic decision-making in schizophrenia

Patients with schizophrenia suffer from deficits in monitoring and controlling their own thoughts. Within these so-called metacognitive impairments, alterations in probabilistic reasoning might be one cognitive phenomenon disposing to delusions. However, so far little is known about alterations in associated brain functionality. A previously established task for functional magnetic resonance imaging (fMRI), which requires a probabilistic decision after a variable amount of stimuli, was applied to 23 schizophrenia patients and 28 healthy controls matched for age, gender and educational levels. We compared activation patterns during decision-making under conditions of certainty versus uncertainty and evaluated the process of final decision-making in ventral striatum (VS) and ventral tegmental area (VTA). We replicated a pre-described extended cortical activation pattern during probabilistic reasoning. During final decision-making, activations in several fronto- and parietocortical areas, as well as in VS and VTA became apparent. In both of these regions schizophrenia patients showed a significantly reduced activation. These results further define the network underlying probabilistic decision-making. The observed hypo-activation in regions commonly associated with dopaminergic neurotransmission fits into current concepts of disrupted prediction error signaling in schizophrenia and suggests functional links to reward anticipation. Forthcoming studies with patients at risk for psychosis and drug-naive first episode patients are necessary to elucidate the development of these findings over time and the interplay with associated clinical symptoms.

Intuitive statistical inferences in chimpanzees and humans follow Weber's law

Humans and nonhuman great apes share a sense for intuitive statistical reasoning, making intuitive probability judgments based on proportional information. This ability is of fundamental importance, in particular for inferring general regularities from finite numbers of observations and, vice versa, for predicting the outcome of single events using prior information. To date it remains unclear which cognitive mechanism underlies and enables this capacity. The aim of the present study was to gain deeper insights into the cognitive structure of intuitive statistics by probing its signatures in chimpanzees and humans. We tested 24 sanctuary-living chimpanzees in a previously established paradigm which required them to reason from populations of food items with different ratios of preferred (peanuts) and non-preferred items (carrot pieces) to randomly drawn samples. In a series of eight test conditions, the ratio between the two ratios to be discriminated (ROR) was systematically varied ranging from 1 (same proportions in both populations) to 16 (high magnitude of difference between populations). One hundred and forty-four human adults were tested in a computerized version of the same task. The main result was that both chimpanzee and human performance varied as a function of the log(ROR) and thus followed Weber's law. This suggests that intuitive statistical reasoning relies on the same cognitive mechanism that is used for comparing absolute quantities, namely the analogue magnitude system.

A description-experience gap in statistical intuitions: Of smart babies, risk-savvy chimps, intuitive statisticians, and stupid grown-ups

Comparison of different lines of research on statistical intuitions and probabilistic reasoning reveals several puzzling contradictions. Whereas babies seem to be intuitive statisticians, surprisingly capable of statistical learning and inference, adults' statistical inferences have been found to be inconsistent with the rules of probability theory and statistics. Whereas researchers in the 1960s concluded that people's probability updating is "conservatively" proportional to normative predictions, probability updating research in the 1970s suggested that people are incapable of following Bayes's rule. And whereas animals appear to be strikingly risk savvy, humans often seem "irrational" when dealing with probabilistic information. Drawing on research on the description-experience gap in risky choice, we integrate and systematize these findings from disparate fields of inquiry that have, to date, operated largely in parallel. Our synthesis shows that a key factor in understanding inconsistencies in statistical intuitions research is whether probabilistic inferences are based on symbolic, abstract descriptions or on the direct experience of statistical information. We delineate this view from other conceptual accounts, consider potential mechanisms by which attributes of first-hand experience can facilitate appropriate statistical inference, and identify conditions under which they improve or impair probabilistic reasoning. To capture the full scope of human statistical intuition, we conclude, research on probabilistic reasoning across the lifespan, across species, and across research traditions must bear in mind that experience and symbolic description of the world may engage systematically distinct cognitive processes.

Interactions between inferential strategies and belief bias

The dual strategy model of reasoning proposed by Verschueren, Schaeken, and d'Ydewalle (Thinking & Reasoning, 11(3), 239-278, 2005a; Memory & Cognition, 33(1), 107-119, 2005b) suggests that people can use either a statistical or a counterexample-based strategy to make deductive inferences. Subsequent studies have supported this distinction and investigated some properties of the two strategies. In the following, we examine the further hypothesis that reasoners using statistical strategies should be more vulnerable to the effects of conclusion belief. In each of three studies, participants were given abstract problems used to determine strategy use and three different forms of syllogism with believable and unbelievable conclusions. Responses, response times, and feeling of rightness (FOR) measures were taken. The results show that participants using a statistical strategy were more prone to the effects of conclusion belief across all three forms of reasoning. In addition, statistical reasoners took less time to make inferences than did counterexample reasoners. Patterns of variation in response times and FOR ratings between believable and unbelievable conclusions were very similar for both strategies, indicating that both statistical and counterexample reasoners were aware of conflict between conclusion belief and premise-based reasoning.

Paranormal belief, thinking style preference and susceptibility to confirmatory conjunction errors

This study examines the extent to which belief in extrasensory perception (ESP), psychokinesis (PK) or life after death (LAD), plus need for cognition (NFC) and faith in intuition (FI), predict the generation of confirmatory conjunction errors. An opportunity sample (n = 261) completed sixteen conjunction problems manipulated across a 2 event type (paranormal vs. non-paranormal) × 2 outcome type (confirmatory vs. disconfirmatory) within subjects design. Three Generalised Linear Mixed Models - one per paranormal belief type - were performed. With respondent gender and age controlled for, ESP, PK and LAD beliefs were all associated with the making (vs. non-making) of conjunction errors both generally and specifically for confirmatory conjunctive outcomes. Event type had no impact. Individuals high in NFC were less likely to commit the fallacy. The role thinking style plays in shaping paranormal believers' susceptibility to confirmatory conjunction biases is discussed. Methodological issues and future research ideas are also considered.

The development of the representativeness heuristic in young children

In classic examinations of the representativeness heuristic, Kahneman and Tversky (1973) presented adult participants with a description of an individual who fit their stereotype of a typical engineer. Importantly, even when participants were told that the individual was drawn from a sample of 70 lawyers and 30 engineers, they estimated that the individual was an engineer at very high levels, showing that they relied almost exclusively on the personality description. Relying on the representativeness heuristic can lead to base-rate neglect and, thus, biased judgments. Two experiments provide insight into the development of the representativeness heuristic in young children using an adaptation of the classic lawyer-engineer problem. Experiment 1 (N = 96) established that 3- to 5-year-olds can use base-rate information on its own, and 4- and 5-year-olds can use individuating information on its own, to make inferences. Experiment 2 (N = 192) varied the relevance of the individuating information across conditions to assess the pervasiveness of this bias early in development. Here 5- and 6-year-olds, much like adults, continue to attempt to rely on individuating information when making classifications even if that information is irrelevant. Together, these experiments reveal how the representativeness heuristic develops across the preschool years and suggest that the bias may strengthen between 4 and 6 years of age.

Glutamatergic Contribution to Probabilistic Reasoning and Jumping to Conclusions in Schizophrenia: A Double-Blind, Randomized Experimental Trial

BACKGROUND

Impaired probabilistic reasoning and the jumping-to-conclusions reasoning bias are hallmark features of schizophrenia (SCZ), yet the neuropharmacological basis of these deficits remains unclear. Here we tested the hypothesis that glutamatergic neurotransmission specifically contributes to jumping to conclusions and impaired probabilistic reasoning in SCZ.

METHODS

A total of 192 healthy participants received either NMDA receptor agonists/antagonists (D-cycloserine/dextromethorphan), dopamine type 2 receptor agonists/antagonists (bromocriptine/haloperidol), or placebo in a randomized, double-blind, between-subjects design. In addition, we tested 32 healthy control participants matched to 32 psychotic inpatients with SCZ-a state associated with compromised probabilistic reasoning due to reduced glutamatergic neurotransmission. All experiments employed two versions of a probabilistic reasoning (beads) task, which required participants to either sample individual amounts of sensory information to infer correct decisions or provide explicit probability estimates for presented sensory information. Our task instantiations assessed both information sampling and explicit probability estimates in different probabilistic contexts (easy vs. difficult conditions) and changing sensory information through random transitions among easy, difficult, and ambiguous trial types.

RESULTS

Following administration of D-cycloserine, haloperidol, and bromocriptine, healthy participants displayed data-gathering behavior that was normal compared with placebo and was adequate in the context of all employed task conditions and trial level difficulties. However, healthy participants receiving dextromethorphan displayed a jumping-to-conclusions bias, abnormally increased probability estimates, and overweighting of sensory information. These effects were mirrored in patients with SCZ performing the same versions of the beads task.

CONCLUSIONS

Our findings provide novel neuropharmacological evidence linking reduced glutamatergic neurotransmission to impaired information sampling and to disrupted probabilistic reasoning, namely to overweighting of sensory evidence, in patients with SCZ.

Prefrontal activity measured by functional near infrared spectroscopy during probabilistic inference in subjects with persecutory delusions

Jumping to conclusions (JTC) is a probabilistic reasoning bias and is thought to contribute to delusion formation. Neurobiological correlates of the JTC bias are not known. We aimed to examine the rostral prefrontal cortex (rPFC) activity with functional near ınfrared spectroscopy during a modified version of the Beads in a Jar Task (BIJT) in subjects with persecutory delusions (N=25). In BIJT participants are presented beads either drawn from one of the two jars with opposite probability ratios (PRs) of colored beads and are required to decide from which jar beads are being drawn. We administered the BIJT with 90/10 and 55/45 PRs. Compared to healthy controls (N=20), patients reached a decision earlier in both conditions. While the medial rPFC regions were more active in the 90/10 condition in controls compared to patients, lateral rPFC activation was higher in the 55/45 condition in patients than controls. Only in the control group, there was a marked decline in the lateral rPFC activation in the 55/45 condition compared to the 90/10 condition. The activity in the lateral rPFC was negatively correlated with the amount of beads drawn in healthy controls but not in subjects with persecutory delusions. Our results suggest that during the BIJT, rPFC does not function as a single unit and rather consists of functional subunits that are organized differently in patients and controls. The failure to deactivate the lateral rPFC may be associated with earlier decisions in subjects with persecutory delusions.

Integration of probabilistic regulatory networks into constraint-based models of metabolism with applications to Alzheimer's disease

Background

Mathematical models of biological networks can provide important predictions and insights into complex disease. Constraint-based models of cellular metabolism and probabilistic models of gene regulatory networks are two distinct areas that have progressed rapidly in parallel over the past decade. In principle, gene regulatory networks and metabolic networks underly the same complex phenotypes and diseases. However, systematic integration of these two model systems remains a fundamental challenge.

Results

In this work, we address this challenge by fusing probabilistic models of gene regulatory networks into constraint-based models of metabolism. The novel approach utilizes probabilistic reasoning in BN models of regulatory networks serves as the “glue” that enables a natural interface between the two systems. Probabilistic reasoning is used to predict and quantify system-wide effects of perturbation to the regulatory network in the form of constraints for flux variability analysis. In this setting, both regulatory and metabolic networks inherently account for uncertainty. Applications leverage constraint-based metabolic models of brain metabolism and gene regulatory networks parameterized by gene expression data from the hippocampus to investigate the role of the HIF-1 pathway in Alzheimer’s disease. Integrated models support HIF-1A as effective target to reduce the effects of hypoxia in Alzheimer’s disease. However, HIF-1A activation is far less effective in shifting metabolism when compared to brain metabolism in healthy controls.

Conclusions

The direct integration of probabilistic regulatory networks into constraint-based models of metabolism provides novel insights into how perturbations in the regulatory network may influence metabolic states. Predictive modeling of enzymatic activity can be facilitated using probabilistic reasoning, thereby extending the predictive capacity of the network. This framework for model integration is generalizable to other systems.

Electronic supplementary material

The online version of this article (10.1186/s12859-019-2872-8) contains supplementary material, which is available to authorized users.

Additional evidence for a dual-strategy model of reasoning: Probabilistic reasoning is more invariant than reasoning about logical validity

One of the major debates concerning the nature of inferential reasoning is between counterexample-based strategies such as mental model theory and the statistical strategies underlying probabilistic models. The dual-strategy model proposed by Verschueren, Schaeken, and d'Ydewalle (2005a, 2005b) suggests that people might have access to both kinds of strategies. One of the postulates of this approach is that statistical strategies correspond to low-cost, intuitive modes of evaluation, whereas counterexample strategies are higher-cost and more variable in use. We examined this hypothesis by using a deductive-updating paradigm. The results of Study 1 showed that individual differences in strategy use predict different levels of deductive updating on inferences about logical validity. Study 2 demonstrated no such variation when explicitly probabilistic inferences were examined. Study 3 showed that presenting updating problems with probabilistic inferences modified performance on subsequent problems using logical validity, whereas the opposite was not true. These results provide clear evidence that the processes used to make probabilistic inferences are less subject to variation than those used to make inferences of logical validity.

Another chance for good reasoning

Disagreement on the "probability status" of chances casts doubt on Girotto and Gonzalez's (2001) conclusion that the human mind can make sound Bayesian inferences involving single-event probabilities. The main objection raised has been that chances are de facto natural frequencies disguised as probabilities. In the present study, we empirically demonstrated that numbers of chances are perceived as being distinct from natural frequencies and that they have a facilitatory effect on Bayesian inference tasks that is completely independent from their (minor) frequentist readings. Overall, therefore, our results strongly disconfirm the hypothesis that natural frequencies are a privileged cognitive representational format for Bayesian inferences and suggest that a significant portion of laypeople adequately handle genuine single-event probability problems once these are rendered computationally more accessible by using numbers of chances.

Judging the Probability of Hypotheses Versus the Impact of Evidence: Which Form of Inductive Inference Is More Accurate and Time-Consistent?

Inductive reasoning requires exploiting links between evidence and hypotheses. This can be done focusing either on the posterior probability of the hypothesis when updated on the new evidence or on the impact of the new evidence on the credibility of the hypothesis. But are these two cognitive representations equally reliable? This study investigates this question by comparing probability and impact judgments on the same experimental materials. The results indicate that impact judgments are more consistent in time and more accurate than probability judgments. Impact judgments also predict the direction of errors in probability judgments. These findings suggest that human inductive reasoning relies more on estimating evidential impact than on posterior probability.

The role of effective connectivity between the task-positive and task-negative network for evidence gathering [Evidence gathering and connectivity]

Reports linking a 'jumping-to-conclusions' bias to delusions have led to growing interest in the neurobiological correlates of probabilistic reasoning. Several brain areas have been implicated in probabilistic reasoning; however, findings are difficult to integrate into a coherent account. The present study aimed to provide additional evidence by investigating, for the first time, effective connectivity among brain areas involved in different stages of evidence gathering. We investigated evidence gathering in 25 healthy individuals using fMRI and a new paradigm (Box Task) designed such as to minimize the effects of cognitive effort and reward processing. Decisions to collect more evidence ('draws') were contrasted to decisions to reach a final choice ('conclusions') with respect to BOLD activity. Psychophysiological interaction analysis was used to investigate effective connectivity. Conclusion events were associated with extensive brain activations in widely distributed brain areas associated with the task-positive network. In contrast, draw events were characterized by higher activation in areas assumed to be part of the task-negative network. Effective connectivity between the two networks decreased during draws and increased during conclusion events. Our findings indicate that probabilistic reasoning may depend on the balance between the task-positive and task-negative network, and that shifts in connectivity between the two may be crucial for evidence gathering. Thus, abnormal connectivity between the two systems may significantly contribute to the jumping-to-conclusions bias.

Dependencies in evidential reports: The case for informational advantages

Whether assessing the accuracy of expert forecasting, the pros and cons of group communication, or the value of evidence in diagnostic or predictive reasoning, dependencies between experts, group members, or evidence have traditionally been seen as a form of redundancy. We demonstrate that this conception of dependence conflates the structure of a dependency network, and the observations across this network. By disentangling these two elements we show, via mathematical proof and specific examples, that there are cases where dependencies yield an informational advantage over independence. More precisely, when a structural dependency exists, but observations are either partial or contradicting, these observations provide more support to a hypothesis than when this structural dependency does not exist, ceteris paribus. Furthermore, we show that lay reasoners endorse sufficient assumptions underpinning these advantageous structures yet fail to appreciate their implications for probability judgments and belief revision.

Functional brain networks underlying probabilistic reasoning and delusions in schizophrenia

Delusions in schizophrenia are false beliefs that are assigned certainty and not afforded the scrutiny that normally gives rise to doubt, even under conditions of weak evidence. The goal of the current functional magnetic resonance imaging (fMRI) study is to identify the brain network(s) involved in gathering information under conditions of weak evidence, in people with schizophrenia experiencing delusions. fMRI activity during probabilistic reasoning in people with schizophrenia experiencing delusions (n = 29) compared to people with schizophrenia not experiencing delusions (n = 41) and healthy controls (n = 41) was observed when participants made judgments based on evidence that weakly or strongly matched (or mismatched) with the focal hypothesis. A brain network involved in visual attention was strongly elicited for conditions of weak evidence for healthy controls and patients not experiencing delusions, but this increase was absent for patients experiencing delusions. This suggests that the state associated with delusions manifests in fMRI as reduced activity in an early visual attentional process whereby weak evidence is incorrectly stamped as conclusive, manifestating as a feeling of fluency and misplaced certainty, short-circuiting the search for evidence, and providing a candidate neural process for 'seeding' delusions.

Categorical Perception of p-Values

Traditional statistics instruction emphasizes a .05 significance level for hypothesis tests. Here, we investigate the consequences of this training for researchers' mental representations of probabilities - whether .05 becomes a boundary, that is, a discontinuity of the mental number line, and alters their reasoning about p-values. Graduate students with statistical training (n = 25) viewed pairs of p-values and judged whether they were "similar" or "different." After controlling for several covariates, participants were more likely and faster to judge p-values as "different" when they crossed the .05 boundary (e.g., .046 vs. .052) compared to when they did not (e.g., .026 vs. .032). This result suggests a categorical perception-like effect for the processing of p-values. It may be a consequence of traditional statistical instruction creating a psychologically real divide between so-called statistical "significance" and "nonsignificance." Such a distortion is undesirable given modern approaches to statistical reasoning that de-emphasize dichotomizing the p-value continuum.

Strategies for selecting and evaluating information

Within the domain of psychology, Optimal Experimental Design (OED) principles have been used to model how people seek and evaluate information. Despite proving valuable as computational-level methods to account for people's behaviour, their descriptive and explanatory powers remain largely unexplored. In a series of experiments, we used a naturalistic crime investigation scenario to examine how people evaluate queries, as well as outcomes, in probabilistic contexts. We aimed to uncover the psychological strategies that people use, not just to assess whether they deviated from OED principles. In addition, we explored the adaptiveness of the identified strategies across both one-shot and stepwise information search tasks. We found that people do not always evaluate queries strictly in OED terms and use distinct strategies, such as by identifying a leading contender at the outset. Moreover, we identified aspects of zero-sum thinking and risk aversion that interact with people's information search strategies. Our findings have implications for building a descriptive account of information seeking and evaluation, accounting for factors that currently lie outside the realm of information-theoretic OED measures, such as context and the learner's own preferences.

An investigation into reasoning biases, mood and cognitive state, and subclinical delusional ideation

Following research on reasoning and the continuum of delusional ideation, the present study attempted to investigate the impact of different experimentally-induced states (stress, paranoia, and neutral) on the jumping-to-conclusions reasoning bias in individuals with varying levels of subclinical delusional ideation (SDI). Participants (N=117) completed a measure of subclinical delusional ideation (the Peters et al. Delusions Inventory or PDI; Peters et al., 1999); and were randomly assigned to receive one of two experimental inductions (stress or paranoia), or no experimental induction; their performance on two probabilistic reasoning tasks--one easy and one challenging--was assessed. Although no differences were found between individuals with high vs. low subclinical delusional ideation in the no induction condition or following the paranoia induction, in the stress-induction condition, individuals with high levels of subclinical delusional ideation were significantly less likely to jump to conclusions on the easy reasoning task. No significant effects emerged on the more challenging task. Assessment of post-test paranoid thinking indicated our paranoia induction did not have its intended effect. Importantly, because there was no pre-test of anxiety, paranoid thinking, or reasoning to determine if they shifted after the inductions, results need to be interpreted with caution.

Can conditionals explain explanations? A modus ponens model of B because A

We suggest a normative model for the evaluation of explanations B because A based on probabilistic conditional reasoning and compare it with empirical data. According to the modus ponens model of explanations, the probability of B because A should equal the joint probability of the conditional if A then B and the explanans A. We argue that B because A expresses the conjunction of A and B as well as positive relevance of A for B. In Study 1, participants (N = 80) judged the subjective probabilities of 20 sets of statements with a focus on belief-based reasoning under uncertainty. In Study 2, participants (N = 376) were assigned to one of six item sets for which we varied the inferential relevance of A for B to explore boundary conditions of our model. We assessed the performance of our model across a range of analyses and report results on the Equation, a fundamental model in research on probabilistic reasoning concerning the evaluation of conditionals. In both studies, results indicate that participants' belief in statements B because A followed model predictions systematically. However, a sizeable proportion of sets of beliefs contained at least one incoherence, indicating deviations from the norms of rationality suggested by our model. In addition, results of Study 2 lend support to the idea that inferential relevance may be relevant for the evaluation of both conditionals and explanations.

Cognitive heterogeneity and complex belief elicitation

The Stochastic Becker-DeGroot-Marschak (SBDM) mechanism is a theoretically elegant way of eliciting incentive-compatible beliefs under a variety of risk preferences. However, the mechanism is complex and there is concern that some participants may misunderstand its incentive properties. We use a two-part design to evaluate the relationship between participants' probabilistic reasoning skills, task complexity, and belief elicitation. We first identify participants whose decision-making is consistent and inconsistent with probabilistic reasoning using a task in which non-Bayesian modes of decision-making lead to violations of stochastic dominance. We then elicit participants' beliefs in both easy and hard decision problems. Relative to Introspection, there is less variation in belief errors between easy and hard problems in the SBDM mechanism. However, there is a greater difference in belief errors between consistent and inconsistent participants. These results suggest that while the SBDM mechanism encourages individuals to think more carefully about beliefs, it is more sensitive to heterogeneity in probabilistic reasoning. In a follow-up experiment, we also identify participants with high and low fluid intelligence with a Raven task, and high and low proclivities for cognitive effort using an extended Cognitive Reflection Test. Although performance on these tasks strongly predict errors in both the SBDM mechanism and Introspection, there is no significant interaction effect between the elicitation mechanism and either ability or effort. Our results suggest that mechanism complexity is an important consideration when using elicitation mechanisms, and that participants' probabilistic reasoning is an important consideration when interpreting elicited beliefs.

A probabilistic approach to incorporating domain knowledge for closed-room people monitoring

We propose a novel probabilistic approach to recognizing people entering and leaving a closed room in human work place or living environment. Specifically, people in the view of a monitoring camera are first tracked and represented using low-level color features. Based on a new color similarity measure, optimal recognition of people leaving and entering the room is carried out by probabilistic reasoning under the constraints imposed by the domain knowledge, e.g., a person currently inside a room cannot enter again without first leaving it, and vice versa. The novelty of our work mainly lies in the development of a systematic way to incorporate the correlation and constraint among a sequence of people observations, and the optimality of recognition is achieved by maximizing a joint posterior probability of the observations. Experimental results of real and synthetic data are presented to show the efficacy of the proposed approach.

The Effect of Evidential Impact on Perceptual Probabilistic Judgments

In a series of three behavioral experiments, we found a systematic distortion of probability judgments concerning elementary visual stimuli. Participants were briefly shown a set of figures that had two features (e.g., a geometric shape and a color) with two possible values each (e.g., triangle or circle and black or white). A figure was then drawn, and participants were informed about the value of one of its features (e.g., that the figure was a "circle") and had to predict the value of the other feature (e.g., whether the figure was "black" or "white"). We repeated this procedure for various sets of figures and, by varying the statistical association between features in the sets, we manipulated the probability of a feature given the evidence of another (e.g., the posterior probability of hypothesis "black" given the evidence "circle") as well as the support provided by a feature to another (e.g., the impact, or confirmation, of evidence "circle" on the hypothesis "black"). Results indicated that participants' judgments were deeply affected by impact, although they only should have depended on the probability distributions over the features, and that the dissociation between evidential impact and posterior probability increased the number of errors. The implications of these findings for lower and higher level cognitive models are discussed.

Natural frequency tree- versus conditional probability formula-based training for medical students' estimation of screening test predictive values: a randomized controlled trial

BACKGROUND

Medical students and professionals often struggle to understand medical test results, which can lead to poor medical decisions. Natural frequency tree-based training (NF-TT) has been suggested to help people correctly estimate the predictive value of medical tests. We aimed to compare the effectiveness of NF-TT with conventional conditional probability formula-based training (CP-FT) and investigate student variables that may influence NF-TT's effectiveness.

METHODS

We conducted a parallel group randomized controlled trial of NF-TT vs. CP-FT in two medical schools in South Korea (a 1:1 allocation ratio). Participants were randomly assigned to watch either NF-TT or CP-FT video at individual computer stations. NF-TT video showed how to translate relevant probabilistic information into natural frequencies using a tree structure to estimate the predictive values of screening tests. CP-FT video showed how to plug the same information into a mathematical formula to calculate predictive values. Both videos were 15 min long. The primary outcome was the accuracy in estimating the predictive value of screening tests assessed using multiple-choice questions at baseline, post-intervention (i.e., immediately after training), and one-month follow-up. The secondary outcome was the accuracy of conditional probabilistic reasoning in non-medical contexts, also assessed using multiple-choice questions, but only at follow-up as a measure of transfer of learning. 231 medical students completed their participation.

RESULTS

Overall, NF-TT was not more effective than CP-FT in improving the predictive value estimation accuracy at post-intervention (NF-TT: 87.13%, CP-FT: 86.03%, p = .86) and follow-up (NF-TT: 72.39%, CP-FT: 68.10%, p = .40) and facilitating transfer of training (NF-TT: 75.54%, CP-FT: 71.43%, p = .41). However, for participants without relevant prior training, NF-TT was more effective than CP-FT in improving estimation accuracy at follow-up (NF-TT: 74.86%, CP-FT: 58.71%, p = .02) and facilitating transfer of learning (NF-TT: 82.86%, CP-FT: 66.13%, p = .04).

CONCLUSIONS

Introducing NF-TT early in the medical school curriculum, before students are exposed to a pervasive conditional probability formula-based approach, would offer the greatest benefit.

TRIAL REGISTRATION

Korea Disease Control and Prevention Agency Clinical Research Information Service KCT0004246 (the date of first trial registration: 27/08/2019). The full trial protocol can be accessed at https://cris.nih.go.kr/cris/search/detailSearch.do?seq=15616&search_page=L .