Improving Understanding of Diagnostic Test Outcomes

Assessment of diagnostic capacity and decision-making based on the 2015 American Thyroid Association ultrasound classification system

BACKGROUND

This study evaluates the American Thyroid Association (ATA) ultrasound (US) classification system for the initial assessment of thyroid nodules to determine if it indeed facilitates clinical decision-making.

AIM

To perform a systematic review and meta-analysis of the diagnostic value of the ATA US classification system for the initial assessment of thyroid nodules.

METHODS

In accordance with the PRISMA statement for diagnostic test accuracy, we selected articles that evaluated the 2015 ATA US pattern guidelines using a diagnostic gold standard. We analyzed these cases using traditional diagnostic parameters, as well as the threshold approach to clinical decision-making and decision curve analysis.

RESULTS

We reviewed 13 articles with 8445 thyroid nodules, which were classified according to 2015 ATA patterns. Of these, 46.62% were malignant. No cancer was found in any of the ATA benign pattern nodules. The Bayesian analysis post-test probability for cancer in each classification was: (1) Very-low suspicion, 0.85%; (2) Low, 2.6%; (3) Intermediate, 6.7%; and (4) High, 40.9%. The net benefit (NB), expressed as avoided interventions, indicated that the highest capacity to avoid unnecessary fine needle aspiration biopsy (FNAB) in the patterns that we studied was 42, 31, 35, and 43 of every 100 FNABs. The NB calculation for a probability threshold of 11% for each of the ATA suspicion patterns studied is less than that of performing FNAB on all nodules.

CONCLUSION

These three types of analysis have shown that only the ATA high-suspicion diagnostic pattern is clinically useful, in which case, FNAB should be performed. However, the curve decision analysis has demonstrated that using the ATA US risk patterns to decide which patients need FNAB does not provide a greater benefit than performing FNAB on all thyroid nodules. Therefore, it is likely that a better way to approach the assessment of thyroid nodules would be to perform FNAB on all non-cystic nodules, as the present analysis has shown the ATA risk patterns do not provide an adequate clinical decision-making framework.

Reference Dependence in Bayesian Reasoning: Value Selection Bias, Congruence Effects, and Response Prompt Sensitivity

This work examines the influence of reference dependence, including value selection bias and congruence effects, on diagnostic reasoning. Across two studies, we explored how dependence on the initial problem structure influences the ability to solve simplified precursors to the more traditional Bayesian reasoning problems. Analyses evaluated accuracy and types of response errors as a function of congruence between the problem presentation and question of interest, amount of information, need for computation, and individual differences in numerical abilities. Across all problem variations, there was consistent and strong evidence of a value selection bias in that incorrect responses almost always conformed to values that were provided in the problem rather than other errors including those related to computation. The most consistent and unexpected error across all conditions in the first experiment was that people were often more likely to utilize the superordinate value (N) as part of their solution rather than the anticipated reference class values. This resulted in a weakened effect of congruence, with relatively low accuracy even in congruent conditions, and a dominant response error of the superordinate value. Experiment 2 confirmed that the introduction of a new sample drew attention away from the provided reference class, increasing reliance on the overall sample size. This superordinate preference error, along with the benefit of repeating the PPV reference class within the question, demonstrated the importance of reference dependence based on the salience of information within the response prompt. Throughout, higher numerical skills were generally associated with higher accuracy, whether calculations were required or not.

Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment

Background. Patients may find clinical prediction models more useful if those models accounted for preferences for false-positive and false-negative predictive errors and for other model characteristics. Methods. We conducted a discrete choice experiment to compare preferences for characteristics of a hypothetical mortality prediction model among community-dwelling patients with chronic lung disease recruited from 3 clinics in Philadelphia. This design was chosen to allow us to quantify "exchange rates" between different characteristics of a prediction model. We provided previously validated educational modules to explain model attributes of sensitivity, specificity, confidence intervals (CI), and time horizons. Patients reported their interest in using prediction models themselves or having their physicians use them. Patients then chose between 2 hypothetical prediction models each containing varying levels of the 4 attributes across 12 tasks. Results. We completed interviews with 200 patients, among whom 95% correctly chose a strictly dominant model in an internal validity check. Patients' interest in predictive information was high for use by themselves (n = 169, 85%) and by their physicians (n = 184, 92%). Interest in maximizing sensitivity and specificity were similar (0.88 percentage points of specificity equivalent to 1 point of sensitivity, 95% CI 0.72 to 1.05). Patients were willing to accept a reduction of 6.10 months (95% CI 3.66 to 8.54) in the predictive time horizon for a 1% increase in specificity. Discussion. Patients with chronic lung disease can articulate their preferences for the characteristics of hypothetical mortality prediction models and are highly interested in using such models as part of their care. Just as clinical care should become more patient centered, so should the characteristics of predictive models used to guide that care.

Physician and Nonphysician Estimates of Positive Predictive Value in Diagnostic v. Mass Screening Mammography: An Examination of Bayesian Reasoning

BACKGROUND

The same test with the same result has different positive predictive values (PPVs) for people with different pretest probability of disease. Representative thinking theory suggests people are unlikely to realize this because they ignore or underweight prior beliefs when given new information (e.g., test results) or due to confusing test sensitivity (probability of positive test given disease) with PPV (probability of disease given positive test). This research examines whether physicians and MBAs intuitively know that PPV following positive mammography for an asymptomatic woman is less than PPV for a symptomatic woman and, if so, whether they correctly perceive the difference.

DESIGN

Sixty general practitioners (GPs) and 84 MBA students were given 2 vignettes of women with abnormal (positive) mammography tests: 1 with prior symptoms (diagnostic test), the other an asymptomatic woman participating in mass screening (screening test). Respondents estimated pretest and posttest probabilities. Sensitivity and specificity were neither provided nor elicited.

RESULTS

Eighty-eight percent of GPs and 46% of MBAs considered base rates and estimated PPV in diagnosis greater than PPV in screening. On average, GPs estimated a 27-point difference and MBAs an 18-point difference, compared to actual of 55 or more points. Ten percent of GPs and 46% of MBAs ignored base rates, incorrectly assessing the 2 PPVs as equal.

CONCLUSIONS

Physicians and patients are better at intuitive Bayesian reasoning than is suggested by studies that make test accuracy values readily available to be confused with PPV. However, MBAs and physicians interpret a positive in screening as more similar to a positive in diagnosis than it is, with nearly half of MBAs and some physicians wrongly equating the two. This has implications for overdiagnosis and overtreatment.