Bayesian clinical reasoning: does intuitive estimation of likelihood ratios on an ordinal scale outperform estimation of sensitivities and specificities?

Interpretation of results of PCR and B-D-glucan for the diagnosis of Pneumocystis Jirovecii Pneumonia in immunocompromised adults with acute respiratory failure

BACKGROUND

The accuracy of a diagnostic test depends on its intrinsic characteristics and the disease incidence. This study aims to depict post-test probability of Pneumocystis pneumonia (PJP), according to results of PCR and Beta-D-Glucan (BDG) tests in patients with acute respiratory failure (ARF).

MATERIALS AND METHODS

Diagnostic performance of PCR and BDG was extracted from literature. Incidence of Pneumocystis pneumonia was assessed in a dataset of 2243 non-HIV immunocompromised patients with ARF. Incidence of Pneumocystis pneumonia was simulated assuming a normal distribution in 5000 random incidence samples. Post-test probability was assessed using Bayes theorem.

RESULTS

Incidence of PJP in non-HIV ARF patients was 4.1% (95%CI 3.3-5). Supervised classification identified 4 subgroups of interest with incidence ranging from 2.0% (No ground glass opacities; 95%CI 1.4-2.8) to 20.2% (hematopoietic cell transplantation, ground glass opacities and no PJP prophylaxis; 95%CI 14.1-27.7). In the overall population, positive post-test probability was 32.9% (95%CI 31.1-34.8) and 22.8% (95%CI 21.5-24.3) for PCR and BDG, respectively. Negative post-test probability of being infected was 0.10% (95%CI 0.09-0.11) and 0.23% (95%CI 0.21-0.25) for PCR and BDG, respectively. In the highest risk subgroup, positive predictive value was 74.5% (95%CI 72.0-76.7) and 63.8% (95%CI 60.8-65.8) for PCR and BDG, respectively.

CONCLUSION

Although both tests yield a high intrinsic performance, the low incidence of PJP in this cohort resulted in a low positive post-test probability. We propose a method to illustrate pre and post-test probability relationship that may improve clinician perception of diagnostic test performance according to disease incidence in predefined clinical settings.

Determining qualitative effect size ratings using a likelihood ratio scatter matrix in diagnostic test accuracy systematic reviews

Background Diagnostic test accuracy (DTA) systematic reviews (SRs) characterize a test's potential for diagnostic quality and safety. However, interpreting DTA measures in the context of SRs is challenging. Further, some evidence grading methods (e.g. Centers for Disease Control and Prevention, Division of Laboratory Systems Laboratory Medicine Best Practices method) require determination of qualitative effect size ratings as a contributor to practice recommendations. This paper describes a recently developed effect size rating approach for assessing a DTA evidence base. Methods A likelihood ratio scatter matrix will plot positive and negative likelihood ratio pairings for DTA studies. Pairings are graphed as single point estimates with confidence intervals, positioned in one of four quadrants derived from established thresholds for test clinical validity. These quadrants support defensible judgments on "substantial", "moderate", or "minimal" effect size ratings for each plotted study. The approach is flexible in relation to a priori determinations of the relative clinical importance of false positive and false negative test results. Results and conclusions This qualitative effect size rating approach was operationalized in a recent SR that assessed effectiveness of test practices for the diagnosis of Clostridium difficile. Relevance of this approach to other methods of grading evidence, and efforts to measure diagnostic quality and safety are described. Limitations of the approach arise from understanding that a diagnostic test is not an isolated element in the diagnostic process, but provides information in clinical context towards diagnostic quality and safety.

How well do health professionals interpret diagnostic information? A systematic review

OBJECTIVE

To evaluate whether clinicians differ in how they evaluate and interpret diagnostic test information.

DESIGN

Systematic review.

DATA SOURCES

MEDLINE, EMBASE and PsycINFO from inception to September 2013; bibliographies of retrieved studies, experts and citation search of key included studies.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Primary studies that provided information on the accuracy of any diagnostic test (eg, sensitivity, specificity, likelihood ratios) to health professionals and that reported outcomes relating to their understanding of information on or implications of test accuracy.

RESULTS

We included 24 studies. 6 assessed ability to define accuracy metrics: health professionals were less likely to identify the correct definition of likelihood ratios than of sensitivity and specificity. -25 studies assessed Bayesian reasoning. Most assessed the influence of a positive test result on the probability of disease: they generally found health professionals' estimation of post-test probability to be poor, with a tendency to overestimation. 3 studies found that approaches based on likelihood ratios resulted in more accurate estimates of post-test probability than approaches based on estimates of sensitivity and specificity alone, while 3 found less accurate estimates. 5 studies found that presenting natural frequencies rather than probabilities improved post-test probability estimation and speed of calculations.

CONCLUSIONS

Commonly used measures of test accuracy are poorly understood by health professionals. Reporting test accuracy using natural frequencies and visual aids may facilitate improved understanding and better estimation of the post-test probability of disease.

Gut instinct: a diagnostic tool?

OBJECTIVES

It is generally accepted that with experience clinicians develop the ability to identify patients who present with malignancy prior to a formal diagnosis. This ability cannot be quantified, nor is it a plausible substitute for investigation. This study aimed to evaluate the association between instinct and head and neck cancer diagnosis.

METHODS

A prospective study of patients requiring urgent diagnostic procedures for suspected cancer between August and December 2010 was performed. Risk factors, symptoms, signs and the clinician's impression were recorded. These were graded and subsequently correlated with histology findings.

RESULTS

Twenty-seven patients, with a mean age of 62.2 years, underwent a diagnostic procedure. Thirty per cent of patients were referred under the two-week pathway and 18.5 per cent had a previous history of head and neck cancer. A diagnosis of cancer was made in 37 per cent of patients. There was a positive correlation between clinical suspicion and cancer diagnosis (Kendall's tau-b = 0.648749).

CONCLUSION

This study highlights the importance of clinical suspicion in cancer diagnosis. Although clinical suspicion cannot be quantified, it should be regarded as an integral part of patient assessment.

The balance beam metaphor: a perspective on clinical diagnosis

Understanding the impact of clinical findings in discriminating between possible causes of a patient's presentation is essential in clinical judgment. A balance beam is a natural physical analogue that can accurately represent the combination of several pieces of evidence with varying ability to discriminate between disease hypotheses. Calculation of Bayes' theorem using log(posterior odds) as a function of log(prior odds) and the logarithms of the evidence's likelihood ratios maps onto the physical forces affecting objects placed on a balance beam. We describe the rules governing the functioning of tokens representing clinical findings in the comparison of 2 competing diseases. The likelihood ratios corresponding to positive (LR+) or negative (LR-) observations for each symptom determine the lateral position at which the symptom's token is placed on the beam, using a weight if the finding is present and a helium balloon if it is absent. We discuss how a balance beam could represent concepts of dynamic specificity (due to changes in competitor diseases' probabilities) and dynamic sensitivity (due to class-conditional independence). Utility-based thresholds for acting on a diagnosis could be represented by moving the balance beam's fulcrum. It is suggested that a balance beam can be a useful aid for students learning clinical diagnosis, allowing them to build on existing intuitive understanding to develop an appreciation of how evidence combines to influence degree of belief. The balance beam could also facilitate exploration of the potential impact of available questions or investigations.

Leveling the playing field: bringing development of biomarkers and molecular diagnostics up to the standards for drug development

Molecular diagnostics are becoming increasingly important in clinical research to stratify or identify molecularly profiled patient cohorts for targeted therapies, to modify the dose of a therapeutic, and to assess early response to therapy or monitor patients. Molecular diagnostics can also be used to identify the pharmacogenetic risk of adverse drug reactions. The articles in this CCR Focus section on molecular diagnosis describe the development and use of markers to guide medical decisions regarding cancer patients. They define sources of preanalytic variability that need to be minimized, as well as the regulatory and financial challenges involved in developing diagnostics and integrating them into clinical practice. They also outline a National Cancer Institute program to assist diagnostic development. Molecular diagnostic clinical tests require rigor in their development and clinical validation, with sensitivity, specificity, and validity comparable to those required for the development of therapeutics. These diagnostics must be offered at a realistic cost that reflects both their clinical value and the costs associated with their development. When genome-sequencing technologies move into the clinic, they must be integrated with and traceable to current technology because they may identify more efficient and accurate approaches to drug development. In addition, regulators may define progressive drug approval for companion diagnostics that requires further evidence regarding efficacy and safety before full approval can be achieved. One way to accomplish this is to emphasize phase IV postmarketing, hypothesis-driven clinical trials with biological characterization that would permit an accurate definition of the association of low-prevalence gene alterations with toxicity or response in large cohorts.