A randomized trial of ways to describe test accuracy: the effect on physicians' post-test probability estimates

Interpretation of results of PCR and B-D-glucan for the diagnosis of Pneumocystis Jirovecii Pneumonia in immunocompromised adults with acute respiratory failure

BACKGROUND

The accuracy of a diagnostic test depends on its intrinsic characteristics and the disease incidence. This study aims to depict post-test probability of Pneumocystis pneumonia (PJP), according to results of PCR and Beta-D-Glucan (BDG) tests in patients with acute respiratory failure (ARF).

MATERIALS AND METHODS

Diagnostic performance of PCR and BDG was extracted from literature. Incidence of Pneumocystis pneumonia was assessed in a dataset of 2243 non-HIV immunocompromised patients with ARF. Incidence of Pneumocystis pneumonia was simulated assuming a normal distribution in 5000 random incidence samples. Post-test probability was assessed using Bayes theorem.

RESULTS

Incidence of PJP in non-HIV ARF patients was 4.1% (95%CI 3.3-5). Supervised classification identified 4 subgroups of interest with incidence ranging from 2.0% (No ground glass opacities; 95%CI 1.4-2.8) to 20.2% (hematopoietic cell transplantation, ground glass opacities and no PJP prophylaxis; 95%CI 14.1-27.7). In the overall population, positive post-test probability was 32.9% (95%CI 31.1-34.8) and 22.8% (95%CI 21.5-24.3) for PCR and BDG, respectively. Negative post-test probability of being infected was 0.10% (95%CI 0.09-0.11) and 0.23% (95%CI 0.21-0.25) for PCR and BDG, respectively. In the highest risk subgroup, positive predictive value was 74.5% (95%CI 72.0-76.7) and 63.8% (95%CI 60.8-65.8) for PCR and BDG, respectively.

CONCLUSION

Although both tests yield a high intrinsic performance, the low incidence of PJP in this cohort resulted in a low positive post-test probability. We propose a method to illustrate pre and post-test probability relationship that may improve clinician perception of diagnostic test performance according to disease incidence in predefined clinical settings.

Serum Mac-2 binding protein glycosylation isomer dynamics in patients achieving sustained virologic response for hepatitis C virus

BACKGROUND AND AIM

Understanding the dynamics of serum Mac-2 binding protein glycosylation isomer (M2BPGi) remains pivotal for hepatitis C virus (HCV) patients' post-sustained virologic response (SVR12) through direct-acting antivirals (DAAs).

METHODS

We compared areas under receiver operating characteristic curves (AUROCs) of M2BPGi, FIB-4, and APRI and assess M2BPGi cutoff levels in predicting fibrosis stages of ≥F3 and F4 utilizing transient elastography in 638 patients. Variations in M2BPGi levels from pretreatment to SVR12 and their association with pretreatment alanine transaminase (ALT) levels and fibrosis stage were investigated.

RESULTS

The AUROCs of M2BPGi were comparable to FIB-4 in predicting ≥F3 (0.914 vs 0.902, P = 0.48) and F4 (0.947 vs 0.915, P = 0.05) but were superior to APRI in predicting ≥F3 (0.914 vs 0.851, P = 0.001) and F4 (0.947 vs 0.857, P < 0.001). Using M2BPGi cutoff values of 2.83 and 3.98, fibrosis stages of ≥F3 and F4 were confirmed with a positive likelihood ratio ≥10. The median M2BPGi change was -0.55. Patients with ALT levels ≥5 times ULN or ≥F3 demonstrated more pronounced median decreases in M2BPGi level compared to those with ALT levels 2-5 times ULN and <2 times ULN (-0.97 vs -0.68 and -0.44; P < 0.001) or with < F3 (-1.52 vs -0.44; P < 0.001).

CONCLUSIONS

Serum M2BPGi is a reliable marker for advanced hepatic fibrosis. Following viral clearance, there is a notable M2BPGi decrease, with the extent of reduction influenced by ALT levels and fibrosis stage.

Family physicians overestimate diagnosis probabilities regardless of the test results

Introduction

As useful tools for clinical decision-making, diagnostic tests require careful interpretation in order to prevent underdiagnosis, overdiagnosis or misdiagnosis. The aim of this study was to explore primary care practitioners' understanding and interpretation of the probability of disease before and after test results for six common clinical scenarios.

Methods

This cross-sectional study was conducted with 414 family physicians who were working at primary care in Istanbul via face-to-face interviews held between November 2021 and March 2022. The participants were asked to estimate the probability of diagnosis in six clinical scenarios provided to them. Clinical scenarios were about three cancer screening cases (breast, cervical and colorectal), and three infectious disease cases (pneumonia, urinary tract infection, and COVID-19). For each scenario participants estimated the probability of the diagnosis before application of a diagnostic test, after a positive test result, and after a negative test result. Their estimates were compared with the true answers derived from relevant guidelines.

Results

For all scenarios, physicians' estimates were significantly higher than the scientific evidence range. The minimum overestimation was positive test result for COVID-19 and maximum was pre-test case for cervical cancer. In the hypothetical control question for prevalence and test accuracy, physicians estimated disease probability as 95.0% for a positive test result and 5.0% for a negative test result while the correct answers were 2.0 and 0%, respectively (p < 0.001).

Discussion

Comparing the scientific evidence, overestimation in all diagnostic scenarios, regardless of if the disease is an acute infection or a cancer, may indicate that the probabilistic approach is not conducted by the family physicians. To prevent inaccurate interpretation of the tests that may lead to incorrect or unnecessary treatments with adverse consequences, evidence-based decision-making capacity must be strengthened.

Serum Mac-2 Binding Protein Glycosylation Isomer to Predict the Severity of Hepatic Fibrosis in Patients with Hepatitis C Virus Infection

Large-scale studies to assess the utility of the Mac-2 binding protein glycosylation isomer (M2BPGi) in predicting hepatic fibrosis in patients with hepatitis C virus (HCV) infection are limited. Serum M2BPGi level determination was performed in 1460 patients with HCV who received liver stiffness measurement (LSM) using transient elastography (TE). The correlation of LSM and grade of hepatic fibrosis as staged by TE with M2BPGi was assessed. Receiver operating characteristic (ROC) curves were constructed to evaluate the diagnostic power of M2BPGi for fibrosis stages of ≥F2, ≥F3, and F4. The selected M2BPGi cutoff values were chosen based on the maximal Youden index, a positive likelihood ratio (LR) ≥ 10, and a negative LR ≤ 0.1. Serum M2BPGi level was highly correlated with LSM (Pearson correlation coefficient: 0.567, p < 0.001) and hepatic fibrosis stage (Spearman’s rank correlation coefficient: 0.772, p < 0.001). The areas under ROC curves (AUROCs) of M2BPGi for ≥F2, ≥F3, and F4 were 0.865 (95% confidence interval [CI]: 0.846−0.884), 0.937 (95 % CI: 0.922−0.952), and 0.962 (95% CI: 0.951−0.972). The maximal Youden indices for ≥F2, ≥F3, and F4 were 1.72, 2.65, and 3.93. By selecting M2BPGi cutoff values with a positive LR ≥ 10 and a negative LR ≤ 0.1, clinicians were able to correctly discriminate F2, F3, and F4 in 69.1%, 77.8%, and 90.1% of patients. In conclusion, serum M2BPGi is a good diagnostic tool to predict the severity of hepatic fibrosis in patients with HCV infection.

Development and Validation of a Mucosal Impedance Contour Analysis System to Distinguish Esophageal Disorders

BACKGROUND & AIMS

Diagnostic testing for chronic esophageal disorders relies on histopathology analysis of biopsies or uncomfortable transnasal catheters or wireless pH monitoring, which capture abnormal intraluminal refluxate. We therefore developed a balloon mucosal impedance (MI) catheter system that instantly detects changes in esophageal mucosal integrity during endoscopy over a long segment of the esophagus. We performed a prospective study to evaluate the ability of a balloon-incorporated MI catheter to detect and evaluate esophageal disorders, including gastroesophageal reflux disease (GERD) and eosinophilic esophagitis (EoE).

METHODS

We performed a prospective study of 69 patients undergoing esophagogastroduodenoscopy with or without wireless pH monitoring. Patients were classified as having GERD (erosive esophagitis or abnormal pH; n = 24), EoE (confirmed with pathology analysis of tissues from both distal and proximal esophagus; n = 21), or non-GERD (normal results from esophagogastroduodenoscopy and pH tests; n = 24). Receiver operating characteristic curves and area under the operating characteristic curve (AUC) were used to compare the accuracy of balloon MI in diagnosis. Probabilities of assignment to each group (GERD, non-GERD, or EoE) were estimated using multinomial logistic regression. Association between MI patterns and diagnoses were validated using data from patients seen at 3 separate institutions.

RESULTS

MI pattern along the esophageal axis differed significantly (P < .01) among patients with GERD, EoE, and non-GERD. Patients with non-GERD had higher MI values along all measured segments. The MI pattern for GERD was easily distinguished from that of EoE: in patients with GERD, MI values were low in the distal esophagus and normalized along the proximal esophagus, whereas in patients with EoE, measurements were low in all segments of the esophagus. Intercept and rate of rise of MI value (slope) as distance increased from the squamocolumnar junction identified patients with GERD with an AUC = 0.67, patients with EoE with an AUC = 0.84, and patients with non-GERD with an AUC = 0.83 in the development cohort. One patient had an adverse event (reported mild chest pain after the procedure) and was discharged from the hospital without further events.

CONCLUSIONS

We developed a balloon MI catheter system that instantly detects changes in esophageal mucosal integrity during endoscopy and found it to be safe and able to identify patients with GERD, EoE, or non-GERD. We validated our findings in a separate cohort for patients. ClinicalTrials.gov ID NCT03103789.

Interactive visualisation for interpreting diagnostic test accuracy study results

Information about the performance of diagnostic tests is typically presented in the form of measures of test accuracy such as sensitivity and specificity. These measures may be difficult to translate directly into decisions about patient treatment, for which information presented in the form of probabilities of disease after a positive or a negative test result may be more useful. These probabilities depend on the prevalence of the disease, which is likely to vary between populations. This article aims to clarify the relationship between pre-test (prevalence) and post-test probabilities of disease, and presents two free, online interactive tools to illustrate this relationship. These tools allow probabilities of disease to be compared with decision thresholds above and below which different treatment decisions may be indicated. They are intended to help those involved in communicating information about diagnostic test performance and are likely to be of benefit when teaching these concepts. A substantive example is presented using C reactive protein as a diagnostic marker for bacterial infection in the older adult population. The tools may also be useful for manufacturers of clinical tests in planning product development, for authors of test evaluation studies to improve reporting and for users of test evaluations to facilitate interpretation and application of the results.

Bayesian reasoning in residents' preliminary diagnoses

Whether and when humans in general, and physicians in particular, use their beliefs about base rates in Bayesian reasoning tasks is a long-standing question. Unfortunately, previous research on whether doctors use their beliefs about the prevalence of diseases in diagnostic judgments has critical limitations. In this study, we assessed whether residents' beliefs about the prevalence of a disease are associated with their judgments of the likelihood of the disease in diagnosis, and whether residents' beliefs about the prevalence of diseases change across the 3 years of residency. Residents were presented with five ambiguous vignettes typical of patients presenting on the inpatient general medicine services. For each vignette, the residents judged the likelihood of five or six possible diagnoses. Afterward, they judged the prevalence within the general medicine services of all the diseases in the vignettes. Most importantly, residents who believed a disease to be more prevalent tended to rate the disease as more likely in the vignette cases, suggesting a rational tendency to incorporate their beliefs about disease prevalence into their diagnostic likelihood judgments. In addition, the residents' prevalence judgments for each disease were assessed over the 3 years of residency. The precision of the prevalence estimates increased across the 3 years of residency, though the accuracy of the prevalence estimates did not. These results imply that residents do have a rational tendency to use prevalence beliefs for diagnosis, and this finding also contributes to a larger question of whether humans intuitively use base rates for making judgments.

How well do health professionals interpret diagnostic information? A systematic review

OBJECTIVE

To evaluate whether clinicians differ in how they evaluate and interpret diagnostic test information.

DESIGN

Systematic review.

DATA SOURCES

MEDLINE, EMBASE and PsycINFO from inception to September 2013; bibliographies of retrieved studies, experts and citation search of key included studies.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Primary studies that provided information on the accuracy of any diagnostic test (eg, sensitivity, specificity, likelihood ratios) to health professionals and that reported outcomes relating to their understanding of information on or implications of test accuracy.

RESULTS

We included 24 studies. 6 assessed ability to define accuracy metrics: health professionals were less likely to identify the correct definition of likelihood ratios than of sensitivity and specificity. -25 studies assessed Bayesian reasoning. Most assessed the influence of a positive test result on the probability of disease: they generally found health professionals' estimation of post-test probability to be poor, with a tendency to overestimation. 3 studies found that approaches based on likelihood ratios resulted in more accurate estimates of post-test probability than approaches based on estimates of sensitivity and specificity alone, while 3 found less accurate estimates. 5 studies found that presenting natural frequencies rather than probabilities improved post-test probability estimation and speed of calculations.

CONCLUSIONS

Commonly used measures of test accuracy are poorly understood by health professionals. Reporting test accuracy using natural frequencies and visual aids may facilitate improved understanding and better estimation of the post-test probability of disease.

Presentation of Diagnostic Information to Doctors May Change Their Interpretation and Clinical Management: A Web-Based Randomised Controlled Trial

BACKGROUND

There is little evidence on how best to present diagnostic information to doctors and whether this makes any difference to clinical management. We undertook a randomised controlled trial to see if different data presentations altered clinicians' decision to further investigate or treat a patient with a fictitious disorder ("Green syndrome") and their ability to determine post-test probability.

METHODS

We recruited doctors registered with the United Kingdom's largest online network for medical doctors between 10 July and 6" November 2012. Participants were randomised to one of four arms: (a) text summary of sensitivity and specificity, (b) Fagan's nomogram, (c) probability-modifying plot (PMP), (d) natural frequency tree (NFT). The main outcome measure was the decision whether to treat, not treat or undertake a brain biopsy on the hypothetical patient and the correct post-test probability. Secondary outcome measures included knowledge of diagnostic tests.

RESULTS

917 participants attempted the survey and complete data were available from 874 (95.3%). Doctors randomized to the PMP and NFT arms were more likely to treat the patient than those randomized to the text-only arm. (ORs 1.49, 95% CI 1.02, 2.16) and 1.43, 95% CI 0.98, 2.08 respectively). More patients randomized to the PMP (87/218-39.9%) and NFT (73/207-35.3%) arms than the nomogram (50/194-25.8%) or text only (30/255-11.8%) arms reported the correct post-test probability (p <0.001). Younger age, postgraduate training and higher self-rated confidence all predicted better knowledge performance. Doctors with better knowledge were more likely to view an optional learning tutorial (OR per correct answer 1.18, 95% CI 1.06, 1.31).

CONCLUSIONS

Presenting diagnostic data using a probability-modifying plot or natural frequency tree influences the threshold for treatment and improves interpretation of tests results compared to text summary of sensitivity and specificity or Fagan's nomogram.

Diagnostic point-of-care tests in resource-limited settings

The aim of diagnostic point-of-care testing is to minimise the time to obtain a test result, thereby allowing clinicians and patients to make a quick clinical decision. Because point-of-care tests are used in resource-limited settings, the benefits need to outweigh the costs. To optimise point-of-care testing in resource-limited settings, diagnostic tests need rigorous assessments focused on relevant clinical outcomes and operational costs, which differ from assessments of conventional diagnostic tests. We reviewed published studies on point-of-care testing in resource-limited settings, and found no clearly defined metric for the clinical usefulness of point-of-care testing. Therefore, we propose a framework for the assessment of point-of-care tests, and suggest and define the term test efficacy to describe the ability of a diagnostic test to support a clinical decision within its operational context. We also propose revised criteria for an ideal diagnostic point-of-care test in resource-limited settings. Through systematic assessments, comparisons between centralised testing and novel point-of-care technologies can be more formalised, and health officials can better establish which point-of-care technologies represent valuable additions to their clinical programmes.

A qualitative study into the difficulties experienced by healthcare decision makers when reading a Cochrane diagnostic test accuracy review

BACKGROUND

Cochrane reviews are one of the best known and most trusted sources of evidence-based information in health care. While steps have been taken to make Cochrane intervention reviews accessible to a diverse readership, little is known about the accessibility of the newcomer to the Cochrane library: diagnostic test accuracy reviews (DTARs). The current qualitative study explored how healthcare decision makers, who varied in their knowledge and experience with test accuracy research and systematic reviews, read and made sense of DTARs.

METHODS

A purposive sample of clinicians, researchers and policy makers (n = 21) took part in a series of think-aloud interviews, using as interview material the first three DTARs published in the Cochrane library. Thematic qualitative analysis of the transcripts was carried out to identify patterns in participants' 'reading' and interpretation of the reviews and the difficulties they encountered.

RESULTS

Participants unfamiliar with the design and methodology of DTARs found the reviews largely inaccessible and experienced a range of difficulties stemming mainly from the mismatch between background knowledge and level of explanation provided in the text. Experience with systematic reviews of interventions did not guarantee better understanding and, in some cases, led to confusion and misinterpretation. These difficulties were further exacerbated by poor layout and presentation, which affected even those with relatively good knowledge of DTARs and had a negative impact not only on their understanding of the reviews but also on their motivation to engage with the text. Comparison between the readings of the three reviews showed that more accessible presentation, such as presenting the results as natural frequencies, significantly increased participants' understanding.

CONCLUSIONS

The study demonstrates that authors and editors should pay more attention to the presentation as well as the content of Cochrane DTARs, especially if the reports are aimed at readers with various levels of background knowledge and experience. It also raises the question as to the anticipated target audience of the reports and suggests that different groups of healthcare decision-makers may require different modes of presentation.

Predictive value of the 4Ts scoring system for heparin-induced thrombocytopenia: a systematic review and meta-analysis

The 4Ts is a pretest clinical scoring system for heparin-induced thrombocytopenia (HIT). Although widely used in clinical practice, its predictive value for HIT in diverse settings and patient populations is unknown. We performed a systematic review and meta-analysis to estimate the predictive value of the 4Ts in patients with suspected HIT. We searched PubMed, Cochrane Database, and ISI Web of Science for studies that included patients with suspected HIT, who were evaluated by both the 4Ts and a reference standard against which the 4Ts could be compared. Quality of eligible studies was assessed by QUADAS-2 criteria. Thirteen studies, collectively involving 3068 patients, fulfilled eligibility criteria. A total of 1712 (55.8%) patients were classified by 4Ts score as having a low probability of HIT. The negative predictive value of a low probability 4Ts score was 0.998 (95% CI, 0.970-1.000) and remained high irrespective of the party responsible for scoring, the prevalence of HIT, or the composition of the study population. The positive predictive value of an intermediate and high probability 4Ts score was 0.14 (0.09-0.22) and 0.64 (0.40-0.82), respectively. A low probability 4Ts score appears to be a robust means of excluding HIT. Patients with intermediate and high probability scores require further evaluation.

Doctors and patients' susceptibility to framing bias: a randomized trial

BACKGROUND

Framing of risk influences the perceptions of treatment benefit.

OBJECTIVE

To determine which risk framing format corresponds best to comprehensive multi-faceted information, and to compare framing bias in doctors and in patients.

DESIGN

Randomized mail surveys.

PARTICIPANTS

One thousand four hundred and thirty-one doctors (56% response rate) and 1121 recently hospitalized patients (65% response rate).

INTERVENTION

Respondents were asked to interpret the results of a hypothetical clinical trial comparing an old and a new drug. They were randomly assigned to the following framing formats: absolute survival (new drug: 96% versus old drug: 94%), absolute mortality (4% versus 6%), relative mortality reduction (reduction by a third) or all three (fully informed condition). The new drug was reported to cause more side-effects.

MAIN MEASURE

Rating of the new drug as more effective than the old drug.

RESULTS

The proportions of doctors who rated the new drug as more effective varied by risk presentation format (abolute survival 51.8%, absolute mortality 68.3%, relative mortality reduction 93.8%, and fully informed condition 69.8%, p < 0.001). In patients these proportions were similar (abolute survival 51.7%, absolute mortality 66.8%, relative mortality reduction 89.3%, and fully informed condition 71.2%, p < 0.001). In both doctors (p = 0.72) and patients (p = 0.23) the fully informed condition was similar to the absolute risk format, but it differed significantly from the other conditions (all p < 0.01). None of the differences between doctors and patients were significant (all p > 0.1). In comparison to the fully informed condition, the odds ratio of greater perceived effectiveness was 0.45 for absolute survival (p < 0.001), 0.89 for absolute mortality (p = 0.29), and 4.40 for relative mortality reduction (p < 0.001).

CONCLUSIONS

Framing bias affects doctors and patients similarly. Describing clinical trial results as absolute risks is the least biased format, for both doctors and patients. Presenting several risk formats (on both absolute and relative scales) should be encouraged.

Defining Biomarker Performance and Clinical Validity

In the evaluation of biomarkers three questions can be answered: what is the analytical validity of the marker, what is its clinical validity, and does the marker have clinical utility? In most cases, clinical validity will be expressed in terms of the marker's accuracy: the degree to which it can be used to correctly identify diseased patients or, more generally, patients with the target condition. Diagnostic accuracy is evaluated in studies in which the biomarker values are compared to the outcome of the clinical reference standard in the same patients. There are several ways in which the results of diagnostic accuracy studies can be summarized, reported, and interpreted. In this paper we summarize and present the available measures. We classify these as error-based measures, information-based measures, and measures of the strength of the association. Clinical validity is linked to clinical utility. If the target condition is well defined and associated with unequivocal downstream management decisions, clinical validity, when defined in comparative terms, may sometimes act as a surrogate outcome measure for clinical utility.

Mixed logit analysis of bicyclist injury severity resulting from motor vehicle crashes at intersection and non-intersection locations

Standard multinomial logit (MNL) and mixed logit (MXL) models are developed to estimate the degree of influence that bicyclist, driver, motor vehicle, geometric, environmental, and crash type characteristics have on bicyclist injury severity, classified as property damage only, possible, nonincapacitating or severe (i.e., incapacitating or fatal) injury. This study is based on 10,029 bicycleinvolved crashes that occurred in the State of Ohio from 2002 to 2008. Results of likelihood ratio tests reveal that some of the factors affecting bicyclist injury severity at intersection and non-intersection locations are substantively different and using a common model to jointly estimate impacts on severity at both types of locations may result in biased or inconsistent estimates. Consequently, separate models are developed to independently assess the impacts of various factors on the degree of bicyclist injury severity resulting from crashes at intersection and non-intersection locations. Several covariates are found to have similar impacts on injury severity at both intersection and non-intersection locations. Conversely, six variables were found to significantly influence injury severity at intersection locations but not non-intersection locations while four variables influenced bicyclist injury severity only at non-intersection locations. In crashes occurring at intersection locations, the likelihood of severe bicyclist injury increases by 14.8 percent if the bicyclist is not wearing a helmet, 82.2 percent if the motorist is under the influence of alcohol, 141.3 percent if the crash-involved motor vehicle is a van, 40.6 percent if the motor vehicle strikes the side of the bicycle, and 182.6 percent if the crash occurs on a horizontal curve with a grade. Results from non-intersection locations show the likelihood of severe injuries increases by 374.5 percent if the bicyclist is under the influence of drugs, 150.1 percent if the motorist is under the influence of alcohol, 53.5 percent if the motor vehicle strikes the side of the bicycle and 99.9 percent if the crash-involved motor vehicle is a heavy-duty truck.

Does prevalence matter to physicians in estimating post-test probability of disease? A randomized trial

BACKGROUND

The probability of a disease following a diagnostic test depends on the sensitivity and specificity of the test, but also on the prevalence of the disease in the population of interest (or pre-test probability). How physicians use this information is not well known.

OBJECTIVE

To assess whether physicians correctly estimate post-test probability according to various levels of prevalence and explore this skill across respondent groups.

DESIGN

Randomized trial.

PARTICIPANTS

Population-based sample of 1,361 physicians of all clinical specialties.

INTERVENTION

We described a scenario of a highly accurate screening test (sensitivity 99% and specificity 99%) in which we randomly manipulated the prevalence of the disease (1%, 2%, 10%, 25%, 95%, or no information).

MAIN MEASURES

We asked physicians to estimate the probability of disease following a positive test (categorized as <60%, 60-79%, 80-94%, 95-99.9%, and >99.9%). Each answer was correct for a different version of the scenario, and no answer was possible in the "no information" scenario. We estimated the proportion of physicians proficient in assessing post-test probability as the proportion of correct answers beyond the distribution of answers attributable to guessing.

KEY RESULTS

Most respondents in each of the six groups (67%-82%) selected a post-test probability of 95-99.9%, regardless of the prevalence of disease and even when no information on prevalence was provided. This answer was correct only for a prevalence of 25%. We estimated that 9.1% (95% CI 6.0-14.0) of respondents knew how to assess correctly the post-test probability. This proportion did not vary with clinical experience or practice setting.

CONCLUSIONS

Most physicians do not take into account the prevalence of disease when interpreting a positive test result. This may cause unnecessary testing and diagnostic errors.

An investigation of dentists' and dental students' estimates of diagnostic probabilities

BACKGROUND

Research in medicine has shown that physicians have difficulty estimating the probability that a patient has a condition on the basis of available diagnostic evidence. They consistently undervalue baseline information about the patient relative to test information and are poor intuitive calculators of probability. The authors could not locate in the literature any studies of diagnostic probability estimates from baseline information and test data for dentists.

METHODS

Using two vignettes that contained different baseline information, dental students and clinical faculty members estimated the probability that the described hypothetical patient had the condition in question. Respondents also commented on the project.

CONCLUSIONS

Both groups of respondents overemphasized the importance of test evidence relative to baseline information, although experienced practitioners did so to a lesser extent than did students. Respondents, especially practitioners, expressed resistance to performing a diagnostic task that required precise estimates of probability.

CLINICAL IMPLICATIONS

Dentists appear to estimate diagnostic probabilities in an intuitive fashion, but they do so imprecisely. Clinical experience provides some protection against the bias of overestimating test evidence compared with baseline information. These findings raise questions about how practitioners use probability estimates and whether other models also may play a role. The incorporation of information from evidence-based dentistry into practice requires better understanding.

Clinical validity: defining biomarker performance

A central phase in the evaluation of a biomarker is the assessment of its clinical validity. In most cases, clinical validity will be expressed in terms of the marker's diagnostic accuracy: the degree to which it can be used to identify diseased patients or, more generally, patients with the target condition. Diagnostic accuracy is evaluated in diagnostic accuracy studies, in which the biomarker values are compared to the outcome of the clinical reference standard in the same patients. There are several ways in which the results of diagnostic accuracy studies can be summarized, reported, and interpreted. We present an overview and a critical commentary of the available measures. We classify them as error-based measures, information-based measures, and measures of the strength of the association.The diagnostic accuracy is not a fixed property of a marker. All accuracy measures may vary between studies, not just through chance, but also with changes in the definition of the target condition, the spectrum of disease, the setting, and the amount of prior testing. We discuss the relativity of the claim that likelihood ratios are a superior way of expressing the accuracy of biomarkers, and defend the use of the sometimes downgraded statistics sensitivity and specificity.

Sensitivity, specificity, positive and negative predictive values: diagnosing purple mange

PURPOSE

To shed light on several epidemiological terms for better understanding of diagnostic testing measures by using a mythical condition, "purple mange."

DATA SOURCES

Scientific literature related to epidemiology and statistical tests.

CONCLUSIONS

Nurse practitioners (NPs) use the concepts of sensitivity (SEN), specificity (SPEC), positive predictive value (PPV), and negative predictive value (NPV) daily in primary care and specialty areas. In addition, PPV and NPV vary with the prevalence of a condition. At times, NPs misunderstand the meaning of these terms.

IMPLICATIONS FOR PRACTICE

In order to develop appropriate treatment plans, an understanding of the concepts of SEN, SPEC, PPV, and NPV is important for interpreting test results. The authors have used this mythical condition purple mange as a teaching tool for NP students.

Reasons for ordering laboratory tests and relationship with frequency of abnormal results

OBJECTIVE

Laboratory tests are ordered on a daily basis, even though disease probability is often very low. Abnormal results, especially mildly abnormal results, can be difficult to interpret in these circumstances. Further insights into the occurrence of abnormalities can help improve rational test ordering and test interpretation. The objective was therefore to examine the frequency of mildly and markedly abnormal results and their relationship with physicians' reasons for ordering tests.

DESIGN

Prospective study. Participants. A total of 87 primary care physicians in the Netherlands collected data on 1775 patients.

MAIN OUTCOME MEASURES

The physicians recorded the reason for ordering the tests, the most probable diagnosis and the pretest probability. The laboratories' reference values and specified "action limits" were used to assess the number of abnormal results and markedly abnormal results, respectively.

RESULTS

Laboratory results were received for 1621 patients and 15,603 tests were reported (mean 9.6). The proportion of abnormal test results increased with increasing pretest probability (from 13.9% to 34.7%) and was 13.4% for tests ordered to reassure the patient and 13.3% for psychosocial diagnoses. The proportion of patients with at least one abnormal test result was high: 53.1% for tests ordered to reassure and 57.7% in patients with low pretest probability. Corresponding values for a marked abnormality were 11.1% and 12.4%, respectively.

CONCLUSION

Abnormal laboratory test results were frequent, even when pretest probability was low. Physicians should therefore carefully consider when tests are necessary. Future research could explore physicians' interpretation of test results and its impact on diagnosis and management.

Pretest expectations strongly influence interpretation of abnormal laboratory results and further management

Background

Abnormal results of diagnostic laboratory tests can be difficult to interpret when disease probability is very low. Although most physicians generally do not use Bayesian calculations to interpret abnormal results, their estimates of pretest disease probability and reasons for ordering diagnostic tests may - in a more implicit manner - influence test interpretation and further management. A better understanding of this influence may help to improve test interpretation and management. Therefore, the objective of this study was to examine the influence of physicians' pretest disease probability estimates, and their reasons for ordering diagnostic tests, on test result interpretation, posttest probability estimates and further management.

Methods

Prospective study among 87 primary care physicians in the Netherlands who each ordered laboratory tests for 25 patients. They recorded their reasons for ordering the tests (to exclude or confirm disease or to reassure patients) and their pretest disease probability estimates. Upon receiving the results they recorded how they interpreted the tests, their posttest probability estimates and further management. Logistic regression was used to analyse whether the pretest probability and the reasons for ordering tests influenced the interpretation, the posttest probability estimates and the decisions on further management.

Results

The physicians ordered tests for diagnostic purposes for 1253 patients; 742 patients had an abnormal result (64%). Physicians' pretest probability estimates and their reasons for ordering diagnostic tests influenced test interpretation, posttest probability estimates and further management. Abnormal results of tests ordered for reasons of reassurance were significantly more likely to be interpreted as normal (65.8%) compared to tests ordered to confirm a diagnosis or exclude a disease (27.7% and 50.9%, respectively). The odds for abnormal results to be interpreted as normal were much lower when the physician estimated a high pretest disease probability, compared to a low pretest probability estimate (OR = 0.18, 95% CI = 0.07-0.52, p < 0.001).

Conclusions

Interpretation and management of abnormal test results were strongly influenced by physicians' estimation of pretest disease probability and by the reason for ordering the test. By relating abnormal laboratory results to their pretest expectations, physicians may seek a balance between over- and under-reacting to laboratory test results.

Diagnostic methods I: sensitivity, specificity, and other measures of accuracy

For most physicians, use of diagnostic tests is part of daily routine. This paper focuses on their usefulness by explaining the different measures of accuracy, the interpretation of test results, and the implementation of a diagnostic strategy. Measures of accuracy include sensitivity and specificity. Although these measures are often considered fixed properties of a diagnostic test, in reality they are subject to multiple sources of variation such as the population case mix and the severity of the disease under study. Furthermore, when evaluating a new diagnostic test, it must be compared to a reference standard, although the latter is usually not perfect. In daily practice diagnostic tests are not used in isolation. Several issues will influence the interpretation of their results. First, clinicians have a prior assumption about the patient's chances of having the disease under investigation, based on the patient's characteristics, symptoms, and the disease prevalence in similar populations. Second, diagnostic tests are usually part of a diagnostic strategy. Therefore, it is not sufficient to determine the accuracy of a single test; one also needs to determine its additional value to the patient's diagnosis, treatment, or outcome as part of a diagnostic strategy.

Interpreting diagnostic test accuracy studies

A central phase in the evaluation of a medical test is the assessment of its diagnostic accuracy: the degree to which the test results correspond to those of the clinical reference standard. There are several methods by which the results of diagnostic accuracy studies can be summarized, reported, and interpreted. Here we provide an overview and a critical commentary of three measures: error-based measures, information-based measures, and measures of the strength of the association. All of these measures may vary between studies, with changes in the definition of the target condition, the spectrum of disease, the setting, and the amount of prior testing. We discuss the relativity of the claim that likelihood ratios are a superior way of expressing diagnostic accuracy, and defend the use of the sometimes downgraded sensitivity and specificity.

Systematic reviews of diagnostic test accuracy

More and more systematic reviews of diagnostic test accuracy studies are being published, but they can be methodologically challenging. In this paper, the authors present some of the recent developments in the methodology for conducting systematic reviews of diagnostic test accuracy studies. Restrictive electronic search filters are discouraged, as is the use of summary quality scores. Methods for meta-analysis should take into account the paired nature of the estimates and their dependence on threshold. Authors of these reviews are advised to use the hierarchical summary receiver-operating characteristic or the bivariate model for the data analysis. Challenges that remain are the poor reporting of original diagnostic test accuracy studies and difficulties with the interpretation of the results of diagnostic test accuracy research.

Bayesian clinical reasoning: does intuitive estimation of likelihood ratios on an ordinal scale outperform estimation of sensitivities and specificities?

RATIONALE

Bedside use of Bayes' theorem for estimating probabilities of diseases is cumbersome. An alternative approach based on five categories of powers of tests from 'useless' to 'very strong' has been proposed. The performance of clinicians using it was assessed.

METHODS

Fifty clinicians attending a course of tropical medicine estimated powers of tests and post-test probabilities using the classical vs. the categorical Bayesian approach. The estimation of post-test probability was assessed for real and dummy diseases in order to avoid the bias of previous knowledge. Accuracy of answers was measured by the difference with reference values obtained from an expert system (Kabisa).

RESULTS

Clinicians estimated positive likelihood ratios (LRs) a median of -1.07 log(10) lower than Kabisa [interquartile range (IQR): -1.47; -0.80] when derived classically and -0.17 (IQR: -0.42; +0.04) when estimated categorically (P < 0.001). For negative LRs the median was +0.39 log(10) higher (IQR: +0.71; +0.08) when derived classically and -0.18 log(10) lower (IQR: +0.03; -0.36) when estimated categorically (P < 0.001). Twenty (40%) disclosed not being able to calculate post-test probabilities using sensitivities and specificities. Regardless the approach post-test probabilities were overestimated both for real and dummy diseases [respectively +1.23 log(10) (IQR: +0,67; +2.08) and +2.03 log(10) (IQR: +0.49; +2.42)] (P = 0277), but the range was wider for the latter (P = 0.001).

CONCLUSIONS

Participants were more accurate in estimating powers with a categorical approach than with sensitivities and specificities. Post-test probabilities were overestimated with both approaches. Knowledge of the disease did not influence the estimation of post-test probabilities. A categorical approach might be an interesting instructional tool, but the effect of training with this approach needs assessment.

Cautionary tales in the clinical interpretation of studies of diagnostic tests

The use of investigational tests in making a diagnosis is a core activity of physicians and one that requires an understanding of the accuracy and usefulness of specific tests in discriminating between several diagnostic possibilities. Studies of diagnostic tests are frequently methodologically flawed and their results are often not well understood or applied in clinical practice. This article defines the performance characteristics of diagnostic tests, describes several commonly encountered deficiencies in study design which may invalidate reports of new diagnostic tests, and explains a Bayesian approach to interpreting test results in terms of disease probability.

We should overcome the barriers to evidence-based clinical diagnosis!

BACKGROUND

The wise integration of evidence from health care research into diagnostic decisions could influence patient outcomes by improving clinical diagnosis, reducing unnecessary testing, and minimizing diagnostic error. Yet for many, this promise does not match reality. Here, we collect and categorize barriers to the use of health care research evidence in diagnostic decisions, examine their potential consequences, and propose potential ways to overcome these impediments.

METHODS

Barriers were derived from observations over years of trying to inform clinical diagnoses with research evidence, and from interpretations of the literature.

RESULTS

Barriers are categorized into those related to the evidence itself, those related to diagnosticians, and those related to health care systems. Tentative solutions are proffered. Data are lacking on the frequency and impact of the identified barriers, as well as on the effectiveness of the proposed solutions.

CONCLUSIONS

Barriers to the sensible use of evidence from health care research in clinical diagnosis can be identified and categorized, and possible solutions can be imagined. We could, and should, muster the will to overcome these barriers.

A Gray Zone Assigned to Inconclusive Results of Quantitative Diagnostic Tests: Application to the Use of Brain Natriuretic Peptide for Diagnosis of Heart Failure in Acute Dyspneic Patients

Background: Most quantitative diagnostic tests do not perfectly differentiate between persons with and without a given disease. We present a simple method to construct a 3-zone partition for quantitative tests results, including positive and negative zones and a gray zone between, and we describe its use in the diagnosis of heart failure by brain natriuretic peptide (BNP) measurement in acute dyspneic patients.

Methods: We conducted a prospective cohort study of 699 consecutive patients with acute dyspnea who were treated at the emergency department of 3 participating hospitals. Heart failure (acute or decompensated) was assessed independently at discharge by cardiologists blind to the results of BNP measurements.

Results: The discriminatory performance of BNP was insufficient to provide a single cutoff value that could be used to correctly diagnose heart failure in clinical practice. Also, the discriminatory performance differed between patients with and without a history of chronic heart failure. The gray zone of inconclusive results was 167–472 ng/L for those without and 0–334 ng/L for those with such a history. Diagnosis of the current episode of heart failure by BNP results and history of heart failure was not enhanced by data from any other sources, including electrocardiography.

Conclusions: The gray zone approach applied to the diagnosis of heart failure by BNP might allow sensible cutoff values to be determined for clinical practice according to relevant subgroups of patients. The gray zone approach might be usefully applied to many other quantitative tests and clinical diagnostic or screening problems.

French academic physicians had a poor knowledge of terms used in clinical epidemiology

OBJECTIVES

To assess academic physicians' understanding and usage of basic epidemiological terms commonly used in medical journals.

STUDY DESIGN AND SETTING

Observational study. A total of 274 physicians, working in a teaching hospital in Paris, France were asked to answer a questionnaire including four vignettes presenting the results of a therapeutic, a diagnostic, a prognostic study and a meta-analysis of clinical trials.

RESULTS

A total of 130 (47%) questionnaires were returned. We observed the highest proportion of good answers for questions about absolute risk reduction (87.7%), sensitivity (84.6%), and specificity (80%); and the lowest for the calculation and use of the likelihood ratio (16.9% and 9.2%, respectively). The global mean score was 5.0/10 (95% confidence interval=4.6-5.4, range 0-9.4). Physicians got higher scores for questions related to treatment than for questions related to diagnosis: mean scores 7.1 (6.6-7.6) vs. 4.2 (3.8-4.6). Regression analysis did not reveal any significant relationship between global performance and physicians' age (r2=0.002, not significant [NS])

CONCLUSION

Physicians demonstrated only moderate knowledge and usage of clinical epidemiology terms used in major medical journals. Their capacity to interpret quantitative data from medical scientific literature may be limited.