Does prevalence matter to physicians in estimating post-test probability of disease? A randomized trial

A fresh look at the SarcoFluor antibody test for the detection of specific antibodies to Sarcocystis neurona for the diagnosis of equine protozoal myeloencephalitis

Equine protozoal myeloencephalitis (EPM) is a challenging disease to diagnose in horses with neurological signs. To optimize contemporary diagnostic testing, including the use of serum:CSF antibody ratios, the SarcoFluor antibody test for Sarcocystis neurona requires revalidation. The SarcoFluor, a previously validated immunofluorescent antibody test (IFAT) for the detection of antibodies specific to S. neurona in serum and cerebrospinal fluid (CSF) of naturally infected horses was analyzed using recent data and considering a serum:CSF antibody ratio threshold. Utilization of serum and CSF phosphorylated neurofilament heavy protein (pNfH) concentrations in support of an EPM diagnosis was also evaluated. 172 horses were divided into three groups: EPM-positive horses (EPM+, n=42), neurological non-EPM horses (n=74) confirmed with non-EPM neurological diseases (cervical vertebral compressive myelopathy, equine neuroaxonal dystrophy/equine degenerative myeloencephalopathy), and control horses (control, n=56) without neurological signs and neurological abnormalities on histology. Logistic regression was used to compare EPM diagnostic regimens. Specifically, EPM+ horses were compared with neurological non-EPM horses showing neurological signs. To consider diagnostic utility, post-test probabilities were calculated by titer. When differentiating between EPM and other neurological diseases, the combination of serum and CSF SarcoFluor testing added more information to the model accuracy than either test alone. Using serum and CSF for pNfH in support of an EPM diagnosis did not identify cutoffs with statistically significant odds ratios but increased the overall model accuracy when used with the IFAT. Utilization of IFAT titers against S. neurona in serum and CSF result in a high post-test probability of detecting EPM+ horses in a clinical setting.

Interpretation of results of PCR and B-D-glucan for the diagnosis of Pneumocystis Jirovecii Pneumonia in immunocompromised adults with acute respiratory failure

BACKGROUND

The accuracy of a diagnostic test depends on its intrinsic characteristics and the disease incidence. This study aims to depict post-test probability of Pneumocystis pneumonia (PJP), according to results of PCR and Beta-D-Glucan (BDG) tests in patients with acute respiratory failure (ARF).

MATERIALS AND METHODS

Diagnostic performance of PCR and BDG was extracted from literature. Incidence of Pneumocystis pneumonia was assessed in a dataset of 2243 non-HIV immunocompromised patients with ARF. Incidence of Pneumocystis pneumonia was simulated assuming a normal distribution in 5000 random incidence samples. Post-test probability was assessed using Bayes theorem.

RESULTS

Incidence of PJP in non-HIV ARF patients was 4.1% (95%CI 3.3-5). Supervised classification identified 4 subgroups of interest with incidence ranging from 2.0% (No ground glass opacities; 95%CI 1.4-2.8) to 20.2% (hematopoietic cell transplantation, ground glass opacities and no PJP prophylaxis; 95%CI 14.1-27.7). In the overall population, positive post-test probability was 32.9% (95%CI 31.1-34.8) and 22.8% (95%CI 21.5-24.3) for PCR and BDG, respectively. Negative post-test probability of being infected was 0.10% (95%CI 0.09-0.11) and 0.23% (95%CI 0.21-0.25) for PCR and BDG, respectively. In the highest risk subgroup, positive predictive value was 74.5% (95%CI 72.0-76.7) and 63.8% (95%CI 60.8-65.8) for PCR and BDG, respectively.

CONCLUSION

Although both tests yield a high intrinsic performance, the low incidence of PJP in this cohort resulted in a low positive post-test probability. We propose a method to illustrate pre and post-test probability relationship that may improve clinician perception of diagnostic test performance according to disease incidence in predefined clinical settings.

Assessing the knot in a noose position by thyrohyoid and cervical spine fracture patterns in suicidal hangings using machine learning algorithms: A new insight into old dilemmas

Hanging is one of the most common suicide methods worldwide. Neck injuries that occur upon such neck compression - fractures of the thyrohyoid complex and cervical spine, occupy forensic pathologists for a long time. However, research failed to identify particular patterns of these injuries corresponding to the force distribution a ligature applies to the neck: the issue of reconstructing the knot in a noose position persists. So far, machine learning (ML) models were not utilized to classify knot positions and reconstruct this event. We conducted a single-institutional, retrospective study on 1235 autopsy cases of suicidal hanging, developed several ML models, and assessed their classification performance in a stepwise manner to discriminate between: 1. typical ('posterior) and atypical ('anterior' and 'lateral') hangings, 2. anterior and lateral hangings, and 3. left and right lateral hangings. The variable coding was based on the presence/absence of fractures of greater hyoid bone horns (GHH), superior thyroid cartilage horns (STH), and cervical spine. Subject age was considered. The models' parameters were optimized by the Genetic Algorithm. The accuracy of ML models in the first step was very modest (c. 60%) but increased subsequently: Multilayer Perceptron - Artificial Neural Network and k-Nearest Neighbors performed excellently discriminating between left and right lateral hangings (accuracy 91.8% and 90.6%, respectively). The latter is of great importance for clarifying probable hanging fracture biomechanics. Alongside the conventional inferential statistical analysis we performed, our results further indicate the association of the knot position with ipsilateral GHH and contralateral STH fractures in lateral hangings. Moreover, odds for unilateral GHH fracture, simultaneous GHH and STH fractures, and cervical spine fracture were significantly higher in atypical ('anterior' and 'lateral') hangings, compared to typical ('posterior') hangings.

Family physicians overestimate diagnosis probabilities regardless of the test results

Introduction

As useful tools for clinical decision-making, diagnostic tests require careful interpretation in order to prevent underdiagnosis, overdiagnosis or misdiagnosis. The aim of this study was to explore primary care practitioners' understanding and interpretation of the probability of disease before and after test results for six common clinical scenarios.

Methods

This cross-sectional study was conducted with 414 family physicians who were working at primary care in Istanbul via face-to-face interviews held between November 2021 and March 2022. The participants were asked to estimate the probability of diagnosis in six clinical scenarios provided to them. Clinical scenarios were about three cancer screening cases (breast, cervical and colorectal), and three infectious disease cases (pneumonia, urinary tract infection, and COVID-19). For each scenario participants estimated the probability of the diagnosis before application of a diagnostic test, after a positive test result, and after a negative test result. Their estimates were compared with the true answers derived from relevant guidelines.

Results

For all scenarios, physicians' estimates were significantly higher than the scientific evidence range. The minimum overestimation was positive test result for COVID-19 and maximum was pre-test case for cervical cancer. In the hypothetical control question for prevalence and test accuracy, physicians estimated disease probability as 95.0% for a positive test result and 5.0% for a negative test result while the correct answers were 2.0 and 0%, respectively (p < 0.001).

Discussion

Comparing the scientific evidence, overestimation in all diagnostic scenarios, regardless of if the disease is an acute infection or a cancer, may indicate that the probabilistic approach is not conducted by the family physicians. To prevent inaccurate interpretation of the tests that may lead to incorrect or unnecessary treatments with adverse consequences, evidence-based decision-making capacity must be strengthened.

The incidence and factors of inappropriate rapid antigen test usage for group A streptococcus

INTRODUCTION

Although rapid antigen tests (RADTs) for group A streptococcus (GAS) can help diagnose group A streptococcal pharyngitis, little is known about the inappropriate use of these RADTs.

METHODS

This retrospective observational study compared the appropriate vs. inappropriate use of RADTs in patients who had a RADT between January 2019 and August 2022. RADTs for patients with a low Centor score of 0-1 point were deemed inappropriate.

RESULTS

Of the 1015 patients, 380 (37.4%) had inappropriate RADTs. Patients with asthma were associated with an increased risk of inappropriate testing. In contrast, during the coronavirus 2019 pandemic, outpatients and residents were associated with a reduced risk of inappropriate testing. Consequent to the inappropriate use of RADTs, 162 (16.0%) patients received potentially inappropriate antibiotics.

CONCLUSIONS

Our results suggest that diagnostic stewardship for pharyngitis, including education for healthcare workers, is needed to reduce inappropriate test ordering and prevent unnecessary care.

Bayesian Statistics to Estimate Diagnostic Probability of Scaphoid Fractures from Clinical Examinations: A Meta-Analysis

BACKGROUND

Management of suspected scaphoid fractures includes repeated evaluation and casting in symptomatic patients with nondiagnostic radiographs. In this systematic review and meta-analysis, the authors compare the diagnostic accuracy of clinical examinations for scaphoid fractures and create a decision guide using Bayesian statistics.

METHODS

The MEDLINE, Embase, and Cumulative Index to Nursing and Allied Health Literature databases were queried for studies that evaluated clinical index tests and their diagnostic accuracies for scaphoid fracture. Summary estimates were achieved by a bivariate random effects model and used in Bayes' theorem. The authors varied the scaphoid fracture prevalence for sensitivity analysis.

RESULTS

Fourteen articles with 22 index tests and 1940 patients were included. Anatomical snuffbox pain/tenderness (11 studies, 1363 patients), pain with axial loading (eight studies, 995 patients), and scaphoid tubercle tenderness (five studies, 953 patients) had sufficient data for pooled analysis. Anatomical snuffbox pain/tenderness was the most sensitive test (0.93; 95 percent CI, 0.87 to 0.97), and pain with axial loading was the most specific test (0.66; 95 percent CI, 0.41 to 0.85), but all three tests had lower estimated specificities compared with sensitivities. In the base case, the probability of fracture was approximately 60 percent when a patient presented with all three findings after acute wrist injury.

CONCLUSIONS

The posttest probability of scaphoid fracture was sensitive to both prevalence and diagnostic accuracy of individual clinical index tests. In a population with a fracture prevalence of 20 percent, patients presenting with concurrent anatomical snuffbox pain/tenderness, pain on axial loading, and scaphoid tubercle tenderness may benefit from early advanced imaging to rule out scaphoid fractures if initial radiographs are nondiagnostic.

CLINICAL QUESTION/LEVEL OF EVIDENCE

Diagnostic, II.

Clinical Prediction of Coronavirus Disease-2019: How Accurate Can One Be?

Background Some models based on clinical information have been reported to predict which patients have Coronavirus Disease-2019 (COVID-19) pneumonia but have failed so far to yield reliable results. We aimed to determine if physicians were able to accurately predict which patients, as described in clinical vignettes, had, or did not have this infection using their clinical acumen and epidemiological data. Methods Of 1177 patients under investigation for COVID-19 admitted, we selected 20 and presented them in a vignette form. We surveyed physicians from different levels of training (<5, and five or more years after graduation from medical school) and included non-medical participants as a control group. We asked all participants to predict the result of the PCR test for COVID-19. We measured the accuracy of responses as a whole, and at three stages of the pandemic associated with a growing incidence of COVID-19 in the community. We calculated the inter-rater reliability, sensitivity, and specificity of the clinical prediction as a whole and by pandemic stage.  Results Between June 8 and August 28, 2020, 82 doctors and 20 non-medical participants completed the survey. The accuracy was 58% (59% for doctors and 52% for non-medical, p=0.002). The lowest accuracy was noted for cases in the pandemic middle stage; years of post-graduate training represented no difference. Of the 2040 total answers, 1176 were accurate and 864 inaccurate (349 false positives and 515 false negatives). Conclusion The influence of symptomatic positivity, confirmation bias, and rapid expertise acquisition on accuracy is discussed, as the disease is new, time after graduation made no difference in the response accuracy. The limited clinical diagnostic capacity emphasizes the need for a reliable diagnostic test.

Critically ill benign EEG variants: Is there such a thing?

Despite growing use of critical care electroencephalography (ccEEG) to detect seizures and status epilepticus in the intensive care unit (ICU), integrating ccEEG findings with traditionally described benign EEG variants (BEVs) is a relatively new concept. BEV-like waveforms are now increasingly encountered in the ICU, and have also been explicitly included in proposed definitions of brief potentially ictal rhythmic discharges (BIRDs) in the ICU, bringing to the fore the question of if and which EEG patterns in critically ill patients can be safely deemed "benign". Though well-characterized as benign in healthy outpatients at low pre-test risk for neurologic disease, the significance of BEVs in the ICU remains largely unknown. Simultaneously, there has been mounting evidence to suggest that certain BEVs can arise from heterogeneous intracranial sources, including some pathologic generators. We conducted an extensive literature review on all known BEVs to assess what is known of BEVs in the ICU. Here we discuss critically ill BEVs and how to interpret them.

Measuring ACPA in the general population or primary care: is it useful?

Rheumatoid arthritis (RA) is associated with a significant disease burden and high costs for society. Because the disease has identifiable preclinical stages, screening and prevention have become a possibility in RA. Anticitrullinated peptide antibodies (ACPAs) are arguably the most likely candidate biomarker to screen for RA. This paper reviews the evidence for the use of ACPAs as a screening test in the broader general population, to identify individuals at high risk of subsequent onset of RA. We will review the diagnostic properties of the test and its positive and negative predictive value in different settings. We will discuss how ACPA testing could effectively be integrated in a broader screening strategy for RA.

Diagnostic performance of modern imaging instruments in glaucoma screening

AIM

To evaluate the applicability of imaging devices (spectral-domain optical coherence tomography (Cirrus SD-OCT), scanning laser polarimetry (GDx) and scanning laser ophthalmoscopy (Heidelberg Retinal Tomograph, HRT3)) for glaucoma screening in a middle-aged unselected population.

METHODS

Participants of the population-based Northern Finland Birth Cohort Eye Study, aged 45 to 49 years, underwent a comprehensive eye examination including modern imaging with five methods (retinal nerve fibre layer (RNFL) and macular ganglion cell layer +inner plexiform layer (GCIPL) analysis and their combination with SD-OCT, GDx and HRT). The performance of the automated classification of the imaging devices was assessed using a clinical glaucoma diagnosis as reference, that is, the '2 out of 3' rule based on the evaluation of optic nerve head and RNFL photographs and visual fields.

RESULTS

We examined 6060 eyes of 3039 subjects; in the clinical evaluation, glaucomatous damage was found in 33 subjects (1.1%) in 43 eyes. The following sensitivities were obtained; RNFL analysis (53%), GCIPL analysis (50%), OCT combination analysis (61%), GDx (56%) and HRT (31%) with corresponding specificities of 95%, 92%, 90%, 88% and 96%. The area under the curve values were 0.76, 0.73, 0.75, 0.75 and 0.73, respectively. Post-test probabilities of glaucoma after positive imaging finding with each of these methods in this unselected population were 11%, 7%, 6%, 5% and 7%, respectively.

CONCLUSION

Screening capabilities of the OCT, GDx and HRT were rather similar. The accuracy of all evaluated parameters was only moderate and thus screening with these parameters alone is not reliable.

Potential for diagnosis of infectious disease from the 100,000 Genomes Project Metagenomic Dataset: Recommendations for reporting results

The identification of microbiological infection is usually a diagnostic investigation, a complex process that is firstly initiated by clinical suspicion. With the emergence of high-throughput sequencing (HTS) technologies, metagenomic analysis has unveiled the power to identify microbial DNA/RNA from a diverse range of clinical samples (1). Metagenomic analysis of whole human genomes at the clinical/research interface bypasses the steps of clinical scrutiny and targeted testing and has the potential to generate unexpected findings relating to infectious and sometimes transmissible disease. There is no doubt that microbial findings that may have a significant impact on a patient’s treatment and their close contacts should be reported to those with clinical responsibility for the sample-donating patient. There are no clear recommendations on how such findings that are incidental, or outside the original investigation, should be handled. Here we aim to provide an informed protocol for the management of incidental microbial findings as part of the 100,000 Genomes Project which may have broader application in this emerging field. As with any other clinical information, we aim to prioritise the reporting of data that are most likely to be of benefit to the patient and their close contacts. We also set out to minimize risks, costs and potential anxiety associated with the reporting of results that are unlikely to be of clinical significance. Our recommendations aim to support the practice of microbial metagenomics by providing a simplified pathway that can be applied to reporting the identification of potential pathogens from metagenomic datasets. Given that the ambition for UK sequenced human genomes over the next 5 years has been set to reach 5 million and the field of metagenomics is rapidly evolving, the guidance will be regularly reviewed and will likely adapt over time as experience develops.

Value and Diagnostic Efficacy of Fetal Morphology Assessment Using Ultrasound in A Poor-Resource Setting

BACKGROUND

Ultrasound is operator-dependent, and its value and efficacy in fetal morphology assessment in a low-resource setting is poorly understood. We assessed the value and efficacy of fetal morphology ultrasound assessment in a Nigerian setting.

MATERIALS AND METHODS

We surveyed fetal morphology ultrasound performed across five facilities and followed-up each fetus to ascertain the outcome. Fetuses were surveyed in the second trimester (18th-22nd weeks) using the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG) guideline. Clinical and surgical reports were used as references to assess the diagnostic efficacy of ultrasound in livebirths, and autopsy reports to confirm anomalies in terminated pregnancies, spontaneous abortions, intrauterine fetal deaths, and still births. We calculated sensitivity, specificity, positive and negative predictive values, Area under the curve (AUC), Youden index, likelihood ratios, and post-test probabilities.

RESULTS

In total, 6520 fetuses of women aged 15-46 years (mean = 31.7 years) were surveyed. The overall sensitivity, specificity, and AUC were 77.1 (95% CI: 68-84.6), 99.5 (95% CI: 99.3-99.7), and 88.3 (95% CI: 83.7-92.2), respectively. Other performance metrics were: positive predictive value, 72.4 (95% CI: 64.7-79.0), negative predictive value, 99.6 (95% CI: 99.5-99.7), and Youden index (77.1%). Abnormality prevalence was 1.67% (95% CI: 1.37-2.01), and the positive and negative likelihood ratios were 254 (95% CI: 107.7-221.4) and 0.23 (95% CI: 0.16-0.33), respectively. The post-test probability for positive test was 72% (95% CI: 65-79).

CONCLUSION

Fetal morphology assessment is valuable in a poor economics setting, however, the variation in the diagnostic efficacy across facilities and the limitations associated with the detection of circulatory system anomalies need to be addressed.

Development and Validation of a Mucosal Impedance Contour Analysis System to Distinguish Esophageal Disorders

BACKGROUND & AIMS

Diagnostic testing for chronic esophageal disorders relies on histopathology analysis of biopsies or uncomfortable transnasal catheters or wireless pH monitoring, which capture abnormal intraluminal refluxate. We therefore developed a balloon mucosal impedance (MI) catheter system that instantly detects changes in esophageal mucosal integrity during endoscopy over a long segment of the esophagus. We performed a prospective study to evaluate the ability of a balloon-incorporated MI catheter to detect and evaluate esophageal disorders, including gastroesophageal reflux disease (GERD) and eosinophilic esophagitis (EoE).

METHODS

We performed a prospective study of 69 patients undergoing esophagogastroduodenoscopy with or without wireless pH monitoring. Patients were classified as having GERD (erosive esophagitis or abnormal pH; n = 24), EoE (confirmed with pathology analysis of tissues from both distal and proximal esophagus; n = 21), or non-GERD (normal results from esophagogastroduodenoscopy and pH tests; n = 24). Receiver operating characteristic curves and area under the operating characteristic curve (AUC) were used to compare the accuracy of balloon MI in diagnosis. Probabilities of assignment to each group (GERD, non-GERD, or EoE) were estimated using multinomial logistic regression. Association between MI patterns and diagnoses were validated using data from patients seen at 3 separate institutions.

RESULTS

MI pattern along the esophageal axis differed significantly (P < .01) among patients with GERD, EoE, and non-GERD. Patients with non-GERD had higher MI values along all measured segments. The MI pattern for GERD was easily distinguished from that of EoE: in patients with GERD, MI values were low in the distal esophagus and normalized along the proximal esophagus, whereas in patients with EoE, measurements were low in all segments of the esophagus. Intercept and rate of rise of MI value (slope) as distance increased from the squamocolumnar junction identified patients with GERD with an AUC = 0.67, patients with EoE with an AUC = 0.84, and patients with non-GERD with an AUC = 0.83 in the development cohort. One patient had an adverse event (reported mild chest pain after the procedure) and was discharged from the hospital without further events.

CONCLUSIONS

We developed a balloon MI catheter system that instantly detects changes in esophageal mucosal integrity during endoscopy and found it to be safe and able to identify patients with GERD, EoE, or non-GERD. We validated our findings in a separate cohort for patients. ClinicalTrials.gov ID NCT03103789.

Physician and Nonphysician Estimates of Positive Predictive Value in Diagnostic v. Mass Screening Mammography: An Examination of Bayesian Reasoning

BACKGROUND

The same test with the same result has different positive predictive values (PPVs) for people with different pretest probability of disease. Representative thinking theory suggests people are unlikely to realize this because they ignore or underweight prior beliefs when given new information (e.g., test results) or due to confusing test sensitivity (probability of positive test given disease) with PPV (probability of disease given positive test). This research examines whether physicians and MBAs intuitively know that PPV following positive mammography for an asymptomatic woman is less than PPV for a symptomatic woman and, if so, whether they correctly perceive the difference.

DESIGN

Sixty general practitioners (GPs) and 84 MBA students were given 2 vignettes of women with abnormal (positive) mammography tests: 1 with prior symptoms (diagnostic test), the other an asymptomatic woman participating in mass screening (screening test). Respondents estimated pretest and posttest probabilities. Sensitivity and specificity were neither provided nor elicited.

RESULTS

Eighty-eight percent of GPs and 46% of MBAs considered base rates and estimated PPV in diagnosis greater than PPV in screening. On average, GPs estimated a 27-point difference and MBAs an 18-point difference, compared to actual of 55 or more points. Ten percent of GPs and 46% of MBAs ignored base rates, incorrectly assessing the 2 PPVs as equal.

CONCLUSIONS

Physicians and patients are better at intuitive Bayesian reasoning than is suggested by studies that make test accuracy values readily available to be confused with PPV. However, MBAs and physicians interpret a positive in screening as more similar to a positive in diagnosis than it is, with nearly half of MBAs and some physicians wrongly equating the two. This has implications for overdiagnosis and overtreatment.

A comprehension scale for central-line associated bloodstream infection: Results of a preliminary survey and factor analysis

BACKGROUND

Central line-associated bloodstream infections (CLABSI) are associated with significant morbidity and mortality. This condition is therefore the focus of quality initiatives, which primarily use audit and feedback to improve performance. However, feedback of quality data inconsistently affects clinician behavior. A hypothesis for this inconsistency is that a lack of comprehension of CLABSI data by decision makers prevents behavior change. In order to rigorously test this hypothesis, a comprehension scale is necessary. Therefore, we sought to develop a scale to assess comprehension of CLABSI quality metric data.

METHODS

The initial instrument was constructed via an exploratory approach, including literature review and iterative item development. The developed instrument was administered to a sample of clinicians, and each item was scored dichotomously as correct or incorrect. Psychometric evaluation via exploratory factor analyses (using tetrachoric correlations) and Cronbach's alpha were used to assess dimensionality and internal consistency.

RESULTS

97 clinicians responded and were included. Factor analyses yielded a scale with one factor containing four items with an eigenvalue of 2.55 and a Cronbach's alpha of 0.82. The final solution was interpreted as an overall CLABSI "comprehension" scale given its unidimensionality and assessment of each piece of data within the CLABSI feedback report. The cohort had a mean performance on the scale of 49% correct (median = 50%).

CONCLUSIONS

We present the first psychometric evaluation of a preliminary scale that assesses clinician comprehension of CLABSI quality metric data. This scale has internal consistency, assesses clinically relevant concepts related to CLABSI comprehension, and is brief, which will assist in response rates. This scale has potential policy relevance as it could aid efforts to make quality metrics more effective in driving practice change.

Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

BACKGROUND

Effective study identification is essential for conducting health research, developing clinical guidance and health policy and supporting health-care decision-making. Methodological search filters (combinations of search terms to capture a specific study design) can assist in searching to achieve this.

OBJECTIVES

This project investigated the methods used to assess the performance of methodological search filters, the information that searchers require when choosing search filters and how that information could be better provided.

METHODS

Five literature reviews were undertaken in 2010/11: search filter development and testing; comparison of search filters; decision-making in choosing search filters; diagnostic test accuracy (DTA) study methods; and decision-making in choosing diagnostic tests. We conducted interviews and a questionnaire with experienced searchers to learn what information assists in the choice of search filters and how filters are used. These investigations informed the development of various approaches to gathering and reporting search filter performance data. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator.

RESULTS

The development of filters most frequently involved using a reference standard derived from hand-searching journals. Most filters were validated internally only. Reporting of methods was generally poor. Sensitivity, precision and specificity were the most commonly reported performance measures and were presented in tables. Aspects of DTA study methods are applicable to search filters, particularly in the development of the reference standard. There is limited evidence on how clinicians choose between diagnostic tests. No published literature was found on how searchers select filters. Interviewing and questioning searchers via a questionnaire found that filters were not appropriate for all tasks but were predominantly used to reduce large numbers of retrieved records and to introduce focus. The Inter Technology Appraisal Support Collaboration (InterTASC) Information Specialists' Sub-Group (ISSG) Search Filters Resource was most frequently mentioned by both groups as the resource consulted to select a filter. Randomised controlled trial (RCT) and systematic review filters, in particular the Cochrane RCT and the McMaster Hedges filters, were most frequently mentioned. The majority indicated that they used different filters depending on the requirement for sensitivity or precision. Over half of the respondents used the filters available in databases. Interviewees used various approaches when using and adapting search filters. Respondents suggested that the main factors that would make choosing a filter easier were the availability of critical appraisals and more detailed performance information. Provenance and having the filter available in a central storage location were also important.

LIMITATIONS

The questionnaire could have been shorter and could have included more multiple choice questions, and the reviews of filter performance focused on only four study designs.

CONCLUSIONS

Search filter studies should use a representative reference standard and explicitly report methods and results. Performance measures should be presented systematically and clearly. Searchers find filters useful in certain circumstances but expressed a need for more user-friendly performance information to aid filter choice. We suggest approaches to use, adapt and report search filter performance. Future work could include research around search filters and performance measures for study designs not addressed here, exploration of alternative methods of displaying performance results and numerical synthesis of performance comparison results.

FUNDING

The National Institute for Health Research (NIHR) Health Technology Assessment programme and Medical Research Council-NIHR Methodology Research Programme (grant number G0901496).

Review of the use of pretest probability for molecular testing in non-small cell lung cancer and overview of new mutations that may affect clinical practice

This article considers the use of pretest probability in non-small cell lung cancer (NSCLC) and how its use in EGFR testing has helped establish clinical guidelines on selecting patients for EGFR testing. With an ever-increasing number of molecular abnormalities being identified and often limited tissue available for testing, the use of pretest probability will need to be increasingly considered in the future for selecting investigations and treatments in patients. In addition we review new mutations that have the potential to affect clinical practice.

Do Clinicians Understand Quality Metric Data? An Evaluation in a Twitter-Derived Sample

OBJECTIVE

Despite significant efforts and cost, quality metrics do not consistently influence practice. While research has focused on improving data through statistical risk-adjustment, whether clinicians understand these data is unknown. Therefore, we assessed clinician comprehension of central line-associated blood stream infection (CLABSI) quality metric data.

DESIGN

Cross-sectional survey with an 11-item test of CLABSI data comprehension. Each question assessed 1 of 3 concepts concerning CLABSI understanding: basic numeracy, risk-adjustment numeracy, and risk-adjustment interpretation. Hypothetical data were used and presented in a validated format.

PARTICIPANTS

Clinicians were recruited from 6 nations via Twitter to take an online survey. Clinician eligibility was confirmed by assessing responses to a question regarding CLABSI.

MAIN MEASURES

The primary outcome was percent correct of attempted questions pertaining to the presented CLABSI data.

RESULTS

Ninety-seven clinicians answered at least 1 item, providing 939 responses; 72 answered all 11 items. The mean percentage of correct answers was 61% (95% confidence interval [CI], 57%-65%). Overall, doctor performance was better than performance by nurses and other respondents (68% [95% CI, 63%-73%] vs. 57% [95% CI, 52%-62%], P = 0.003). In basic numeracy, mean percent correct was 82% (95% CI, 77%-87%). For risk-adjustment numeracy, the mean percent correct was 70% (95% CI, 64%-76%). Risk-adjustment interpretation had the lowest average percent correct, 43% (95% CI, 37%-49%). All pairwise differences between concepts were statistically significant at P <0.05.

CONCLUSIONS

CLABSI quality metric comprehension appears low and varies substantially among clinicians. These findings may contribute to the limited impact of quality metric reporting programs, and further research is needed. Journal of Hospital Medicine 2017;12:18-22.

Bayesian reasoning in residents' preliminary diagnoses

Whether and when humans in general, and physicians in particular, use their beliefs about base rates in Bayesian reasoning tasks is a long-standing question. Unfortunately, previous research on whether doctors use their beliefs about the prevalence of diseases in diagnostic judgments has critical limitations. In this study, we assessed whether residents' beliefs about the prevalence of a disease are associated with their judgments of the likelihood of the disease in diagnosis, and whether residents' beliefs about the prevalence of diseases change across the 3 years of residency. Residents were presented with five ambiguous vignettes typical of patients presenting on the inpatient general medicine services. For each vignette, the residents judged the likelihood of five or six possible diagnoses. Afterward, they judged the prevalence within the general medicine services of all the diseases in the vignettes. Most importantly, residents who believed a disease to be more prevalent tended to rate the disease as more likely in the vignette cases, suggesting a rational tendency to incorporate their beliefs about disease prevalence into their diagnostic likelihood judgments. In addition, the residents' prevalence judgments for each disease were assessed over the 3 years of residency. The precision of the prevalence estimates increased across the 3 years of residency, though the accuracy of the prevalence estimates did not. These results imply that residents do have a rational tendency to use prevalence beliefs for diagnosis, and this finding also contributes to a larger question of whether humans intuitively use base rates for making judgments.

How well do health professionals interpret diagnostic information? A systematic review

OBJECTIVE

To evaluate whether clinicians differ in how they evaluate and interpret diagnostic test information.

DESIGN

Systematic review.

DATA SOURCES

MEDLINE, EMBASE and PsycINFO from inception to September 2013; bibliographies of retrieved studies, experts and citation search of key included studies.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Primary studies that provided information on the accuracy of any diagnostic test (eg, sensitivity, specificity, likelihood ratios) to health professionals and that reported outcomes relating to their understanding of information on or implications of test accuracy.

RESULTS

We included 24 studies. 6 assessed ability to define accuracy metrics: health professionals were less likely to identify the correct definition of likelihood ratios than of sensitivity and specificity. -25 studies assessed Bayesian reasoning. Most assessed the influence of a positive test result on the probability of disease: they generally found health professionals' estimation of post-test probability to be poor, with a tendency to overestimation. 3 studies found that approaches based on likelihood ratios resulted in more accurate estimates of post-test probability than approaches based on estimates of sensitivity and specificity alone, while 3 found less accurate estimates. 5 studies found that presenting natural frequencies rather than probabilities improved post-test probability estimation and speed of calculations.

CONCLUSIONS

Commonly used measures of test accuracy are poorly understood by health professionals. Reporting test accuracy using natural frequencies and visual aids may facilitate improved understanding and better estimation of the post-test probability of disease.

Presentation of Diagnostic Information to Doctors May Change Their Interpretation and Clinical Management: A Web-Based Randomised Controlled Trial

BACKGROUND

There is little evidence on how best to present diagnostic information to doctors and whether this makes any difference to clinical management. We undertook a randomised controlled trial to see if different data presentations altered clinicians' decision to further investigate or treat a patient with a fictitious disorder ("Green syndrome") and their ability to determine post-test probability.

METHODS

We recruited doctors registered with the United Kingdom's largest online network for medical doctors between 10 July and 6" November 2012. Participants were randomised to one of four arms: (a) text summary of sensitivity and specificity, (b) Fagan's nomogram, (c) probability-modifying plot (PMP), (d) natural frequency tree (NFT). The main outcome measure was the decision whether to treat, not treat or undertake a brain biopsy on the hypothetical patient and the correct post-test probability. Secondary outcome measures included knowledge of diagnostic tests.

RESULTS

917 participants attempted the survey and complete data were available from 874 (95.3%). Doctors randomized to the PMP and NFT arms were more likely to treat the patient than those randomized to the text-only arm. (ORs 1.49, 95% CI 1.02, 2.16) and 1.43, 95% CI 0.98, 2.08 respectively). More patients randomized to the PMP (87/218-39.9%) and NFT (73/207-35.3%) arms than the nomogram (50/194-25.8%) or text only (30/255-11.8%) arms reported the correct post-test probability (p <0.001). Younger age, postgraduate training and higher self-rated confidence all predicted better knowledge performance. Doctors with better knowledge were more likely to view an optional learning tutorial (OR per correct answer 1.18, 95% CI 1.06, 1.31).

CONCLUSIONS

Presenting diagnostic data using a probability-modifying plot or natural frequency tree influences the threshold for treatment and improves interpretation of tests results compared to text summary of sensitivity and specificity or Fagan's nomogram.

Multi-centered investigation of a point-of-care NT-proBNP ELISA assay to detect moderate to severe occult (pre-clinical) feline heart disease in cats referred for cardiac evaluation

OBJECTIVE

To prospectively evaluate the diagnostic accuracy of a point-of-care (POC) N-terminal pro-B-type natriuretic peptide (NT-proBNP) ELISA to assess the likelihood of moderate to severe occult heart disease (OcHD) in a clinical population of cats suspected to have heart disease.

ANIMALS

One hundred and forty-six asymptomatic client-owned cats with a heart murmur, gallop rhythm, arrhythmia, or cardiomegaly.

METHODS

Physical examination, blood pressure measurement and echocardiography were performed prospectively. Point-of-care ELISA was visually assessed as either positive or negative by a reader blinded to the echocardiographic results.

RESULTS

Forty-three healthy cats, 50 mild OcHD, 31 moderate OcHD, 6 severe OcHD, and 16 cats equivocal for OcHD were examined. Cats with OcHD included 65 with hypertrophic cardiomyopathy, 6 with restrictive or unclassified cardiomyopathy, 1 with arrhythmogenic right ventricular cardiomyopathy, and 15 with non-cardiomyopathic forms of heart disease. Point-of-care ELISA differentiated cats with moderate or severe OcHD with sensitivity/specificity of 83.8%/82.6% and overall accuracy of 82.9%. Positive POC ELISA increased likelihood of moderate or severe OcHD by a factor of 4.8 vs. those that tested negative. Point-of-care ELISA differentiated cats with moderate or severe cardiomyopathic OcHD with sensitivity/specificity of 88.6%/81.3% and overall accuracy of 83.2%.

CONCLUSION

In a select sample of cats referred for cardiac evaluation, positive POC NT-proBNP ELISA increases likelihood of moderate to severe OcHD while negative POC NT-proBNP ELISA result excludes moderate to severe OcHD.

Twelve tips for teaching evidence-based physical examination

BACKGROUND

Practicing evidence-based physical examination (EBPE) requires clinicians to apply the diagnostic accuracy of PE findings in relation to a suspected disease. Though it is important to effectively teach EBPE, clinicians often find the topic challenging.

AIMS

There are few resources available to guide clinicians on strategies to teach EBPE. We seek to fill that need by presenting tips for effectively teaching EBPE in the clinical context.

METHODS

This report is based primarily on the authors' experience and is supported by the available literature.

RESULTS

We present 12 practical tips targeting the clinician educator. The first six tips condense key preparatory steps for the teacher, including basic statistics underpinning EBPE. The final six tips provide specific guidance on how to teach EBPE in the clinical environment.

CONCLUSIONS

By practicing the 12 tips provided, clinicians will develop the confidence needed to effectively teach EBPE in inpatient or outpatient settings.

A high resolution computer tomography scoring system to predict culture-positive pulmonary tuberculosis in the emergency department

This study evaluated the use of high-resolution computed tomography (HRCT) to predict the presence of culture-positive pulmonary tuberculosis (PTB) in adult patients with pulmonary lesions in the emergency department (ED). The study included a derivation phase and validation phase with a total of 8,245 patients with pulmonary disease. There were 132 patients with culture-positive PTB in the derivation phase and 147 patients with culture-positive PTB in the validation phase. Imaging evaluation of pulmonary lesions included morphology and segmental distribution. The post-test probability ratios between both phases in three prevalence areas were analyzed. In the derivation phase, a multivariate analysis model identified cavitation, consolidation, and clusters/nodules in right or left upper lobe (except anterior segment) and consolidation of the superior segment of the right or left lower lobe as independent positive factors for culture-positive PTB, while consolidation of the right or left lower lobe (except superior segment) were independent negative factors. An ideal cutoff point based on the receiver operating characteristic (ROC) curve analysis was obtained at a score of 1. The sensitivity, specificity, positivity predictive value, and negative predictive value from derivation phase were 98.5% (130/132), 99.7% (3997/4008), 92.2% (130/141), and 99.9% (3997/3999). Based on the predicted positive likelihood ratio value of 328.33 in derivation phase, the post-test probability was observed to be 91.5% in the derivation phase, 92.5% in the validation phase, 94.5% in a high TB prevalence area, 91.0% in a moderate prevalence area, and 76.8% in moderate-to-low prevalence area. Our model using HRCT, which is feasible to perform in the ED, can promptly diagnose culture-positive PTB in moderate and moderate-to-low prevalence areas.

Effect of numbering of return envelopes on participation, explicit refusals, and bias: experiment and meta-analysis

BACKGROUND

Tracing mail survey responses is useful for the management of reminders but may cause concerns about anonymity among prospective participants. We examined the impact of numbering return envelopes on the participation and the results of a survey on a sensitive topic among hospital staff.

METHODS

In a survey about regrets associated with providing healthcare conducted among hospital-based doctors and nurses, two randomly drawn subsamples were provided numbered (N = 1100) and non-numbered (N = 500) envelopes for the return of completed questionnaires. Participation, explicit refusals, and item responses were compared. We also conducted a meta-analysis of the effect of questionnaire/envelope numbering on participation in health surveys.

RESULTS

The participation rate was lower in the "numbered" group than in the "non-numbered" group (30.3% vs. 35.0%, p = 0.073), the proportion of explicit refusals was higher in the "numbered" group (23.1% vs 17.5%, p = 0.016), and the proportion of those who never returned the questionnaire was similar (46.6% vs 47.5%, p = 0.78). The means of responses differed significantly for 12 of 105 items (11.4%), which did not differ significantly from the expected frequency of type 1 errors, i.e., 5% (permutation test, p = 0.078). The meta-analysis of 7 experimental surveys (including this one) indicated that numbering is associated with a 2.4% decrease in the survey response rate (95% confidence interval 0.3% to 4.4%).

CONCLUSIONS

Numbered return envelopes may reduce the response rate and increase explicit refusals to participate in a sensitive survey. Reduced participation was confirmed by a meta-analysis of randomized health surveys. There was no strong evidence of bias.

Comparative performance of electrochemiluminescence immunoassay and EIA for HIV screening in a multiethnic region of China

Background

The recent approval of 4th generation HIV tests has forced many laboratories to decide whether to shift from 3rd to these tests. There are limited published studies on the comparative evaluation of these two different assays. We compare the performance of fourth-generation electrochemiluminescence immunoassay (ChIA) and third-generation enzyme linked immunosorbent assay (EIA) for human immunodeficiency virus (HIV) screening and gauge whether the shift from EIA to ChIA could be better in a multiethnic region of China.

Methodology/Principal Findings

We identified a large number of routine specimens (345,492) using two different assays from Jan 2008 to Aug 2011 in a teaching hospital with high sample throughput. Of the 344,596 specimens with interpretable HIV test results, 526(0.23%) of 228,761 using EIA and 303(0.26%) of 115,835 using ChIA were HIV-1 positive. The false-positive rate of EIA was lower than that of ChIA [0.03% vs. 0.08%, odds ratio 0.33 (95% confidence interval 0.24, 0.45)]. The positive predictive value (PPV) of EIA (89.6%) was significantly higher than that of ChIA (76.1%) (<0.001), reflecting the difference between the two assays. The clinical sensitivities of two assays in this study were 99.64% for EIA and 99.88% for ChIA.

Conclusion

Caution is needed before shifting from 3rd to 4th generation HIV tests. Since none of these tests are perfect, different geographic and ethnic area probably require different considerations with regard to HIV testing methods, taking into account the local conditions.

Doctors and patients' susceptibility to framing bias: a randomized trial

BACKGROUND

Framing of risk influences the perceptions of treatment benefit.

OBJECTIVE

To determine which risk framing format corresponds best to comprehensive multi-faceted information, and to compare framing bias in doctors and in patients.

DESIGN

Randomized mail surveys.

PARTICIPANTS

One thousand four hundred and thirty-one doctors (56% response rate) and 1121 recently hospitalized patients (65% response rate).

INTERVENTION

Respondents were asked to interpret the results of a hypothetical clinical trial comparing an old and a new drug. They were randomly assigned to the following framing formats: absolute survival (new drug: 96% versus old drug: 94%), absolute mortality (4% versus 6%), relative mortality reduction (reduction by a third) or all three (fully informed condition). The new drug was reported to cause more side-effects.

MAIN MEASURE

Rating of the new drug as more effective than the old drug.

RESULTS

The proportions of doctors who rated the new drug as more effective varied by risk presentation format (abolute survival 51.8%, absolute mortality 68.3%, relative mortality reduction 93.8%, and fully informed condition 69.8%, p < 0.001). In patients these proportions were similar (abolute survival 51.7%, absolute mortality 66.8%, relative mortality reduction 89.3%, and fully informed condition 71.2%, p < 0.001). In both doctors (p = 0.72) and patients (p = 0.23) the fully informed condition was similar to the absolute risk format, but it differed significantly from the other conditions (all p < 0.01). None of the differences between doctors and patients were significant (all p > 0.1). In comparison to the fully informed condition, the odds ratio of greater perceived effectiveness was 0.45 for absolute survival (p < 0.001), 0.89 for absolute mortality (p = 0.29), and 4.40 for relative mortality reduction (p < 0.001).

CONCLUSIONS

Framing bias affects doctors and patients similarly. Describing clinical trial results as absolute risks is the least biased format, for both doctors and patients. Presenting several risk formats (on both absolute and relative scales) should be encouraged.