Intermediate, indeterminate, and uninterpretable diagnostic test results

A trivariate meta-analysis of diagnostic studies accounting for prevalence and non-evaluable subjects: re-evaluation of the meta-analysis of coronary CT angiography studies

BACKGROUND

A recent paper proposed an intent-to-diagnose approach to handle non-evaluable index test results and discussed several alternative approaches, with an application to the meta-analysis of coronary CT angiography diagnostic accuracy studies. However, no simulation studies have been conducted to test the performance of the methods.

METHODS

We propose an extended trivariate generalized linear mixed model (TGLMM) to handle non-evaluable index test results. The performance of the intent-to-diagnose approach, the alternative approaches and the extended TGLMM approach is examined by extensive simulation studies. The meta-analysis of coronary CT angiography diagnostic accuracy studies is re-evaluated by the extended TGLMM.

RESULTS

Simulation studies showed that the intent-to-diagnose approach under-estimate sensitivity and specificity. Under the missing at random (MAR) assumption, the TGLMM gives nearly unbiased estimates of test accuracy indices and disease prevalence. After applying the TGLMM approach to re-evaluate the coronary CT angiography meta-analysis, overall median sensitivity is 0.98 (0.967, 0.993), specificity is 0.875 (0.827, 0.923) and disease prevalence is 0.478 (0.379, 0.577).

CONCLUSIONS

Under MAR assumption, the intent-to-diagnose approach under-estimate both sensitivity and specificity, while the extended TGLMM gives nearly unbiased estimates of sensitivity, specificity and prevalence. We recommend the extended TGLMM to handle non-evaluable index test subjects.

Understanding the direction of bias in studies of diagnostic test accuracy

Ordering and interpreting diagnostic tests is a critical part of emergency medicine (EM). In evaluating a study of diagnostic test accuracy, emergency physicians (EPs) need to recognize whether the study uses case-control or cross-sectional sampling and account for common biases. The authors group biases in studies of test accuracy into five categories: incorporation bias, partial verification bias, differential verification bias, imperfect gold standard bias, and spectrum bias. Other named biases are either equivalent to these biases or subtypes within these broader categories. The authors go beyond identifying a bias and predict the direction of its effect on sensitivity and specificity, providing numerical examples from published test accuracy studies. Understanding the direction of a bias may permit useful inferences from even a flawed study of test accuracy.

Comparison of PET-CT and conventional imaging in staging pediatric rhabdomyosarcoma

BACKGROUND

Over the past decade, PET-CT has been used to assess rhabdomyosarcoma (RMS) in children. However, the role of PET-CT in staging RMS is unknown.

PROCEDURE

Thirty subjects with RMS, median age 7.3 years, underwent PET-CT before therapy. PET-CTs and conventional imaging (CI) were independently reviewed by two radiologists and two nuclear medicine physicians to determine the presence of metastases. Accuracy, sensitivity, and specificity of PET-CT for detecting metastases were compared to CI using biopsy and clinical follow-up as reference standards. Maximum standardized uptake values (SUV(max)) of primary tumors, lymph nodes, and pulmonary nodules were measured.

RESULTS

Primary tumors had an average SUV(max) of 7.2 (range, 2.5-19.2). Accuracy rates for 17 subjects with nodal disease were 95% for PET-CT and 49% for CI. PET-CT had 94% sensitivity and 100% specificity for nodal disease. Of seven pulmonary nodules detected by CI, three were not identified by PET-CT, two were indeterminate, and one was malignant with a SUV(max) (3.4) > twice that of benign nodules. Two subjects had bone disease; both were identified by PET-CT but only one by CI. Four subjects had bone marrow disease, two had positive PET-CTs but none had positive CI. Two subjects had soft tissue metastases detected by PET-CT but not CI.

CONCLUSIONS

PET-CT performed better than CI in identifying nodal, bone, bone marrow, and soft tissue disease in children with RMS. CI remains essential for detection of pulmonary nodules. We recommend PET-CT for staging of children with RMS. CI with Tc(99m) bone scan can be eliminated.

Stage IB1 cervical cancer: role of preoperative MR imaging in selection of patients for fertility-sparing radical trachelectomy

PURPOSE

To determine whether magnetic resonance (MR) imaging evaluation of key morphologic tumor characteristics can improve patient selection for radical trachelectomy.

MATERIALS AND METHODS

The institutional review board approved and waived informed consent for this study of 62 patients (mean age, 32 years; age range, 23-42 years) with International Federation of Gynecology and Obstetrics stage IB1 cervical carcinoma who underwent attempted radical trachelectomy between November 2001 and January 2011 and had preoperative MR imaging. Retrospectively, two radiologists reviewed MR images for tumor presence and size, distance between tumor and internal os, and presence of deep cervical stromal invasion. Associations between MR imaging findings and surgery type were tested.

RESULTS

Sensitivity and specificity of tumor detection were, respectively, 87% and 100% (reader 1) and 76% and 95% (reader 2). Six of six patients with negative cone biopsy margins and no tumor at postconization MR imaging were without tumor at trachelectomy pathologic analysis. Mean differences between MR imaging and histologic tumor sizes were 0.7 mm (range, -15 to 11 mm) for reader 1 and 2.2 mm (range, -9 to 15 mm) for reader 2. Sensitivities for deep cervical stromal invasion were 75% (reader 1) and 50% (reader 2). For each reader, nine of nine (100%) patients with tumor 5 mm or less from the internal os and three of five (60%) patients with tumor 6-9 mm from the internal os at MR imaging needed radical hysterectomy. For both readers, tumor size of 2 cm or larger (P < .001) and deep cervical stromal invasion (P ≤ .003) at MR imaging were associated with increased chance of radical hysterectomy.

CONCLUSION

Pretrachelectomy MR imaging can help identify high-risk patients likely to need radical hysterectomy or confirm the absence of residual tumor in the cervix after a cone biopsy with negative margins.

Psychometric properties and osteoprotective behaviors among type 2 diabetic patients: osteoporosis self-efficacy scale Malay version (OSES-M)

In type 2 diabetic patients (T2DM), only 22 % have normal bone mineral density and almost three quarters of the sample population had low self-efficacy towards osteoporosis. These results reflect the need for screening and educational programs to increase the awareness of T2DM towards osteoporosis.

INTRODUCTION

Our aim was to translate and examine the psychometric properties of the Malay version of the osteoporosis self-efficacy scale (OSES-M) among T2DM and to determine the best cut-off value with optimum sensitivity and specificity. In addition, to assess factors that affects diabetic patients' osteoporosis self-efficacy.

METHODS

A standard "forward-backward" procedure was used to translate the OSES into Malay language, which was then validated with a convenience sample of 250 T2DM. The sensitivity and specificity of the OSES-M was calculated using receiver operating characteristic curve analysis. Bivariate and multivariate approaches were used to examine multiple independent variables on each dependent variable.

RESULTS

The mean score of OSES-M was 731.74 ± 197.15. Fleiss' kappa, content validity ratio range, and content validity index were 0.99, 0.75-1, and 0.96, respectively. Two factors were extracted from exploratory factor analysis and were confirmed through confirmatory factor analysis. Internal consistency and test-retest reliability were 0.92 and 0.86, respectively. The optimum cut-off point of OSES-M to predict osteoporosis/osteopenia was 858. Regression analysis revealed that knowledge, health belief, and some demographic data had an impact on OSES-M.

CONCLUSIONS

The results show that the OSES-M is a reliable and valid instrument for measuring osteoporosis self-efficacy in the Malaysian clinical setting.

Evaluating accuracy of diagnostic tests with intermediate results in the absence of a gold standard

Intermediate test results often occur with diagnostic tests. When assessing diagnostic accuracy, it is important to properly report and account for these results. In the literature, these results are commonly discarded prior to analysis or treated as either a positive or a negative result. Although such adjustments allow sensitivity and specificity to be computed in the standard way, these forced decisions limit the interpretability and usefulness of the results. Estimation of diagnostic accuracy is further complicated when tests are evaluated without a gold standard. Although traditional latent class modeling can be readily applied to analyze these data and account for intermediate results, these models assume that tests are independent conditional on the true disease status, which is rarely valid in practice. We extend both the log-linear latent class model and the probit latent class model to accommodate the conditional dependence among tests while taking the intermediate results into consideration. We illustrate our methods using a simulation study and a published medical study on the detection of epileptiform activity in the brain.

Use of 3x2 tables with an intention to diagnose approach to assess clinical performance of diagnostic tests: meta-analytical evaluation of coronary CT angiography studies

OBJECTIVE

To determine whether a 3 × 2 table, using an intention to diagnose approach, is better than the "classic" 2 × 2 table at handling transparent reporting and non-evaluable results, when assessing the accuracy of a diagnostic test.

DESIGN

Based on a systematic search for diagnostic accuracy studies of coronary computed tomography (CT) angiography, full texts of relevant studies were evaluated to determine whether they could calculate an alternative 3 × 2 table. To quantify an overall effect, we pooled diagnostic accuracy values according to a meta-analytical approach.

DATA SOURCES

Medline (via PubMed), Embase (via Ovid), and ISI Web of Science electronic databases.

ELIGIBILITY CRITERIA

Prospective English or German language studies comparing coronary CT with conventional coronary angiography in all patients and providing sufficient data for a patient level analysis.

RESULTS

120 studies (10,287 patients) were eligible. Studies varied greatly in their approaches to handling non-evaluable findings. We found 26 studies (including 2298 patients) that allowed us to calculate both 2 × 2 tables and 3 × 2 tables. Using a bivariate random effects model, we compared the 2 × 2 table with the 3 × 2 table, and found significant differences for pooled sensitivity (98.2 (95% confidence interval 96.7 to 99.1) v 92.7 (88.5 to 95.3)), area under the curve (0.99 (0.98 to 1.00) v 0.93 (0.91 to 0.95)), positive likelihood ratio (9.1 (6.2 to 13.3) v 4.4 (3.3 to 6.0)), and negative likelihood ratio (0.02 (0.01 to 0.04) v 0.09 (0.06 to 0.15); (P<0.05)).

CONCLUSION

Parameters for diagnostic performance significantly decrease if non-evaluable results are included by a 3 × 2 table for analysis (intention to diagnose approach). This approach provides a more realistic picture of the clinical potential of diagnostic tests.

Point-of-care urine tests for smoking status and isoniazid treatment monitoring in adult patients

BACKGROUND

Poor adherence to isoniazid (INH) preventive therapy (IPT) is an impediment to effective control of latent tuberculosis (TB) infection. TB patients who smoke are at higher risk of latent TB infection, active disease, and TB mortality, and may have lower adherence to their TB medications. The objective of our study was to validate IsoScreen and SmokeScreen (GFC Diagnostics, UK), two point-of-care tests for monitoring INH intake and determining smoking status. The tests could be used together in the same individual to help identify patients with a high-risk profile and provide a tailored treatment plan that includes medication management, adherence interventions, and smoking cessation programs.

METHODOLOGY/PRINCIPAL FINDINGS

200 adult outpatients attending the TB and/or the smoking cessation clinic were recruited at the Montreal Chest Institute. Sensitivity and specificity were measured for each test against the corresponding composite reference standard. Test reliability was measured using kappa statistic for intra-rater and inter-rater agreement. Univariate and multivariate logistic regression models were used to explore possible covariates that might be related to false-positive and false-negative test results. IsoScreen had a sensitivity of 93.2% (95% confidence interval [CI] 80.3, 98.2) and specificity of 98.7% (94.8, 99.8). IsoScreen had intra-rater agreement (kappa) of 0.75 (0.48, 0.94) and inter-rater agreement of 0.61 (0.27, 0.90). SmokeScreen had a sensitivity of 69.2% (56.4, 79.8), specificity of 81.6% (73.0, 88.0), intra-rater agreement of 0.77 (0.56, 0.94), and inter-rater agreement of 0.66 (0.42, 0.88). False-positive SmokeScreen tests were strongly associated with INH treatment.

CONCLUSIONS

IsoScreen had high validity and reliability, whereas SmokeScreen had modest validity and reliability. SmokeScreen tests did not perform well in a population receiving INH due to the association between INH treatment and false-positive SmokeScreen test results. Development of the next generation SmokeScreen assay should account for this potential interference.

Intra- and interlaboratory reproducibility of an ELISA serological test for Lyme disease

OBJECTIVES

Lyme disease has been increasingly diagnosed throughout North America since the late 1970s. The clinical diagnosis and epidemiological monitoring of Lyme disease are aided by serological testing for the etiological agent, Borrelia burgdorferi. Numerous authorities have questioned the reproducibility of these serological tests. This study assessed the intra- and interlaboratory reproducibility of an elisa used to aid in the diagnosis of Lyme disease.

METHODS

Twenty-seven sera from cases and noncases were tested by three laboratories. Two of the laboratories repeated the tests once. These testings were part of the 1991 quality control assessment of provincial laboratories carried out by the Laboratory Centre for Disease Control (lcdc), Ottawa.

RESULTS

The mean weighted kappa statistics were 0.87 for interlaboratory comparisons and 0.89 for intralaboratory comparisons.

CONCLUSIONS

Overall, the elisa assessed in this study demonstrated good to excellent intra- and interlaboratory reproducibility in the lcdc 1991 quality control assessment when the data were assessed in the categorical scale using the weighted kappa statistic. Generalization of these findings to clinical laboratory settings must be done with caution.

Beyond SpPIN and SnNOUT: Considerations with Dichotomous Tests During Assessment of Diagnostic Accuracy

Paramount to efficient and effective care is the determination of an accurate diagnosis that leads to the proper referral and/or intervention. In an effort to improve the clinical utility of diagnostic accuracy calculations, researchers have promoted the use of the mnemonics SpPIn (if specificity is high, a positive test rules in pathology) and SnNOut (if sensitivity is high, a negative test rules out pathology). Using examples from diagnostic accuracy studies and a review of pertinent literature, this clinimetrics corner outlines additional considerations for clinicians when consuming research in this area. The paper has three foci. First, sensitivity, specificity, and other estimates of the diagnostic accuracy of dichotomous physical examination tests should be viewed as estimates with confidence when those estimates are expressed as confidence intervals. Second, appropriate power must be considered when evaluating each study. Last, the quality of a diagnostic accuracy study can affect the generalizibility of the results to practice environments.

The value of MR imaging when the site of uterine cancer origin is uncertain

PURPOSE

To assess the value of magnetic resonance (MR) imaging in determining the site of origin of newly diagnosed uterine cancer (corpus vs cervix) when clinical and/or histologic evaluation is indeterminate.

MATERIALS AND METHODS

The Institutional Review Board approved and waived informed consent for this HIPAA-compliant study of 59 women (median age, 59 years; range, 28-84 years) who underwent pelvic MR imaging to determine the anatomic origin of uterine cancer. Two radiologists independently retrospectively assessed all MR imaging studies. In 48 patients who underwent hysterectomy, surgical pathologic findings were the reference standard, and overall test yields and diagnostic likelihood ratios were measured. Accuracy in detecting invasion of adjacent structures was also calculated. For the remaining patients, imaging and biopsy findings are presented descriptively.

RESULTS

At hysterectomy, 32 patients had uterine corpus cancer and 16 had cervical cancer. Overall test yields for reader 1 and reader 2 were 0.85 and 0.88, respectively. When a reader attributed a tumor's origin to either the uterine corpus or cervix, the odds of the tumor originating from that site were 4.80-6.35 times greater than they would have been if no other information were available. Accuracy levels in detecting invasion of the myometrium, cervical stroma, parametria and/or adnexae, and vagina, respectively, were 72%, 69%, 74%, and 85% for reader 1 and 78%, 77%, 76%, and 85% for reader 2.

CONCLUSION

MR imaging is useful for determining the anatomic origin of uterine cancer and provides helpful information regarding invasion of adjacent structures.

Gadolinium-enhanced magnetic resonance angiography for pulmonary embolism: a multicenter prospective study (PIOPED III)

BACKGROUND

The accuracy of gadolinium-enhanced magnetic resonance pulmonary angiography and magnetic resonance venography for diagnosing pulmonary embolism has not been determined conclusively.

OBJECTIVE

To investigate performance characteristics of magnetic resonance angiography, with or without magnetic resonance venography, for diagnosing pulmonary embolism.

DESIGN

Prospective, multicenter study from 10 April 2006 to 30 September 2008.

SETTING

7 hospitals and their emergency services.

PATIENTS

371 adults with diagnosed or excluded pulmonary embolism.

MEASUREMENTS

Sensitivity, specificity, and likelihood ratios were measured by comparing independently read magnetic resonance imaging with the reference standard for diagnosing pulmonary embolism. Reference standard diagnosis or exclusion was made by using various tests, including computed tomographic angiography and venography, ventilation-perfusion lung scan, venous ultrasonography, d-dimer assay, and clinical assessment.

RESULTS

Magnetic resonance angiography, averaged across centers, was technically inadequate in 25% of patients (92 of 371). The proportion of technically inadequate images ranged from 11% to 52% at various centers. Including patients with technically inadequate images, magnetic resonance angiography identified 57% (59 of 104) with pulmonary embolism. Technically adequate magnetic resonance angiography had a sensitivity of 78% and a specificity of 99%. Technically adequate magnetic resonance angiography and venography had a sensitivity of 92% and a specificity of 96%, but 52% of patients (194 of 370) had technically inadequate results.

LIMITATION

A high proportion of patients with suspected embolism was not eligible or declined to participate.

CONCLUSION

Magnetic resonance pulmonary angiography should be considered only at centers that routinely perform it well and only for patients for whom standard tests are contraindicated. Magnetic resonance pulmonary angiography and magnetic resonance venography combined have a higher sensitivity than magnetic resonance pulmonary angiography alone in patients with technically adequate images, but it is more difficult to obtain technically adequate images with the 2 procedures.

Direct comparison of whole-heart navigator-gated magnetic resonance coronary angiography and 40- and 64-slice multidetector row computed tomography to detect the coronary artery stenosis in patients scheduled for conventional coronary angiography

BACKGROUND

Both whole-heart magnetic resonance coronary angiography (WH-MRCA) and multidetector computed tomography (MDCT) have been proposed for the noninvasive identification of the coronary stenosis. The authors sought to directly compare the diagnostic accuracy of these noninvasive imaging techniques using the invasive quantitative coronary angiography as a reference standard.

METHODS AND RESULTS

Seventy-seven consecutive patients (56 men, 61+/-14 years) prospectively underwent WH-MRCA and 40- or 64-slice MDCT before the quantitative coronary angiography. Diagnostic accuracy of WH-MRCA and MDCT for the visual identification of >50% diameter stenosis in segments of >1.5 mm size was compared using the quantitative coronary angiography as a reference. According to the quantitative coronary angiography, 49 of 992 coronary segments >1.5 mm diameter had >50% diameter stenosis. MDCT had a higher success rate (100% versus 88%, P<0.001) and enabled identification of more segments (963 versus 726, P<0.001) than did WH-MRCA. On a per-segment basis, WH-MRCA had similar sensitivity (47/49 or 96% versus 48/49 or 98%, P=0.9) but significantly lower specificity (644/943 or 68% versus 863/943 or 92%, P<0.001) and accuracy (691/992 or 70% versus 911/992 or 92%, P<0.001) for the detection of >50% diameter stenosis than did MDCT. On a per-patient basis, the sensitivity was similar (17/17 or 100% versus 16/17 or 94%, P=0.9), but specificity (43/60 or 72% versus 53/60 or 88%, P=0.024) and diagnostic accuracy (60/77 or 78%, versus 69/77 or 90%, P=0.044) of WH-MRCA for the detection of >50% diameter stenosis were significantly lower than of MDCT.

CONCLUSIONS

Because of the higher success rate and higher number of interpretable segments, 40- or 64-slice MDCT performs better than WH-MRCA.

The STARD statement for reporting diagnostic accuracy studies: application to the history and physical examination

OBJECTIVE

The Standards for Reporting of Diagnostic Accuracy (STARD) statement provided guidelines for investigators conducting diagnostic accuracy studies. We reviewed each item in the statement for its applicability to clinical examination diagnostic accuracy research, viewing each discrete aspect of the history and physical examination as a diagnostic test.

SETTING

Nonsystematic review of the STARD statement.

INTERVENTIONS

Two former STARD Group participants and 1 editor of a journal series on clinical examination research reviewed each STARD item. Suggested interpretations and comments were shared to develop consensus.

MEASUREMENTS AND MAIN RESULTS

The STARD Statement applies generally well to clinical examination diagnostic accuracy studies. Three items are the most important for clinical examination diagnostic accuracy studies, and investigators should pay particular attention to their requirements: describe carefully the patient recruitment process, describe participant sampling and address if patients were from a consecutive series, and describe whether the clinicians were masked to the reference standard tests and whether the interpretation of the reference standard test was masked to the clinical examination components or overall clinical impression. The consideration of these and the other STARD items in clinical examination diagnostic research studies would improve the quality of investigations and strengthen conclusions reached by practicing clinicians.

CONCLUSIONS

The STARD statement provides a very useful framework for diagnostic accuracy studies. The group correctly anticipated that there would be nuances applicable to studies of the clinical examination. We offer guidance that should enhance their usefulness to investigators embarking on original studies of a patient's history and physical examination.

Influence of heart rate on diagnostic accuracy and image quality of 16-slice CT coronary angiography: comparison of multisegment and halfscan reconstruction approaches

The lower the heart rate the better image quality in multislice computed tomography (MSCT) coronary angiography. We prospectively assessed the influence of heart rate on per-patient diagnostic accuracy and image quality of MSCT coronary angiography and compared adaptive multisegment and standard halfscan reconstruction. A consecutive cohort of 126 patients scheduled to undergo conventional coronary angiography was examined with 16-slice CT. For all heart rate groups, per-patient diagnostic accuracy was significantly higher for multisegment than halfscan reconstruction with values of 95 vs. 79% (p < 0.05, <65 bpm, 38 patients), 85 vs. 66% (p < 0.05, 65-74 bpm, 47 patients), and 78% vs. 41% (p < 0.001, >74 bpm, 41 patients). Differences in diagnostic accuracy between adjacent heart rate groups were only significant for halfscan reconstruction for the comparison between the 65-74 and >74 bpm group (p < 0.05). The vessel lengths free of motion artifacts were significantly longer with multisegment reconstruction in all heart rate groups and for all coronary arteries (p < 0.005). For noninvasive MSCT coronary angiography, both per-patient diagnostic accuracy and image quality decline with increasing heart rate, and multisegment reconstruction at high heart rates yields similar results as standard halfscan reconstruction at low heart rates.

CT coronary angiography: influence of different cardiac reconstruction intervals on image quality and diagnostic accuracy

PURPOSE

To prospectively analyze image quality and diagnostic accuracy of different reconstruction intervals of coronary angiography using multislice computed tomography (MSCT).

MATERIALS AND METHODS

For each of 47 patients, 10 ECG-gated MSCT reconstructions were generated throughout the RR interval from 0 to 90%, resulting in altogether 470 datasets. These datasets were randomly analyzed for image quality and accuracy and compared with conventional angiography. Statistical comparison of intervals was performed using nonparametric analysis for repeated measurements to account for clustering of arteries within patients.

RESULTS

Image reconstruction intervals centered at 80, 70, and 40% of the RR interval resulted (in that order) in the best overall image quality for all four main coronary vessels. Eighty percent reconstructions also yielded the highest diagnostic accuracy of all intervals. The combination of the three best intervals (80, 70, and 40%) significantly reduced the nondiagnostic rate as compared with 80% alone (p=0.005). However, the optimal reconstruction interval combination achieved significantly improved specificities and nondiagnostic rates (p<0.05). The optimal combination consisted of 1.7+/-0.9 reconstruction intervals on average. In approximately half of the patients (49%, 23/47) a single reconstruction was optimal. In 18 (38%), 3 (6%), and 3 (6%) patients one, two, and three additional reconstruction intervals were required, respectively, to achieve optimal quality. In 28% of the patients the optimal combination consisted of reconstructions other than the three best intervals (80, 70, and 40%).

CONCLUSION

Multiple image reconstruction intervals are essential to ensure high image quality and accuracy of CT coronary angiography.

Validation of the indirect TaSP enzyme-linked immunosorbent assay for diagnosis of Theileria annulata infection in cattle

An ELISA based on a recombinant Theileria annulata surface protein (TaSP) was evaluated for detection of antibodies in sera from cattle exposed to tropical theileriosis in Sudan. The reference positive samples, used in this study, were from Theileria-infected populations and consisted of 80 cattle from an endemic area in Khartoum State, with high antibody titers in the indirect fluorescent antibody test (IFAT). The reference negative samples were taken from non-exposed populations and consisted of 120 cattle maintained under strict tick control at a commercial farm in Sudan. The cut-off value determined by Two-Graph Receiver-Operating Characteristic (TG-ROC) curves was set at 31.6%, based on the positive reference samples. Further diagnostic validation was performed, which consisted of the measurement of the area under the ROC (AUC) and by valid range proportion (VRP), which was 0.97 and 0.98 for the cut-off, respectively. There were no cross-reactions with antibodies raised against Babesia spp. It is concluded that the TaSP ELISA is a useful test for the diagnosis of T. annulata infection in cattle under field conditions.

The quality of reporting of diagnostic accuracy studies published in ophthalmic journals

AIM

To evaluate the quality of reporting of all diagnostic studies published in five major ophthalmic journals in the year 2002 using the Standards for Reporting of Diagnostic Accuracy (STARD) initiative parameters.

METHODS

Manual searching was used to identify diagnostic studies published in 2002 in five leading ophthalmic journals, the American Journal of Ophthalmology (AJO), Archives of Ophthalmology (Archives), British Journal of Ophthalmology (BJO), Investigative Ophthalmology and Visual Science (IOVS), and Ophthalmology. The STARD checklist of 25 items and flow chart was used to evaluate the quality of each publication.

RESULTS

A total of 16 publications were included (AJO = 5, Archives = 1, BJO = 2, IOVS = 2, and Ophthalmology = 6). More than half of the studies (n = 9) were related to glaucoma diagnosis. Other specialties included retina (n = 4) cornea (n = 2), and neuro-ophthalmology (n = 1). The most common description of diagnostic accuracy was sensitivity and specificity values, published in 13 articles. The number of fully reported items in evaluated studies ranged from eight to 19. Seven studies reported more than 50% of the STARD items.

CONCLUSIONS

The current standards of reporting of diagnostic accuracy tests are highly variable. The STARD initiative may be a useful tool for appraising the strengths and weaknesses of diagnostic accuracy studies.

Comparison of whole-body FDG-PET to bone scan for detection of bone metastases in patients with a new diagnosis of lung cancer

The purpose of this study was to compare the accuracy and agreement of whole-body positron-emission tomography (PET) scan to bone scintigraphy for the detection of bony metastases in staging patients with newly diagnosed lung cancer. The tumor registry and nuclear medicine database at our institution were queried and identified all patients between July 1998 and August 2002 with a new diagnosis of lung cancer, a whole-body 2-deoxy-2-[18F]fluoro-D-glucose (FDG)-PET scan, and a bone scan prior to therapy. All of these patients' radiologic reports were then retrospectively reviewed, and confirmation of bone metastases was determined by consideration of all available clinical information. The sensitivity, specificity, and accuracy for each study were then calculated. Two hundred and fifty-seven patients fulfilled the entrance criteria. One hundred and four patients (40%) presented with stage IV disease, and bone metastases were confirmed in 57 (22%) patients. The accuracies of PET and bone scan were 94 and 85% (P < 0.05), sensitivity values were 91 and 75%, and specificity values were 96 and 95%, respectively. The weighted-kappa statistic suggested moderate agreement between the two modalities KW = 0.510, 95% CI, 0.402-0.618). The use of both whole-body PET and bone scintigraphy as initial staging studies in lung cancer patients provides redundant information about the presence of bony metastases. The improvement in accuracy and sensitivity with PET suggests bone scan can be eliminated from the staging evaluation at presentation. Due to its retrospective nature, the results of this study are subject to several forms of bias including selection bias, verification bias, test review bias, and incorporation bias. A prospective trial with appropriate verification of bony metastases is suggested to confirm the results.

Accuracy of virtual bronchoscopy to detect endobronchial lesions

BACKGROUND

Virtual bronchoscopy (VB) could obviate flexible bronchoscopy (FB) if no endobronchial lesion is detected in patients presenting with a suspicion of malignancy. Our objectives were to evaluate the accuracy (in terms of sensitivity and specificity) of VB in detecting endobronchial lesions, and to determine the anatomical limit of detection of endobronchial lesions by VB.

METHODS

This study involved, in a blind comparison of VB and FB, consecutive patients presenting with symptoms or plain chest radiography abnormalities raising the suspicion of pulmonary neoplasm. After the standard chest computed tomography (CT), additional helical CT data were acquired from the aortic arch to the origin of the segmental bronchi of the inferior lobes in one 20-second breath hold using an helicoidal CT scan (3.0-mm collimation with a pitch of 1.5 and 1.5-mm reconstruction intervals).

RESULTS

One hundred ninety patients were enrolled; 136 patients (including 63 with an endobronchial lesion at FB) contributed to the primary analysis. The sensitivity and specificity of VB to detect endobronchial lesions were 68% (95% confidence interval [CI]: 55% to 79%) and 90% (95% CI: 81% to 96%), respectively. Overall, the agreement between VB and FB regarding the location on endobronchial lesions was substantial (weighted kappa: 0.66). However, VB detected only 26 of the 34 lobar lesions (sensitivity: 76%; CI: 59% to 89%) and 11 of the 23 segmental lesions (sensitivity: 48%; CI: 27% to 69%).

CONCLUSIONS

Beyond the mainstem bronchi, VB is not accurate enough to detect endobronchial lesions and to obviate FB in patients presenting with a suspicion of malignancy.

Staphylococcus aureus bacteremia after median sternotomy: clinical utility of blood culture results in the identification of postoperative mediastinitis

BACKGROUND

Mediastinitis is a complication of coronary artery bypass graft surgery (CABG) that can be difficult to diagnose. This study evaluated the utility of blood culture results in identifying patients with mediastinitis.

METHODS AND RESULTS

All unique patients undergoing CABG at our institution over a 60-month study period (n=5500) and all blood cultures performed on these patients <or=90 days after CABG were identified. Mediastinitis was identified by prospective active infection control surveillance. Eight hundred fifty-five (15.5%) patients had >or=1 blood culture drawn within 90 days of CABG. Mediastinitis occurred in 46 of 60 (76.7%) patients with blood cultures positive for Staphylococcus aureus, 15 of 126 (11.9%) patients with blood cultures positive for other pathogens, 37 of 669 (5.5%) patients with blood cultures with no growth, and 44 of 4645 (0.9%) patients with no blood cultures obtained. The isolation of S aureus from even 1 blood culture drawn after <or=90 days of CABG was strongly associated with mediastinitis (likelihood ratio [LR], 25; 95% CI, 14.7 to 44.4). Bacteremia attributable to other organisms did not alter pretest suspicion for mediastinitis (LR, 1.0; 95% CI, 0.6 to 1.7). Patients with negative blood cultures were less likely to have mediastinitis (LR, 0.45; 95% CI, 0.35 to 0.58). The association between S aureus bacteremia and mediastinitis remained highly significant when all unique patients undergoing CABG were analyzed in a logistic regression model and when a case-control analysis was used to evaluate patients with >or=1 blood culture obtained after CABG.

CONCLUSIONS

Among patients with blood cultures drawn after CABG, S aureus bacteremia strongly suggests the presence of mediastinitis.

Diagnostic accuracy of flow rate testing in urology

OBJECTIVE

To highlight the importance of methodological standards in diagnostic testing in urology, by reviewing the extent of compliance with these standards in reports of published evaluations.

METHODS

Sixteen papers evaluating the diagnostic accuracy of urinary flow-rate testing were independently assessed by two reviewers for compliance with the following methodological standards: specification of study population, analysis of pertinent subgroups, avoidance of evaluation (verification) bias, avoidance of review bias, reporting of precision of estimates of test accuracy, indeterminate test results, and test reproducibility. Compliance with each standard was expressed as a percentage with a 95% confidence interval.

RESULTS

Compliance ranged from just 12.5 (1.5-38)% for reporting of test reproducibility to 87.5 (62-98)% for specification of the study population. Only nine of the 16 evaluations complied with four or more of the methodological standards; one paper did not comply with any.

CONCLUSIONS

This evaluation of urological flow-rate testing show poor compliance with accepted methodological standards. The study design, reporting and evaluation of new diagnostic tests can be improved by adhering to these standards. Policy-makers, purchasers and providers would be less likely to adopt new diagnostic tests inappropriately if they were to appraise published evaluations against the standards. A reduction in the inappropriate use of tests would in turn lead to more cost-effective use of healthcare resources.

The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration

The quality of reporting of studies of diagnostic accuracy is less than optimal. Complete and accurate reporting is necessary to enable readers to assess the potential for bias in the study and to evaluate the generalisability of the results. A group of scientists and editors has developed the STARD (Standards for Reporting of Diagnostic Accuracy) statement to improve the reporting the quality of reporting of studies of diagnostic accuracy. The statement consists of a checklist of 25 items and flow diagram that authors can use to ensure that all relevant information is present. This explanatory document aims to facilitate the use, understanding and dissemination of the checklist. The document contains a clarification of the meaning, rationale and optimal use of each item on the checklist, as well as a short summary of the available evidence on bias and applicability. The STARD statement, checklist, flowchart and this explanation and elaboration document should be useful resources to improve reporting of diagnostic accuracy studies. Complete and informative reporting can only lead to better decisions in healthcare.

Evaluation of two direct immunoassays for rapid detection of petroleum products on marine birds

To reduce rehabilitation time and improve survival, a practical, objective test is needed to determine if marine birds are contaminated with oil before they enter captivity. The RaPID Assays and EnviroGard immunoassays for detection of polycyclic aromatic hydrocarbons (PAHs) in soil were evaluated for their ability to detect petroleum on feather samples from 30 intermediate fuel oil contaminated and 30 uncontaminated common murres (Uria aalge). Sensitivity, specificity, positive and negative predictive value, precision, and time required to run each assay were determined. The RaPID Assays was 96.7% sensitive and specific, while the EnviroGard assay was 93.3% sensitive, and 90.0% specific. Sensitivity decreased at higher dilutions for both assays. Intra-replicate variation was less than 20%. Our evaluation showed that these immunoassays are rapid and cost-effective methods for detecting oil-contamination on the plumage of seabirds, with the EnviroGard assay being more practical in most oil spill response situations due to ease of use and rapidity of results.

Review article: methodological standards in radiographer plain film reading performance studies

The objectives of this paper are to raise awareness of the methodological standards that can affect the quality of radiographer plain-film reading performance studies and to determine the frequency with which these standards are fulfilled. Multiple search methods identified 30 such studies from between 1971 and the end of June 1999. The percentage of studies that fulfilled criteria for the 10 methodological standards were as follows. (1) Performance of a sample size calculation, 3%; (2) definition of a normal and abnormal report, 97%; (3) description of the sequence of events through which films passed before reporting, 94%; (4) analysis of individual groups of observers within a combination of groups, 50% (5) appropriate choice of reference standard, 80%; (6) appropriate choice of arbiter, 57%; (7) appropriate use of a control, 22%; (8) analysis of pertinent clinical subgroups, e.g. body areas, patient type, 44%; (9) availability of data for re-calculation, 59%; and (10) presentation of indeterminate results, 69%. These findings indicate variation in the application of the methodological standards to studies of radiographer's film reading performance. Careful consideration of these standards is an essential component of study quality and hence the validity of the evidence base used to underpin radiographic reporting policy.

Durkan gauge and carpal compression test: accuracy and diagnostic test properties

STUDY DESIGN

A prospective, criterion-based validity study.

OBJECTIVES

To assess the diagnostic properties of the carpal compression test (CCT) when performed with the Durkan carpal tunnel syndrome (CTS) gauge, and to determine the measurement validity of the gauge.

BACKGROUND

The CCT has been reported to be highly sensitive (.87-.89) and specific (.93-1.0) in the diagnosis of CTS when it is done with thumb pressure. The accuracy of measurements with the Durkan CTS gauge, however, has not been established and the diagnostic sensitivity and specificity of the CCT when the gauge is used has not been independently confirmed.

METHODS AND MEASURES

The study sample included 33 women and 19 men, aged 18 to 85 years (45.7 +/- 13.5 years). The accuracy of the gauge was determined with a force dynamometer and holding frame. Standard nerve conduction studies (NCS) and the CCT were performed on the symptomatic extremity of all subjects. A compatible history and the NCS results were used to confirm CTS.

RESULTS

The Durkan gauge registered pressures of 11.94 psi and 15.25 psi at the 12 and 15 psi gauge marks, respectively. Test sensitivity and specificity were .36 (95% CI = .17-.54) and .57 (95% CI = .39-.74), respectively.

CONCLUSIONS

Pressure measurements obtained with the Durkan CTS gauge were accurate. The CCT when performed with the Durkan gauge, however, was neither sensitive or specific for the diagnosis of CTS.

Validation of the indirect MAP1-B enzyme-linked immunosorbent assay for diagnosis of experimental Cowdria ruminantium infection in small ruminants

The major antigenic protein 1 fragment B (MAP1-B) enzyme-linked immunosorbent assay (ELISA) for the diagnosis of Cowdria ruminantium infections was validated to determine cutoff values and evaluate its diagnostic performance with sheep and goat sera. Cowdria-infected populations consisted of 48 sheep and 44 goats, while the noninfected populations consisted of 64 sheep and 107 goats. Cutoff values were determined by two-graph receiver-operating characteristic (TG-ROC) curves. The cutoff value was set at 31 and 26.6% of the positive control reference samples for sheep and goat sera, respectively. The test's diagnostic performance was evaluated with measurements of the area under the concentration-time curve (AUC) of the ROC curves and by the valid range proportion (VRP). The AUCs were 0.978 for sheep sera and 0.989 for goat sera. The VRP for both sheep and goat sera was approximately 1.0. The intermediate range (IR), which defines results that are neither positive nor negative, was 0 for goat sera and 2.81 for sheep sera. In an ideal test, the AUC and VRP would be 1.0 and the IR would be 0. In this study these parameters were close to those of an ideal test. It is concluded that the MAP1-B ELISA is a useful test for the diagnosis of C. ruminantium infection in small ruminants.

Evaluating bias and variability in diagnostic test reports

Diagnostic testing is an important component of modern medical care. Unfortunately, many diagnostic tests are not rigorously evaluated before general application. Studies examining test characteristics often have methodologic flaws that impair their ability to provide reliable information on test performance. These flaws can introduce systematic nonrandom errors (biases) that distort measures of test accuracy. Other design errors can make it difficult to generalize the results of individual studies. These problems may enhance the apparent performance of poor tests while obscuring the performance of good tests, and they may result in the widespread use of tests with uncertain or limited efficacy. This article explores the ways in which studies of diagnostic test efficacy can be affected by bias and variability.

Quantitative assessments from the clinical examination. How should clinicians integrate the numerous results?

OBJECTIVE

To describe strategies for using multiple clinical examination items to estimate disease probabilities; and to evaluate the diagnostic accuracy of each strategy.

DESIGN

Prospective observational study.

SETTING

Medical preoperative evaluation clinic at a university-affiliated Veterans Affairs Medical Center.

PATIENTS

Previously reported consecutive series of patients referred for outpatient medical preoperative risk assessment.

MEASUREMENTS AND MAIN RESULTS

Pulmonary clinical examination and spirometry were the measurements. A strategy of using likelihood ratios (LRs) from seven clinical examination items was least accurate (p < .0001). Three alternative strategies were equivalent in diagnostic accuracy (p > or = .2): (1) using the single best clinical examination item and its LR, (2) using the LRs from three clinical examination items chosen by logistic regression, and (3) using the adjusted LRs chosen in strategy 2. When compared with using LRs from all seven items, the strategies of using three LRs chosen by logistic regression or using adjusted likelihood ratios better discriminated patients with airflow limitation from those without (receiver operating characteristic [ROC] areas 0.79 vs 0.69; p = .02). Using the single best clinical finding did not statistically degrade the clinical examination's discriminating ability (ROC areas 0.79 vs 0.75; p = .20).

CONCLUSIONS

Describing the rational clinical examination requires evaluating conditional independence of examination components. Conditional independence assumptions were violated when seven clinical examination items were used to estimate posterior probability of airflow limitation. Focusing on clinical examination items identified through logistic models overcame violations of independence; further statistical adjustment did not improve diagnostic accuracy. Clinicians can use the single most predictive clinical examination finding to avoid inaccuracy from violating the independence assumption.

The meaning of kappa: probabilistic concepts of reliability and validity revisited

A framework--the "agreement concept"--is developed to study the use of Cohen's kappa as well as alternative measures of chance-corrected agreement in a unified manner. Focusing on intrarater consistency it is demonstrated that for 2 x 2 tables an adequate choice between different measures of chance-corrected agreement can be made only if the characteristics of the observational setting are taken into account. In particular, a naive use of Cohen's kappa may lead to strikingly overoptimistic estimates of chance-corrected agreement. Such bias can be overcome by more elaborate study designs that allow for an unrestricted estimation of the probabilities at issue. When Cohen's kappa is appropriately applied as a measure of chance-corrected agreement, its values prove to be a linear--and not a parabolic--function of true prevalence. It is further shown how the validity of ratings is influenced by lack of consistency. Depending on the design of a validity study, this may lead, on purely formal grounds, to prevalence-dependent estimates of sensitivity and specificity. Proposed formulas for "chance-corrected" validity indexes fail to adjust for this phenomenon.

Two-graph receiver operating characteristic (TG-ROC): a Microsoft-EXCEL template for the selection of cut-off values in diagnostic tests

TG-ROC, a template for Microsoft-EXCEL, represents a novel, easy-to-handle approach for selecting cut-off values in quantitative diagnostic tests. In addition to graphical representations of test efficiency, Youden index and likelihood ratios as functions of the preselected cut-off value, the software supports the definition of an intermediate range of test results. For this purpose, two cut-off values are established that realise a pre-selected accuracy level (e.g., 90 or 95% sensitivity and specificity) which can be specified by the user.

A modified ROC analysis for the selection of cut-off values and the definition of intermediate results of serodiagnostic tests

A total number of 50 sera from clinically confirmed cases of canine Borrelia (B.) burgdorferi infection and 44 negative control sera were tested with a B. burgdorferi specific antibody ELISA. The data were submitted to the 'two-graph receiver operating characteristic' (TG-ROC) analysis which is a plot of the test sensitivity (Se) and specificity (Sp) against the threshold (cut-off) value assuming the latter to be an independent variable. Thus, in contrast to the conventional ROC analysis, valid pairs of Se and Sp can be read for pre-assigned threshold values directly from the TG-ROC plots. A cut-off that realises equal test parameters (Se = Sp = theta 0 (theta-zero)) can be obtained as the intersection point of the two graphs. Since the value for theta 0 is below a preselected accuracy level (95% or 90%), two cut-off values are selected that represent the bounds of an 'intermediate range' (IR). IR can be considered as a 'borderline' range for the clinical interpretation of test results. The proportion of the measurement range (MR) that gives unambiguous test results can be expressed using IR as the 'valid range proportion' (VRP = (MR-IR)/MR). VRP and theta 0 are useful parameters for test comparison since they do not depend upon the selection of a single cut-off point. In addition, the selection of cut-off values is supported by graphical displays of efficiency, Youden's index and likelihood ratios which can be considered as functions of the pre-assigned cut-off value. TG-ROC was derived as a user-defined template for a commercially available spreadsheet programme (MS-EXCEL, Microsoft).

Methodological issues in assessing reproducibility--a comparative study of various indices of reproducibility applied to repeat ELISA serologic tests for Lyme disease

This study applies and evaluates a variety of different measures of reproducibility. As an example, repeat enzyme-linked immunosorbent assays (ELISA) for antibodies to Borrelia burgdorferi, the etiologic agent of Lyme disease, are used. These repeat tests were part of the 1991 Quality Control Assessment of provincial laboratories that was carried out by the Laboratory Centre for Disease Control (Ottawa). Twenty-seven sera from cases and non-cases were tested by three laboratories, and two laboratories repeated the tests once. Methodological issues discussed include: different methods of assessing reproducibility in the continuous scale; whether reproducibility should be assessed with data in continuous or categorical form; problems assessing the reproducibility of data that has been standardized using a calibration-regression line; and problems with external generalizability of reproducibility studies of rare diseases. The authors conclude that the statistical method used to assess the reproducibility of a test must be adjusted to suit individual study designs and data, and the usage of the test.

Should physicians screen for oral disease? A physical examination study of the oral cavity

OBJECTIVE

To assess how well primary care clinicians select patients needing early referral to a dentist based on an oral cavity examination.

DESIGN

Prospective comparison of the screening oral cavity examinations performed by primary care clinicians with that performed by a dentist.

SETTING

General medicine clinic and dental clinic of a Veterans Affairs Medical Center.

PATIENTS

A sample of 86 consecutive patients attending regularly scheduled appointments in a primary care medical clinic.

MEASURES

Each patient was examined independently by two primary care clinicians and then one dentist. History and physical findings, clinical impression, and plan based on the oral cavity examination were recorded. The reference standard was the examination by a dentist blinded to the patient's history.

RESULTS

The prevalences of lesions suspicious for premalignancy, periodontal disease, calculus, and caries were 23%, 37%, 54%, and 18%, respectively. When the primary care clinicians noticed a lesion suspicious for premalignancy, the likelihood that such a lesion was present increased significantly (LR+ = 2.7 to 6.6). However, a normal examination by the primary care clinicians did not significantly lower the likelihood of a premalignant lesion (LR- = 0.7 to 0.8). The primary care clinicians were more efficient at evaluating dental and gingival conditions (LR+ = 2.7 to 5.8, LR- = 0.2 to 0.7).

CONCLUSIONS

Screening for oral disease by primary care clinicians is justified but should not replace routine screening by dentists.

A new system of visual presentation of analysis of test performance: the "double-ring" diagram

Substitution of graphic representation for extensive lists of numerical statistical data is highly desirable by both editors and readers of medical journals, faced with an exploding abundance of contemporary medical literature. A novel graphic tool, the "double-ring diagram", is described herein which permits visual representation of information regarding certain statistical variables used to describe the performance of a test or physical sign in the diagnosis of a disease. The diagram is relatively easy to construct on the basis of a number of primary data such as the prevalence and the true positive, true negative, false positive and false negative test results. These values are reflected in the diagram along with the values of other statistical variables derived from them, such as the sensitivity, specificity, predictive values for positive and negative test result, and accuracy. This diagram may be useful in visualizing a test's performance and facilitating visual comparison of performance of two or more tests.

Diagnosis of obstructive airways disease from the clinical examination

OBJECTIVE

To determine the operating characteristics of history and physical examination items for pulmonary airflow obstruction.

DESIGN

Prospective observational study.

SETTING

Medical Preoperative Evaluation Clinic at the Durham Veterans Affairs Medical Center.

PATIENTS/PARTICIPANTS

Consecutive patients referred for outpatient medical preoperative risk assessment.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Number of years the patient had smoked cigarettes, patient-reported wheezing [LR+ (likelihood ratio for finding present) = 3.1; LR- (likelihood ratio for finding absent) = 0.58], and auscultated wheezing (LR+ = 12; LR- = 0.87) were independent predictors of obstructive airways disease from the history and physical examination. Forced expiratory time and peak expiratory flow rate, both measured by the clinician at the bedside, were additional independent predictors of airflow obstruction. A nomogram using patient-reported wheezing, number of years the patient had smoked, and auscultated wheezing was developed and validated (area under receiver operating characteristic curve = 0.78; p = 0.0001) for the bedside prediction of obstructive airways disease. Peak expiratory flow rate can be substituted for auscultated wheezing with similar predictive ability.

CONCLUSIONS

The results of bedside clinical examinations predict the presence of obstructive airways disease. A nomogram based on a combination of four bedside findings predicts airflow obstruction as well as clinicians' overall clinical impressions.

Likelihood ratios for continuous test results--making the clinicians' job easier or harder?

Clinicians' paradigms for considering diagnostic test results require decisions based on the actual test value. However, when the test result is reported on a continuous scale each possible outcome may not result in unique actions. To simplify decision making, clinicians often break down the continuous scale into dichotomous or ordered outcomes. Likelihood ratios, reported with the test outcome, help summarize the impact of diagnostic tests. Although commonly applied to dichotomous outcomes, likelihood ratios can also be applied to ordinal or continuous results. This application allows investigators to consider the effect of clinically simplifying continuous data into dichotomous or ordinal categories. The parameters of a simple logistic regression equation summarize continuous likelihood ratios, evaluate covariates, generate likelihood ratio lines, and help assess the statistical significance of more complex models. Having visually inspected likelihood ratio lines and considered statistical differences, the investigator should choose the test report format that best accounts the realities driving clinical decisions.