Verification bias an underrecognized source of error in assessing the efficacy of medical imaging

Diagnostic Accuracy Studies in Radiology: How to Recognize and Address Potential Sources of Bias

Accuracy is an important parameter of a diagnostic test. Studies that attempt to determine a test's accuracy can suffer from various forms of bias. As radiology is a diagnostic specialty, many radiologists may design a diagnostic accuracy study or review one to understand how it may apply to their practice. Radiologists also frequently serve as consultants to other physicians regarding the selection of the most appropriate diagnostic exams. In these roles, understanding how to critically appraise the literature is important for all radiologists. The purpose of this review is to provide a framework for evaluating potential sources of study design biases that are found in diagnostic accuracy studies and to explain their impact on sensitivity and specificity estimates. To help the reader understand these biases, we also present examples from the radiology literature.

Imaging of the post-operative shoulder: does injection of iodinated contrast in addition to MR contrast during arthrography improve diagnostic accuracy and patient throughput?

OBJECTIVE

Post-operative shoulder patients are often difficult to image due to scar tissue, metallic artifact, and residual irregularity of anatomic structures. The purpose of this study was to assess MR versus MR arthrography versus CT arthrography in the post-operative shoulder. Also, we assessed whether injecting CT contrast in addition to MR contrast during arthrography improved patient care.

METHODS AND MATERIALS

One hundred consecutive post-operative conventional shoulder MR and MR arthrography exams performed on the same patients were reviewed retrospectively by two musculoskeletal radiologists. A combination of gadolinium and CT contrast was injected at arthrography so CT imaging could be performed post arthrography if metallic artifact precluded MR imaging. Twenty-two of these patients also had CT exams performed post-arthrography due to metallic artifact on MR exam. Exams were assessed for labral tears and supraspinatus tendon tears. All patients went on to arthroscopy.

RESULTS

Of these 100 patients, 35 had SLAP (superior labral anterior to posterior) tears, 22 had posterior labral tears, 24 had anterior labral tears, and 46 had full-thickness supraspinatus tendon tears on conventional MR exam. On MR arthrography, 48 patients had SLAP tears, 26 had posterior labral tears, 27 had anterior labral tears, and 54 had full-thickness supraspinatus tendon tears. MR arthrogram detected 12 SLAP tears, three posterior labral tears, three anterior labral tears, and nine supraspinatus tendon tears not detected on conventional MR exam. Twenty-two patients had additional imaging performed with CT arthrography due to metallic artifacts precluding MR assessment of shoulder pathology. There were four SLAP tears, six posterior labral tears, five anterior labral tears, and five supraspinatus tendon tear seen on CT arthrography not seen on MR exam.

CONCLUSIONS

MR arthrography is more accurate than conventional MR in assessment of post-operative shoulder pathology. CT arthrography can detect additional pathology when there is metallic artifact in post-operative patients. It is beneficial to inject a combination of gadolinium and CT contrast at arthrography so CT imaging can be performed post-arthrography if metallic artifact precludes imaging shoulder pathology by MR.

Accuracy of Two-Dimensional Transvaginal Sonography and Office Hysteroscopy for Detection of Uterine Abnormalities in Patients with Repeated Implantation Failures or Recurrent Pregnancy Loss

Background

We sought to compare diagnostic values of two-dimensional transvaginal sonography (2D TVS) and office hysteroscopy (OH) for evaluation of endometrial pathologies in cases with repeated implantation failure (RIF) or recurrent pregnancy loss (RPL).

MATERIALS AND METHODS

This prospective study was performed at Royan Institute from December 2013 to January 2015. TVS was performed before hysteroscopy as part of the routine diagnostic work-up in 789 patients with RIF or RPL. Uterine biopsy was performed in cases with abnormal diagnosis in TVS and/or hysteroscopy. We compared the diagnostic accuracy values of TVS in detection of uterine abnormalities with OH by receiver operating characteristic (ROC) curve analysis.

RESULTS

TVS examination detected 545 (69%) normal cases and 244 (31%) pathologic cases, which included 84 (10.6%) endometrial polyps, 15 (1.6%) uterine fibroids, 10 (1.3%) Asherman's syndrome, 9 (1.1%) endometrial hypertrophy, and 126 (15.9%) septate and arcuate uterus. TVS and OH concurred in 163 pathologic cases, although TVS did not detect some pathology cases (n=120). OH had 94% sensitivity, 95% specificity, 62% positive predictive value (PPV), and 99% negative predictive value (NPV) for detection of endometrial polyps. In the diagnosis of myoma, sensitivity, specificity, PPV, and NPV were 100%. TVS had a sensitivity of 50% and specificity of 98% for the diagnosis of myoma. For polyps, TVS had a sensitivity of 54% and specificity of 80%. Area under the ROC curve (AUROC) was 70.69% for the accuracy of TVS compared to OH.

CONCLUSION

TVS had high specificity and low sensitivity for detection of uterine pathologies in patients with RIF or RPL compared with OH. OH should be considered as a workup method prior to treatment in patients with normal TVS findings.

[Systematic errors in clinical studies : A comprehensive survey]

Systematic errors and related phenomena represent an intrinsic challenge to the quality of clinical research. As a consequence even otherwise methodologically demanding studies may produce results that systematically differ from the true values. Systematic errors relating to investigative medicine are divided into six groups according to their affiliation with the consecutive chronological sections of the study. Bias can occur in preliminary literature research in the field, specifying the study design and selecting the study sample, measuring exposure and outcome, analyzing the data, interpreting the analyses and publishing the results. The most important systematic errors that concern diagnostic and interventional studies are created by access to the data of previous tests, calculated study design, preselection of the participants, comparison with non-contemporaneous controls, antedating the time of diagnosis and overdiagnosis of slowly progressive forms of diseases examined. Checking the measured values often leads to a mosaic of several biases with one being more or less dominant. Even by exercising due care in the preparation and performance of the study, the majority of distortions cannot be eliminated but only diminished. It is essential to consider each detected bias as a potential full or partial argument in support of an observed correlation. The control of systematic errors and related phenomena is both a significant element of the discussion of the study report and a key element for assessment of its scientific value.

[Systematic errors in clinical studies : A comprehensive survey]

Systematic errors and related phenomena represent an intrinsic challenge to the quality of clinical research. As a consequence even otherwise methodologically demanding studies may produce results that systematically differ from the true values. Systematic errors relating to investigative medicine are divided into six groups according to their affiliation with the consecutive chronological sections of the study. Bias can occur in preliminary literature research in the field, specifying the study design and selecting the study sample, measuring exposure and outcome, analyzing the data, interpreting the analyses and publishing the results. The most important systematic errors that concern diagnostic and interventional studies are created by access to the data of previous tests, calculated study design, preselection of the participants, comparison with non-contemporaneous controls, antedating the time of diagnosis and overdiagnosis of slowly progressive forms of diseases examined. Checking the measured values often leads to a mosaic of several biases with one being more or less dominant. Even by exercising due care in the preparation and performance of the study, the majority of distortions cannot be eliminated but only diminished. It is essential to consider each detected bias as a potential full or partial argument in support of an observed correlation. The control of systematic errors and related phenomena is both a significant element of the discussion of the study report and a key element for assessment of its scientific value.

A diagnostic accuracy meta-analysis of endoanal ultrasound and MRI for perianal fistula assessment

BACKGROUND

Imaging modalities such as endoanal ultrasound or MRI can be useful preoperative adjuncts before the appropriate surgical intervention for perianal fistulas.

OBJECTIVES

We present a systematic review of published literature comparing endoanal ultrasound with MRI for the assessment of idiopathic and Crohn's perianal fistulas.

DESIGN

A meta-analysis was performed to obtain pooled values for specificity and sensitivity.

SETTINGS

Electronic databases were searched from January 1970 to October 2010 for published studies.

PATIENTS AND INTERVENTIONS

Four studies were used in our analysis. There were 241 fistulas in the ultrasound group and 240 in the magnetic resonance group.

RESULTS

The combined sensitivity and specificity of magnetic resonance for fistula detection were 0.87 (95% CI: 0.63-0.96) and 0.69 (95% CI: 0.51-0.82). There was a high degree of heterogeneity between studies reporting on MRI sensitivity (df = 3, I = 93%). This compares to a sensitivity and specificity for endoanal ultrasound of 0.87 (95% CI: 0.70-0.95) and 0.43 (95% CI: 0.21-0.69). There was a high degree of heterogeneity between studies reporting on endoanal ultrasound sensitivity (df = 3, I = 92%).

CONCLUSIONS

From the available literature, the summarized performance characteristics for MRI and endoanal ultrasound demonstrate comparable sensitivities at detecting perianal fistulas, although the specificity for MRI was higher than that for endoanal ultrasound. Both specificity values are considered to be diagnostically poor, however. The high degree of data heterogeneity and the shortage of applicable studies precludes any firm conclusions being made for clinical practice. Future trials with improved study design (including prospective data collection and consideration of verification bias) may help to further clarify the role of MRI in the assessment and treatment response monitoring of perianal fistulas (particularly in patients with Crohn's disease).

Verification bias: an under-recognized source of error in assessing the efficacy of MRI of the meniscii

RATIONALE AND OBJECTIVES

The sensitivity and specificity of magnetic resonance imaging (MRI) for diagnosis of meniscal tears has been studied extensively, with tears usually verified by surgery. However, surgically unverified cases are often not considered in these studies, leading to verification bias, which can falsely increase the sensitivity and decrease the specificity estimates. Our study suggests that such bias may be very common in the meniscal MRI literature, and illustrates techniques to detect and correct for such bias.

MATERIALS AND METHODS

PubMed was searched for articles estimating sensitivity and specificity of MRI for meniscal tears. These were assessed for verification bias, deemed potentially present if a study included any patients whose MRI findings were not surgically verified. Retrospective global sensitivity analysis (GSA) was performed when possible.

RESULTS

Thirty-nine of the 314 studies retrieved from PubMed specifically dealt with meniscal tears. All 39 included unverified patients, and hence, potential verification bias. Only seven articles included sufficient information to perform GSA. Of these, one showed definite verification bias, two showed no bias, and four others showed bias within certain ranges of disease prevalence. Only 9 of 39 acknowledged the possibility of verification bias.

CONCLUSION

Verification bias is underrecognized and potentially common in published estimates of the sensitivity and specificity of MRI for the diagnosis of meniscal tears. When possible, it should be avoided by proper study design. If unavoidable, it should be acknowledged. Investigators should tabulate unverified as well as verified data. Finally, verification bias should be estimated; if present, corrected estimates of sensitivity and specificity should be used. Our online web-based calculator makes this process relatively easy.