An interactive web-based tool for detecting verification (work-up) bias in studies of the efficacy of diagnostic imaging

The Accuracy of Computed Tomography Angiography Compared With Technetium-99m Labelled Red Blood Cell Scintigraphy for the Diagnosis and Localization of Acute Gastrointestinal Bleeding: A Systematic Review and Meta-Analysis

Imaging tests are commonly used as an initial or early investigation for patients presenting with suspected acute gastrointestinal bleeding (AGIB). However, controversy remains regarding which of two frequently used modalities, computed tomography angiography (CTA) or technetium-99m labelled red blood cell scintigraphy (RBCS), is most accurate. This systematic review and meta-analysis was performed to compare the accuracy of CTA and RBCS for the detection and localization of AGIB. Five electronic databases were searched with additional manual searching of reference lists of relevant publications identified during the search. Two reviewers independently performed screening, data extraction and methodological assessments. Where appropriate, the bivariate model was used for meta-analysis of sensitivities and specificities for the detection of bleeding and Freeman-Tukey double-arcsine transformation used for meta-analysis of proportions of correctly localized bleeding sites. Forty-four unique primary studies were included: twenty-two investigating CTA, seventeen investigating RBCS and five investigating both modalities. Meta-analysis produced similar pooled sensitivities; 0.83 (95% CI 0.74-0.90) and 0.84 (0.68-0.92) for CTA and RBCS respectively. Pooled specificity for CTA was higher than RBCS; 0.90 (0.72-0.97) and 0.84 (0.71-0.91) respectively. However, differences were not statistically significant. CTA was superior to RBCS in correctly localizing bleeding; pooled proportions of 1.00 (0.98-1.00) and 0.90 (0.83-0.96) respectively (statistically significant difference, P < 0.001). There is no evidence that CTA and RBCS have different diagnostic performance with respect to the detection of AGIB. However, CTA is superior to RBCS in terms of correctly localising the bleeding site, supporting usage of CTA over RBCS as the first line imaging investigation.

Hidden sources of bias in diagnostic studies: the example of visceral leishmaniasis in east Africa

A clear understanding of the accuracy of diagnostic tests is essential for the management of infections such as visceral leishmaniasis in endemic areas. Using both published and unpublished datasets from field diagnostic trials of visceral leishmaniasis in the past ten years, we show the potential effects of unrecognised sources of bias including work-up bias, referral bias, and reference test bias on the results. We outline why these biases, which can occur in diagnostic studies of any disease, can go unrecognised despite adherence to current STARD and QUADAS-2 guidelines. Using these examples and referring to others seen in studies of bacterial and viral infections, we make specific recommendations on how these biases might be avoided through specific steps in study design, study reporting, and the locations where studies are conducted.

Deep convolutional neural network-based detection of meniscus tears: comparison with radiologists and surgery as standard of reference

OBJECTIVE

To clinically validate a fully automated deep convolutional neural network (DCNN) for detection of surgically proven meniscus tears.

MATERIALS AND METHODS

One hundred consecutive patients were retrospectively included, who underwent knee MRI and knee arthroscopy in our institution. All MRI were evaluated for medial and lateral meniscus tears by two musculoskeletal radiologists independently and by DCNN. Included patients were not part of the training set of the DCNN. Surgical reports served as the standard of reference. Statistics included sensitivity, specificity, accuracy, ROC curve analysis, and kappa statistics.

RESULTS

Fifty-seven percent (57/100) of patients had a tear of the medial and 24% (24/100) of the lateral meniscus, including 12% (12/100) with a tear of both menisci. For medial meniscus tear detection, sensitivity, specificity, and accuracy were for reader 1: 93%, 91%, and 92%, for reader 2: 96%, 86%, and 92%, and for the DCNN: 84%, 88%, and 86%. For lateral meniscus tear detection, sensitivity, specificity, and accuracy were for reader 1: 71%, 95%, and 89%, for reader 2: 67%, 99%, and 91%, and for the DCNN: 58%, 92%, and 84%. Sensitivity for medial meniscus tears was significantly different between reader 2 and the DCNN (p = 0.039), and no significant differences existed for all other comparisons (all p ≥ 0.092). The AUC-ROC of the DCNN was 0.882, 0.781, and 0.961 for detection of medial, lateral, and overall meniscus tear. Inter-reader agreement was very good for the medial (kappa = 0.876) and good for the lateral meniscus (kappa = 0.741).

CONCLUSION

DCNN-based meniscus tear detection can be performed in a fully automated manner with a similar specificity but a lower sensitivity in comparison with musculoskeletal radiologists.

Imaging of the post-operative shoulder: does injection of iodinated contrast in addition to MR contrast during arthrography improve diagnostic accuracy and patient throughput?

OBJECTIVE

Post-operative shoulder patients are often difficult to image due to scar tissue, metallic artifact, and residual irregularity of anatomic structures. The purpose of this study was to assess MR versus MR arthrography versus CT arthrography in the post-operative shoulder. Also, we assessed whether injecting CT contrast in addition to MR contrast during arthrography improved patient care.

METHODS AND MATERIALS

One hundred consecutive post-operative conventional shoulder MR and MR arthrography exams performed on the same patients were reviewed retrospectively by two musculoskeletal radiologists. A combination of gadolinium and CT contrast was injected at arthrography so CT imaging could be performed post arthrography if metallic artifact precluded MR imaging. Twenty-two of these patients also had CT exams performed post-arthrography due to metallic artifact on MR exam. Exams were assessed for labral tears and supraspinatus tendon tears. All patients went on to arthroscopy.

RESULTS

Of these 100 patients, 35 had SLAP (superior labral anterior to posterior) tears, 22 had posterior labral tears, 24 had anterior labral tears, and 46 had full-thickness supraspinatus tendon tears on conventional MR exam. On MR arthrography, 48 patients had SLAP tears, 26 had posterior labral tears, 27 had anterior labral tears, and 54 had full-thickness supraspinatus tendon tears. MR arthrogram detected 12 SLAP tears, three posterior labral tears, three anterior labral tears, and nine supraspinatus tendon tears not detected on conventional MR exam. Twenty-two patients had additional imaging performed with CT arthrography due to metallic artifacts precluding MR assessment of shoulder pathology. There were four SLAP tears, six posterior labral tears, five anterior labral tears, and five supraspinatus tendon tear seen on CT arthrography not seen on MR exam.

CONCLUSIONS

MR arthrography is more accurate than conventional MR in assessment of post-operative shoulder pathology. CT arthrography can detect additional pathology when there is metallic artifact in post-operative patients. It is beneficial to inject a combination of gadolinium and CT contrast at arthrography so CT imaging can be performed post-arthrography if metallic artifact precludes imaging shoulder pathology by MR.

Verification bias: an under-recognized source of error in assessing the efficacy of MRI of the meniscii

RATIONALE AND OBJECTIVES

The sensitivity and specificity of magnetic resonance imaging (MRI) for diagnosis of meniscal tears has been studied extensively, with tears usually verified by surgery. However, surgically unverified cases are often not considered in these studies, leading to verification bias, which can falsely increase the sensitivity and decrease the specificity estimates. Our study suggests that such bias may be very common in the meniscal MRI literature, and illustrates techniques to detect and correct for such bias.

MATERIALS AND METHODS

PubMed was searched for articles estimating sensitivity and specificity of MRI for meniscal tears. These were assessed for verification bias, deemed potentially present if a study included any patients whose MRI findings were not surgically verified. Retrospective global sensitivity analysis (GSA) was performed when possible.

RESULTS

Thirty-nine of the 314 studies retrieved from PubMed specifically dealt with meniscal tears. All 39 included unverified patients, and hence, potential verification bias. Only seven articles included sufficient information to perform GSA. Of these, one showed definite verification bias, two showed no bias, and four others showed bias within certain ranges of disease prevalence. Only 9 of 39 acknowledged the possibility of verification bias.

CONCLUSION

Verification bias is underrecognized and potentially common in published estimates of the sensitivity and specificity of MRI for the diagnosis of meniscal tears. When possible, it should be avoided by proper study design. If unavoidable, it should be acknowledged. Investigators should tabulate unverified as well as verified data. Finally, verification bias should be estimated; if present, corrected estimates of sensitivity and specificity should be used. Our online web-based calculator makes this process relatively easy.