Meta-analysis of clinical studies of diagnostic tests: developments in how the receiver operating characteristic "works"

Visualizing and diagnosing spillover within randomized concurrent controlled trials through the application of diagnostic test assessment methods

BACKGROUND

Spillover of effect, whether positive or negative, from intervention to control group patients invalidates the Stable Unit Treatment Variable Assumption (SUTVA). SUTVA is critical to valid causal inference from randomized concurrent controlled trials (RCCT). Spillover of infection prevention is an important population level effect mediating herd immunity. This herd effect, being additional to any individual level effect, is subsumed within the overall effect size (ES) estimate derived by contrast-based techniques from RCCT's. This herd effect would manifest only as increased dispersion among the control group infection incidence rates above background.

METHODS AND RESULTS

The objective here is to explore aspects of spillover and how this might be visualized and diagnosed. I use, for illustration, data from 190 RCCT's abstracted in 13 Cochrane reviews of various antimicrobial versus non-antimicrobial based interventions to prevent pneumonia in ICU patients. Spillover has long been postulated in this context. Arm-based techniques enable three approaches to identify increased dispersion, not available from contrast-based techniques, which enable the diagnosis of spillover within antimicrobial versus non-antimicrobial based infection prevention RCCT's. These three approaches are benchmarking the pneumonia incidence rates versus a clinically relevant range, comparing the dispersion in pneumonia incidence among the control versus the intervention groups and thirdly, visualizing the incidence dispersion within summary receiver operator characteristic (SROC) plots. By these criteria there is harmful spillover effects to concurrent control group patients.

CONCLUSIONS

Arm-based versus contrast-based techniques lead to contrary inferences from the aggregated RCCT's of antimicrobial based interventions despite similar summary ES estimates. Moreover, the inferred relationship between underlying control group risk and ES is 'flipped'.

Diagnosis of Pneumonia Due to Invasive Molds

Pneumonia is the most common presentation of invasive mold infections (IMIs), and is pathogenetically characterized as angioinvasion by hyphae, resulting in tissue infarction and necrosis. Aspergillus species are the typical etiologic cause of mold pneumonia, with A. fumigatus in most cases, followed by the Mucorales species. Typical populations at risk include hematologic cancer patients on chemotherapy, bone marrow and solid organ transplant patients, and patients on immunosuppressive medications. Invasive lung disease due to molds is challenging to definitively diagnose based on clinical features and imaging findings alone, as these methods are nonspecific. Etiologic laboratory testing is limited to insensitive culture techniques, non-specific and not readily available PCR, and tissue biopsies, which are often difficult to obtain and impact on the clinical fragility of patients. Microbiologic/mycologic analysis has limited sensitivity and may not be sufficiently timely to be actionable. Due to the inadequacy of current diagnostics, clinicians should consider a combination of diagnostic modalities to prevent morbidity in patients with mold pneumonia. Diagnosis of IMIs requires improvement, and the availability of noninvasive methods such as fungal biomarkers, microbial cell-free DNA sequencing, and metabolomics-breath testing could represent a new era of timely diagnosis and early treatment of mold pneumonia.

Could simulation methods solve the curse of sparse data within clinical studies of antibiotic resistance?

Infectious disease (ID) physicians and ID pharmacists commonly confront therapeutic questions relating to antibiotic resistance. Randomized controlled trial data are few and meta-analytic-based approaches to develop the evidence-base from several small studies that might relate to an antibiotic resistance question are not simple. The overriding challenge is the sparsity of data which is problematic for traditional frequentist methods, being the paradigm underlying the derivation of ‘P value’ inferential statistics. In other sparse data contexts, simulation methods enable answers to key questions that are meaningful, quantitative and potentially relevant. How these simulation methods ‘work’ and how Bayesian-based methods, being not ‘P value based’, can facilitate simulation are reviewed. These methods are becoming increasingly accessible. This review highlights why sparse data is less of an issue within Bayesian versus frequentist paradigms. A fictional pharmacokinetic study with sparse data illustrates a simplistic application of Bayesian and simulation methods to antibiotic dosing. Whether within epidemiological projections or clinical studies, simulation methods are likely to play an increasing role in antimicrobial resistance research within both hospital and community studies of either rare infectious disease or infections within specific population groups.

How the Cluster-randomized Trial "Works"

Cluster-randomized trials (CRTs) are able to address research questions that randomized controlled trials (RCTs) of individual patients cannot answer. Of great interest for infectious disease physicians and infection control practitioners are research questions relating to the impact of interventions on infectious disease dynamics at the whole-of-population level. However, there are important conceptual differences between CRTs and RCTs relating to design, analysis, and inference. These differences can be illustrated by the adage "peas in a pod." Does the question of interest relate to the "peas" (the individual patients) or the "pods" (the clusters)? Several examples of recent CRTs of community and intensive care unit infection prevention interventions are used to illustrate these key concepts. Examples of differences between the results of RCTs and CRTs on the same topic are given.

Thiopurine S-methyltransferase testing for averting drug toxicity: a meta-analysis of diagnostic test accuracy

Thiopurine S-methyltransferase (TPMT) deficiency increases the risk of serious adverse events in persons receiving thiopurines. The objective was to synthesize reported sensitivity and specificity of TPMT phenotyping and genotyping using a latent class hierarchical summary receiver operating characteristic meta-analysis. In 27 studies, pooled sensitivity and specificity of phenotyping for deficient individuals was 75.9% (95% credible interval (CrI), 58.3-87.0%) and 98.9% (96.3-100%), respectively. For genotype tests evaluating TPMT*2 and TPMT*3, sensitivity and specificity was 90.4% (79.1-99.4%) and 100.0% (99.9-100%), respectively. For individuals with deficient or intermediate activity, phenotype sensitivity and specificity was 91.3% (86.4-95.5%) and 92.6% (86.5-96.6%), respectively. For genotype tests evaluating TPMT*2 and TPMT*3, sensitivity and specificity was 88.9% (81.6-97.5%) and 99.2% (98.4-99.9%), respectively. Genotyping has higher sensitivity as long as TPMT*2 and TPMT*3 are tested. Both approaches display high specificity. Latent class meta-analysis is a useful method for synthesizing diagnostic test performance data for clinical practice guidelines.The Pharmacogenomics Journal advance online publication, 24 May 2016; doi:10.1038/tpj.2016.37.

Underutilization of Meta-analysis in Diagnostic Pathology

CONTEXT

Underutilization of Meta-analysis No studies to our knowledge have investigated citations and utilization of meta-analysis in diagnostic pathology (DP).

OBJECTIVE

To characterize meta-analyses in DP compared with meta-analyses in medicine.

DESIGN

We searched PubMed for meta-analyses in 12 major DP journals without specifying years and in 4 major medicine journals in both 2006 and 2011. We compared articles' adjusted citation ratios (ACRs), defined as an article's citation count divided by the mean citations for the meta-analysis, review, and original research articles published in the same journal in the same year.

RESULTS

Forty-one of 76 DP articles, 74 of 125 medicine articles in 2011, and 52 of 83 medicine articles in 2006 were qualified meta-analyses as identified by PubMed. The ACRs of DP meta-analysis articles were higher than those of original research articles (2.62 ± 2.31 versus 0.92 ± 0.84, P < .001) and similar to those of review articles in 2006 (2.62 ± 2.31 versus 1.95 ± 1.59, P = .50), but they were similar to both in 2011 (1.85 ± 1.39 versus 0.99 ± 1.43, P = .11; 1.85 ± 1.39 versus 1.12 ± 1.43, P = .21, respectively). Diagnostic pathology and medicine meta-analyses had similar ACRs (1.85 ± 1.39 versus 1.57 ± 1.35 in 2011, P = .60; and 2.62 ± 2.31 versus 1.85 ± 1.90 in 2006, P = .50, respectively). However, although DP journals published fewer meta-analyses (0.97% versus 6.66% in 2011 and 0.67% versus 4.40% in 2006, P < .001 for both), they published more meta-analyses using both original and published data than medicine (21.95% versus 1.59%, P < .001). They also published more meta-analyses per year in 2011-2014 than in 2000-2010 (6.4 ± 1.29 versus 1.36 ± 1.03 articles per year, P < .001).

CONCLUSIONS

We found underutilization of meta-analyses in DP, despite their high ACRs and recently increased utilization. More DP meta-analyses are needed.

Underutilization of Meta-analysis in Diagnostic Pathology

Context

No studies to our knowledge have investigated citations and utilization of meta-analysis in diagnostic pathology (DP).

Objective

To characterize meta-analyses in DP compared with meta-analyses in medicine.

Design

We searched PubMed for meta-analyses in 12 major DP journals without specifying years and in 4 major medicine journals in both 2006 and 2011. We compared articles' adjusted citation ratios (ACRs), defined as an article's citation count divided by the mean citations for the meta-analysis, review, and original research articles published in the same journal in the same year.

Results

Forty-one of 76 DP articles, 74 of 125 medicine articles in 2011, and 52 of 83 medicine articles in 2006 were qualified meta-analyses as identified by PubMed. The ACRs of DP meta-analysis articles were higher than those of original research articles (2.62 ± 2.31 versus 0.92 ± 0.84, P &lt; .001) and similar to those of review articles in 2006 (2.62 ± 2.31 versus 1.95 ± 1.59, P = .50), but they were similar to both in 2011 (1.85 ± 1.39 versus 0.99 ± 1.43, P = .11; 1.85 ± 1.39 versus 1.12 ± 1.43, P = .21, respectively). Diagnostic pathology and medicine meta-analyses had similar ACRs (1.85 ± 1.39 versus 1.57 ± 1.35 in 2011, P = .60; and 2.62 ± 2.31 versus 1.85 ± 1.90 in 2006, P = .50, respectively). However, although DP journals published fewer meta-analyses (0.97% versus 6.66% in 2011 and 0.67% versus 4.40% in 2006, P &lt; .001 for both), they published more meta-analyses using both original and published data than medicine (21.95% versus 1.59%, P &lt; .001). They also published more meta-analyses per year in 2011–2014 than in 2000–2010 (6.4 ± 1.29 versus 1.36 ± 1.03 articles per year, P &lt; .001).

Conclusions

We found underutilization of meta-analyses in DP, despite their high ACRs and recently increased utilization. More DP meta-analyses are needed.

Endotoxemia as a diagnostic tool for patients with suspected bacteremia caused by gram-negative organisms: a meta-analysis of 4 decades of studies

The clinical significance of endotoxin detection in blood has been evaluated for a broad range of patient groups in over 40 studies published over 4 decades. The influences of Gram-negative (GN) bacteremia species type and patient inclusion criteria on endotoxemia detection rates in published studies remain unclear. Studies were identified after a literature search and manual reviews of article bibliographies, together with a direct approach to authors of potentially eligible studies for data clarifications. The concordance between GN bacteremia and endotoxemia expressed as the summary diagnostic odds ratios (DORs) was derived for three GN bacteremia categories across eligible studies by using a hierarchical summary receiver operating characteristic (HSROC) method. Forty-two studies met broad inclusion criteria, with between 2 and 173 GN bacteremias in each study. Among all 42 studies, the DORs (95% confidence interval) were 3.2 (1.7 to 6.0) and 5.8 (2.4 to 13.7) in association with GN bacteremias with Escherichia coli and those with Pseudomonas aeruginosa, respectively. Among 12 studies of patients with sepsis, the proportion of endotoxemia positivity (95% confidence interval) among patients with P. aeruginosa bacteremia (69% [57 to 79%]; P=0.004) or with Proteus bacteremia (76% [51 to 91%]; P=0.04) was significantly higher than that among patients without GN bacteremia (49% [33 to 64%]), but this was not so for patients bacteremic with E. coli (57% [40 to 73%]; P=0.55). Among studies of the sepsis patient group, the concordance of endotoxemia with GN bacteremia was surprisingly weak, especially for E. coli GN bacteremia.

Ventilator-associated pneumonia prevention methods using topical antibiotics: herd protection or herd peril?

Ventilator-associated pneumonia (VAP) develops in approximately 20% of patients in the ICU receiving prolonged mechanical ventilation (MV). Among the range of methods for preventing VAP, the evidence base for topical antibiotics (TAs), including selective digestive decontamination, appears to be the most compelling. However, several observations are puzzling, and the contextual influence resulting from concurrent use of both topical placebo and TA within an ICU remains untested. As with herd protection conferred by vaccination, contextual influences resulting from a population-based intervention cannot be estimated at the level of a single trial. Estimating contextual effects requires multilevel random-effects methods. In this way the dispersion in VAP incidence across groups from 206 studies, as cited in various-source systematic reviews, was calibrated. The benchmark mean VAP incidence derived from 49 observational groups of patients receiving MV is 23.7% (95% CI, 20.6%-27.2%). In contrast, for 20 and 15 concurrent control groups from the TA evidence base that did vs did not receive topical placebo, respectively, this incidence is 38% (95% CI, 29%-48%) and 33% (95% CI, 20%-50%). This contextual influence remains significant in a meta-regression model adjusted for group-level variables, such as within a trauma ICU context. The mean VAP incidence for five other categories of control groups from the broader evidence base is within four percentage points of the benchmark. The contextual effect of TA is paradoxic, peculiar, potent, perfidious, and potentially perilous. The TA evidence base requires reappraisal to consider this herd peril.