No evidence of bias in the process of publication of diagnostic accuracy studies in stroke submitted as abstracts

Association of Accuracy, Conclusions, and Reporting Completeness With Acceptance by Radiology Conferences and Journals

BACKGROUND

Preferential publication of studies with positive findings can lead to overestimation of diagnostic test accuracy (i.e. publication bias). Understanding the contribution of the editorial process to publication bias could inform interventions to optimize the evidence guiding clinical decisions.

PURPOSE/HYPOTHESIS

To evaluate whether accuracy estimates, abstract conclusion positivity, and completeness of abstract reporting are associated with acceptance to radiology conferences and journals.

STUDY TYPE

Meta-research.

POPULATION

Abstracts submitted to radiology conferences (European Society of Gastrointestinal and Abdominal Radiology (ESGAR) and International Society for Magnetic Resonance in Medicine (ISMRM)) from 2008 to 2018 and manuscripts submitted to radiology journals (Radiology, Journal of Magnetic Resonance Imaging [JMRI]) from 2017 to 2018. Primary clinical studies evaluating sensitivity and specificity of a diagnostic imaging test in humans with available editorial decisions were included.

ASSESSMENT

Primary variables (Youden's index [YI > 0.8 vs. <0.8], abstract conclusion positivity [positive vs. neutral/negative], number of reported items on the Standards for Reporting of Diagnostic Accuracy Studies [STARD] for Abstract guideline) and confounding variables (prospective vs. retrospective/unreported, sample size, study duration, interobserver agreement assessment, subspecialty, modality) were extracted.

STATISTICAL TESTS

Multivariable logistic regression to obtain adjusted odds ratio (OR) as a measure of the association between the primary variables and acceptance by radiology conferences and journals; 95% confidence intervals (CIs) and P-values were obtained; the threshold for statistical significance was P < 0.05.

RESULTS

A total of 1000 conference abstracts (500 ESGAR and 500 ISMRM) and 1000 journal manuscripts (505 Radiology and 495 JMRI) were included. Conference abstract acceptance was not significantly associated with YI (adjusted OR = 0.97 for YI > 0.8; CI = 0.70-1.35), conclusion positivity (OR = 1.21 for positive conclusions; CI = 0.75-1.90) or STARD for Abstracts adherence (OR = 0.96 per unit increase in reported items; CI = 0.82-1.18). Manuscripts with positive abstract conclusions were less likely to be accepted by radiology journals (OR = 0.45; CI = 0.24-0.86), while YI (OR = 0.85; CI = 0.56-1.29) and STARD for Abstracts adherence (OR = 1.06; CI = 0.87-1.30) showed no significant association. Positive conclusions were present in 86.7% of submitted conference abstracts and 90.2% of journal manuscripts.

DATA CONCLUSION

Diagnostic test accuracy studies with positive findings were not preferentially accepted by the evaluated radiology conferences or journals.

EVIDENCE LEVEL

3 TECHNICAL EFFICACY: Stage 2.

Comparison of Centor and McIsaac scores in primary care: a meta-analysis over multiple thresholds

Background

Centor and McIsaac scores are both used to diagnose group A beta-haemolytic streptococcus (GABHS) infection, but have not been compared through meta-analysis.

Aim

To compare the performance of Centor and McIsaac scores at diagnosing patients with GABHS presenting to primary care with pharyngitis.

Design and setting

A meta-analysis of diagnostic test accuracy studies conducted in primary care was performed using a novel model that incorporates data at multiple thresholds.

Method

MEDLINE, EMBASE, and PsycINFO were searched for studies published between January 1980 and February 2019. Included studies were: cross-sectional; recruited patients with sore throats from primary care; used the Centor or McIsaac score; had GABHS infection as the target diagnosis; used throat swab culture as the reference standard; and reported 2 × 2 tables across multiple thresholds. Selection and data extraction were conducted by two independent reviewers. QUADAS-2 was used to assess study quality. Summary receiver operating characteristic (SROC) curves were synthesised. Calibration curves were used to assess the transferability of results into practice.

Results

Ten studies using the Centor score and eight using the McIsaac score were included. The prevalence of GABHS ranged between 4% and 44%. The areas under the SROC curves for McIsaac and Centor scores were 0.7052 and 0.6888, respectively. The P-value for the difference (0.0164) was 0.419, suggesting the SROC curves for the tests are equivalent. Both scores demonstrated poor calibration.

Conclusion

Both Centor and McIsaac scores provide only fair discrimination of those with and without GABHS, and appear broadly equivalent in performance. The poor calibration for a positive test result suggests other point-of-care tests are required to rule in GABHS; however, with both Centor and McIsaac scores, a score of ≤0 may be sufficient to rule out infection.

Searching practices and inclusion of unpublished studies in systematic reviews of diagnostic accuracy

Introduction

Many diagnostic accuracy studies are never reported in full in a peer‐reviewed journal. Searching for unpublished studies may avoid bias due to selective publication, enrich the power of systematic reviews, and thereby help to reduce research waste. We assessed searching practices among recent systematic reviews of diagnostic accuracy.

Methods

We extracted data from 100 non‐Cochrane systematic reviews of diagnostic accuracy indexed in MEDLINE and published between October 2017 and January 2018 and from all 100 Cochrane systematic reviews of diagnostic accuracy published by December 2018, irrespective of whether meta‐analysis had been performed.

Results

Non‐Cochrane and Cochrane reviews searched a median of 4 (IQR 3‐5) and 6 (IQR 5‐9) databases, respectively; most often MEDLINE/PubMed (n = 100 and n = 100) and EMBASE (n = 81 and n = 100). Additional efforts to identify studies beyond searching bibliographic databases were performed in 76 and 98 reviews, most often through screening reference lists (n = 71 and n = 96), review/guideline articles (n = 18 and n = 52), or citing articles (n = 3 and n = 42). Specific sources of unpublished studies were searched in 22 and 68 reviews, for example, conference proceedings (n = 4 and n = 18), databases only containing conference abstracts (n = 2 and n = 33), or trial registries (n = 12 and n = 39). At least one unpublished study was included in 17 and 23 reviews. Overall, 39 of 2082 studies (1.9%) included in non‐Cochrane reviews were unpublished, and 64 of 2780 studies (2.3%) in Cochrane reviews, most often conference abstracts (97/103).

Conclusion

Searching practices vary considerably across systematic reviews of diagnostic accuracy. Unpublished studies are a minimal fraction of the evidence included in recent reviews.

Publication bias in diagnostic imaging: conference abstracts with positive conclusions are more likely to be published

OBJECTIVE

To evaluate whether imaging diagnostic test accuracy conference abstracts with positive conclusions or titles are more likely to reach full-text publication than those with negative (or neutral) conclusions or titles.

METHODS

Diagnostic accuracy research abstracts were included if they were presented at the 2011 or 2012 Radiological Society of North America conference. Full-text publication status at 5 years post conference abstract submission was determined. Conclusion and title positivity of conference abstracts were extracted, as well as potential confounding factors. The associations of conclusion and title positivity with publication status at 5 years post conference abstract submission were assessed using a multivariable logistic regression model. Conditional odds ratios were calculated to express the strength of associations, adjusting for the confounders.

RESULTS

In total, 282/400 (71%) of included conference abstracts reached full-text publication. A total of 246 out of 337 (74%) conference abstracts with positive conclusions resulted in full-text publications, compared with 26/48 (54%) with neutral conclusions and 5/15 (33%) with negative conclusions. In multivariable logistic regression, conclusion positivity was significantly associated with full-text publication (odds ratio 3.6; 95% CI 1.9-6.7 for conference abstracts with positive conclusions, compared with those with non-positive conclusions); this did not apply to title positivity (odds ratio 1.2; 95% CI 0.47-3.0).

CONCLUSION

Imaging conference abstracts with positive conclusions were more likely to be published as full-text articles. Title positivity was not associated with publication. This preferential publication pattern may lead to an overrepresentation of positive studies in the literature. An overrepresentation of positive studies may contribute to inflated estimates of test accuracy and has the potential to adversely influence patient care.

KEY POINTS

• Imaging diagnostic test accuracy conference abstracts with positive conclusions were more likely to be reported as full-text articles than those with non-positive conclusions. • The majority (75%) of imaging diagnostic test accuracy conference abstracts with positive conclusions were published, compared with only 53% and 33% with neutral and negative conclusions, respectively. • Conclusion positivity remained associated with the full-text publication of conference abstracts when controlling for multiple potential confounding variables.

Publication bias may exist among prognostic accuracy studies of middle cerebral artery Doppler ultrasound

OBJECTIVES

The objective of this study was to assess if there is evidence of publication bias in prognostic accuracy studies of middle cerebral artery (MCA) or cerebroplacental ratio (CPR) for adverse perinatal outcome.

STUDY DESIGN AND SETTING

We queried PubMed, EMBASE, the Cochrane Library, and ClinicalTrials.gov and searched abstract books of five perinatal conferences (1989-2017). We included prognostic accuracy studies on MCA and/or CPR. Highest reported accuracy estimates, sample size, study design, and conclusion positivity were extracted and compared.

RESULTS

We included 127 full-text articles and 51 conference abstracts, 29 of which had not been reported as full-text article. In conference abstracts not reported in full, median negative predictive value was significantly lower compared to full-text articles (0.79 [interquartile range 0.67-0.97] vs. 0.95 [0.89-0.99]; P < 0.001). No significant difference was identified for positive predictive value (0.62 vs. 0.59; P = 0.827), sensitivity (0.67 vs. 0.71; P = 0.159), and specificity (0.86 vs. 0.86; P = 0.632). Study design differed significantly as well (P = 0.030), with fewer prospective studies in conference abstracts not reported in full compared to full-text articles (28% vs. 54%). We found no significant differences in sample size or conclusion positivity.

CONCLUSION

Possibly, a publication bias in previously published meta-analyses of MCA and CPR has led to overly generous estimates of prognostic performance.

Publication Bias: Association of Diagnostic Accuracy in Radiology Conference Abstracts with Full-Text Publication

Background Recent investigations have identified a faster time to publication for imaging studies with higher diagnostic test accuracy (DTA), but it is unknown whether such studies are more likely to be published. A higher probability of full-text publication for studies with higher DTA could have negative consequences on clinical decision making and patient care. Purpose To evaluate the proportion of imaging diagnostic accuracy studies presented as conference abstracts that reach full-text publication and to identify whether there is an association between diagnostic accuracy and full-text publication in peer-reviewed journals within 5 years after abstract submission. Materials and Methods Diagnostic accuracy research abstracts presented at the Radiological Society of North America (RSNA) Annual Meeting in 2011 and 2012 were evaluated between September 1, 2017, and January 11, 2018. Sensitivity and specificity from the abstracts were used to calculate the Youden index (sensitivity + specificity-1); additional abstract characteristics were extracted. To identify full-text publications within 5 years after abstract submission, PubMed and Google Scholar were searched, and authors were contacted. Logistic regression analysis was used to assess for associations between higher diagnostic accuracy and full-text publication. Results A total of 7970 abstracts were evaluated, and 405 were included. Of these, 288 (71%) reached full-text publication within 5 years after abstract submission. Logistic regression analysis accounting for several confounding variables failed to show an association between reported Youden index in the conference abstract and probability of full-text publication (odds ratio, 1.01; 95% confidence interval: 0.99, 1.02; P = .21). Conclusion More than a quarter of abstracts presented at the RSNA Annual Meeting do not reach full-text publication in peer-reviewed journals. The magnitude of reported diagnostic accuracy was not associated with full-text publication, which is consistent with results of diagnostic accuracy studies in other medical specialties. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Fielding in this issue.

Full publication of results initially presented in abstracts

BACKGROUND

Abstracts of presentations at scientific meetings are usually available only in conference proceedings. If subsequent full publication of results reported in these abstracts is based on the magnitude or direction of the results, publication bias may result. Publication bias creates problems for those conducting systematic reviews or relying on the published literature for evidence about health and social care.

OBJECTIVES

To systematically review reports of studies that have examined the proportion of meeting abstracts and other summaries that are subsequently published in full, the time between meeting presentation and full publication, and factors associated with full publication.

SEARCH METHODS

We searched MEDLINE, Embase, the Cochrane Library, Science Citation Index, reference lists, and author files. The most recent search was done in February 2016 for this substantial update to our earlier Cochrane Methodology Review (published in 2007).

SELECTION CRITERIA

We included reports of methodology research that examined the proportion of biomedical results initially presented as abstracts or in summary form that were subsequently published. Searches for full publications had to be at least two years after meeting presentation.

DATA COLLECTION AND ANALYSIS

Two review authors extracted data and assessed risk of bias. We calculated the proportion of abstracts published in full using a random-effects model. Dichotomous variables were analyzed using risk ratio (RR), with multivariable models taking into account various characteristics of the reports. We assessed time to publication using Kaplan-Meier survival analyses.

MAIN RESULTS

Combining data from 425 reports (307,028 abstracts) resulted in an overall full publication proportion of 37.3% (95% confidence interval (CI), 35.3% to 39.3%) with varying lengths of follow-up. This is significantly lower than that found in our 2007 review (44.5%. 95% CI, 43.9% to 45.1%). Using a survival analyses to estimate the proportion of abstracts that would be published in full by 10 years produced proportions of 46.4% for all studies; 68.7% for randomized and controlled trials and 44.9% for other studies. Three hundred and fifty-three reports were at high risk of bias on one or more items, but only 32 reports were considered at high risk of bias overall.Forty-five reports (15,783 abstracts) with 'positive' results (defined as any 'significant' result) showed an association with full publication (RR = 1.31; 95% CI 1.23 to 1.40), as did 'positive' results defined as a result favoring the experimental treatment (RR =1.17; 95% CI 1.07 to 1.28) in 34 reports (8794 abstracts). Results emanating from randomized or controlled trials showed the same pattern for both definitions (RR = 1.21; 95% CI 1.10 to 1.32 (15 reports and 2616 abstracts) and RR = 1.17; 95% CI, 1.04 to 1.32 (13 reports and 2307 abstracts), respectively.Other factors associated with full publication include oral presentation (RR = 1.46; 95% CI 1.40 to 1.52; studied in 143 reports with 115,910 abstracts); acceptance for meeting presentation (RR = 1.65; 95% CI 1.48 to 1.85; 22 reports with 22,319 abstracts); randomized trial design (RR = 1.51; 95% CI 1.36 to 1.67; 47 reports with 28,928 abstracts); and basic research (RR = 0.78; 95% CI 0.74 to 0.82; 92 reports with 97,372 abstracts). Abstracts originating at an academic setting were associated with full publication (RR = 1.60; 95% CI 1.34 to 1.92; 34 reports with 16,913 abstracts), as were those considered to be of higher quality (RR = 1.46; 95% CI 1.23 to 1.73; 12 reports with 3364 abstracts), or having high impact (RR = 1.60; 95% CI 1.41 to 1.82; 11 reports with 6982 abstracts). Sensitivity analyses excluding reports that were abstracts themselves or classified as having a high risk of bias did not change these findings in any important way.In considering the reports of the methodology research that we included in this review, we found that reports published in English or from a native English-speaking country found significantly higher proportions of studies published in full, but that there was no association with year of report publication. The findings correspond to a proportion of abstracts published in full of 31.9% for all reports, 40.5% for reports in English, 42.9% for reports from native English-speaking countries, and 52.2% for both these covariates combined.

AUTHORS' CONCLUSIONS

More than half of results from abstracts, and almost a third of randomized trial results initially presented as abstracts fail to be published in full and this problem does not appear to be decreasing over time. Publication bias is present in that 'positive' results were more frequently published than 'not positive' results. Reports of methodology research written in English showed that a higher proportion of abstracts had been published in full, as did those from native English-speaking countries, suggesting that studies from non-native English-speaking countries may be underrepresented in the scientific literature. After the considerable work involved in adding in the more than 300 additional studies found by the February 2016 searches, we chose not to update the search again because additional searches are unlikely to change these overall conclusions in any important way.

Reporting bias in imaging: higher accuracy is linked to faster publication

OBJECTIVES

The objective of this study was to evaluate whether higher reported accuracy estimates are associated with shorter time to publication among imaging diagnostic accuracy studies.

METHODS

We included primary imaging diagnostic accuracy studies, included in meta-analyses from systematic reviews published in 2015. For each primary study, we extracted accuracy estimates, participant recruitment periods and publication dates. Our primary outcome was the association between Youden's index (sensitivity + specificity - 1, a single measure of diagnostic accuracy) and time to publication.

RESULTS

We included 55 systematic reviews and 781 primary studies. Study completion dates were missing for 238 (30%) studies. The median time from completion to publication in the remaining 543 studies was 20 months (IQR 14-29). Youden's index was negatively correlated with time from completion to publication (rho = -0.11, p = 0.009). This association remained significant in multivariable Cox regression analyses after adjusting for seven study characteristics: hazard ratio of publication was 1.09 (95% CI 1.03-1.16, p = 0.004) per unit increase for logit-transformed estimates of Youden's index. When dichotomizing Youden's index by a median split, time from completion to publication was 20 months (IQR 13-33) for studies with a Youden's index below the median, and 19 months (14-27) for studies with a Youden's index above the median (p = 0.104).

CONCLUSION

Imaging diagnostic accuracy studies with higher accuracy estimates were weakly associated with a shorter time to publication.

KEY POINTS

• Higher accuracy estimates are weakly associated with shorter time to publication. • Lag in time to publication remained significant in multivariate Cox regression analyses. • No correlation between accuracy and time from submission to publication was identified.

Magnetic resonance perfusion for differentiating low-grade from high-grade gliomas at first presentation

BACKGROUND

Gliomas are the most common primary brain tumour. They are graded using the WHO classification system, with Grade II-IV astrocytomas, oligodendrogliomas and oligoastrocytomas. Low-grade gliomas (LGGs) are WHO Grade II infiltrative brain tumours that typically appear solid and non-enhancing on magnetic resonance imaging (MRI) scans. People with LGG often have little or no neurologic deficit, so may opt for a watch-and-wait-approach over surgical resection, radiotherapy or both, as surgery can result in early neurologic disability. Occasionally, high-grade gliomas (HGGs, WHO Grade III and IV) may have the same MRI appearance as LGGs. Taking a watch-and-wait approach could be detrimental for the patient if the tumour progresses quickly. Advanced imaging techniques are increasingly used in clinical practice to predict the grade of the tumour and to aid clinical decision of when to intervene surgically. One such advanced imaging technique is magnetic resonance (MR) perfusion, which detects abnormal haemodynamic changes related to increased angiogenesis and vascular permeability, or "leakiness" that occur with aggressive tumour histology. These are reflected by changes in cerebral blood volume (CBV) expressed as rCBV (ratio of tumoural CBV to normal appearing white matter CBV) and permeability, measured by K^trans.

OBJECTIVES

To determine the diagnostic test accuracy of MR perfusion for identifying patients with primary solid and non-enhancing LGGs (WHO Grade II) at first presentation in children and adults. In performing the quantitative analysis for this review, patients with LGGs were considered disease positive while patients with HGGs were considered disease negative.To determine what clinical features and methodological features affect the accuracy of MR perfusion.

SEARCH METHODS

Our search strategy used two concepts: (1) glioma and the various histologies of interest, and (2) MR perfusion. We used structured search strategies appropriate for each database searched, which included: MEDLINE (Ovid SP), Embase (Ovid SP), and Web of Science Core Collection (Science Citation Index Expanded and Conference Proceedings Citation Index). The most recent search for this review was run on 9 November 2016.We also identified 'grey literature' from online records of conference proceedings from the American College of Radiology, European Society of Radiology, American Society of Neuroradiology and European Society of Neuroradiology in the last 20 years.

SELECTION CRITERIA

The titles and abstracts from the search results were screened to obtain full-text articles for inclusion or exclusion. We contacted authors to clarify or obtain missing/unpublished data.We included cross-sectional studies that performed dynamic susceptibility (DSC) or dynamic contrast-enhanced (DCE) MR perfusion or both of untreated LGGs and HGGs, and where rCBV and/or K^trans values were reported. We selected participants with solid and non-enhancing gliomas who underwent MR perfusion within two months prior to histological confirmation. We excluded studies on participants who received radiation or chemotherapy before MR perfusion, or those without histologic confirmation.

DATA COLLECTION AND ANALYSIS

Two review authors extracted information on study characteristics and data, and assessed the methodological quality using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. We present a summary of the study characteristics and QUADAS-2 results, and rate studies as good quality when they have low risk of bias in the domains of reference standard of tissue diagnosis and flow and timing between MR perfusion and tissue diagnosis.In the quantitative analysis, LGGs were considered disease positive, while HGGs were disease negative. The sensitivity refers to the proportion of LGGs detected by MR perfusion, and specificity as the proportion of detected HGGs. We constructed two-by-two tables with true positives and false negatives as the number of correctly and incorrectly diagnosed LGG, respectively, while true negatives and false positives are the number of correctly and incorrectly diagnosed HGG, respectively.Meta-analysis was performed on studies with two-by-two tables, with further sensitivity analysis using good quality studies. Limited data precluded regression analysis to explore heterogeneity but subgroup analysis was performed on tumour histology groups.

MAIN RESULTS

Seven studies with small sample sizes (4 to 48) met our inclusion criteria. These were mostly conducted in university hospitals and mostly recruited adult patients. All studies performed DSC MR perfusion and described heterogeneous acquisition and post-processing methods. Only one study performed DCE MR perfusion, precluding quantitative analysis.Using patient-level data allowed selection of individual participants relevant to the review, with generally low risks of bias for the participant selection, reference standard and flow and timing domains. Most studies did not use a pre-specified threshold, which was considered a significant source of bias, however this did not affect quantitative analysis as we adopted a common rCBV threshold of 1.75 for the review. Concerns regarding applicability were low.From published and unpublished data, 115 participants were selected and included in the meta-analysis. Average rCBV (range) of 83 LGGs and 32 HGGs were 1.29 (0.01 to 5.10) and 1.89 (0.30 to 6.51), respectively. Using the widely accepted rCBV threshold of <1.75 to differentiate LGG from HGG, the summary sensitivity/specificity estimates were 0.83 (95% CI 0.66 to 0.93)/0.48 (95% CI 0.09 to 0.90). Sensitivity analysis using five good quality studies yielded sensitivity/specificity of 0.80 (95% CI 0.61 to 0.91)/0.67 (95% CI 0.07 to 0.98). Subgroup analysis for tumour histology showed sensitivity/specificity of 0.92 (95% CI 0.55 to 0.99)/0.42 (95% CI 0.02 to 0.95) in astrocytomas (6 studies, 55 participants) and 0.77 (95% CI 0.46 to 0.93)/0.53 (95% CI 0.14 to 0.88) in oligodendrogliomas+oligoastrocytomas (6 studies, 56 participants). Data were too sparse to investigate any differences across subgroups.

AUTHORS' CONCLUSIONS

The limited available evidence precludes reliable estimation of the performance of DSC MR perfusion-derived rCBV for the identification of grade in untreated solid and non-enhancing LGG from that of HGG. Pooled data yielded a wide range of estimates for both sensitivity (range 66% to 93% for detection of LGGs) and specificity (range 9% to 90% for detection of HGGs). Other clinical and methodological features affecting accuracy of the technique could not be determined from the limited data. A larger sample size of both LGG and HGG, preferably using a standardised scanning approach and with an updated reference standard incorporating molecular profiles, is required for a definite conclusion.

Rapid antigen detection test for group A streptococcus in children with pharyngitis

BACKGROUND

Group A streptococcus (GAS) accounts for 20% to 40% of cases of pharyngitis in children; the remaining cases are caused by viruses. Compared with throat culture, rapid antigen detection tests (RADTs) offer diagnosis at the point of care (within five to 10 minutes).

OBJECTIVES

To determine the diagnostic accuracy of RADTs for diagnosing GAS in children with pharyngitis. To assess the relative diagnostic accuracy of the two major types of RADTs (enzyme immunoassays (EIA) and optical immunoassays (OIA)) by indirect and direct comparison.

SEARCH METHODS

We searched CENTRAL, MEDLINE, EMBASE, Web of Science, CDSR, DARE, MEDION and TRIP (January 1980 to July 2015). We also conducted related citations tracking via PubMed, handsearched reference lists of included studies and relevant review articles, and screened all articles citing included studies via Google Scholar.

SELECTION CRITERIA

We included studies that compared RADT for GAS pharyngitis with throat culture on a blood agar plate in a microbiology laboratory in children seen in ambulatory care.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened titles and abstracts for relevance, assessed full texts for inclusion, and carried out data extraction and quality assessment using the QUADAS-2 tool. We used bivariate meta-analysis to estimate summary sensitivity and specificity, and to investigate heterogeneity across studies. We compared the accuracy of EIA and OIA tests using indirect and direct evidence.

MAIN RESULTS

We included 98 unique studies in the review (116 test evaluations; 101,121 participants). The overall methodological quality of included studies was poor, mainly because many studies were at high risk of bias regarding patient selection and the reference standard used (in 73% and 43% of test evaluations, respectively). In studies in which all participants underwent both RADT and throat culture (105 test evaluations; 58,244 participants; median prevalence of participants with GAS was 29.5%), RADT had a summary sensitivity of 85.6%; 95% confidence interval (CI) 83.3 to 87.6 and a summary specificity of 95.4%; 95% CI 94.5 to 96.2. There was substantial heterogeneity in sensitivity across studies; specificity was more stable. There was no evidence of a trade-off between sensitivity and specificity. Heterogeneity in accuracy was not explained by study-level characteristics such as whether an enrichment broth was used before plating, mean age and clinical severity of participants, and GAS prevalence. The sensitivity of EIA and OIA tests was comparable (summary sensitivity 85.4% versus 86.2%). Sensitivity analyses showed that summary estimates of sensitivity and specificity were stable in low risk of bias studies.

AUTHORS' CONCLUSIONS

In a population of 1000 children with a GAS prevalence of 30%, 43 patients with GAS will be missed. Whether or not RADT can be used as a stand-alone test to rule out GAS will depend mainly on the epidemiological context. The sensitivity of EIA and OIA tests seems comparable. RADT specificity is sufficiently high to ensure against unnecessary use of antibiotics. Based on these results, we would expect that amongst 100 children with strep throat, 86 would be correctly detected with the rapid test while 14 would be missed and not receive antibiotic treatment.

Reported estimates of diagnostic accuracy in ophthalmology conference abstracts were not associated with full-text publication

OBJECTIVE

To assess whether conference abstracts that report higher estimates of diagnostic accuracy are more likely to reach full-text publication in a peer-reviewed journal.

STUDY DESIGN AND SETTING

We identified abstracts describing diagnostic accuracy studies, presented between 2007 and 2010 at the Association for Research in Vision and Ophthalmology (ARVO) Annual Meeting. We extracted reported estimates of sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and diagnostic odds ratio (DOR). Between May and July 2015, we searched MEDLINE and EMBASE to identify corresponding full-text publications; if needed, we contacted abstract authors. Cox regression was performed to estimate associations with full-text publication, where sensitivity, specificity, and AUC were logit transformed, and DOR was log transformed.

RESULTS

A full-text publication was found for 226/399 (57%) included abstracts. There was no association between reported estimates of sensitivity and full-text publication (hazard ratio [HR] 1.09 [95% confidence interval {CI} 0.98, 1.22]). The same applied to specificity (HR 1.00 [95% CI 0.88, 1.14]), AUC (HR 0.91 [95% CI 0.75, 1.09]), and DOR (HR 1.01 [95% CI 0.94, 1.09]).

CONCLUSION

Almost half of the ARVO conference abstracts describing diagnostic accuracy studies did not reach full-text publication. Studies in abstracts that mentioned higher accuracy estimates were not more likely to be reported in a full-text publication.

Time to publication among completed diagnostic accuracy studies: associated with reported accuracy estimates

Background

Previous evaluations have documented that studies evaluating the effectiveness of therapeutic interventions are not always reported, and that those with statistically significant results are published more rapidly than those without. This can lead to reporting bias in systematic reviews and other literature syntheses. We evaluated whether diagnostic accuracy studies that report promising results about the performance of medical tests are also published more rapidly.

Methods

We obtained all primary diagnostic accuracy studies included in meta-analyses of Medline-indexed systematic reviews that were published between September 2011 and January 2012. For each primary study, we extracted estimates of diagnostic accuracy (sensitivity, specificity, Youden’s index), the completion date of participant recruitment, and the publication date. We calculated the time from completion to publication and assessed associations with reported accuracy estimates.

Results

Forty-nine systematic reviews were identified, containing 92 meta-analyses and 924 unique primary studies, of which 756 could be included. Study completion dates were missing for 285 (38 %) of these. Median time from completion to publication in the remaining 471 studies was 24 months (IQR 16 to 35). Primary studies that reported higher estimates of sensitivity (Spearman’s rho = −0.14; p = 0.003), specificity (rho = −0.17; p < 0.001), and Youden’s index (rho = −0.22; p < 0.001) had significantly shorter times to publication. When comparing time to publication in studies reporting accuracy estimates above versus below the median, the median number of months was 23 versus 25 for sensitivity (p = 0.046), 22 versus 27 for specificity (p = 0.001), and 22 versus 27 for Youden’s index (p < 0.001). These differential time lags remained significant in multivariable Cox regression analyses with adjustment for other study characteristics, with hazard ratios of publication of 1.06 (95 % CI 1.02 to 1.11; p = 0.007) for logit-transformed estimates of sensitivity, 1.09 (95 % CI 1.04 to 1.14; p < 0.001) for logit-transformed estimates of specificity, and 1.09 (95 % CI 1.03 to 1.14; p = 0.001) for logit-transformed estimates of Youden’s index.

Conclusions

Time to publication was significantly shorter for studies reporting higher estimates of diagnostic accuracy compared to those reporting lower estimates. This suggests that searching and analyzing the published literature, rather than all completed studies, can produce a biased view of the performance of medical tests.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0177-4) contains supplementary material, which is available to authorized users.

Reporting Weaknesses in Conference Abstracts of Diagnostic Accuracy Studies in Ophthalmology

IMPORTANCE

Conference abstracts present information that helps clinicians and researchers to decide whether to attend a presentation. They also provide a source of unpublished research that could potentially be included in systematic reviews. We systematically assessed whether conference abstracts of studies that evaluated the accuracy of a diagnostic test were sufficiently informative.

OBSERVATIONS

We identified all abstracts describing work presented at the 2010 Annual Meeting of the Association for Research in Vision and Ophthalmology. Abstracts were eligible if they included a measure of diagnostic accuracy, such as sensitivity, specificity, or likelihood ratios. Two independent reviewers evaluated each abstract using a list of 21 items, selected from published guidance for adequate reporting. A total of 126 of 6310 abstracts presented were eligible. Only a minority reported inclusion criteria (5%), clinical setting (24%), patient sampling (10%), reference standard (48%), whether test readers were masked (7%), 2 × 2 tables (16%), and confidence intervals around accuracy estimates (16%). The mean number of items reported was 8.9 of 21 (SD, 2.1; range, 4-17).

CONCLUSIONS AND RELEVANCE

Crucial information about study methods and results is often missing in abstracts of diagnostic studies presented at the Association for Research in Vision and Ophthalmology Annual Meeting, making it difficult to assess risk for bias and applicability to specific clinical settings.

Thirty percent of abstracts presented at dental conferences are published in full: a systematic review

OBJECTIVES

To review the publication fate of abstracts presented at dental conferences and investigate the association between full publication proportion (FPP) and abstract characteristics, conference characteristics, and methodological quality of primary studies.

STUDY DESIGN AND SETTING

PubMed, EMBASE, and Google Scholar were searched up to November 2014 for studies that reported at least one FPP of abstracts presented at dental conferences, with a follow-up length of no less than 48 months.

RESULTS

Sixteen studies involving 10,365 abstracts presented at 52 conferences were included. The pooled FPP was 29.62% (95% confidence interval: 22.90%, 36.81%) for all presented abstracts and 51.97% (95% confidence interval: 43.19%, 60.70%) for randomized controlled trial abstracts. Abstract characteristics significantly associated with higher FPP included reporting of statistical analysis (P < 0.001), oral presentation (P < 0.001), basic science research (P = 0.047), and reporting of financial support (P = 0.009). Abstracts with positive (P = 0.29) or statistically significant results (P = 0.33) were not published more often than negative or nonsignificant results, respectively. In multivariable meta-regression analysis, conferences held in Asia (P < 0.001) and at a continental rather than national level (P < 0.001) were significantly associated with higher FPP.

CONCLUSIONS

Less than one-third of abstracts presented at dental conferences were published in full more than 4 years after conference presentation.

Literature survey of high-impact journals revealed reporting weaknesses in abstracts of diagnostic accuracy studies

OBJECTIVES

Informative journal abstracts are crucial for the identification and initial appraisal of studies. We aimed to evaluate the informativeness of abstracts of diagnostic accuracy studies.

STUDY DESIGN AND SETTING

PubMed was searched for reports of studies that had evaluated the diagnostic accuracy of a test against a clinical reference standard, published in 12 high-impact journals in 2012. Two reviewers independently evaluated the information contained in included abstracts using 21 items deemed important based on published guidance for adequate reporting and study quality assessment.

RESULTS

We included 103 abstracts. Crucial information on study population, setting, patient sampling, and blinding as well as confidence intervals around accuracy estimates were reported in <50% of the abstracts. The mean number of reported items per abstract was 10.1 of 21 (standard deviation 2.2). The mean number of reported items was significantly lower for multiple-gate (case-control type) studies, in reports in specialty journals, and for studies with smaller sample sizes and lower abstract word counts. No significant differences were found between studies evaluating different types of tests.

CONCLUSION

Many abstracts of diagnostic accuracy study reports in high-impact journals are insufficiently informative. Developing guidelines for such abstracts could help the transparency and completeness of reporting.

Systematic reviews and meta-analyses of diagnostic test accuracy

Systematic reviews of diagnostic test accuracy summarize the accuracy, e.g. the sensitivity and specificity, of diagnostic tests in a systematic and transparent way. The aim of such a review is to investigate whether a test is sufficiently specific or sensitive to fit its role in practice, to compare the accuracy of two or more diagnostic tests, or to investigate where existing variation in results comes from. The search strategy should be broad and preferably fully reported, to enable readers to assess the completeness of it. Included studies usually have a cross-sectional design in which the tests of interest, ideally both the index test and its comparator, are evaluated against the reference standard. They should be a reflection of the situation that the review question refers to. The quality of included studies is assessed with the Quality Assessment of Diagnostic Accuracy Studies-2 checklist, containing items such as a consecutive and all-inclusive patient selection process, blinding of index test and reference standard assessment, a valid reference standard, and complete verification of all included participants. Studies recruiting cases separately from (healthy) controls are regarded as bearing a high risk of bias. For meta-analysis, the bivariate model or the hierarchical summary receiver operating characteristic model is used. These models take into account potential threshold effects and the correlation between sensitivity and specificity. They also allow addition of covariates for investigatation of potential sources of heterogeneity. Finally, the results from the meta-analyses should be explained and interpreted for the reader, to be well understood.

Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study

Background

The validity of a meta-analysis can be understood better in light of the possible impact of publication bias. The majority of the methods to investigate publication bias in terms of small study-effects are developed for meta-analyses of intervention studies, leaving authors of diagnostic test accuracy (DTA) systematic reviews with limited guidance. The aim of this study was to evaluate if and how publication bias was assessed in meta-analyses of DTA, and to compare the results of various statistical methods used to assess publication bias.

Methods

A systematic search was initiated to identify DTA reviews with a meta-analysis published between September 2011 and January 2012. We extracted all information about publication bias from the reviews and the two-by-two tables. Existing statistical methods for the detection of publication bias were applied on data from the included studies.

Results

Out of 1,335 references, 114 reviews could be included. Publication bias was explicitly mentioned in 75 reviews (65.8%) and 47 of these had performed statistical methods to investigate publication bias in terms of small study-effects: 6 by drawing funnel plots, 16 by statistical testing and 25 by applying both methods. The applied tests were Egger’s test (n = 18), Deeks’ test (n = 12), Begg’s test (n = 5), both the Egger and Begg tests (n = 4), and other tests (n = 2). Our own comparison of the results of Begg’s, Egger’s and Deeks’ test for 92 meta-analyses indicated that up to 34% of the results did not correspond with one another.

Conclusions

The majority of DTA review authors mention or investigate publication bias. They mainly use suboptimal methods like the Begg and Egger tests that are not developed for DTA meta-analyses. Our comparison of the Begg, Egger and Deeks tests indicated that these tests do give different results and thus are not interchangeable. Deeks’ test is recommended for DTA meta-analyses and should be preferred.

Infrequent and incomplete registration of test accuracy studies: analysis of recent study reports

OBJECTIVES

To identify the proportion of articles reporting on test accuracy for which the corresponding study had been registered.

DESIGN

Analysis of a consecutive sample of published study reports.

PARTICIPANTS

PubMed was searched for publications in journals with an impact factor of 5 or higher in May and June 2012. Articles were included if they reported on original studies evaluating the accuracy of one or more diagnostic or prognostic tests or markers against a clinical reference standard in humans.

PRIMARY AND SECONDARY OUTCOME MEASURES

Primary outcome was registration of the reported test accuracy study. We additionally explored study characteristics associated with registration.

RESULTS

We found 1941 references; 351 study reports fulfilled the inclusion criteria, of which 52 studies (15%) had been registered. Of these, 27 (52%) provided a registration number in the publication, and 12 (23%) provided a reference to the publication in the registry. Registration rates were similar for studies on diagnostic versus those on prognostic tests, and among studies on imaging tests versus those on laboratory techniques. Studies reporting some form of industry involvement were more often registered (33%) than studies reporting another source of funding (11%), and studies without a (reported) source of (external) funding (9%; p<0.001). Of the registered studies, 8 (15%) had been registered after completion, 14 were registered before initiation (27%) and 30 (58%) between initiation and completion. Only 16 (31%; 5% of the total sample) had registered the published primary outcome measures before completion.

CONCLUSIONS

Few test accuracy studies published in higher impact journals are registered. Only 1 in 22 of such studies register their primary outcomes before study completion. Owing to the reasons for registering studies that investigate the cause-and-effect relationship between health-related interventions and health outcomes also apply to test accuracy studies, prospective study registration of these studies should be further promoted among investigators and journal editors.

Publication and reporting of test accuracy studies registered in ClinicalTrials.gov

BACKGROUND

Failure to publish and selective reporting are recognized problems in the biomedical literature, but their extent in the field of diagnostic testing is unknown. We aimed to identify nonpublication and discrepancies between registered records and publications among registered test accuracy studies.

METHODS

We identified studies evaluating a test's accuracy against a reference standard that were registered in ClinicalTrials.gov between January 2006 and December 2010. We included studies if their completion date was set before October 2011, allowing at least 18 months until publication. We searched PubMed, EMBASE, and Web of Science and contacted investigators for publications.

RESULTS

We included 418 studies, of which 224 (54%) had been published by mid-2013. Among studies that had been completed at least 30 months before our analyses, 45% were published within 30 months after their completion. Publication rates were high in studies registered after study completion (76%) and low for studies with an unknown (rather than completed) study status (36%). After we excluded these 2 categories, study duration was the only characteristic significantly associated with publication, with lower rates in studies lasting up to 1 year (39%) compared to studies of 13-24 months (62%) or longer (67%) (P = 0.01). In the 153 published studies that had been registered before completion, 49 (32%) showed discrepancies between the registry and publication regarding inclusion criteria (n = 19), test/threshold (n = 9), and outcomes (n = 32).

CONCLUSIONS

Failure to publish and selective reporting are prevalent in test accuracy studies. Their registration should be further promoted among researchers and journal editors.

Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation

We explored the value of handsearching to identify diagnostic test accuracy (DTA) studies to inform systematic reviews of DTA. Handsearching was conducted alongside a systematic review of the DTA of 18F-fluorodeoxyglucose positron emission tomography-computed tomography (FDG PET-CT) for colorectal cancer. Ten journals, identified by frequency analysis of studies from six imaging reviews, were handsearched for reports of DTA studies. The numbers of studies identified by handsearching and by database searching were compared. A total of 573 journal issues from ten journals were handsearched in 185 h. A total of 936 potential reports of DTA studies were identified: 25 were relevant to the FDG PET-CT review. 7/25 FDG PET-CT papers had not been identified by database searches. No papers met the systematic review inclusion criteria. The FDG-PET systematic review included 30 papers, from 24 different journals. Handsearching two of those journals identified 211 potential reports of DTA studies for all topics and 18 for FDG PET-CT. Handsearching identified previously unseen papers but did not yield unique relevant DTA studies for the specific review. DTA imaging studies are widely distributed, and it may be more efficient to choose journals to handsearch after the identification of some relevant studies. Copyright © 2012 John Wiley & Sons, Ltd.

Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies

BACKGROUND

The validity of research synthesis is threatened if published studies comprise a biased selection of all studies that have been conducted. We conducted a meta-analysis to ascertain the strength and consistency of the association between study results and formal publication.

METHODS

The Cochrane Methodology Register Database, MEDLINE and other electronic bibliographic databases were searched (to May 2009) to identify empirical studies that tracked a cohort of studies and reported the odds of formal publication by study results. Reference lists of retrieved articles were also examined for relevant studies. Odds ratios were used to measure the association between formal publication and significant or positive results. Included studies were separated into subgroups according to starting time of follow-up, and results from individual cohort studies within the subgroups were quantitatively pooled.

RESULTS

We identified 12 cohort studies that followed up research from inception, four that included trials submitted to a regulatory authority, 28 that assessed the fate of studies presented as conference abstracts, and four cohort studies that followed manuscripts submitted to journals. The pooled odds ratio of publication of studies with positive results, compared to those without positive results (publication bias) was 2.78 (95% CI: 2.10 to 3.69) in cohorts that followed from inception, 5.00 (95% CI: 2.01 to 12.45) in trials submitted to regulatory authority, 1.70 (95% CI: 1.44 to 2.02) in abstract cohorts, and 1.06 (95% CI: 0.80 to 1.39) in cohorts of manuscripts.

CONCLUSION

Dissemination of research findings is likely to be a biased process. Publication bias appears to occur early, mainly before the presentation of findings at conferences or submission of manuscripts to journals.