Methodology in Diagnostic Laboratory Test Research in Clinical Chemistry and Clinical Chemistry and Laboratory Medicine

A survey of the awareness, knowledge, policies and views of veterinary journal Editors-in-Chief on reporting guidelines for publication of research

BACKGROUND

Wider adoption of reporting guidelines by veterinary journals could improve the quality of published veterinary research. The aims of this study were to assess the knowledge and views of veterinary Editors-in-Chief on reporting guidelines, identify the policies of their journals, and determine their information needs. Editors-in-Chief of 185 journals on the contact list for the International Association of Veterinary Editors (IAVE) were surveyed in April 2012 using an online questionnaire which contained both closed and open questions.

RESULTS

The response rate was 36.8% (68/185). Thirty-six of 68 editors (52.9%) stated they knew what a reporting guideline was before receiving the questionnaire. Editors said they had found out about reporting guidelines primarily through articles in other journals, via the Internet and through their own journal. Twenty of 57 respondents (35.1%) said their journal referred to reporting guidelines in its instructions to authors. CONSORT, REFLECT, and ARRIVE were the most frequently cited. Forty-four of 68 respondents (68.2%) believed that reporting guidelines should be adopted by all refereed veterinary journals. Qualitative analysis of the open questions revealed that lack of knowledge, fear, resistance to change, and difficulty in implementation were perceived as barriers to the adoption of reporting guidelines by journals. Editors suggested that reporting guidelines be promoted through communication and education of the veterinary community, with roles for the IAVE and universities. Many respondents believed a consensus policy on guideline implementation was needed for veterinary journals.

CONCLUSIONS

Further communication and education about reporting guidelines for editors, authors and reviewers has the potential to increase their adoption by veterinary journals in the future.

Procalcitonin as a marker of postoperative complications

OBJECTIVE

Procalcitonin (PCT) is a 116 amino acid peptide that functions as a pro-hormone for calcitonin in the C cells of the thyroid gland. Large quantities of intact PCT are present in the blood of patients with sepsis, particularly when organ dysfunction occurs. PCT has been proposed as an early marker of postoperative complications. The aim of this study was to examine the diagnostic accuracy of PCT as a marker of postoperative complications by systematically reviewing the existing literature.

MATERIAL AND METHODS

The databases PubMed, Embase and the Cochrane Library were searched to find studies on the diagnostic accuracy of PCT in the postoperative phase. Primary studies were retrieved using specific inclusion and exclusion criteria.

RESULTS

A total of nine studies were included. These studies were heterogeneous regarding the spectrum of patients, complications, design and methodological quality according to QUADAS (quality assessment of studies of diagnostic accuracy). This could explain the marked variation in diagnostic accuracy. Considering all types of complications the sensitivity ranged from 37% to 100% and the specificity from 70% to 100%. On examining the infectious complications separately, it was found that the sensitivity ranged from 70% to 86% and the specificity from 45% to 98%.

CONCLUSIONS

Owing to a pronounced heterogeneity among the existing studies, the diagnostic accuracy of PCT as a marker for postoperative complications is not yet sufficiently clarified.

Overinterpretation of Clinical Applicability in Molecular Diagnostic Research

Background: We evaluated whether articles on molecular diagnostic tests interpret appropriately the clinical applicability of their results.

Methods: We selected original-research articles published in 2006 that addressed the diagnostic value of a molecular test. We defined overinterpretation of clinical applicability by means of prespecified rules that evaluated study design, conclusions regarding applicability, presence of statements suggesting the need for further clinical evaluation of the test, and diagnostic accuracy. Two reviewers independently evaluated the articles; consensus was reached after discussion and arbitration by a third reviewer.

Results: Of 108 articles included in the study, 82 (76%) used a design that used healthy controls or alternative-diagnosis controls, only 15 (11%) addressed a clinically relevant population similar to that in which the test might be applied in practice, 104 articles (96%) made definitely favorable or promising statements regarding clinical applicability, and 61 (56%) of the articles apparently overinterpreted the clinical applicability of their findings. Articles published in journals with higher impact factors were more likely to overinterpret their results than those with lower impact factors (adjusted odds ratio, 1.71 per impact factor quartile; 95% CI, 1.09–2.69; P = 0.020). Overinterpretation was more common when authors were based in laboratories than in clinical settings (adjusted odds ratio, 18.7; 95% CI, 1.41–249; P = 0.036).

Conclusions: Although expectations are high for new diagnostic tests based on molecular techniques, the majority of published research has involved preclinical phases of research. Overinterpretation of the clinical applicability of findings for new molecular diagnostic tests is common.

[Meta-analysis for the pooled sensitivity and specificity of hepatitis B surface antigen rapid tests]

BACKGROUND

Although hepatitis B surface antigen (HBsAg) rapid test based on immunochromatographic assay (ICA) is now widely used, the test has not been evaluated sufficiently enough to validate its performance. Thus, it is important to summarize the clinical performance of the test kits. In this study, we performed meta-analysis for the performance of the HBsAg rapid tests.

METHODS

PubMed database was searched using keywords about the accuracy of diagnostic tests for hepatitis B virus (HBV) infection. Two investigators assessed methodological quality utilizing standards for reporting of diagnostic accuracy studies (STARD) checklist. After performing a heterogeneity test, we obtained pooled sensitivity and specificity. Positive and negative predictive values (PPV and NPV) were simulated according to HBV prevalence.

RESULTS

A total of 38 studies was selected from 10 papers. The quality scores ranged from 3 to 13 (median, 8). Kappa value was good (0.85). The performance of the 38 studies was heterogeneous. When 33 studies with better quality from 7 papers were re-selected, the pooled sensitivity and specificity were 98.07% (95% confidence interval, CI: 97.67-98.47%) and 99.56% (95% CI: 99.21-99.91%), respectively. With an HBV prevalence of 5%, PPV and NPV were predicted to be 92.14% and 99.90%, respectively.

CONCLUSIONS

In view of high HBV prevalence in Korea, it is thought that the HBsAg rapid test can be used for HBV screening in small-sized laboratories or for epidemiologic studies. This study should be helpful in establishing a guideline for the proper performance evaluation of the HBsAg rapid tests.

QUADOMICS: an adaptation of the Quality Assessment of Diagnostic Accuracy Assessment (QUADAS) for the evaluation of the methodological quality of studies on the diagnostic accuracy of '-omics'-based technologies

OBJECTIVES

To adapt the Quality Assessment of Diagnostic Accuracy Assessment (QUADAS) to the particular methodological challenges posed by research on '-omics'-based diagnostic tests.

DESIGN AND METHODS

We generated new guidelines by appraising the suitability of each criterion from QUADAS to '-omics'-based diagnostic research, and by adding new items that addressed specific sources of error. In addition, we defined four phases in the evaluation of a diagnostic test.

RESULTS

Twelve of the 14 criteria from QUADAS were retained in the new tool. The items relating to selection criteria and the description of the test were reformulated, and the criteria about external validation and the availability of clinical data were applied only in studies in the last research phase. Four new items were incorporated to QUADOMICS related to pre-analytical conditions and methods to avoid overfitting.

CONCLUSIONS

QUADOMICS is an adaptation of QUADAS to the special nature of '-omics'-based diagnostic research. The tool adds new items that assess quality issues specific to this research, and may enhance the application of '-omics'-based discoveries to clinical \and public health practice.

[Development of a Web-based Program to Calculate Sample Size for Evaluating the Performance of In Vitro Diagnostic Kits.]

BACKGROUND

Many studies evaluating the performance of in vitro diagnostic kits have been criticized for the lack of reliability. To attain reliability those evaluation studies should be preceded by sample size calculation ensuring statistical power. This study was intended to develop a web-based system to estimate the sample size, which was often neglected because it would require expert knowledge in statistics.

METHODS

For sample size calculation, we extracted essential parameters from the performance studies on the 3rd generation anti-hepatitis C virus (HCV) kits reported in the literature. We developed a system with PHP web-script language and MySQL. The statistical models used in this system were as follows; one sample without power consideration (model 1), one sample with power consideration (model 2), and two samples with power consideration (model 3).

RESULTS

Among the articles published between 1989 and 2005, 13 articles that evaluated the performance of anti-HCV kits were identified by searching with Medical Subject Headings (MeSH). The diagnostic sensitivity was 83-100% with a median of 145 samples (range; 12-1,091) and the specificity was 97-100% with a median of 1,025 samples (range; 33-4,381). The estimated sample size would be 280 in the model 1, 817 in the model 2, and 1,510 in the model 3, when we set 2% prevalence of HCV infection, 95% sensitivity of a conventional kit, 97% sensitivity of a new kit , 95% significance level (two-sided test), 2% allowable error, and 80% power.

CONCLUSIONS

Our study indicates that an insufficient sample size is still a problem in performance evaluation. Our system should be helpful in increasing the reliability of performance evaluation by providing an appropriate sample size.

Quality of reporting of test accuracy studies in reproductive medicine: impact of the Standards for Reporting of Diagnostic Accuracy (STARD) initiative

OBJECTIVE

To evaluate the extent to which test accuracy studies published in two leading reproductive medicine journals in the years 1999 and 2004 adhered to the Standards for Reporting of Diagnostic Accuracy (STARD) initiative parameters, and to explore whether the introduction of the STARD statement has led to an improved quality of reporting.

DESIGN

Structured literature search. Articles that reported on the diagnostic performance of a test in comparison with a reference standard were eligible for inclusion. For each article we scored how well the 25 items of the STARD checklist were reported. These items deal with the study question, study participants, study design, test methods, reference standard, statistical methods, reporting of results, and conclusions. We calculated the total number of reported STARD items per article, summary scores for each STARD item, and the average number of reported STARD items per publication year.

SETTING

Not applicable.

PATIENT(S)

Not applicable.

INTERVENTION(S)

Not applicable.

MAIN OUTCOME MEASURE(S)

Quality of reporting.

RESULT(S)

We found 24 studies reporting on test accuracy in reproductive medicine in 1999 and 27 studies in 2004. The mean number of reported STARD items for articles published in 1999 was 12.1 +/- 3.3 (range 6.5-20) and 12.4 +/- 3.2 (range 7-17.5) in 2004, after publication of the STARD statement. Overall, less than half of the studies reported adequately on 50% or more of the STARD items. The reporting of individual items showed a wide variation. There was no significant improvement in mean number of reported items for the articles published after the introduction of the STARD statement.

CONCLUSION(S)

Authors of test accuracy studies in the two leading fertility journals poorly report the design, conduct, methodology, and statistical analysis of their study. Strict adherence to the STARD guidelines should be encouraged.

Evaluation of the research methodology in genetic, molecular and proteomic tests

INTRODUCTION

The advances in genomic analysis technologies have conducted to the development of new diagnostic tests in the clinical practice. As well as it happened in other diagnostic fields, the knowledge of the main flaws affecting genetic investigation will facilitate the application of the results.

METHODS

We included 44 original articles that evaluate diagnostic exactitude of genetic and molecular tests (including proteomic), published from 2002 to June 2005 in five international publications: JAMA, Lancet, New England Journal of Medicine, Cancer Research and Clinical Cancer Research. We examined adherence to 24 methodological criteria included in the guide STARD (Standards for Reporting of Diagnostic Accuracy.

RESULTS

The mean number of methodological criteria satisfied was 9.8 (95% CI 8.8-10.6); the greater deficiencies were in aspects related to the description of patient's selection 9 (20%), the treatment of indeterminate results 5 (11%) and the determination of test reproducibility 6 (13%). It was observed a high fulfillment in the description of the gold standard 39 (87%) and in the methodology of the test 28 (62%).

DISCUSSION

The methodologic quality of the evaluated articles is lower than the quality observed in other research fields. The methodologic aspects that most need improvement are those linked to the clinical information of the populations studied and the reproducibility of the tests. The research and development of new genetic-molecular technologies requires a better fulfillment of the epidemiological and clinical criteria already followed by other diagnostic fields.

Laboratory variables and stratification of metastatic colorectal cancer patients: recommendations for therapeutic trials and for clinical practice guidelines

OBJECTIVE

To identify, through a systematic review of the literature, the laboratory variables that, in addition to performance status and to the degree of tumor invasion, would allow a more accurate stratification of metastatic colorectal cancer patients who participate in chemotherapy trials, with or without radiotherapy. SECONDARY AIM: To compare the results of our systematic review with the recommendations made in current clinical practice guidelines, and with the results of related systematic reviews.

METHODS

Update of two recently published systematic reviews, without metaanalysis, following the recommendations of the International Federation of Clinical Chemistry and Laboratory Medicine, and taking into account the Consolidated Standards of Reporting Trials statement.

RESULTS

Of 877 publications retrieved, reasonable exclusion and inclusion criteria allow us to include 15 studies in our systematic review, thus confirming the low quality of clinical research in laboratory medicine. Four variables were most often found "significant" in multivariate statistical analysis: pretherapeutic levels of laboratory tests (13/15, 87%), degree of tumor invasion (9/13, 69%), treatment, or response to treatment (6/9, 67%), and performance status (8/13, 62%). The laboratory variable whose measurements are quite often recommended in the 10 clinical practice guidelines or in the four related systematic reviews that we retrieved are carcinoembryonic antigen (CEA), and liver function tests to a lesser extent.

CONCLUSIONS

Available evidence supports the recommendation that in all metastatic colorectal cancer patients who participate in therapeutic trials, the following pretreatment laboratory variables should be systematically measured: blood cell counts, and haemoglobin, plasma prothrombin time, serum alkaline phosphatase (ALP), lactate dehydrogenase, transaminases, albumin, bilirubin, and CEA. If other tests were to be added, gamma glutamyl transferase, and erythrocyte sedimentation rate might perhaps be proposed. Further studies would be necessary to support the addition to this list, of other tests [e.g., cancer antigen (CA) 19-9]. Rather than using laboratory variables according to arbitrary thresholds, it seems recommendable to use them as continuous variables, and if possible, in terms of kinetics. Many clinical practice guidelines do not use levels of evidence in order to grade the strength of their recommendations, but rather seem to be based on experts opinions which are not always in agreement with the results of systematic reviews.