Reporting quality of diagnostic accuracy studies: a systematic review and meta-analysis of investigations on adherence to STARD

Point-of-Care Lung Ultrasound in Emergency Medicine: A Scoping Review With an Interactive Database

BACKGROUND

This scoping review was conducted to provide an overview of the evidence of point-of-care lung ultrasound (LUS) in emergency medicine. By emphasizing clinical topics, time trends, study designs, and the scope of the primary outcomes, a map is provided for physicians and researchers to guide their future initiatives.

RESEARCH QUESTION

Which study designs and primary outcomes are reported in published studies of LUS in emergency medicine?

STUDY DESIGN AND METHODS

We performed a systematic search in the PubMed/MEDLINE, Embase, Web of Science, Scopus, and Cochrane Library databases for LUS studies published prior to May 13, 2023. Study characteristics were synthesized quantitatively. The primary outcomes in all papers were categorized into the hierarchical Fryback and Thornbury levels.

RESULTS

A total of 4,076 papers were screened and, following selection and handsearching, 406 papers were included. The number of publications doubled from January 2020 to May 2023 (204 to 406 papers). The study designs were primarily observational (n = 375 [92%]), followed by randomized (n = 18 [4%]) and case series (n = 13 [3%]). The primary outcome measure concerned diagnostic accuracy in 319 papers (79%), diagnostic thinking in 32 (8%), therapeutic changes in 4 (1%), and patient outcomes in 14 (3%). No increase in the proportions of randomized controlled trials or the scope of primary outcome measures was observed with time. A freely available interactive database was created to enable readers to search for any given interest (https://public.tableau.com/app/profile/blinded/viz/LUSinEM_240216/INFO).

INTERPRETATION

Observational diagnostic studies have been produced in abundance, leaving a paucity of research exploring clinical utility. Notably, research exploring whether LUS causes changes to clinical decisions is imperative prior to any further research being made into patient benefits.

Assessment of Advanced Diagnostic Bronchoscopy Outcomes for Peripheral Lung Lesions: A Delphi Consensus Definition of Diagnostic Yield and Recommendations for Patient-centered Study Designs. An Official American Thoracic Society/American College of Chest Physicians Research Statement

Background: Advanced diagnostic bronchoscopy targeting the lung periphery has developed at an accelerated pace over the last two decades, whereas evidence to support introduction of innovative technologies has been variable and deficient. A major gap relates to variable reporting of diagnostic yield, in addition to limited comparative studies. Objectives: To develop a research framework to standardize the evaluation of advanced diagnostic bronchoscopy techniques for peripheral lung lesions. Specifically, we aimed for consensus on a robust definition of diagnostic yield, and we propose potential study designs at various stages of technology development. Methods: Panel members were selected for their diverse expertise. Workgroup meetings were conducted in virtual or hybrid format. The cochairs subsequently developed summary statements, with voting proceeding according to a modified Delphi process. The statement was cosponsored by the American Thoracic Society and the American College of Chest Physicians. Results: Consensus was reached on 15 statements on the definition of diagnostic outcomes and study designs. A strict definition of diagnostic yield should be used, and studies should be reported according to the STARD (Standards for Reporting Diagnostic Accuracy Studies) guidelines. Clinical or radiographic follow-up may be incorporated into the reference standard definition but should not be used to calculate diagnostic yield from the procedural encounter. Methodologically robust comparative studies, with incorporation of patient-reported outcomes, are needed to adequately assess and validate minimally invasive diagnostic technologies targeting the lung periphery. Conclusions: This American Thoracic Society/American College of Chest Physicians statement aims to provide a research framework that allows greater standardization of device validation efforts through clearly defined diagnostic outcomes and robust study designs. High-quality studies, both industry and publicly funded, can support subsequent health economic analyses and guide implementation decisions in various healthcare settings.

A guide to conducting systematic reviews of clinical laboratory tests

Clinical laboratory professionals have an instrumental role in supporting clinical decision making with the optimal use of laboratory testing for screening, risk stratification, diagnostic, prognostic, treatment selection and monitoring of different states of health and disease. Delivering evidence-based laboratory medicine relies on review of available data and literature. The information derived, supports many national policies to improve patient care through clinical practice guidelines or best practice recommendations. The quality, validity and bias of this literature is variable. Hence, there is a need to collate similar studies and data and analyse them critically. Systematic review, thus, becomes the most important source of evidence. A systematic review, unlike a scoping or narrative review, involves a thorough understanding of the procedure involved and a stepwise methodology. There are nuances that need some consideration for laboratory medicine systematic reviews. The purpose of this article is to describe the process of performing a systematic review in the field of laboratory medicine, describing the available methodologies, tools and software packages that can be used to facilitate this process.

STROBE, CONSORT, PRISMA, MOOSE, STARD, SPIRIT, and other guidelines - Overview and application

The purpose of research is to seek answers and new knowledge. When conducted properly and systematically, research adds to humanity's corpus of knowledge and hence to our general advancement. However, this is only possible if reported research is accurate and transparent. Guidelines for all the major types of studies (STROBE, CONSORT, PRISMA, MOOSE, STARD, and SPIRIT) have been developed and refined over the years, and their inception, development, and application are briefly discussed in this paper. Indeed, there are currently over 250 of these guidelines for various types of medical research, and these are published by the EQUATOR network. This paper will also briefly review progress in acceptance and adoption of these guidelines.

Authors arbitrarily used methodological approaches to analyze the quality of reporting in research reports: a meta-research study

OBJECTIVES

Many authors used reporting checklists as an assessment tool to analyze the reporting quality of diverse types of evidence. We aimed to analyze methodological approaches used by researchers assessing reporting quality of evidence in randomized controlled trials, systematic reviews, and observational studies.

STUDY DESIGN AND SETTING

We analyzed articles reporting quality assessment of evidence with Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA), CONsolidated Standards of Reporting Trials (CONSORT), or the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) checklists published up to 18 July 2021. We analyzed methods used for assessing reporting quality.

RESULTS

Among 356 analyzed articles, 293 (88%) investigated a specific thematic field. The CONSORT checklist (N = 225; 67%) was most often used, in its original, modified, partial form, or its extension. Numerical scores were given for adherence to checklist items in 252 articles (75%), of which 36 articles (11%) used various reporting quality thresholds. In 158 (47%) articles, predictors of adherence to reporting checklist were analyzed. The most studied factor associated with adherence to reporting checklist was the year of article publication (N = 82; 52%).

CONCLUSION

The methodology used for assessing reporting quality of evidence varied considerably. The research community needs a consensus on a consistent methodology for assessing the quality of reporting.

Has the quality of reporting improved since it became mandatory to use the Standards for Reporting Diagnostic Accuracy?

OBJECTIVES

To investigate whether making the Standards for Reporting Diagnostic Accuracy (STARD) mandatory by the leading journal 'Radiology' in 2016 improved the quality of reporting of diagnostic accuracy studies.

METHODS

A validated search term was used to identify diagnostic accuracy studies published in Radiology in 2015 and 2019. STARD adherence was assessed by two independent reviewers. Each item was scored as yes (1 point) if adequately reported or as no (0 points) if not. The total STARD score per article was calculated. Wilcoxon-Mann-Whitney tests were used to evaluate differences of the total STARD scores between 2015 and 2019. In addition, the total STARD score was compared between studies stratified by study design, citation rate, and data collection.

RESULTS

The median number of reported STARD items for the total of 66 diagnostic accuracy studies from 2015 to 2019 was 18.5 (interquartile range [IQR] 17.5-20.0) of 29. Adherence to the STARD checklist significantly improved the STARD score from a median of 18.0 (IQR 15.5-19.5) in 2015 to a median of 19.5 (IQR 18.5-21.5) in 2019 (p < 0.001). No significant differences were found between studies stratified by mode of data collection (prospective vs. retrospective studies, p = 0.68), study design (cohort vs. case-control studies, p = 0.81), and citation rate (two groups divided by median split [< 0.56 citations/month vs. ≥ 0.56 citations/month], p = 0.54).

CONCLUSIONS

Making use of the STARD checklist mandatory significantly increased the adherence with reporting standards for diagnostic accuracy studies and should be considered by editors and publishers for widespread implementation.

CRITICAL RELEVANCE STATEMENT

Editors may consider making reporting guidelines mandatory to improve the scientific quality.

Recommendations for reporting measures of diagnostic accuracy

This article serves as the third in a series that offers recommendations for optimal data reporting, specifically focusing on the statistical methods most frequently reported in Cytopathology articles. Measures of diagnostic accuracy were among the most commonly reported results in Cytopathology, with sensitivity and/or specificity reported in 32% of the reviewed articles. This article will provide a brief overview of common measures of diagnostic accuracy, including sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, overall diagnostic accuracy, receiver operating characteristic (ROC) curve, and area under the curve (AUC). Reporting recommendations for these measures will be reviewed, including the suggestion of providing a 2 × 2 contingency table when possible, or numerator and denominator values for calculations when all values needed for a contingency table are not known, and calculation of ROC and AUC if relevant. Additionally, paired measures should be reported, including sensitivity with specificity, positive with negative predictive values, and positive with negative likelihood ratios. Calculating 95% confidence intervals (CI) for the measures is recommended, with several methods to choose from, including the Wald interval, Wilson interval, Clopper-Pearson interval, Agresti-Coull interval, and Bayesian highest posterior density (HPD) interval. Since there are various methods for CI calculations, the author encourages the reader to consult with a trained statistician to identify the most appropriate method based on the data, which should be reported in the methods section of the resulting write-up.

The reporting of prognostic prediction models for obstetric care was poor: a cross-sectional survey of 10-year publications

BACKGROUND

To investigate the reporting of prognostic prediction model studies in obstetric care through a cross-sectional survey design.

METHODS

PubMed was searched to identify prognostic prediction model studies in obstetric care published from January 2011 to December 2020. The quality of reporting was assessed by the TRIPOD checklist. The overall adherence by study and the adherence by item were calculated separately, and linear regression analysis was conducted to explore the association between overall adherence and prespecified study characteristics.

RESULTS

A total of 121 studies were included, while no study completely adhered to the TRIPOD. The results showed that the overall adherence was poor (median 46.4%), and no significant improvement was observed after the release of the TRIPOD (43.9 to 46.7%). Studies including both model development and external validation had higher reporting quality versus those including model development only (68.1% vs. 44.8%). Among the 37 items required by the TRIPOD, 10 items were reported adequately with an adherence rate over of 80%, and the remaining 27 items had an adherence rate ranging from 2.5 to 79.3%. In addition, 11 items had a report rate lower than 25.0% and even covered key methodological aspects, including blinding assessment of predictors (2.5%), methods for model-building procedures (4.5%) and predictor handling (13.5%), how to use the model (13.5%), and presentation of model performance (14.4%).

CONCLUSIONS

In a 10-year span, prognostic prediction studies in obstetric care continued to be poorly reported and did not improve even after the release of the TRIPOD checklist. Substantial efforts are warranted to improve the reporting of obstetric prognostic prediction models, particularly those that adhere to the TRIPOD checklist are highly desirable.

Systematic review identifies deficiencies in reporting of diagnostic test accuracy among clinical decision support systems

OBJECTIVES

This systematic review assesses the reporting quality and risk of bias in studies evaluating the diagnostic test accuracy (DTA) of clinical decision support systems (CDSS).

STUDY DESIGN AND SETTING

The Cochrane Library, PubMed/MEDLINE, Scopus, and Web of Science were searched for studies, published between January 1, 2016 and May 31, 2021, evaluating the DTA of CDSS for human patients. Articles using a patient's self-diagnosis, assessing disease severity, focusing on treatment/follow-up, or comparing pre-post CDSS implementation periods were excluded. All eligible studies were assessed for reporting quality using STARD 2015 and for risk of bias using QUADAS-2. Item ratings were presented using heat maps. This study was reported as per PRISMA-DTA.

RESULTS

In total, 158 of 2,820 screened articles were included in the analysis. The studies were heterogeneous in terms of study characteristics, reporting quality, risk of biases, and applicability concerns with few highly rated studies. Mostly the overall quality was deficient for items addressing the domains 'methodology,' 'results,' and 'other information'.

CONCLUSION

Our analysis revealed shortcomings in critical domains of reporting quality and risk of bias, indicating the need for additional guidance and training in an interdisciplinary scientific field with mixed biostatistical expertise.

Reporting of Artificial Intelligence Diagnostic Accuracy Studies in Pathology Abstracts: Compliance with STARD for Abstracts Guidelines

Artificial intelligence (AI) research is transforming the range tools and technologies available to pathologists, leading to potentially faster, personalized and more accurate diagnoses for patients. However, to see the use of tools for patient benefit and achieve this safely, the implementation of any algorithm must be underpinned by high quality evidence from research that is understandable, replicable, usable and inclusive of details needed for critical appraisal of potential bias. Evidence suggests that reporting guidelines can improve the completeness of reporting of research, especially with good awareness of guidelines. The quality of evidence provided by abstracts alone is profoundly important, as they influence the decision of a researcher to read a paper, attend a conference presentation or include a study in a systematic review. AI abstracts at two international pathology conferences were assessed to establish completeness of reporting against the STARD for Abstracts criteria. This reporting guideline is for abstracts of diagnostic accuracy studies and includes a checklist of 11 essential items required to accomplish satisfactory reporting of such an investigation. A total of 3488 abstracts were screened from the United States & Canadian Academy of Pathology annual meeting 2019 and the 31st European Congress of Pathology (ESP Congress). Of these, 51 AI diagnostic accuracy abstracts were identified and assessed against the STARD for Abstracts criteria for completeness of reporting. Completeness of reporting was suboptimal for the 11 essential criteria, a mean of 5.8 (SD 1.5) items were detailed per abstract. Inclusion was variable across the different checklist items, with all abstracts including study objectives and no abstracts including a registration number or registry. Greater use and awareness of the STARD for Abstracts criteria could improve completeness of reporting and further consideration is needed for areas where AI studies are vulnerable to bias.

Machine learning prediction models in orthopedic surgery: A systematic review in transparent reporting

Machine learning (ML) studies are becoming increasingly popular in orthopedics but lack a critically appraisal of their adherence to peer-reviewed guidelines. The objective of this review was to (1) evaluate quality and transparent reporting of ML prediction models in orthopedic surgery based on the transparent reporting of multivariable prediction models for individual prognosis or diagnosis (TRIPOD), and (2) assess risk of bias with the Prediction model Risk Of Bias ASsessment Tool. A systematic review was performed to identify all ML prediction studies published in orthopedic surgery through June 18th, 2020. After screening 7138 studies, 59 studies met the study criteria and were included. Two reviewers independently extracted data and discrepancies were resolved by discussion with at least two additional reviewers present. Across all studies, the overall median completeness for the TRIPOD checklist was 53% (interquartile range 47%-60%). The overall risk of bias was low in 44% (n = 26), high in 41% (n = 24), and unclear in 15% (n = 9). High overall risk of bias was driven by incomplete reporting of performance measures, inadequate handling of missing data, and use of small datasets with inadequate outcome numbers. Although the number of ML studies in orthopedic surgery is increasing rapidly, over 40% of the existing models are at high risk of bias. Furthermore, over half incompletely reported their methods and/or performance measures. Until these issues are adequately addressed to give patients and providers trust in ML models, a considerable gap remains between the development of ML prediction models and their implementation in orthopedic practice.

Training sample selection: Impact on screening automation in diagnostic test accuracy reviews

When performing a systematic review, researchers screen the articles retrieved after a broad search strategy one by one, which is time‐consuming. Computerised support of this screening process has been applied with varying success. This is partly due to the dependency on large amounts of data to develop models that predict inclusion. In this paper, we present an approach to choose which data to use in model training and compare it with established approaches. We used a dataset of 50 Cochrane diagnostic test accuracy reviews, and each was used as a target review. From the remaining 49 reviews, we selected those that most closely resembled the target review's clinical topic using the cosine similarity metric. Included and excluded studies from these selected reviews were then used to develop our prediction models. The performance of models trained on the selected reviews was compared against models trained on studies from all available reviews. The prediction models performed best with a larger number of reviews in the training set and on target reviews that had a research subject similar to other reviews in the dataset. Our approach using cosine similarity may reduce computational costs for model training and the duration of the screening process.

Predictors of plasma leakage among dengue patients in Thailand: A plasma-leak score analysis

Delayed plasma leakage recognition could lead to improper fluid administration resulting in dengue shock syndrome, subsequently, multi-organ failure, and death. This prospective observational study was conducted in Bangkok, Thailand, between March 2018 and February 2020 to determine predictors of plasma leakage and develop a plasma leakage predictive score among dengue patients aged ≥15 years. Of 667 confirmed dengue patients, 318 (47.7%) developed plasma leakage, and 349 (52.3%) had no plasma leakage. Multivariate analysis showed three independent factors associated with plasma leakage, including body mass index ≥25.0 kg/m2 (odds ratio [OR] = 1.784; 95% confidence interval [CI] = 1.040-3.057; P = 0.035), platelet count <100,000/mm3 on fever days 3 to 4 (OR = 2.151; 95% CI = 1.269-3.647; P = 0.004), and aspartate aminotransferase or alanine aminotransferase ≥100 U/l on fever days 3 to 4 (OR = 2.189; 95% CI = 1.231-3.891; P = 0.008). Because these three parameters had evidence of equality, each independent factor was weighted to give a score of 1 with a total plasma-leak score of 3. Higher scores were associated with increased plasma leakage occurrence, with ORs of 2.017 (95% CI = 1.052-3.869; P = 0.035) for score 1, 6.158 (95% CI = 2.914-13.015; P <0.001) for score 2, and 6.300 (95% CI = 2.419-16.407; P <0.001) for score 3. The area under the receiver operating characteristics curves for predicting plasma leakage was good (0.677 [95% CI = 0.616-0.739]). Patients with a plasma-leak score ≥1 had high sensitivity (88.8%), and those with a plasma-leak score of 3 had high specificity (93.4%) for plasma leakage occurrence. This simple and easily accessible clinical score might help physicians provide early and timely appropriate clinical dengue management in endemic areas.

Validity of negative bone biopsy in suspicious bone lesions

Background

The presence of malignant cells in bone biopsies is considered gold standard to verify occurrence of cancer, whereas a negative bone biopsy can represent a false negative, with a risk of increasing patient morbidity and mortality and creating misleading conclusions in cancer research. However, a paucity of literature documents the validity of negative bone biopsy as an exclusion criterion for the presence of skeletal malignancies.

Purpose

To investigate the validity of a negative bone biopsy in bone lesions suspicious of malignancy.

Material and Method

A retrospective cohort of 215 consecutive targeted non-malignant skeletal biopsies from 207 patients (43% women, 57% men, median age 64, and range 94) representing suspicious focal bone lesions, collected from January 1, 2011, to July 31, 2013, was followed over a 2-year period to examine any additional biopsy, imaging, and clinical follow-up information to categorize the original biopsy as truly benign, malignant, or equivocal. Standard deviations and 95% confidence intervals were calculated.

Results

210 of 215 biopsies (98%; 95% CI 0.94-0.99) showed to be truly benign 2 years after initial biopsy. Two biopsies were false negatives (1%; 95% CI 0.001-0.03), and three were equivocal (lack of imaging description).

Conclusion

Our study documents negative bone biopsy as a valid criterion for the absence of bone metastasis. Since only 28% had a confirmed diagnosis of prior cancer and not all patients received adequately sensitive imaging, our results might not be applicable to all cancer patients with suspicious bone lesions.

Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol

INTRODUCTION

Standards for Reporting of Diagnostic Accuracy Study (STARD) was developed to improve the completeness and transparency of reporting in studies investigating diagnostic test accuracy. However, its current form, STARD 2015 does not address the issues and challenges raised by artificial intelligence (AI)-centred interventions. As such, we propose an AI-specific version of the STARD checklist (STARD-AI), which focuses on the reporting of AI diagnostic test accuracy studies. This paper describes the methods that will be used to develop STARD-AI.

METHODS AND ANALYSIS

The development of the STARD-AI checklist can be distilled into six stages. (1) A project organisation phase has been undertaken, during which a Project Team and a Steering Committee were established; (2) An item generation process has been completed following a literature review, a patient and public involvement and engagement exercise and an online scoping survey of international experts; (3) A three-round modified Delphi consensus methodology is underway, which will culminate in a teleconference consensus meeting of experts; (4) Thereafter, the Project Team will draft the initial STARD-AI checklist and the accompanying documents; (5) A piloting phase among expert users will be undertaken to identify items which are either unclear or missing. This process, consisting of surveys and semistructured interviews, will contribute towards the explanation and elaboration document and (6) On finalisation of the manuscripts, the group's efforts turn towards an organised dissemination and implementation strategy to maximise end-user adoption.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the Joint Research Compliance Office at Imperial College London (reference number: 19IC5679). A dissemination strategy will be aimed towards five groups of stakeholders: (1) academia, (2) policy, (3) guidelines and regulation, (4) industry and (5) public and non-specific stakeholders. We anticipate that dissemination will take place in Q3 of 2021.

EVIDENCE Publication Checklist for Studies Evaluating Connected Sensor Technologies: Explanation and Elaboration

The EVIDENCE (EValuatIng connecteD sENsor teChnologiEs) checklist was developed by a multidisciplinary group of content experts convened by the Digital Medicine Society, representing the clinical sciences, data management, technology development, and biostatistics. The aim of EVIDENCE is to promote high quality reporting in studies where the primary objective is an evaluation of a digital measurement product or its constituent parts. Here we use the terms digital measurement product and connected sensor technology interchangeably to refer to tools that process data captured by mobile sensors using algorithms to generate measures of behavioral and/or physiological function. EVIDENCE is applicable to 5 types of evaluations: (1) proof of concept; (2) verification, (3) analytical validation, and (4) clinical validation as defined by the V3 framework; and (5) utility and usability assessments. Using EVIDENCE, those preparing, reading, or reviewing studies evaluating digital measurement products will be better equipped to distinguish necessary reporting requirements to drive high-quality research. With broad adoption, the EVIDENCE checklist will serve as a much-needed guide to raise the bar for quality reporting in published literature evaluating digital measurements products.

Machine learning for identifying relevant publications in updates of systematic reviews of diagnostic test studies

Updating systematic reviews is often a time-consuming process that involves a lot of human effort and is therefore not conducted as often as it should be. The aim of our research project was to explore the potential of machine learning methods to reduce human workload. Furthermore, we evaluated the performance of deep learning methods in comparison to more established machine learning methods. We used three available reviews of diagnostic test studies as the data set. In order to identify relevant publications, we used typical text pre-processing methods. The reference standard for the evaluation was the human-consensus based on binary classification (inclusion, exclusion). For the evaluation of the models, various scenarios were generated using a grid of combinations of data preprocessing steps. Moreover, we evaluated each machine learning approach with an approach-specific predefined grid of tuning parameters using the Brier score metric. The best performance was obtained with an ensemble method for two of the reviews, and by a deep learning approach for the other review. Yet, the final performance of approaches strongly depends on data preparation. Overall, machine learning methods provided reasonable classification. It seems possible to reduce human workload in updating systematic reviews by using machine learning methods. Yet, as the influence of data preprocessing on the final performance seems to be at least as important as choosing the specific machine learning approach, users should not blindly expect a good performance by solely using approaches from a popular class, such as deep learning.

STARD Adherence in an Interventional Radiology Guideline for Diagnostic Arteriography

Objective: To assess the quality of reporting in diagnostic accuracy studies (DAS) referenced by the Quality Improvement Guidelines for Diagnostic Arteriography and their adherence to the Standards for Reporting of Diagnostic Accuracy (STARD) statement.Materials and Methods: Citations specific to the Society of Interventional Radiology's Quality Improvement Guidelines for Diagnostic Arteriography were collected. Using the 34-item STARD checklist, two authors in a duplicate and blinded fashion documented the number of items reported per diagnostic accuracy study. Authors met, and any discrepancies were resolved in a resolution meeting.Results: Of the 26 diagnostic accuracy studies included, the mean number of STARD items reported was 17.8 (SD ± 3.1). The median adherence was 18 (IQR, 17-19) items. Ten articles were published prior to 2003, the original date of STARD publication, and 16 articles were published after 2003. The mean number of reported items for the articles published prior to STARD 2003, and after STARD 2003 was 17.4 (SD ± 2.4) and 18.1 (SD ± 3.5), respectively. There were 14 STARD items that demonstrated an adherence of < 25%, and 13 an adherence > 75%.Conclusion: The dichotomous distribution of adherence to the STARD statement by DAS investigated demonstrates that areas of deficient reporting may be present and require attention to ensure complete and transparent reporting in the future.

Systematic Review and STARD Scoring of Renal Cell Carcinoma Circulating Diagnostic Biomarker Manuscripts

Background

No validated molecular biomarkers exist to help guide diagnosis of renal cell carcinoma (RCC) patients. We seek to evaluate the quality of published RCC circulating diagnostic biomarker manuscripts using the Standards for Reporting of Diagnostic Accuracy Studies (STARD) guidelines.

Methods

The phrase “(renal cell carcinoma OR renal cancer OR kidney cancer OR kidney carcinoma) AND circulating AND (biomarkers OR cell free DNA OR tumor DNA OR methylated cell free DNA OR methylated tumor DNA)” was searched in Embase, MEDLINE, and PubMed in March 2018. Relevant manuscripts were scored using 41 STARD subcriteria for a maximal score of 26 points. All tests of statistical significance were 2 sided.

Results

The search identified 535 publications: 27 manuscripts of primary research were analyzed. The median STARD score was 11.5 (range = 7-16.75). All manuscripts had appropriate abstracts, introductions, and distribution of alternative diagnoses. None of the manuscripts stated how indeterminant data were handled or if adverse events occurred from performing the index test or reference standard. Statistically significantly higher STARD scores were present in manuscripts reporting receiver operator characteristic curves (P < .001), larger sample sizes (P = .007), and after release of the original STARD statement (P = .005).

Conclusions

Most RCC circulating diagnostic biomarker manuscripts poorly adhere to the STARD guidelines. Future studies adhering to STARD guidelines may address this unmet need.

Reporting Quality of Diagnostic Accuracy Studies in Laboratory Medicine: Adherence to Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015

Background

Poor reporting quality in diagnostic accuracy studies hampers an adequate judgment of the validity of the study. The Standards for Reporting of Diagnostic Accuracy Studies (STARD) statement was published to improve the reporting quality of diagnostic accuracy studies. This study aimed to evaluate the adherence of diagnostic accuracy studies published in Annals of Laboratory Medicine (ALM) to STARD 2015 and to identify directions for improvement in the reporting quality of these studies.

Methods

Two independent authors assessed articles published in ALM between 2012–2018 for compliance with 30 STARD 2015 checklist items to identify all eligible diagnostic accuracy studies published during this period. We included 66 diagnostic accuracy studies. A total of the fulfilled STARD items were calculated, and adherence was analyzed on an individual-item basis.

Results

The overall mean±SD number of STARD items reported for the included studies was 11.2±2.7. Only five (7.6%) studies adhered to more than 50% of the 30 items. No study satisfied more than 80% of the items. Large variability in adherence to reporting standards was detected across items, ranging from 0% to 100%.

Conclusions

Adherence to STARD 2015 is suboptimal among diagnostic accuracy studies published in ALM. Our study emphasizes the necessity of adherence to STARD to improve the reporting quality of future diagnostic accuracy studies to be published in ALM.

An analysis reveals differences between pragmatic and explanatory diagnostic accuracy studies

OBJECTIVES

The objective of this study was to clarify a difference between two approaches while evaluating the diagnostic accuracy of medical tests, labeled here as "pragmatic" vs. "explanatory" studies.

METHODS

Using the definitions and characteristics described by Schwartz and Lellouch for randomized trials of interventions, and Schwartz' more general distinction between a pragmatic and an explanatory approach in medical research, we define a similar continuum for diagnostic accuracy studies. Explanatory studies aim to better understand the behavior of a test; pragmatic ones are done to support recommendations or decisions about using the test in clinical practice.

RESULTS

Pragmatic test accuracy studies differ from explanatory test accuracy studies in several ways. The difference in aims has implications for key elements of study design, such as the study eligibility criteria, the recruitment of patients, the reference standard, and the choice of the statistical analysis. Explanatory accuracy studies are often designed to test a hypothesis. They are typically selective in recruitment, may include "healthy controls," with a small sample size, often recruited at a single center. They ignore testing failures in the analysis and more often present their results as ROC curves. By contrast, pragmatic studies are designed to guide decision making. They ideally will recruit a single, large, and representative group of patients at multiple sites and will more often present their results as estimates of sensitivity and specificity or predictive values at a prespecified threshold.

CONCLUSION

Distinguishing between a pragmatic and an explanatory approach can help in the design, analysis, and interpretation of diagnostic accuracy studies. It can clarify debates about the appropriateness of design features to the study purpose and about the validity and applicability of study findings.

Reporting quality in systematic reviews of in vitro studies: a systematic review

Background: Systematic reviews (SRs) and/or meta-analyses of in vitro research have an important role in establishing the foundation for clinical studies. In this study, we aimed to evaluate the reporting quality of SRs of in vitro studies using the PRISMA checklist.Method: Four databases were searched including PubMed, Virtual Health Library (VHL), Web of Science (ISI) and Scopus. The search was limited from 2006 to 2016 to include all SRs and/or meta-analyses (MAs) of pure in vitro studies. The evaluation of reporting quality was done using the PRISMA checklist.Results: Out of 7702 search results, 65 SRs were included and evaluated with the PRISMA checklist. Overall, the mean overall quality score of reported items of the PRISMA checklist was 68%. We have noticed an increasing pattern in the numbers of published SRs of in vitro studies over the last 10 years. In contrast, the reporting quality was not significantly improved over the same period (p = .363). There was a positive but not significant correlation between the overall quality score and the journal impact factor of the included studies.Conclusions: The adherence of SRs of in vitro studies to the PRISMA guidelines was poor. Therefore, we believe that using reporting guidelines and journals paying attention to this fact will improve the quality of SRs of in vitro studies.

Choosing an Adequate Test to Determine Fitness for Air Travel in Obese Individuals

BACKGROUND

Air travel is physically demanding and, because obesity is rising, physicians increasingly need to assess whether such patients can fly safely. Our aim was to compare the diagnostic accuracy of two routinely used exercise tests, 50-m walk test and 6-min walk test, and hypoxic challenge testing (HCT) in obese individuals. We further explored the diagnostic potential of perceived dyspnea as measured with the Borg scale because this is often recorded subsequent to walking tests.

METHODS

In this prospective study, we examined 21 obese participants (10 women, age 51 ± 15 [mean  ±  SD], BMI 36 ± 5  kg/m²). The most prevalent comorbidity was COPD (n = 11). The reference standard for in-flight hypoxia, defined as oxygen saturation below 90%, was established in an altitude chamber. Diagnostic accuracy of each index test was estimated by area under the receiver operating characteristic curve (AUC).

RESULTS

Of the 21 participants, 13 (9 with COPD) were identified with in-flight hypoxia. HCT was the only test separating the reference groups significantly with AUC 0.87 (95% CI,  0.62-0.96). Neither of the walking tests predicted noticeably above chance level: 50 m walk test had an AUC of 0.63 (0.36-0.84) and 6MWT had an AUC of 0.64 (0.35-0.86). We further observed good prognostic ability of subjective dyspnea assessment when recorded after 6MWT with an AUC of 0.80 (0.55-0.93).

CONCLUSIONS

In-flight hypoxia in obese individuals can be predicted by HCT but not by simple walking tests.

Quality of science and reporting of radiomics in oncologic studies: room for improvement according to radiomics quality score and TRIPOD statement

OBJECTIVES

To evaluate radiomics studies according to radiomics quality score (RQS) and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) to provide objective measurement of radiomics research.

MATERIALS AND METHODS

PubMed and Embase were searched for studies published in high clinical imaging journals until December 2018 using the terms "radiomics" and "radiogenomics." Studies were scored against the items in the RQS and TRIPOD guidelines. Subgroup analyses were performed for journal type (clinical vs. imaging), intended use (diagnostic vs. prognostic), and imaging modality (CT vs. MRI), and articles were compared using Fisher's exact test and Mann-Whitney analysis.

RESULTS

Seventy-seven articles were included. The mean RQS score was 26.1% of the maximum (9.4 out of 36). The RQS was low in demonstration of clinical utility (19.5%), test-retest analysis (6.5%), prospective study (3.9%), and open science (3.9%). None of the studies conducted a phantom or cost-effectiveness analysis. The adherence rate for TRIPOD was 57.8% (mean) and was particularly low in reporting title (2.6%), stating study objective in abstract and introduction (7.8% and 16.9%), blind assessment of outcome (14.3%), sample size (6.5%), and missing data (11.7%) categories. Studies in clinical journals scored higher and more frequently adopted external validation than imaging journals.

CONCLUSIONS

The overall scientific quality and reporting of radiomics studies is insufficient. Scientific improvements need to be made to feature reproducibility, analysis of clinical utility, and open science categories. Reporting of study objectives, blind assessment, sample size, and missing data is deemed to be necessary.

KEY POINTS

• The overall scientific quality and reporting of radiomics studies is insufficient. • The RQS was low in demonstration of clinical utility, test-retest analysis, prospective study, and open science. • Room for improvement was shown in TRIPOD in stating study objective in abstract and introduction, blind assessment of outcome, sample size, and missing data categories.

Medical Diagnostic Tests: A Review of Test Anatomy, Phases, and Statistical Treatment of Data

Diagnostic tests are approaches used in clinical practice to identify with high accuracy the disease of a particular patient and thus to provide early and proper treatment. Reporting high-quality results of diagnostic tests, for both basic and advanced methods, is solely the responsibility of the authors. Despite the existence of recommendation and standards regarding the content or format of statistical aspects, the quality of what and how the statistic is reported when a diagnostic test is assessed varied from excellent to very poor. This article briefly reviews the steps in the evaluation of a diagnostic test from the anatomy, to the role in clinical practice, and to the statistical methods used to show their performances. The statistical approaches are linked with the phase, clinical question, and objective and are accompanied by examples. More details are provided for phase I and II studies while the statistical treatment of phase III and IV is just briefly presented. Several free online resources useful in the calculation of some statistics are also given.

Uniformity in measuring adherence to reporting guidelines: the example of TRIPOD for assessing completeness of reporting of prediction model studies

To promote uniformity in measuring adherence to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, a reporting guideline for diagnostic and prognostic prediction model studies, and thereby facilitate comparability of future studies assessing its impact, we transformed the original 22 TRIPOD items into an adherence assessment form and defined adherence scoring rules. TRIPOD specific challenges encountered were the existence of different types of prediction model studies and possible combinations of these within publications. More general issues included dealing with multiple reporting elements, reference to information in another publication, and non-applicability of items. We recommend our adherence assessment form to be used by anyone (eg, researchers, reviewers, editors) evaluating adherence to TRIPOD, to make these assessments comparable. In general, when developing a form to assess adherence to a reporting guideline, we recommend formulating specific adherence elements (if needed multiple per reporting guideline item) using unambiguous wording and the consideration of issues of applicability in advance.

Determining qualitative effect size ratings using a likelihood ratio scatter matrix in diagnostic test accuracy systematic reviews

Background Diagnostic test accuracy (DTA) systematic reviews (SRs) characterize a test's potential for diagnostic quality and safety. However, interpreting DTA measures in the context of SRs is challenging. Further, some evidence grading methods (e.g. Centers for Disease Control and Prevention, Division of Laboratory Systems Laboratory Medicine Best Practices method) require determination of qualitative effect size ratings as a contributor to practice recommendations. This paper describes a recently developed effect size rating approach for assessing a DTA evidence base. Methods A likelihood ratio scatter matrix will plot positive and negative likelihood ratio pairings for DTA studies. Pairings are graphed as single point estimates with confidence intervals, positioned in one of four quadrants derived from established thresholds for test clinical validity. These quadrants support defensible judgments on "substantial", "moderate", or "minimal" effect size ratings for each plotted study. The approach is flexible in relation to a priori determinations of the relative clinical importance of false positive and false negative test results. Results and conclusions This qualitative effect size rating approach was operationalized in a recent SR that assessed effectiveness of test practices for the diagnosis of Clostridium difficile. Relevance of this approach to other methods of grading evidence, and efforts to measure diagnostic quality and safety are described. Limitations of the approach arise from understanding that a diagnostic test is not an isolated element in the diagnostic process, but provides information in clinical context towards diagnostic quality and safety.

Targeted test evaluation: a framework for designing diagnostic accuracy studies with clear study hypotheses

Most randomized controlled trials evaluating medical interventions have a pre-specified hypothesis, which is statistically tested against the null hypothesis of no effect. In diagnostic accuracy studies, study hypotheses are rarely pre-defined and sample size calculations are usually not performed, which may jeopardize scientific rigor and can lead to over-interpretation or "spin" of study findings. In this paper, we propose a strategy for defining meaningful hypotheses in diagnostic accuracy studies. Based on the role of the index test in the clinical pathway and the downstream consequences of test results, the consequences of test misclassifications can be weighed, to arrive at minimally acceptable criteria for pre-defined test performance: levels of sensitivity and specificity that would justify the test's intended use. Minimally acceptable criteria for test performance should form the basis for hypothesis formulation and sample size calculations in diagnostic accuracy studies.

Intelligence Assessment Instruments in Adult Prison Populations: A Systematic Review

Detection of intellectual disability (ID) in the penitentiary system is important for the following reasons: (a) to provide assistance to people with ID in understanding their legal rights and court proceedings; (b) to facilitate rehabilitation programs tailored to ID patients, which improves the enhancement of their quality of life and reduces their risk of reoffending; and (c) to provide a reliable estimate of the risk of offence recidivism. It requires a short assessment instrument that provides a reliable estimation of a person's intellectual functioning at the earliest possible stage of this process. The aim of this systematic review is (a) to provide an overview of recent short assessment instruments that provide a full-scale IQ score in adult prison populations and (b) to achieve a quality measurement of the validation studies regarding these instruments to determine which tests are most feasible in this target population. The Preferred Reporting Items for Systematic reviews and Meta-Analyses Statement is used to ensure reliability. The Satz-Mögel, an item-reduction short form of the Wechsler Adult Intelligence Scale, shows the highest correlation with the golden standard and is described to be most reliable. Nevertheless, when it comes to applicability in prison populations, the shorter and less verbal Quick Test can be preferred over others. Without affecting these conclusions, major limitations emerge from the present systematic review, which give rise to several important recommendations for further research.

Poor reporting of multivariable prediction model studies: towards a targeted implementation strategy of the TRIPOD statement

BACKGROUND

As complete reporting is essential to judge the validity and applicability of multivariable prediction models, a guideline for the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) was introduced. We assessed the completeness of reporting of prediction model studies published just before the introduction of the TRIPOD statement, to refine and tailor its implementation strategy.

METHODS

Within each of 37 clinical domains, 10 journals with the highest journal impact factor were selected. A PubMed search was performed to identify prediction model studies published before the launch of TRIPOD in these journals (May 2014). Eligible publications reported on the development or external validation of a multivariable prediction model (either diagnostic or prognostic) or on the incremental value of adding a predictor to an existing model.

RESULTS

We included 146 publications (84% prognostic), from which we assessed 170 models: 73 (43%) on model development, 43 (25%) on external validation, 33 (19%) on incremental value, and 21 (12%) on combined development and external validation of the same model. Overall, publications adhered to a median of 44% (25th-75th percentile 35-52%) of TRIPOD items, with 44% (35-53%) for prognostic and 41% (34-48%) for diagnostic models. TRIPOD items that were completely reported for less than 25% of the models concerned abstract (2%), title (5%), blinding of predictor assessment (6%), comparison of development and validation data (11%), model updating (14%), model performance (14%), model specification (17%), characteristics of participants (21%), model performance measures (methods) (21%), and model-building procedures (24%). Most often reported were TRIPOD items regarding overall interpretation (96%), source of data (95%), and risk groups (90%).

CONCLUSIONS

More than half of the items considered essential for transparent reporting were not fully addressed in publications of multivariable prediction model studies. Essential information for using a model in individual risk prediction, i.e. model specifications and model performance, was incomplete for more than 80% of the models. Items that require improved reporting are title, abstract, and model-building procedures, as they are crucial for identification and external validation of prediction models.

Accuracy of Computed Tomography in Diagnosis of Intra-abdominal Injuries in Stable Patients With Anterior Abdominal Stab Wounds: A Systematic Review and Meta-analysis

BACKGROUND

Workup for patients presenting to the emergency department (ED) following an anterior abdominal stab wound (AASW) has been debated since the 1960s. Experts agree that patients with peritonitis, evisceration, or hemodynamic instability should undergo immediate laparotomy (LAP); however, workup of stable, asymptomatic or nonperitoneal patients is not clearly defined.

OBJECTIVES

The objective was to evaluate the accuracy of computed tomography of abdomen and pelvis (CTAP) for diagnosis of intraabdominal injuries requiring therapeutic laparotomy (THER-LAP) in ED patients with AASW. Is a negative CT scan without a period of observation sufficient to safely discharge a hemodynamically stable, asymptomatic AASW patient?

METHODS

We searched PubMed, Embase, and Scopus from their inception until May 2017 for studies on ED patients with AASW. We defined the reference standard test as LAP for patients who were managed surgically and inpatient observation in those who were managed nonoperatively. In those who underwent LAP, THER-LAP was considered as disease positive. We used the Quality Assessment Tool for Diagnostic Accuracy Studies (QUADAS-2) to evaluate the risk of bias and assess the applicability of the included studies. We attempted to compute the pooled sensitivity, specificity, positive likelihood ratio (LR+), and negative likelihood ratio (LR-) using a random-effects model with MetaDiSc software and calculate testing and treatment thresholds for CT scan applying the Pauker and Kassirer model.

RESULTS

Seven studies were included encompassing 575 patients. The weighted prevalence of THER-LAP was 34.3% (95% confidence interval [CI] = 30.5%-38.2%). Studies had variable quality and the inclusion criteria were not uniform. The operating characteristics of CT scan were as follows: sensitivity = 50% to 100%, specificity = 39% to 97%, LR+ = 1.0 to 15.7, and LR- = 0.07 to 1.0. The high heterogeneity (I²  > 75%) of the operating characteristics of CT scan prevented pooling of the data and therefore the testing and treatment thresholds could not be estimated.

DISCUSSION

The articles revealed a high prevalence (8.7%, 95% CI = 6.1%-12.2%) of injuries requiring THER-LAP in patients with a negative CT scan and almost half (47%, 95% CI = 30%-64%) of those injuries involved the small bowel.

CONCLUSIONS

In stable AASW patients, a negative CT scan alone without an observation period is inadequate to exclude significant intraabdominal injuries.

Prognostic role of neutrophils to lymphocytes ratio in patients with acute pulmonary embolism: a systematic review and meta-analysis of the literature

The prognostic assessment of patients with acute pulmonary embolism (PE) is essential to drive its management. The search for new prognostic factors is a central issue for a more accurate estimate of short-term adverse events. Circulating neutrophils/lymphocytes ratio (NLR) has been suggested as prognostic biomarker for different cardiovascular diseases. Given the central role of inflammation, and in particular of neutrophils in the pathogenesis of VTE and its clinical history, NLR might represent a prognostic tool also in this setting. We performed a systematic review and meta-analysis of the literature to assess the prognostic role of NLR in patients with acute PE. MEDLINE and EMBASE were searched up to 2017, week 21. A bivariate random-effects regression approach was used to obtain summary estimate of accuracy of the high NLR adjusting for inter-study variability. Six studies for a total of 1424 patient are included. High NLR has a weighted mean sensitivity of 77% (95% CI 68-83) and a weighted mean specificity of 74% (95% CI 68-79). High NLR positive and negative predictive values are 24.4% (95% CI 20.4-28.3) and 96.7% (95% CI 95.6-97.8), respectively. The relevant impact of NLR on short-term mortality after an acute PE makes it a promising biomarker to better stratify patient prognosis.

Reporting quality of trial abstracts-improved yet suboptimal: A systematic review and meta-analysis

OBJECTIVES

We conducted a systematic review and meta-analysis of literature to determine if the publication of the Consolidated Standards of Reporting Trials (CONSORT) abstract guideline in 2008 was followed by change in reporting quality of randomized controlled trial (RCT) abstracts.

STUDY DESIGN AND SETTINGS

Evaluations were included if they compared reporting quality of RCT abstracts before and after the publication of CONSORT-abstract guideline. The literature search was performed (January 2008 to April 2017) in Medline (Ovid), EMbase, CINAHL plus and Cochrane methodologies register. We assessed study validity with a special validity tool, adapted from a previous Cochrane review.

RESULTS

Initial search identified 4142 articles, of which total 10 evaluations including 5184 abstracts were included. Total 22 outcomes related to individual items of CONSORT-abstract guideline were assessed, and 14 showed significant effect sizes favoring CONSORT-abstract guideline. Despite significant effect size, the overall post-CONSORT reporting (PCR) was suboptimal for ten items: title (RR = 1.40, 95% CI 1.23 to 1.59, PCR = 53.4%), participants (RR = 1.58, 95% CI 1.11 to 2.26, PCR = 24.5%), primary outcome (RR = 1.12, 95% CI 1.02 to 1.23, PCR = 65%), blinding (RR = 2.13, 95% CI 1.20 to 3.76, PCR = 13.9%), trial status (RR = 1.81, 95% 1.39 to 2.35, PCR = 10.6%), numbers analyzed (RR = 1.51, 95% CI 1.15 to 1.98, PCR = 26.5%), outcome (RR = 1.40, 95% 1.05 to 1.86, PCR = 21.9%), effect size and precision (RR = 1.59, 95% CI 1.15 to 2.19, PCR = 58.9%), harms (RR = 1.24, 95% CI 1.04 to 1.48, PCR = 41.8%), trial registration (RR = 2.02, 95% CI 1.63 to 2.50, PCR = 33.8%). Three items with favorable effect size in addition had wide CIs: randomization (RR = -4.28, 95% CI 1.56 to 11.75, PCR = -3.3%), allocation concealment (RR = -19.89, 95% CI 1.54 to 256.69, PCR = -5.7%), and funding (RR = -22.61, 95% CI 8.13 to 62.67, PCR = -11.32%).

CONCLUSION

The change in reporting quality of RCT abstracts is far from satisfactory, as evidenced by suboptimal post-CONSORT rates and wide CIs of effect sizes for majority of improved items. Mere publication of CONSORT-abstract guideline, without strict endorsement has failed to translate into good quality abstracts.

Diagnostic accuracy research in glaucoma is still incompletely reported: An application of Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015

Background

Research has shown a modest adherence of diagnostic test accuracy (DTA) studies in glaucoma to the Standards for Reporting of Diagnostic Accuracy Studies (STARD). We have applied the updated 30-item STARD 2015 checklist to a set of studies included in a Cochrane DTA systematic review of imaging tools for diagnosing manifest glaucoma.

Methods

Three pairs of reviewers, including one senior reviewer who assessed all studies, independently checked the adherence of each study to STARD 2015. Adherence was analyzed on an individual-item basis. Logistic regression was used to evaluate the effect of publication year and impact factor on adherence.

Results

We included 106 DTA studies, published between 2003–2014 in journals with a median impact factor of 2.6. Overall adherence was 54.1% for 3,286 individual rating across 31 items, with a mean of 16.8 (SD: 3.1; range 8–23) items per study. Large variability in adherence to reporting standards was detected across individual STARD 2015 items, ranging from 0 to 100%. Nine items (1: identification as diagnostic accuracy study in title/abstract; 6: eligibility criteria; 10: index test (a) and reference standard (b) definition; 12: cut-off definitions for index test (a) and reference standard (b); 14: estimation of diagnostic accuracy measures; 21a: severity spectrum of diseased; 23: cross-tabulation of the index and reference standard results) were adequately reported in more than 90% of the studies. Conversely, 10 items (3: scientific and clinical background of the index test; 11: rationale for the reference standard; 13b: blinding of index test results; 17: analyses of variability; 18; sample size calculation; 19: study flow diagram; 20: baseline characteristics of participants; 28: registration number and registry; 29: availability of study protocol; 30: sources of funding) were adequately reported in less than 30% of the studies. Only four items showed a statistically significant improvement over time: missing data (16), baseline characteristics of participants (20), estimates of diagnostic accuracy (24) and sources of funding (30).

Conclusions

Adherence to STARD 2015 among DTA studies in glaucoma research is incomplete, and only modestly increasing over time.

Preanalytical investigations of phlebotomy: methodological aspects, pitfalls and recommendations

Phlebotomy is often addressed as a crucial process in the pre-analytical phase, in which a large part of laboratory errors take place, but to date there is not yet a consolidated methodological paradigm. Seeking literature, we found 36 suitable investigations issued between 1996 and 2016 (April) dealing with the investigation of pre-analytical factors related to phlebotomy. We found that the largest part of studies had a cohort of healthy volunteers (22/36) or outpatients (11/36), with the former group showing a significantly smaller median sample size (N = 20, IQR: 17.5-30 and N = 88, IQR: 54.5-220.5 respectively, P < 0.001). Moreover, the largest part investigated one pre-analytical factor (26/36) and regarded more than one laboratory test (29/36), and authors preferably used paired Student’s t-test (17/36) or Wilcoxon’s test (11/36), but calibration (i.e. sample size calculation for a detectable effect) was addressed only in one manuscript. The Bland-Altman plot was often the preferred method used to estimate bias (12/36), as well as the Passing-Bablok regression for agreement (8/36). However, often papers did assess neither bias (12/36) nor agreement (24/36). Clinical significance of bias was preferably assessed comparing to a database value (16/36), and it resulted uncorrelated with the size of the effect produced by the factor (P = 0.142). However, the median effect size (ES) resulted significantly larger if the associated factor was clinically significant instead of non-significant (ES = 1.140, IQR: 0.815-1.700 and ES = 0.349, IQR: 0.228-0.531 respectively, P < 0.001). On these evidences, we discussed some recommendations for improving methodological consistency, delivering reliable results, as well as ensuring accessibility to practical evidences.

STARD 2015 was reproducible in a large set of studies on glaucoma

Aim

To investigate the reproducibility of the updated Standards for the Reporting of Diagnostic Accuracy Studies tool (STARD 2015) in a set of 106 studies included in a Cochrane diagnostic test accuracy (DTA) systematic review of imaging tests for diagnosing manifest glaucoma.

Methods

One senior rater with DTA methodological and clinical expertise used STARD 2015 on all studies, and each of three raters with different training profiles assessed about a third of the studies.

Results

Raw agreement was very good or almost perfect between the senior rater and an ophthalmology resident with DTA methods training, acceptable with a clinical rater with little DTA methods training, and only moderate with a pharmacology researcher with general, but not DTA, systematic review training and no clinical expertise. The relationship between adherence with STARD 2015 and methodological quality with QUADAS 2 was only partial and difficult to investigate, suggesting that raters used substantial context knowledge in risk of bias assessment.

Conclusions

STARD 2015 proved to be reproducible in this specific research field, provided that both clinical and DTA methodological expertise are achieved through training of its users.

Adherence to Standards for Reporting Diagnostic Accuracy in Emergency Medicine Research

BACKGROUND

Diagnostic tests are used frequently in the emergency department (ED) to guide clinical decision making and, hence, influence clinical outcomes. The Standards for Reporting of Diagnostic Accuracy (STARD) criteria were developed to ensure that diagnostic test studies are performed and reported to best inform clinical decision making in the ED.

OBJECTIVE

The objective was to determine the extent to which diagnostic studies published in emergency medicine journals adhered to STARD 2003 criteria.

METHODS

Diagnostic studies published in eight MEDLINE-listed, peer-reviewed, emergency medicine journals over a 5-year period were reviewed for compliance to STARD criteria.

RESULTS

A total of 12,649 articles were screened and 114 studies were included in our study. Twenty percent of these were randomly selected for assessment using STARD 2003 criteria. Adherence to STARD 2003 reporting standards for each criteria ranged from 8.7% adherence (criteria-reporting adverse events from performing index test or reference standard) to 100% (multiple criteria).

CONCLUSION

Just over half of STARD criteria are reported in more than 80% studies. As poorly reported studies may negatively impact their clinical usefulness, it is essential that studies of diagnostic test accuracy be performed and reported adequately. Future studies should assess whether studies have improved compliance with the STARD 2015 criteria amendment.

A guide to aid the selection of diagnostic tests

In recent years, a wide range of diagnostic tests has become available for use in resource-constrained settings. Accordingly, a huge number of guidelines, performance evaluations and implementation reports have been produced. However, this wealth of information is unstructured and of uneven quality, which has made it difficult for end-users, such as clinics, laboratories and health ministries, to determine which test would be best for improving clinical care and patient outcomes in a specific context. This paper outlines a six-step guide to the selection and implementation of in vitro diagnostic tests based on Médecins Sans Frontières' practical experience: (i) define the test's purpose; (ii) review the market; (iii) ascertain regulatory approval; (iv) determine the test's diagnostic accuracy under ideal conditions; (v) determine the test's diagnostic accuracy in clinical practice; and (vi) monitor the test's performance in routine use. Gaps in the information needed to complete these six steps and gaps in regulatory systems are highlighted. Finally, ways of improving the quality of diagnostic tests are suggested, such as establishing a model list of essential diagnostics, establishing a repository of information on the design of diagnostic studies and improving quality control and postmarketing surveillance.

How to write an original radiological research manuscript

Many scientific manuscripts submitted for publication are limited by fundamental mistakes in their preparation, leading to rejection. We describe how to write a well-organized radiological research manuscript containing all of the important ingredients for effective communication of a hypothesis-driven scientific study in the context of medical imaging.

KEY POINTS

• Mistakes in the preparation of scientific manuscripts lead to rejection. • Scientific writing, like any important skill, can be learned. • A well-developed approach will improve the quality of scientific writing. • High-quality scientific writing is essential to communicate research results. • A well-organized manuscript effectively communicates a hypothesis-driven scientific study.

Diagnostic accuracy of nucleic acid amplification tests (NAATs) in urine for genitourinary tuberculosis: a systematic review and meta-analysis

BACKGROUND

Genitourinary tuberculosis is the third most common form of extrapulmonary tuberculosis. Diagnosis is difficult because of unspecific clinical manifestations and low accuracy of conventional tests. Unfortunately, the delayed diagnosis impacts the urinary tract severely. Nucleic acid amplification tests yield fast results, and among these, new technologies can also detect drug resistance. There is lack of consensus regarding the use of these tests in genitourinary tuberculosis; we therefore aimed to assess the accuracy of nucleic acid amplification tests in the diagnosis of genitourinary tuberculosis and to evaluate the heterogeneity between studies.

METHODS

We did a systematic review and meta-analysis of research articles comparing the accuracy of a reference standard and a nucleic acid amplification test for diagnosis of urinary tract tuberculosis. We searched Medline, EMBASE, Web of Science, LILACS, Cochrane Library, and Scopus for articles published between Jan 1, 1990, and Apr 14, 2016. Two investigators identified eligible articles and extracted data for individual study sites. We analyzed data in groups with the same index test. Then, we generated pooled summary estimates (95% CIs) for sensitivity and specificity by use of random-effects meta-analysis when studies were not heterogeneous.

RESULTS

We identified eleven relevant studies from ten articles, giving information on PCR, LCR and Xpert MTB/RIF tests. All PCR studies were "in-house" tests, with different gene targets and had several quality concerns therefore we did not proceed with a pooled analysis. Only one study used LCR. Xpert studies were of good quality and not heterogeneous, pooled sensitivity was 0·87 (0·66-0·96) and specificity was 0·91 (0·84-0·95).

CONCLUSION

PCR studies were highly heterogeneous. Among Xpert MTB/RIF studies, specificity was favorable with an acceptable confidence interval, however new studies can update meta-analysis and get more precise estimates. Further high-quality studies are urgently needed to improve diagnosis of genitourinary tuberculosis.

PROTOCOL REGISTRATION

PROSPERO CRD42016039020.

Waste, Leaks, and Failures in the Biomarker Pipeline

BACKGROUND

The large, expanding literature on biomarkers is characterized by almost ubiquitous significant results, with claims about the potential importance, but few of these discovered biomarkers are used in routine clinical care.

CONTENT

The pipeline of biomarker development includes several specific stages: discovery, validation, clinical translation, evaluation, implementation (and, in the case of nonutility, deimplementation). Each of these stages can be plagued by problems that cause failures of the overall pipeline. Some problems are nonspecific challenges for all biomedical investigation, while others are specific to the peculiarities of biomarker research. Discovery suffers from poor methods and incomplete and selective reporting. External independent validation is limited. Selection for clinical translation is often shaped by nonrational choices. Evaluation is sparse and the clinical utility of many biomarkers remains unknown. The regulatory environment for biomarkers remains weak and guidelines can reach biased or divergent recommendations. Removing inefficient or even harmful biomarkers that have been entrenched in clinical care can meet with major resistance.

SUMMARY

The current biomarker pipeline is too prone to failures. Consideration of clinical needs should become a starting point for the development of biomarkers. Improvements can include the use of more stringent methodology, better reporting, larger collaborative studies, careful external independent validation, preregistration, rigorous systematic reviews and umbrella reviews, pivotal randomized trials, and implementation and deimplementation studies. Incentives should be aligned toward delivering useful biomarkers.