Does the Reporting Quality of Diagnostic Test Accuracy Studies, as Defined by STARD 2015, Affect Citation?

Has the quality of reporting improved since it became mandatory to use the Standards for Reporting Diagnostic Accuracy?

OBJECTIVES

To investigate whether making the Standards for Reporting Diagnostic Accuracy (STARD) mandatory by the leading journal 'Radiology' in 2016 improved the quality of reporting of diagnostic accuracy studies.

METHODS

A validated search term was used to identify diagnostic accuracy studies published in Radiology in 2015 and 2019. STARD adherence was assessed by two independent reviewers. Each item was scored as yes (1 point) if adequately reported or as no (0 points) if not. The total STARD score per article was calculated. Wilcoxon-Mann-Whitney tests were used to evaluate differences of the total STARD scores between 2015 and 2019. In addition, the total STARD score was compared between studies stratified by study design, citation rate, and data collection.

RESULTS

The median number of reported STARD items for the total of 66 diagnostic accuracy studies from 2015 to 2019 was 18.5 (interquartile range [IQR] 17.5-20.0) of 29. Adherence to the STARD checklist significantly improved the STARD score from a median of 18.0 (IQR 15.5-19.5) in 2015 to a median of 19.5 (IQR 18.5-21.5) in 2019 (p < 0.001). No significant differences were found between studies stratified by mode of data collection (prospective vs. retrospective studies, p = 0.68), study design (cohort vs. case-control studies, p = 0.81), and citation rate (two groups divided by median split [< 0.56 citations/month vs. ≥ 0.56 citations/month], p = 0.54).

CONCLUSIONS

Making use of the STARD checklist mandatory significantly increased the adherence with reporting standards for diagnostic accuracy studies and should be considered by editors and publishers for widespread implementation.

CRITICAL RELEVANCE STATEMENT

Editors may consider making reporting guidelines mandatory to improve the scientific quality.

A radiomics feature-based machine learning models to detect brainstem infarction (RMEBI) may enable early diagnosis in non-contrast enhanced CT

OBJECTIVES

Magnetic resonance imaging has high sensitivity in detecting early brainstem infarction (EBI). However, MRI is not practical for all patients who present with possible stroke and would lead to delayed treatment. The detection rate of EBI on non-contrast computed tomography (NCCT) is currently very low. Thus, we aimed to develop and validate the radiomics feature-based machine learning models to detect EBI (RMEBIs) on NCCT.

METHODS

In this retrospective observational study, 355 participants from a multicentre multimodal database established by Huashan Hospital were randomly divided into two data sets: a training cohort (70%) and an internal validation cohort (30%). Fifty-seven participants from the Second Affiliated Hospital of Xuzhou Medical University were included as the external validation cohort. Brainstems were segmented by a radiologist committee on NCCT and 1781 radiomics features were automatically computed. After selecting the relevant features, 7 machine learning models were assessed in the training cohort to predict early brainstem infarction. Accuracy, sensitivity, specificity, positive predictive value, negative predictive value, F1-score, and the area under the receiver operating characteristic curve (AUC) were used to evaluate the performance of the prediction models.

RESULTS

The multilayer perceptron (MLP) RMEBI showed the best performance (AUC: 0.99 [95% CI: 0.96-1.00]) in the internal validation cohort. The AUC value in external validation cohort was 0.91 (95% CI: 0.82-0.98).

CONCLUSIONS

RMEBIs have the potential in routine clinical practice to enable accurate computer-assisted diagnoses of early brainstem infarction in patients with NCCT, which may have important clinical value in reducing therapeutic decision-making time.

KEY POINTS

• RMEBIs have the potential to enable accurate diagnoses of early brainstem infarction in patients with NCCT. • RMEBIs are suitable for various multidetector CT scanners. • The patient treatment decision-making time is shortened.

Has the STARD statement improved the quality of reporting of diagnostic accuracy studies published in European Radiology?

OBJECTIVES

To investigate whether encouraging authors to follow the Standards for Reporting Diagnostic Accuracy (STARD) guidelines improves the quality of reporting of diagnostic accuracy studies.

METHODS

In mid-2017, European Radiology started encouraging its authors to follow the STARD guidelines. Our MEDLINE search identified 114 diagnostic accuracy studies published in European Radiology in 2015 and 2019. The quality of reporting was evaluated by two independent reviewers using the revised STARD statement. Item 11 was excluded because a meaningful decision about adherence was not possible. Student's t test for independent samples was used to analyze differences in the mean number of reported STARD items between studies published in 2015 and in 2019. In addition, we calculated differences related to the study design, data collection, and citation rate.

RESULTS

The mean total number of reported STARD items for all 114 diagnostic accuracy studies analyzed was 15.9 ± 2.6 (54.8%) of 29 items (range 9.5-22.5). The quality of reporting of diagnostic accuracy studies was significantly better in 2019 (mean ± standard deviation (SD), 16.3 ± 2.7) than in 2015 (mean ± SD, 15.1 ± 2.3; p < 0.02). No significant differences in the reported STARD items were identified in relation to study design (p = 0.13), data collection (p = 0.87), and citation rate (p = 0.09).

CONCLUSION

The quality of reporting of diagnostic accuracy studies according to the STARD statement was moderate with a slight improvement since European Radiology started to recommend its authors to follow the STARD guidelines.

KEY POINTS

• The quality of reporting of diagnostic accuracy studies was moderate with a mean total number of reported STARD items of 15.9 ± 2.6. • The adherence to STARD was significantly better in 2019 than in 2015 (16.3 ± 2.7 vs. 15.1 ± 2.3; p = 0.016). • No significant differences in the reported STARD items were identified in relation to study design (p = 0.13), data collection (p = 0.87), and citation rate (p = 0.09).

Bias and Reporting Quality of Clinical Prognostic Models for Idiopathic Pulmonary Fibrosis: A Cross-Sectional Study

Objective

This study aims to evaluate the risk of bias (ROB) and reporting quality of idiopathic pulmonary fibrosis (IPF) prediction models by assessing characteristics of these models.

Methods

The development and/or validation of IPF prognostic models were identified via an electronic search of PubMed, Embase, and Web of Science (from inception to 12 August, 2021). Two researchers independently assessed the risk of bias (ROB) and reporting quality of IPF prediction models based on the Prediction model Risk Of Bias Assessment Tool (PROBAST) and Transparent Reporting of a multivariable prognostic model for Individual Prognosis or Diagnosis (TRIPOD) checklist.

Results

Twenty prognostic model studies for IPF were included, including 7 (35%) model development and external validation studies, 8 (40%) development studies, and 5 (25%) external validation studies. According to PROBAST, all studies were appraised with high ROB, because of deficient reporting in the domains of participants (45.0%) and analysis (67.3%), and at least 55% studies were susceptible to 4 of 20 sources of bias. For the reporting quality, none of them completely adhered to the TRIPOD checklist, with the lowest mean reporting score for the methods and results domains (46.6% and 44.7%). For specific items, eight sub-items had a reporting rate ≥80% and adhered to the TRIPOD checklist, and nine sub-items had a very poor reporting rate, less than 30%.

Conclusion

Studies adhering to PROBAST and TRIPOD checklists are recommended in the future. The reproducibility and transparency can be improved when studies completely adhere to PROBAST and TRIPOD checklists.

Histogram analysis of arterial spin labeling perfusion data to determine the human papillomavirus status of oropharyngeal squamous cell carcinomas

PURPOSE

To evaluate the correlation between histogram parameters derived from pseudo-continuous arterial spin labeling (PCASL) and human papillomavirus (HPV) status in patients with oropharyngeal squamous cell carcinoma (OPSCC).

METHODS

This study included a total of 58 patients (HPV-positive: n = 45; -negative: n = 13) from a prospective cohort of consecutive patients aged ≥ 18 years, who were newly diagnosed with oropharyngeal squamous cell carcinoma. All patients were required to have undergone pre-treatment MRI with PCASL to measure regional perfusion. The region of interest was drawn by two radiologists, encompassing the entire tumor volume on all corresponding slices. Differences in the histogram parameters derived from tumor blood flow (TBF) in ASL were assessed for HPV-positive and -negative patients. Receiver operating characteristic curve analysis was performed to determine the best differentiating parameters, and a leave-one-out cross-validation was used.

RESULTS

Patients with HPV-positive OPSCC showed a significantly lower overall standard deviation and 95th percentile value of tumor blood flow (P < .007). The standard deviation of TBF was the single best predictive parameter. Leave-one-out cross-validation tests revealed that the area under the receiver operating characteristic curve, accuracy, sensitivity, and specificity were 0.745, 75.9%, 75.6%, and 76.9%, respectively.

CONCLUSION

PCASL revealed differences in perfusion parameters according to HPV status in patients with OPSCC, reflecting their distinct histopathology.

Diagnostic Performance of the Modified Korean Thyroid Imaging Reporting and Data System for Thyroid Malignancy: A Multicenter Validation Study

OBJECTIVE

To evaluate the diagnostic performance of the modified Korean Thyroid Imaging Reporting and Data System (K-TIRADS), and compare it with the 2016 version of K-TIRADS using the Thyroid Imaging Network of Korea.

MATERIALS AND METHODS

Between June and September 2015, 5708 thyroid nodules (≥ 1.0 cm) from 5081 consecutive patients who had undergone thyroid ultrasonography at 26 institutions were retrospectively evaluated. We used a biopsy size threshold of 2 cm for K-TIRADS 3 and 1 cm for K-TIRADS 4 (modified K-TIRADS 1) or 1.5 cm for K-TIRADS 4 (modified K-TIRADS 3). The modified K-TIRADS 2 subcategorized the K-TIRADS 4 into 4A and 4B, and the cutoff sizes for the biopsies were defined as 1 cm for K-TIRADS 4B and 1.5 cm for K-TIRADS 4A. The diagnostic performance and the rate of unnecessary biopsies of the modified K-TIRADS for detecting malignancy were compared with those of the 2016 K-TIRAD, which were stratified by nodule size (with a threshold of 2 cm).

RESULTS

A total of 1111 malignant nodules and 4597 benign nodules were included. The sensitivity, specificity, and unnecessary biopsy rate of the benign nodules were 94.9%, 24.4%, and 60.9% for the 2016 K-TIRADS; 91.0%, 39.7%, and 48.6% for the modified K-TIRADS 1; 84.9%, 45.9%, and 43.5% for the modified K-TIRADS 2; and 76.1%, 50.2%, and 40.1% for the modified K-TIRADS 3. For small nodules (1-2 cm), the diagnostic sensitivity of the modified K-TIRADS decreased by 5.2-25.6% and the rate of unnecessary biopsies reduced by 19.2-32.8% compared with those of the 2016 K-TIRADS (p < 0.001). For large nodules (> 2 cm), the modified K-TIRADSs maintained a very high sensitivity for detecting malignancy (98%).

CONCLUSION

The modified K-TIRADSs significantly reduced the rate of unnecessary biopsies for small (1-2 cm) nodules while maintaining a very high sensitivity for malignancy for large (> 2 cm) nodules.

Radiomics and "radi-…omics" in cancer immunotherapy: a guide for clinicians

In recent years the concept of precision medicine has become a popular topic particularly in medical oncology. Besides the identification of new molecular prognostic and predictive biomarkers and the development of new targeted and immunotherapeutic drugs, imaging has started to play a central role in this new era. Terms such as "radiomics", "radiogenomics" or "radi…-omics" are becoming increasingly common in the literature and soon they will represent an integral part of clinical practice. The use of artificial intelligence, imaging and "-omics" data can be used to develop models able to predict, for example, the features of the tumor immune microenvironment through imaging, and to monitor the therapeutic response beyond the standard radiological criteria. The aims of this narrative review are to provide a simplified guide for clinicians to these concepts, and to summarize the existing evidence on radiomics and "radi…-omics" in cancer immunotherapy.

Reporting Quality of Diagnostic Accuracy Studies in Laboratory Medicine: Adherence to Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015

Background

Poor reporting quality in diagnostic accuracy studies hampers an adequate judgment of the validity of the study. The Standards for Reporting of Diagnostic Accuracy Studies (STARD) statement was published to improve the reporting quality of diagnostic accuracy studies. This study aimed to evaluate the adherence of diagnostic accuracy studies published in Annals of Laboratory Medicine (ALM) to STARD 2015 and to identify directions for improvement in the reporting quality of these studies.

Methods

Two independent authors assessed articles published in ALM between 2012–2018 for compliance with 30 STARD 2015 checklist items to identify all eligible diagnostic accuracy studies published during this period. We included 66 diagnostic accuracy studies. A total of the fulfilled STARD items were calculated, and adherence was analyzed on an individual-item basis.

Results

The overall mean±SD number of STARD items reported for the included studies was 11.2±2.7. Only five (7.6%) studies adhered to more than 50% of the 30 items. No study satisfied more than 80% of the items. Large variability in adherence to reporting standards was detected across items, ranging from 0% to 100%.

Conclusions

Adherence to STARD 2015 is suboptimal among diagnostic accuracy studies published in ALM. Our study emphasizes the necessity of adherence to STARD to improve the reporting quality of future diagnostic accuracy studies to be published in ALM.

Quality of science and reporting of radiomics in oncologic studies: room for improvement according to radiomics quality score and TRIPOD statement

OBJECTIVES

To evaluate radiomics studies according to radiomics quality score (RQS) and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) to provide objective measurement of radiomics research.

MATERIALS AND METHODS

PubMed and Embase were searched for studies published in high clinical imaging journals until December 2018 using the terms "radiomics" and "radiogenomics." Studies were scored against the items in the RQS and TRIPOD guidelines. Subgroup analyses were performed for journal type (clinical vs. imaging), intended use (diagnostic vs. prognostic), and imaging modality (CT vs. MRI), and articles were compared using Fisher's exact test and Mann-Whitney analysis.

RESULTS

Seventy-seven articles were included. The mean RQS score was 26.1% of the maximum (9.4 out of 36). The RQS was low in demonstration of clinical utility (19.5%), test-retest analysis (6.5%), prospective study (3.9%), and open science (3.9%). None of the studies conducted a phantom or cost-effectiveness analysis. The adherence rate for TRIPOD was 57.8% (mean) and was particularly low in reporting title (2.6%), stating study objective in abstract and introduction (7.8% and 16.9%), blind assessment of outcome (14.3%), sample size (6.5%), and missing data (11.7%) categories. Studies in clinical journals scored higher and more frequently adopted external validation than imaging journals.

CONCLUSIONS

The overall scientific quality and reporting of radiomics studies is insufficient. Scientific improvements need to be made to feature reproducibility, analysis of clinical utility, and open science categories. Reporting of study objectives, blind assessment, sample size, and missing data is deemed to be necessary.

KEY POINTS

• The overall scientific quality and reporting of radiomics studies is insufficient. • The RQS was low in demonstration of clinical utility, test-retest analysis, prospective study, and open science. • Room for improvement was shown in TRIPOD in stating study objective in abstract and introduction, blind assessment of outcome, sample size, and missing data categories.

Medical Diagnostic Tests: A Review of Test Anatomy, Phases, and Statistical Treatment of Data

Diagnostic tests are approaches used in clinical practice to identify with high accuracy the disease of a particular patient and thus to provide early and proper treatment. Reporting high-quality results of diagnostic tests, for both basic and advanced methods, is solely the responsibility of the authors. Despite the existence of recommendation and standards regarding the content or format of statistical aspects, the quality of what and how the statistic is reported when a diagnostic test is assessed varied from excellent to very poor. This article briefly reviews the steps in the evaluation of a diagnostic test from the anatomy, to the role in clinical practice, and to the statistical methods used to show their performances. The statistical approaches are linked with the phase, clinical question, and objective and are accompanied by examples. More details are provided for phase I and II studies while the statistical treatment of phase III and IV is just briefly presented. Several free online resources useful in the calculation of some statistics are also given.

Age of Data in Contemporary Research Articles Published in Representative General Radiology Journals

Objective

To analyze and compare the age of data in contemporary research articles published in representative general radiology journals.

Materials and Methods

We searched for articles reporting original research studies analyzing patient data that were published in the print issues of the Korean Journal of Radiology (KJR), European Radiology (ER), and Radiology in 2017. Eligible articles were reviewed to extract data collection period (time from first patient recruitment to last patient follow-up) and age of data (time between data collection end and publication). The journals were compared in terms of the proportion of articles reporting the data collection period to the level of calendar month and regarding the age of data.

Results

There were 50, 492, and 254 eligible articles in KJR, ER, and Radiology, respectively. Of these, 44 (88%; 95% confidence interval [CI]: 75.8-94.8%), 359 (73%; 95% CI: 68.9-76.7%), and 211 (83.1%; 95% CI: 78-87.2%) articles, respectively, provided enough details of data collection period, revealing a significant difference between ER and Radiology (p = 0.002). The age of data was significantly greater in KJR (median age: 826 days; range: 299-2843 days) than in ER (median age: 570 days; range: 56-4742 days; p < 0.001) and Radiology (median age: 618; range: 75-4271 days; p < 0.001).

Conclusion

Korean Journal of Radiology did not fall behind ER or Radiology in reporting of data collection period, but showed a significantly greater age of data than ER and Radiology, suggesting that KJR should take measures to improve the timeliness of its data.

Diagnostic accuracy research in glaucoma is still incompletely reported: An application of Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015

Background

Research has shown a modest adherence of diagnostic test accuracy (DTA) studies in glaucoma to the Standards for Reporting of Diagnostic Accuracy Studies (STARD). We have applied the updated 30-item STARD 2015 checklist to a set of studies included in a Cochrane DTA systematic review of imaging tools for diagnosing manifest glaucoma.

Methods

Three pairs of reviewers, including one senior reviewer who assessed all studies, independently checked the adherence of each study to STARD 2015. Adherence was analyzed on an individual-item basis. Logistic regression was used to evaluate the effect of publication year and impact factor on adherence.

Results

We included 106 DTA studies, published between 2003–2014 in journals with a median impact factor of 2.6. Overall adherence was 54.1% for 3,286 individual rating across 31 items, with a mean of 16.8 (SD: 3.1; range 8–23) items per study. Large variability in adherence to reporting standards was detected across individual STARD 2015 items, ranging from 0 to 100%. Nine items (1: identification as diagnostic accuracy study in title/abstract; 6: eligibility criteria; 10: index test (a) and reference standard (b) definition; 12: cut-off definitions for index test (a) and reference standard (b); 14: estimation of diagnostic accuracy measures; 21a: severity spectrum of diseased; 23: cross-tabulation of the index and reference standard results) were adequately reported in more than 90% of the studies. Conversely, 10 items (3: scientific and clinical background of the index test; 11: rationale for the reference standard; 13b: blinding of index test results; 17: analyses of variability; 18; sample size calculation; 19: study flow diagram; 20: baseline characteristics of participants; 28: registration number and registry; 29: availability of study protocol; 30: sources of funding) were adequately reported in less than 30% of the studies. Only four items showed a statistically significant improvement over time: missing data (16), baseline characteristics of participants (20), estimates of diagnostic accuracy (24) and sources of funding (30).

Conclusions

Adherence to STARD 2015 among DTA studies in glaucoma research is incomplete, and only modestly increasing over time.

Selection and Reporting of Statistical Methods to Assess Reliability of a Diagnostic Test: Conformity to Recommended Methods in a Peer-Reviewed Journal

Objective

To evaluate the frequency and adequacy of statistical analyses in a general radiology journal when reporting a reliability analysis for a diagnostic test.

Materials and Methods

Sixty-three studies of diagnostic test accuracy (DTA) and 36 studies reporting reliability analyses published in the Korean Journal of Radiology between 2012 and 2016 were analyzed. Studies were judged using the methodological guidelines of the Radiological Society of North America-Quantitative Imaging Biomarkers Alliance (RSNA-QIBA), and COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative. DTA studies were evaluated by nine editorial board members of the journal. Reliability studies were evaluated by study reviewers experienced with reliability analysis.

Results

Thirty-one (49.2%) of the 63 DTA studies did not include a reliability analysis when deemed necessary. Among the 36 reliability studies, proper statistical methods were used in all (5/5) studies dealing with dichotomous/nominal data, 46.7% (7/15) of studies dealing with ordinal data, and 95.2% (20/21) of studies dealing with continuous data. Statistical methods were described in sufficient detail regarding weighted kappa in 28.6% (2/7) of studies and regarding the model and assumptions of intraclass correlation coefficient in 35.3% (6/17) and 29.4% (5/17) of studies, respectively. Reliability parameters were used as if they were agreement parameters in 23.1% (3/13) of studies. Reproducibility and repeatability were used incorrectly in 20% (3/15) of studies.

Conclusion

Greater attention to the importance of reporting reliability, thorough description of the related statistical methods, efforts not to neglect agreement parameters, and better use of relevant terminology is necessary.