Mickenautsch S, Miletić I, Rupf S, Renteria J, Göstemeyer G. The Composite Quality Score (CQS) as a trial appraisal tool: inter-rater reliability and rating time.
Clin Oral Investig 2021;
25:6015-6023. [PMID:
34379191 DOI:
10.1007/s00784-021-04099-w]
[Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2021] [Accepted: 07/22/2021] [Indexed: 01/10/2023]
Abstract
OBJECTIVE
To establish the CQS inter-rater reliability and rating time and to compare both against that of the Jadad scale and Cochrane's Risk of Bias Tool (ROBT).
MATERIAL AND METHODS
Four independent raters rated 45 trial reports. The inter-rater reliability was established by use of the Brennan-Prediger coefficient (BPC). The coefficients were compared using the two-sample z-test. Secondary analysis included comparison of the inter-rater reliability of the randomization component of all tools, as well as of the allocation concealment component of the CQS to that of the ROBT. The mean rating time with standard deviation (SD) for each tool was determined using one-way repeated measures analysis of variance. Post hoc comparisons were made using the Tukey-Kramer adjustment for three pair-wise multiple comparisons.
RESULTS
The inter-rater reliability was significantly higher for the CQS (BPC, 95% CI: 0.95, 0.87-1.00) compared to Jadad (0.70, 0.58-0.82) (adjusted p = 0.0005) and most components of ROBT. The mean (SD) time to complete the CQS (4.0 (1.0) min) did not differ significantly from that of the Jadad scale 4.8 (1.1) min (adjusted p = 0.11), but was significantly shorter compared to that of the ROBT 15.3 (5.9) min (adjusted p < 0.0001).
CONCLUSIONS
The results suggest the CQS to be a very reliable and fast trial appraisal tool. Clinical relevance The higher the inter-rater reliability, the higher the probability that trial results reflect therapeutic truth. The CQS will need to take further bias sources into consideration, in order to increase its utility.
Collapse