Are Most of the Published Clinical Trial Results in Restorative Dentistry Invalid? An Empirical Investigation

Inter-rater reliability of the extended Composite Quality Score (CQS-2)

Aim

To establish the inter-rater reliability of the Composite Quality Score (CQS-2) and to test the null hypothesis that it did not differ significantly from that of the first CQS version (CQS-1).

Materials and methods

Four independent raters were selected to rate 45 clinical trial reports using CQS-1 and CQS-2. The raters remained unaware of each other's participation in this study until all rating had been completed. Each rater received only one rating template at a time in a random sequence for CQS-1 and CQS-2 rating. Raters completed each template and sent these back to the principal investigator. Each rater received their next template 2 weeks after submission of the completed previous template. The inter-rater reliabilities for the overall appraisal score of the CQS-1 and the CQS-2 were established by using the Brennan-Prediger coefficient (BPC). The coefficients of both CQS versions were compared by using the two-sample z-test. During secondary analysis, the BPCs for every criterion and each corroboration level for both CQS versions were established.

Results

The BPC for the CQS-1 was 0.85 (95% CI: 0.64-1.00) and for the CQS-2 it was 1.00 (95% CI: 0.94-1.00), suggesting a very high inter-rater reliability for both. The difference between the two CQS versions was statistically not significant (p = 0.17). The null hypothesis was accepted.

Conclusion

The CQS-2 is still under development, This study shows that it is associated with a very high inter-rater reliability, which did not statistically significantly differ from that of the CQS-1. The promising results of this study warrant further investigation in the applicability of the CQS-2 as an appraisal tool for prospective controlled clinical therapy trials.

The composite quality score for the appraisal of prospective controlled clinical therapy trials in systematic reviews and its limits

Systematic reviews of prospective controlled clinical therapy trials are one of the most important sources of information in modern medicine. Besides the systematic search for and statistical pooling of current clinical trial data for a particular type of therapy, systematic reviews also have the task of appraising the quality of trial results. The quality of trial results may be diminished by low internal trial validity, due to systematic error (bias). A high risk of bias may likely cause the reported trial results to be diverted from the actual true therapeutic effect and thus render it unsuitable for clinical guidance. According to the Cochrane Collaboration, the risk of bias in clinical therapy trials should be assessed using its Risk of Bias tool, Version 2 (RoB 2). However, the tool has been established to have poor inter-rater reliability, with a limited empirical evidence base and described as complex and demanding. Against this background, the composite quality score (CQS) has been developed as a possible alternative trial appraisal tool, characterised by high epistemic rigour, empirical evidence base, inter-rater reliability and ease of use. This article presents the current evidence of the CQS and its limitations.

Extension of the Composite Quality Score (CQS) as an appraisal tool for prospective, controlled clinical therapy trials-A systematic review of meta-epidemiological evidence

AIM

To conduct a survey of current meta-epidemiological studies to identify additional trial design characteristics that may be associated with significant over- or underestimation of the treatment effect and to use such identified characteristics as a basis for the formulation of new CQS appraisal criteria.

MATERIALS AND METHODS

We retrieved eligible studies from two systematic reviews on this topic (latest search May 2015) and searched the databases PubMed and Embase for further studies from June 2015 -March 2022. All data were extracted by one author and verified by another. Sufficiently homogeneous estimates from single studies were pooled using random-effects meta-analysis. Trial design characteristics associated with statistically significant estimates from single datasets (which could not be pooled) and meta-analyses were used as a basis to formulate new or amend existing CQS criteria.

RESULTS

A total of 38 meta-epidemiological studies were identified. From these, seven trial design characteristics associated with statistically significant over- or underestimation of the true therapeutic effect were found.

CONCLUSION

One new criterion concerning double-blinding was added to the CQS, and the original criteria for concealing the random allocation sequence and for minimum sample size were amended.

The Composite Quality Score (CQS) as a trial appraisal tool: inter-rater reliability and rating time

OBJECTIVE

To establish the CQS inter-rater reliability and rating time and to compare both against that of the Jadad scale and Cochrane's Risk of Bias Tool (ROBT).

MATERIAL AND METHODS

Four independent raters rated 45 trial reports. The inter-rater reliability was established by use of the Brennan-Prediger coefficient (BPC). The coefficients were compared using the two-sample z-test. Secondary analysis included comparison of the inter-rater reliability of the randomization component of all tools, as well as of the allocation concealment component of the CQS to that of the ROBT. The mean rating time with standard deviation (SD) for each tool was determined using one-way repeated measures analysis of variance. Post hoc comparisons were made using the Tukey-Kramer adjustment for three pair-wise multiple comparisons.

RESULTS

The inter-rater reliability was significantly higher for the CQS (BPC, 95% CI: 0.95, 0.87-1.00) compared to Jadad (0.70, 0.58-0.82) (adjusted p = 0.0005) and most components of ROBT. The mean (SD) time to complete the CQS (4.0 (1.0) min) did not differ significantly from that of the Jadad scale 4.8 (1.1) min (adjusted p = 0.11), but was significantly shorter compared to that of the ROBT 15.3 (5.9) min (adjusted p < 0.0001).

CONCLUSIONS

The results suggest the CQS to be a very reliable and fast trial appraisal tool. Clinical relevance The higher the inter-rater reliability, the higher the probability that trial results reflect therapeutic truth. The CQS will need to take further bias sources into consideration, in order to increase its utility.