Central adjudication of serious adverse events did not affect trial's safety results: Data from the Efficacy of Nitric Oxide in Stroke (ENOS) trial

Clinical Adjudication of Hemodialysis Catheter-Related Bloodstream Infections: Findings from the REDUCCTION Trial

Key Points

The inter-rater reliability of reporting hemodialysis catheter-related infectious events between site investigators and trial adjudicators in Australia and New Zealand was substantial. The high concordance level in reporting catheter infections improves confidence in using site-level bacteremia rates as a clinical metric for quality benchmarking and future pragmatic clinical trials. A rigorous adjudication protocol may not be needed if clearly defined criteria to ascertain catheter-associated bacteremia are used.

Background

Hemodialysis catheter-related bloodstream infection (HD-CRBSI) are a significant source of morbidity and mortality among dialysis patients, but benchmarking remains difficult because of varying definitions of HD-CRBSI. This study explored the effect of clinical adjudication process on HD-CRBSI reporting.

Methods

The REDUcing the burden of Catheter ComplicaTIOns: a National approach trial implemented an evidence-based intervention bundle using a stepped-wedge design to reduce HD-CRBSI rates in 37 Australian kidney services. Six New Zealand services participated in an observational capacity. Adult patients with a new hemodialysis catheter between December 2016 and March 2020 were included. HD-CRBSI events reported were compared with the adjudicated outcomes using the end point definition and adjudication processes of the REDUcing the burden of Catheter ComplicaTIOns: a National approach trial. The concordance level was estimated using Gwet agreement coefficient (AC1) adjusted for service-level effects and implementation tranches (Australia only), with the primary outcome being the concordance of confirmed HD-CRBSI.

Results

A total of 744 hemodialysis catheter-related infectious events were reported among 7258 patients, 12,630 catheters, and 1.3 million catheter-exposure days. The majority were confirmed HD-CRBSI, with 77.9% agreement and substantial concordance (AC1=0.77; 95% confidence interval [CI], 0.73 to 0.81). Exit site infections have the highest concordance (AC1=0.85; 95% CI, 0.78 to 0.91); the greatest discordance was in events classified as other (AC1=0.33; 95% CI, 0.16 to 0.49). The concordance of all hemodialysis catheter infectious events remained substantial (AC1=0.80; 95% CI, 0.76 to 0.83) even after adjusting for the intervention tranches in Australia and overall service-level clustering.

Conclusions

There was a substantial level of concordance in overall and service-level reporting of confirmed HD-CRBSI. A standardized end point definition of HD-CRBSI resulted in comparable hemodialysis catheter infection rates in Australian and New Zealand kidney services. Consistent end point definition could enable reliable benchmarking outside clinical trials without the need for independent clinical adjudication.

Should we adjudicate outcomes in stroke trials? A systematic review

BACKGROUND

Central adjudication of outcomes is common in randomized clinical trials in stroke. The rationale for adjudication is clear; centrally adjudicated outcomes should have less random and systematic errors than outcomes assessed locally by site investigators. However, adjudication brings added complexities to a clinical trial and can be costly.

AIM

To assess the evidence for outcome adjudication in stroke trials.

SUMMARY OF REVIEW

We identified 12 studies evaluating central adjudication in stroke trials. The majority of these were secondary analyses of trials, and the results of all of these would have remained unchanged had central adjudication not taken place, even for trials without sufficient blinding. The largest differences between site-assessed and adjudicator-assessed outcomes were between the most subjective outcomes, such as causality of serious adverse events. We found that the cost of adjudication could be upward of £100,000 for medium to large prevention trials. These findings suggest that the cost of central adjudication may outweigh the advantages it brings in many cases. However, through simulation, we found that only a small amount of bias is required in site investigators' outcome assessments before adjudication becomes important.

CONCLUSION

Central adjudication may not be necessary in stroke trials with blinded outcome assessment. However, for open-label studies, central adjudication may be more important.

Comparative Reductions in Investigator-Reported and Adjudicated Ischemic Events in REDUCE-IT

BACKGROUND

REDUCE-IT (Reduction of Cardiovascular Events With Icosapent Ethyl-Intervention Trial) randomized statin-treated patients with elevated triglycerides to icosapent ethyl (IPE) or placebo. There was a significant reduction in adjudicated events, including the primary endpoint (cardiovascular [CV] death, myocardial infarction [MI], stroke, coronary revascularization, unstable angina requiring hospitalization) and key secondary endpoint (CV death, MI, stroke) with IPE.

OBJECTIVES

The purpose of this study was to determine the effects of IPE on investigator-reported events.

METHODS

Potential endpoints were collected by blinded site investigators and subsequently adjudicated by a blinded Clinical Endpoint Committee (CEC) according to a prespecified charter. Investigator-reported events were compared with adjudicated events for concordance.

RESULTS

There was a high degree of concordance between investigator-reported and adjudicated endpoints. The simple Kappa statistic between CEC-adjudicated vs site-reported events for the primary endpoint was 0.89 and for the key secondary endpoint was 0.90. Based on investigator-reported events in 8,179 randomized patients, IPE significantly reduced the rate of the primary endpoint (19.1% vs 24.6%; HR: 0.74 [95% CI: 0.67-0.81]; P < 0.0001) and the key secondary endpoint (10.5% vs 13.6%; HR: 0.75 [95% CI: 0.66-0.85]; P < 0.0001). Among adjudicated events, IPE similarly reduced the rate of the primary and key secondary endpoints.

CONCLUSIONS

IPE led to consistent, significant reductions in CV events, including MI and coronary revascularization, as determined by independent, blinded CEC adjudication as well as by blinded investigator-reported assessment. These results highlight the robust evidence for the substantial CV benefits of IPE seen in REDUCE-IT and further raise the question of whether adjudication of CV outcome trial endpoints is routinely required in blinded, placebo-controlled trials. (Evaluation of the Effect of AMR101 on Cardiovascular Health and Mortality in Hypertriglyceridemic Patients With Cardiovascular Disease or at High Risk for Cardiovascular Disease: REDUCE-IT [Reduction of Cardiovascular Events With EPA - Intervention Trial]; NCT01492361).

Cost-benefit of outcome adjudication in nine randomised stroke trials

BACKGROUND

Central adjudication of outcomes is common for randomised trials and should control for differential misclassification. However, few studies have estimated the cost of the adjudication process.

METHODS

We estimated the cost of adjudicating the primary outcome in nine randomised stroke trials (25,436 participants). The costs included adjudicators' time, direct payments to adjudicators, and co-ordinating centre costs (e.g. uploading cranial scans and general set-up costs). The number of events corrected after adjudication was our measure of benefit. We calculated cost per corrected event for each trial and in total.

RESULTS

The primary outcome in all nine trials was either stroke or a composite that included stroke. In total, the adjudication process associated with this primary outcome cost in excess of £100,000 for a third of the trials (3/9). Mean cost per event corrected by adjudication was £2295.10 (SD: £1482.42).

CONCLUSIONS

Central adjudication is a time-consuming and potentially costly process. These costs need to be considered when designing a trial and should be evaluated alongside the potential benefits adjudication brings to determine whether they outweigh this expense.

Outcome assessment by central adjudicators in randomised stroke trials: Simulation of differential and non-differential misclassification

INTRODUCTION

Adjudication of the primary outcome in randomised trials is thought to control misclassification. We investigated the amount of misclassification needed before adjudication changed the primary trial results.Patients (or materials) and methods: We included data from five randomised stroke trials. Differential misclassification was introduced for each primary outcome until the estimated treatment effect was altered. This was simulated 1000 times. We calculated the between-simulation mean proportion of participants that needed to be differentially misclassified to alter the treatment effect. In addition, we simulated hypothetical trials with a binary outcome and varying sample size (1000-10,000), overall event rate (10%-50%) and treatment effect (0.67-0.90). We introduced non-differential misclassification until the treatment effect was non-significant at 5% level.

RESULTS

For the five trials, the range of unweighted kappa values were reduced from 0.89-0.97 to 0.65-0.85 before the treatment effect was altered. This corresponded to 2.1%-6% of participants misclassified differentially for trials with a binary outcome. For the hypothetical trials, those with a larger sample size, stronger treatment effect and overall event rate closer to 50% needed a higher proportion of events non-differentially misclassified before the treatment effect became non-significant.

DISCUSSION

We found that only a small amount of differential misclassification was required before adjudication altered the primary trial results, whereas a considerable proportion of participants needed to be misclassified non-differentially before adjudication changed trial conclusions. Given that differential misclassification should not occur in trials with sufficient blinding, these results suggest that central adjudication is of most use in studies with unblinded outcome assessment.

CONCLUSION

For trials without adequate blinding, central adjudication is vital to control for differential misclassification. However, for large blinded trials, adjudication is of less importance and may not be necessary.

Assessment of the End Point Adjudication Process on the Results of the Platelet-Oriented Inhibition in New TIA and Minor Ischemic Stroke (POINT) Trial: A Secondary Analysis

IMPORTANCE

Debate continues about the value of event adjudication in clinical trials and whether independent centralized assessments improve reliability and validity of study results in masked randomized trials compared with local, investigator-assessed end points.

OBJECTIVE

To assess the results of the adjudicated end point process in the Platelet-Oriented Inhibition in New TIA and Minor Ischemic Stroke (POINT) trial by comparing end points assessed by local site investigators with centrally adjudicated end points.

DESIGN, SETTING, AND PARTICIPANTS

This is an ad hoc secondary analysis of a randomized, double-blind clinical trial comparing safety and effectiveness of clopidogrel bisulphate plus aspirin vs placebo plus aspirin. Patients received either 600 mg of clopidogrel bisulphate on day 1, then 75 mg per day through day 90 plus 50 to 325 mg of aspirin per day, or the same range of dosages of placebo plus aspirin. Investigators reported all potential end points; independent masked adjudicators were randomly assigned to review using definitions specified in the study protocol. This was a multicenter study; 269 international sites in 10 countries enrolled from May 28, 2010, to December 19, 2017. The study enrolled 4881 patients 18 years or older with transient ischemic attack or minor acute ischemic stroke within 12 hours of symptom onset and followed for 90 days from randomization; last follow-up was completed in March 2018.

MAIN OUTCOMES AND MEASURES

Independent adjudicators external to the study and masked to study treatment assignment adjudicated 467 primary and secondary effectiveness outcomes and major and minor bleeding events, including the primary composite end point, which was the risk of a composite of major ischemic events at 90 days, defined as ischemic stroke, myocardial infarction, or death from an ischemic vascular event. The primary safety end point was major hemorrhage. All components of the primary and safety outcomes were adjudicated.

RESULTS

In this secondary analysis of an international randomized clinical trial, a total of 269 sites worldwide randomized 4881 patients (median age, 65.0 years; interquartile range, 55-74 years); 55.0% were male. The primary results have been published previously. The hazard ratios for clopidogrel plus aspirin vs placebo plus aspirin for the primary composite end point were 0.75 (95% CI, 0.59-0.95) for adjudicator-assessed events and 0.76 (95% CI, 0.60-0.95) for investigator-assessed events. Agreement between adjudicator and investigator assessments was 90.7%. The hazard ratios for clopidogrel plus aspirin vs placebo plus aspirin for the primary safety end point were 2.32 (95% CI, 1.10-4.87) for adjudicator-assessed events and 2.58 (95% CI, 1.19-5.58) for investigator-assessed events, with an agreement rate of 77.5%.

CONCLUSIONS AND RELEVANCE

Independent end point adjudication did not substantially alter estimates of the primary treatment effectiveness in the POINT trial.

TRIAL REGISTRATION

ClinicalTrials.gov identifier: NCT00991029.

Outcome Assessment by Central Adjudicators Versus Site Investigators in Stroke Trials: A Systematic Review and Meta-Analysis

Background and Purpose- In randomized stroke trials, central adjudication of a trial's primary outcome is regularly implemented. However, recent evidence questions the importance of central adjudication in randomized trials. The aim of this review was to compare outcomes assessed by central adjudicators with outcomes assessed by site investigators. Methods- We included randomized stroke trials where the primary outcome had undergone an assessment by site investigators and central adjudicators. We searched MEDLINE, EMBASE, CENTRAL (Cochrane Central Register of Controlled Trials), Web of Science, PsycINFO, and Google Scholar for eligible studies. We extracted information about the adjudication process as well as the treatment effect for the primary outcome, assessed both by central adjudicators and by site investigators. We calculated the ratio of these treatment effects so that a ratio of these treatment effects >1 indicated that central adjudication resulted in a more beneficial treatment effect than assessment by the site investigator. A random-effects meta-analysis model was fitted to estimate a pooled effect. Results- Fifteen trials, comprising 69 560 participants, were included. The primary outcomes included were stroke (8/15, 53%), a composite event including stroke (6/15, 40%) and functional outcome after stroke measured on the modified Rankin Scale (1/15, 7%). The majority of site investigators were blind to treatment allocation (9/15, 60%). On average, there was no difference in treatment effect estimates based on data from central adjudicators and site investigators (pooled ratio of these treatment effects=1.02; 95% CI, [0.95-1.09]). Conclusions- We found no evidence that central adjudication of the primary outcome in stroke trials had any impact on trial conclusions. This suggests that potential advantages of central adjudication may not outweigh cost and time disadvantages in stroke studies if the primary purpose of adjudication is to ensure validity of trial findings.