Comparing classifications of death in the Mode Selection Trial: agreement and disagreement among site investigators and a clinical events committee

Endpoint adjudication in cardiovascular clinical trials

Endpoint adjudication (EA) is a common feature of contemporary randomized controlled trials (RCTs) in cardiovascular medicine. Endpoint adjudication refers to a process wherein a group of expert reviewers, known as the clinical endpoint committee (CEC), verify potential endpoints identified by site investigators. Events that are determined by the CEC to meet pre-specified trial definitions are then utilized for analysis. The rationale behind the use of EA is that it may lessen the potential misclassification of clinical events, thereby reducing statistical noise and bias. However, it has been questioned whether this is universally true, especially given that EA significantly increases the time, effort, and resources required to conduct a trial. Herein, we compare the summary estimates obtained using adjudicated vs. non-adjudicated site designated endpoints in major cardiovascular RCTs in which both were reported. Based on these data, we lay out a framework to determine which trials may warrant EA and where it may be redundant. The value of EA is likely greater when cardiovascular trials have nuanced primary endpoints, endpoint definitions that align poorly with practice, sub-optimal data completeness, greater operator variability, and lack of blinding. EA may not be needed if the primary endpoint is all-cause death or all-cause hospitalization. In contrast, EA is likely merited for more nuanced endpoints such as myocardial infarction, bleeding, worsening heart failure as an outpatient, unstable angina, or transient ischaemic attack. A risk-based approach to adjudication can potentially allow compromise between costs and accuracy. This would involve adjudication of a small proportion of events, with further adjudication done if inconsistencies are detected.

Comparison of Outcome Adjudication by Investigators and by a Central End Point Committee in Heart Failure Trials

Background:

The usefulness of adjudication by central end point committees (CECs) is poorly assessed in heart failure (HF) trials. We aimed to assess its impact on the outcome of the SHIFT trial (Systolic HF Treatment With the If Inhibitor Ivabradine Trial).

Methods:

SHIFT was a randomized placebo-controlled trial investigating the effect of ivabradine in 6505 HF patients with reduced ejection fraction. Prespecified end points, reported by investigators (all cardiologists) using specific case report form pages, included all-cause and specific causes of deaths and hospitalizations. The primary end point was a composite of cardiovascular deaths or hospitalizations for worsening HF. We compared the adjudication of prespecified end points made by investigators and by the CEC.

Results:

Investigators identified 7529 prespecified end points, 6793 of which were confirmed by the CEC: 98.1% of cardiovascular deaths, 88.6% of all hospitalizations, and 84.4% of hospitalizations for worsening HF. These differences had no meaningful impact on the study results; hazard ratio for the primary composite end point: investigators, 0.83 (95% CI, 0.76–0.91) versus CEC, 0.82 (95% CI, 0.75–0.90), with similar results for each component of the primary end point (hazard ratio of 0.92 versus 0.91 for cardiovascular death and 0.78 versus 0.74 for hospitalization for worsening HF).

Conclusions:

Central adjudication by a CEC in the SHIFT study confirmed most of cardiovascular deaths and worsening HF hospitalizations assessed by cardiologists and did not result in a significant change of the final result as compared to investigator judgment. In this context, the benefits of CEC in blinded HF trials should be reconsidered.

Registration:

URL: https://www.clinicaltrials.gov ; Unique identifier: NCT02441218. URL: http://www.isrctn.com/ISRCTN70429960 ; Unique identifier: ISRCTN70429960.

Scientific independence and objectivity: many questions linger about treatment of type 2 diabetes, such as scientific study design, optimal glucose control and the safety of injecting exogenous insulin

Whilst clinical guidelines exist for the treatment of people with type 2 diabetes, many underlying assumptions are still not qualified by convincing evidence. In this commentary, it is argued that fundamental issues still cloud clinical practice, such as biases in the design of clinical studies, the association between glucose control & clinical outcomes, and the safety of exposure to exogenous insulin and other glucose-lowering drugs. Relevant scientific evidence and alternative opinions about important issues continue to be largely ignored, and no effort has been made to resolve these questions. This may have had serious consequences, such as stifling innovation because there are no further benefits to be achieved in relation to glucose control.

Outcome Assessment by Central Adjudicators Versus Site Investigators in Stroke Trials: A Systematic Review and Meta-Analysis

Background and Purpose- In randomized stroke trials, central adjudication of a trial's primary outcome is regularly implemented. However, recent evidence questions the importance of central adjudication in randomized trials. The aim of this review was to compare outcomes assessed by central adjudicators with outcomes assessed by site investigators. Methods- We included randomized stroke trials where the primary outcome had undergone an assessment by site investigators and central adjudicators. We searched MEDLINE, EMBASE, CENTRAL (Cochrane Central Register of Controlled Trials), Web of Science, PsycINFO, and Google Scholar for eligible studies. We extracted information about the adjudication process as well as the treatment effect for the primary outcome, assessed both by central adjudicators and by site investigators. We calculated the ratio of these treatment effects so that a ratio of these treatment effects >1 indicated that central adjudication resulted in a more beneficial treatment effect than assessment by the site investigator. A random-effects meta-analysis model was fitted to estimate a pooled effect. Results- Fifteen trials, comprising 69 560 participants, were included. The primary outcomes included were stroke (8/15, 53%), a composite event including stroke (6/15, 40%) and functional outcome after stroke measured on the modified Rankin Scale (1/15, 7%). The majority of site investigators were blind to treatment allocation (9/15, 60%). On average, there was no difference in treatment effect estimates based on data from central adjudicators and site investigators (pooled ratio of these treatment effects=1.02; 95% CI, [0.95-1.09]). Conclusions- We found no evidence that central adjudication of the primary outcome in stroke trials had any impact on trial conclusions. This suggests that potential advantages of central adjudication may not outweigh cost and time disadvantages in stroke studies if the primary purpose of adjudication is to ensure validity of trial findings.

Central masked adjudication of stroke diagnosis at trial entry offered no advantage over diagnosis by local clinicians: Secondary analysis and simulation

BACKGROUND

Central adjudication of stroke type is commonly implemented in large multicentre clinical trials. We investigated the effect of central adjudication of diagnosis of stroke type at trial entry in the Efficacy of Nitric Oxide in Stroke (ENOS) trial.

METHODS

ENOS recruited patients with acute ischaemic or haemorrhagic stroke, and diagnostic adjudication was carried out using cranial scans. For this study, diagnoses made by local site clinicians were compared with those by central, masked adjudicators using kappa statistics. The trial primary analysis and subgroup analysis by stroke type were re-analysed using stroke diagnosis made by local clinicians, and simulations were used to assess the impact of increased non-differential misclassification and subgroup effects.

RESULTS

Agreement on stroke type (Ischaemic, Intracerebral Haemorrhage, Unknown stroke type, No-stroke) was high (κ = 0.92). Adjudication of stroke type had no impact on the primary outcome or subgroup analysis by stroke type. With misclassification increased to 10 times the level observed in ENOS and a simulated subgroup effect present, adjudication would have affected trial conclusions.

CONCLUSIONS

Stroke type at trial entry was diagnosed accurately by local clinicians in ENOS. Adjudication of stroke type by central adjudicators had no measurable effect on trial conclusions. Diagnostic adjudication may be important if diagnosis is complex and a treatment-diagnosis interaction is expected.

A comparison of approaches for adjudicating outcomes in clinical trials

Background

Incorrect classification of outcomes in clinical trials can lead to biased estimates of treatment effect and reduced power. Ensuring appropriate adjudication methods to minimize outcome misclassification is therefore essential. While there are many reported adjudication approaches, there is little consensus over which approach is best.

Methods

Under the assumption of non-differential assessment (i.e. that misclassification rates are the same in each treatment arm, as would typically be the case when outcome assessors are blinded), we use simulation and theoretical results to address four different questions about outcome adjudication: (a) How many assessors should be used? (b) When is it better to use onsite or central assessment? (c) Should central assessors adjudicate all outcomes, or only suspected events? (d) Should central assessment with multiple assessors be done independently or through group consensus?

Results

No one adjudication approach performs optimally in all settings. The optimal approach depends on the misclassification rates of site and central assessors, and the correlation between assessors. We found: (a) there will generally be little incremental benefit to using more than three assessors and, for outcomes with very high correlation between assessors, using one assessor is sufficient; (b) when choosing between site and central assessors, the assessor with the smallest misclassification rate should be chosen; when these rates are unknown, a combination of one site assessor and two central assessors will provide good results across a range of scenarios; (c) having central assessors adjudicate only suspected events will typically increase bias, and should be avoided, unless the threshold for sending outcomes for central assessment is extremely low; (d) central assessors can adjudicate either independently or in a group, and the preferred option should be dictated by whichever is expected to have the lowest misclassification rate.

Conclusions

Outcome adjudication is of critical importance to ensure validity of trial results, although no one approach is optimal across all settings. Investigators should choose the best strategy based on the specific characteristics of their trial. Regardless of the adjudication strategy chosen, assessors should be qualified and receive appropriate training.

Reproducibility of clinical events adjudications in a trial of venous thromboembolism prevention

Essentials The reproducibility of Clinical Events Committee (CEC) adjudications is almost unexplored. A random selection of events from a venous thromboembolism trial was blindly re-adjudicated. 'Unexplained sudden deaths' (possible fatal embolism) explained most discordant adjudications. A precise definition for CEC adjudication of this type of events is needed and proposed.

SUMMARY

Background When clinical trials use clinical endpoints, establishing independent Clinical Events Committees (CECs) is recommended to homogenize the interpretation of investigators' data. However, the reproducibility of CEC adjudications is almost unexplored. Objectives To assess the reproducibility of CEC adjudications in a trial of venous thromboembolism (VTE) prevention. Methods The PREVENU trial, a multicenter trial of VTE prevention, included 15 351 hospitalized medical patients. The primary endpoint was the composite of symptomatic VTE, major bleeding or unexplained sudden death (interpreted as possible fatal pulmonary embolism [PE]) at 3 months. The CEC comprised a chairman and four pairs of adjudicators. Of 2970 adjudicated clinical events, a random selection of 179 events (121 deaths, 40 bleeding events, and 18 VTE events) was blindly resubmitted to the CEC. Kappa values and their 95% confidence intervals (CIs) were calculated to measure adjudication agreement. Results Overall, 18 of 179 (10.1%, 95% CI 6.5-15.3%) adjudications proved discordant. Agreement for the PREVENU composite primary endpoint was good (kappa = 0.73, 95% CI 0.61-0.85). When analyzed separately, agreements were very good for non-fatal VTE events (1, 95% CI not applicable), moderate for all (fatal and non-fatal) VTE events (0.58, 95% CI 0.34-0.82), good for fatal and non-fatal major bleeding events (0.71, 95% CI 0.55-0.88), and moderate for all fatal events (0.60, 95% CI 0.40-0.81). Unexplained sudden death interpreted as possible fatal PE was responsible for nine of 18 (50%) discordant adjudications. Conclusion The reproducibility of CEC adjudications was good or very good for non-fatal VTE and bleeding events, but insufficient for VTE-related deaths, for which more precise and widely accepted definitions are needed.

Lack of Concordance Between Local Investigators, Angiographic Core Laboratory, and Clinical Event Committee in the Assessment of Stent Thrombosis: Results From the TRACER Angiographic Substudy

BACKGROUND

Stent thrombosis (ST) is an important end point in cardiovascular clinical trials. Adjudication is traditionally based on clinical event committee (CEC) review of case report forms and source documentation rather than angiograms. However, the degree to which this method of adjudication is concordant with the review of independent angiographic core laboratories (ACLs) has not been studied. This report represents the first assessment of variability between local investigators (LIs), a CEC, and an ACL.

METHODS AND RESULTS

Serial angiograms of 329 patients with acute coronary syndrome without ST-segment-elevation who underwent percutaneous coronary intervention at entry in the Trial to Assess the Effects of Vorapaxar in Preventing Heart Attack and Stroke in Particpants With Acute Coronary Syndrome (TRACER) and who met criteria for possible ST subsequent to the index event were reviewed by an ACL. The ACL was blinded to the assessment by both LIs and the CEC regarding the presence or absence of ST. CEC adjudication was based on Academic Research Consortium definitions of ST, using case report form data and source documents, including catheterization laboratory reports. The ACL, CEC, and LIs agreed on the presence or absence of ST in 52.9% events (κ=0.32; 95% confidence interval, 0.26-0.39). The ACL and CEC agreed on 82.7% of events (κ=0.57; 95% confidence interval, 0.47-0.67); the ACL and LIs agreed on 61.1% of events (κ=0.25; 95% confidence interval, 0.16-0.34); and the CEC and LIs agreed on 62% of events (κ=0.28; 95% confidence interval, 0.21-0.36).

CONCLUSIONS

ST reporting by an ACL, a CEC, and LIs is discordant. The assessment of ST is more often detected by direct review of angiograms by an ACL.

CLINICAL TRIAL REGISTRATION

URL: http://www.clinicaltrials.gov. Unique identifier: NCT00527943.

Comparison of central adjudication of outcomes and onsite outcome assessment on treatment effect estimates

BACKGROUND

Assessment of events by adjudication committees (ACs) is recommended in multicentre randomised controlled trials (RCTs). However, its usefulness has been questioned.

OBJECTIVES

The aim of this systematic review was to compare 1) treatment effect estimates of subjective clinical events assessed by onsite assessors versus by AC, and 2) treatment effect estimates according to the blinding status of the onsite assessor as well as the process used to select events to adjudicate.

SEARCH METHODS

We searched Cochrane Central Register of Controlled Trials (CENTRAL), PubMed, EMBASE, PsycINFO, CINAHL and Google Scholar (25 August 2015 as the last updated search date), using a combination of terms to retrieve RCTs with commonly used terms to describe ACs.

SELECTION CRITERIA

We included all reports of RCTs and the published RCTs included in reviews and meta-analyses that reported the same subjective outcome event assessed by both an onsite assessor and an AC.

DATA COLLECTION AND ANALYSIS

We extracted the odds ratio (OR) from onsite assessment and the corresponding OR from AC assessment and calculated the ratio of the odds ratios (ROR). A ratio of odds ratios < 1 indicated that onsite assessors generated larger effect estimates in favour of the experimental treatment than ACs.

MAIN RESULTS

Data from 47 RCTs (275,078 patients) were used in the meta-analysis. We excluded 11 RCTs because of incomplete outcome data to calculate the OR for onsite and AC assessments. On average, there was no difference in treatment effect estimates from onsite assessors and AC (combined ROR: 1.00, 95% confidence interval (CI) 0.97 to 1.04; I(2) = 0%, 47 RCTs). The combined ROR was 1.00 (95% CI 0.96 to 1.04; I(2) = 0%, 35 RCTs) when onsite assessors were blinded; 0.76 (95% CI 0.48 to 1.12, I(2) = 0%, two RCTs) when AC assessed events identified independently from unblinded onsite assessors; and 1.11 (95% CI 0.96 to 1.27, I(2) = 0%, 10 RCTs) when AC assessed events identified by unblinded onsite assessors. However, there was a statistically significant interaction between these subgroups (P = 0.03) AUTHORS' CONCLUSIONS: On average, treatment effect estimates for subjective outcome events assessed by onsite assessors did not differ from those assessed by ACs. Results of subgroup analysis showed an interaction according to the blinded status of onsite assessors and the process used to submit data to AC. These results suggest that the use of ACs might be most important when onsite assessors are not blinded and the risk of misclassification is high. Furthermore, research is needed to explore the impact of the different procedures used to select events to adjudicate.

Alignment of site versus adjudication committee–based diagnosis with patient outcomes: Insights from the Providing Rapid Out of Hospital Acute Cardiovascular Treatment 3 trial

Background:

Adjudication by an adjudication committee in clinical trials plays an important role in the assessment of outcomes. Controversy exists regarding the utility of adjudication committee versus site-based assessments and their relationship to subsequent clinical events.

Methods:

This study is a secondary analysis of the Providing Rapid Out of Hospital Acute Cardiovascular Treatment-3 trial, which randomized patients with chest pain or shortness of breath for biomarker testing in the ambulance. The emergency department physician diagnosis at the time of emergency department disposition was compared with an adjudicated diagnosis assigned by an adjudication committee. The level of agreement between emergency department and adjudication committee diagnosis was evaluated using kappa coefficient and compared to clinical outcomes (30-day re-hospitalization, 30-day and 1-year mortality).

Results:

Of the 477 patients, 49.3% were male with a median age of 70 years; hospital admission rate was 31.2%. The emergency department physicians and the adjudication committee disagreed in 55 cases (11.5%) with a kappa of 0.71 (95% confidence interval: 0.64, 0.78). The 30-day re-hospitalization, 30-day mortality, and 1-year mortality were 22%, 1.9%, and 9.4%, respectively. Although there were similar rates of re-hospitalization irrespective of adjudication, in cases of disagreement compared to agreement between adjudication committee and emergency department diagnosis, there was a higher 30-day (7.3% vs 1.2%, p = 0.002) and 1-year mortality (27.3% vs 7.1%, p < 0.001).

Conclusion:

Despite substantial agreement between the diagnosis of emergency department physicians and adjudication committee, in the subgroup of patients where there was disagreement, there was significantly worse short-term and long-term mortality.

Centralized adjudication of cardiovascular end points in cardiovascular and noncardiovascular pharmacologic trials: a report from the Cardiac Safety Research Consortium

This white paper provides a summary of presentations and discussions at a cardiovascular (CV) end point adjudication think tank cosponsored by the Cardiac Safety Research Committee and the US Food and Drug Administration (FDA) that was convened at the FDA's White Oak headquarters on November 6, 2013. Attention was focused on the lack of clarity concerning the need for end point adjudication in both CV and non-CV trials: there is currently an absence of widely accepted academic or industry standards and a definitive regulatory policy on how best to structure and use clinical end point committees (CECs). This meeting therefore provided a forum for leaders in the fields of CV clinical trials and CV safety to develop a foundation of initial best practice recommendations for use in future CEC charters. Attendees included representatives from pharmaceutical companies, regulatory agencies, end point adjudication specialist groups, clinical research organizations, and active, academically based adjudicators. The manuscript presents recommendations from the think tank regarding when CV end point adjudication should be considered in trials conducted by cardiologists and by noncardiologists as well as detailing key issues in the composition of a CEC and its charter. In addition, it presents several recommended best practices for the establishment and operation of CECs. The science underlying CV event adjudication is evolving, and suggestions for additional areas of research will be needed to continue to advance this science. This manuscript does not constitute regulatory guidance.

Adjudication-related processes are underreported and lack standardization in clinical trials of venous thromboembolism: a systematic review

OBJECTIVES

Although the use of an adjudication committee (AC) for outcomes is recommended in randomized controlled trials, there are limited data on the process of adjudication. We therefore aimed to assess whether the reporting of the adjudication process in venous thromboembolism (VTE) trials meets existing quality standards and which characteristics of trials influence the use of an AC.

STUDY DESIGN AND SETTING

We systematically searched MEDLINE and the Cochrane Library from January 1, 2003, to June 1, 2012, for randomized controlled trials on VTE. We abstracted information about characteristics and quality of trials and reporting of adjudication processes. We used stepwise backward logistic regression model to identify trial characteristics independently associated with the use of an AC.

RESULTS

We included 161 trials. Of these, 68.9% (111 of 161) reported the use of an AC. Overall, 99.1% (110 of 111) of trials with an AC used independent or blinded ACs, 14.4% (16 of 111) reported how the adjudication decision was reached within the AC, and 4.5% (5 of 111) reported on whether the reliability of adjudication was assessed. In multivariate analyses, multicenter trials [odds ratio (OR), 8.6; 95% confidence interval (CI): 2.7, 27.8], use of a data safety-monitoring board (OR, 3.7; 95% CI: 1.2, 11.6), and VTE as the primary outcome (OR, 5.7; 95% CI: 1.7, 19.4) were associated with the use of an AC. Trials without random allocation concealment (OR, 0.3; 95% CI: 0.1, 0.8) and open-label trials (OR, 0.3; 95% CI: 0.1, 1.0) were less likely to report an AC.

CONCLUSION

Recommended processes of adjudication are underreported and lack standardization in VTE-related clinical trials. The use of an AC varies substantially by trial characteristics.

Clinical outcome endpoints in heart failure trials: a European Society of Cardiology Heart Failure Association consensus document

Endpoint selection is a critically important step in clinical trial design. It poses major challenges for investigators, regulators, and study sponsors, and it also has important clinical and practical implications for physicians and patients. Clinical outcomes of interest in heart failure trials include all-cause mortality, cause-specific mortality, relevant non-fatal morbidity (e.g., all-cause and cause-specific hospitalization), composites capturing both morbidity and mortality, safety, symptoms, functional capacity, and patient-reported outcomes. Each of these endpoints has strengths and weaknesses that create controversies regarding which is most appropriate in terms of clinical importance, sensitivity, reliability, and consistency. Not surprisingly, a lack of consensus exists within the scientific community regarding the optimal endpoint(s) for both acute and chronic heart failure trials. In an effort to address these issues, the Heart Failure Association of the European Society of Cardiology (HFA-ESC) convened a group of expert heart failure clinical investigators, biostatisticians, regulators, and pharmaceutical industry scientists (Nice, France, 12-13 February 2012) to evaluate the challenges of defining heart failure endpoints in clinical trials and to develop a consensus framework. This report summarizes the group's recommendations for achieving common views on heart failure endpoints in clinical trials.

Effects of the endpoint adjudication process on the results of a randomised controlled trial: the ADVANCE trial

Background

Endpoint adjudication committees (EPAC) are widely used in clinical trials. The aim of the present analysis is to assess the effects of the endpoint adjudication process on the main findings of the ADVANCE trial (Trial registration: ClinicalTrials.gov NCT00145925).

Methods and Findings

The ADVANCE trial was a multicentre, 2×2 factorial randomised controlled trial of blood pressure lowering and intensive blood glucose control in 11140 patients with type 2 diabetes. Primary outcomes were major macrovascular (nonfatal myocardial infarction, nonfatal stroke and cardiovascular death) and microvascular (new or worsening nephropathy and retinopathy) events. Suspected primary outcomes were initially reported by the investigators at the 215 sites with subsequent adjudication by the EPAC. The EPAC also adjudicated upon potential events identified directly by ongoing screening of all reported events. Over a median follow-up of 5 years, the site investigators reported one or more primary outcomes among 2443 participants. After adjudication these events were confirmed for 2077 (85%) with 48 further events added through the EPAC-led database screening process. The estimated relative risk reductions (95% confidence intervals) in the primary outcome for the blood pressure lowering comparison were 8% (−1 to 15%) based on the investigator-reported events and 9% (0 to 17%) based on the EPAC-based events (P for homogeneity = 0.70). The corresponding findings for the glucose comparison were 8% (1 to 15%) and 10% (2% to 18%) (P for homogeneity = 0.60). The effect estimates were also highly comparable when studied separately for macrovascular events and microvascular events for both comparisons (all P for homogeneity>0.6).

Conclusions

The endpoint adjudication process had no discernible impact on the main findings in ADVANCE. These data highlight the need for careful consideration of the likely impact of an EPAC on the findings and conclusions of clinical trials prior to their establishment.

Adjudication of bleeding outcomes in an international thromboprophylaxis trial in critical illness

INTRODUCTION

Measuring bleeding in critical care trials is challenging. We determined the reliability of adjudicated bleeding assessments in a large thromboprophylaxis trial in the intensive care unit (ICU).

MATERIALS AND METHODS

PROphylaxis for ThromboEmbolism in Critical Care Trial (PROTECT) was an international randomized controlled trial that compared dalteparin to unfractionated heparin for the prevention of deep vein thrombosis in the ICU. Daily bleeding data were collected prospectively using a validated tool. Bleeds were adjudicated in duplicate by 2 of 4 members comprising a central adjudication committee. Bleeds were stratified by severity and study drug, then randomly assigned to adjudicator pairs. Adjudicators were blinded to treatment allocation, study centre and peer-assessments. We calculated agreement on bleeding severity and examined the effect of adjudication on overall trial results.

RESULTS

In PROTECT, 491 patients had bleeding events including 208 with major bleeding and 283 with minor bleeding only. Of 491 patients, 446 were adjudicated in duplicate: 182 with major, 250 with minor and 14 with no bleeding. After adjudication, 52 of 244 bleeds were downgraded to minor; whereas only 15 of 244 were upgraded to major. Overall agreement among adjudicators was excellent (crude agreement=86.3%; kappa=0.76). Hazard ratios for major or any bleeding with dalteparin or unfractionated heparin were similar when analyzed using non-adjudicated events.

CONCLUSIONS

Major bleeds were sometimes over-called by research coordinators in a large ICU thromboprohylaxis trial. Adjudicator agreement was excellent. Central adjudication allowed reliable bleeding assessment and enhanced the rigor and validity of this major safety outcome.

Adjudicating bleeding events in a platelet dose study: impact on outcome results and challenges

BACKGROUND

In the SToP platelet dose study, the World Health Organization (WHO) bleeding grade was assigned using adjudication. This study describes the challenges associated with adjudicating bleeding events and compares the adjudicated and bedside results for bleeding grade.

STUDY DESIGN AND METHODS

To categorize bleeding, the following information was provided to adjudicators: daily bleeding assessments, interventions to stop or control bleeding, daily blood counts, and transfused blood components. Each daily assessment was sent to two adjudicators who independently assigned a grade and anatomic site of bleeding. Discordant cases where disagreement occurred were sent to a third adjudicator and subsequently to a fourth or fifth adjudicator in an attempt to reach agreement. Disagreement after five adjudicators was resolved by consensus. The final adjudicated grade was compared with the grade of bleeding assigned at the bedside by study personnel.

RESULTS

A total of 1150 case report forms were adjudicated. Disagreement on grade of bleeding was common: 31.2% after the first two adjudicators, 4.0% after the third adjudicator, 0.7% after four, and 0.05% after five. Disagreement on anatomic site was less but still occurred in 17% of cases after two adjudicators. The frequency of bleeding (≥ Grade 2) based on adjudication was higher than bedside grading (standard-dose arm, 47.5% vs. 34.4%; low-dose arm, 50.0% vs. 43.1%).

CONCLUSION

The frequency of WHO bleeding varies depending on the method used to assign grade. Adjudication to assign bleeding grade resulted in significant disagreement when two adjudicators were used.

Strategic lessons from the clinical event classification process for the Assessment of Pexelizumab in Acute Myocardial Infarction (APEX-AMI) trial

BACKGROUND

Independent adjudication of clinical trial events is traditionally performed by physicians on a clinical event classification (CEC) committee.

OBJECTIVES

The experience of the centralized CEC group of the APEX-AMI trial is described. This group adjudicated key secondary pre-specified outcome measures of congestive heart failure (CHF) and cardiogenic shock through 90 days using an algorithmic approach for some events.

METHODS

Data were collected via an electronic data capture (EDC) tool on all subjects, and additional information was provided via EDC for patients identified by site investigators with CHF or shock. Two strategies were used to adjudicate potential events: 1) a computer algorithm (followed by physician confirmation) analyzed data to determine whether events met trial end point definitions; or 2) physician review was used if EDC data were inadequate to allow classification by algorithm.

RESULTS

Of 5745 patients, 282 suspected cardiogenic shock and 465 suspected CHF events were identified. The computer algorithm or physicians confirmed 196/282 cardiogenic shock and 277/465 CHF end points. Overall, 242/742 (32.6%) of suspected events were classified by algorithm. Of the 500 events not resolved by computer algorithm, the CEC physicians agreed with site investigator assessments in 126/277 (45%) of CHF and 151/196 (77%) of cardiogenic shock events. The CEC committee completed adjudication of all suspected 30- and 90-day CHF and cardiogenic shock events within 7 days of the last patient 30-day follow-up visit and within 1 day of the last patient 90-day follow-up visit. Only 27% of patients required source document collection in addition to EDC-collected information.

CONCLUSIONS

A complementary approach of a computerized assessment and physician review was used in the CEC effort of the APEX-AMI trial. The algorithm categorized approximately one third of suspected CHF/cardiogenic shock events. The APEX-AMI CEC experience shows that an algorithmic approach may be a useful strategy for end point evaluation but requires validation.

Determination of hospitalization type by investigator case report form or adjudication committee in a large heart failure clinical trial (β-Blocker Evaluation of Survival Trial [BEST])

BACKGROUND

End point committees are routinely used to adjudicate efficacy and safety end points in clinical trials. The 2,708-patient β-Blocker Evaluation of Survival Trial (BEST) originally determined hospitalization type via investigator case report forms (CRFs), which captured whether a hospitalization was due to worsening heart failure (HF). Recently, the BEST End Points Committee (EPC) completed a blinded adjudication of all hospitalizations, allowing a comparison of the CRF method to the EPC method of determining hospitalization type. We sought to compare the investigator-determined mode of hospitalizations with the adjudicated events, to quantify the degree of agreement, and to compare the clinical trial results by method of event classification.

METHODS

The BEST EPC reviewed all 5,086 hospitalizations that occurred in BEST. Events were identified using investigator-reported hospitalizations, as well as those documented by FDA Form 3500 (MedWatch) reports.

RESULTS

The investigators identified more HF hospitalization events than adjudication (2,466 vs 1,729, P < .0001, paired analysis). Eight hundred thirty-four (34%) HF hospitalizations identified in CRFs were not confirmed by adjudication. Ninety-seven (6%) adjudicated events were not identified by the investigator reported method. One thousand six hundred thirty-two events were similarly identified by both methods.

CONCLUSIONS

The EPC adjudication identified fewer HF hospitalizations than did the investigator reported method with no change in the hazard ratio for this end point. Our findings suggest that independent end point committees may improve reliability through reduced variance, thus providing similar outcome results with fewer events and no increase in CIs.

Evaluating the benefit of event adjudication of cardiovascular outcomes in large simple RCTs

BACKGROUND

Event adjudication in randomized controlled trials is thought to be a necessary step to remove noise and potential bias from the results [1,2]. However, this hypothesis has not been widely evaluated. We conducted a meta-analysis of a series of cardiovascular outcomes trials and estimated the effect of adjudication on treatment estimates and on the number of outcomes included the trials.

METHODS

Data were retrieved from all cardiovascular outcomes trials conducted at the Population Health Research Institute (PHRI) between 1993 and 2006. These data included 10 trials with over 9000 events from 95,038 individuals. Differences in the log odds ratios between adjudicated and reported outcomes were analyzed and summarized using a ratio of odds ratios. Both masked and unmasked trials were included in this analysis.

RESULTS

There were no effects of event adjudication on the estimates of treatment effect for the primary outcomes, myocardial infarction (MI), stroke, or cardiovascular/vascular death. For the trial primary outcomes, the effect of adjudication vs. reported events was OR ratio = 1.00 [95% confidence interval (CI): 0.97-1.02]. There were also no significant differences in the number of outcomes included in the trials. Results were the same for masked and unmasked trials.

LIMITATIONS

The number of unmasked trials were small, and this analysis was restricted to cardiovascular endpoints reported from trials managed by a single coordinating center, with similar sets of procedures. Individual patient data were not used for the analysis.

CONCLUSIONS

This systematic meta-analysis failed to detect any effect of event adjudication on study conclusions and the numbers of events included in the final analyses were minimally changed. Given the considerable effort required to perform adjudication, there is a need to demonstrate that this process does indeed increase the sensitivity of trials. There is a need to conduct more systematic analyses of the effect of event adjudication in other trials to determine if this process is truly worthwhile.

End points for clinical trials in acute heart failure syndromes

Acute heart failure syndromes (AHFS) remain a major cause of morbidity and mortality, in part because the development of new therapies for these disorders has been marked by frequent failure and little success. The heterogeneity of current approaches to AHFS drug development, particularly with regard to end points, remains a major potential barrier to progress in the field. End points involving hemodynamic status, biomarkers, symptoms, hospital stay, end organ function, and mortality have all been employed either alone or in combination in recent randomized clinical trials in AHFS. In this review, we will discuss the various end point domains from both a clinical and a statistical perspective, summarize the wide variety of end points used in completed and ongoing AHFS studies, and suggest steps for greater standardization of end points across AHFS trials.

A web-based endpoint adjudication system for interim analyses in clinical trials

BACKGROUND

A data monitoring committee (DMC) is often employed to assess trial progress and review safety data and efficacy endpoints throughout a trail. Interim analyses performed for the DMC should use data that are as complete and verified as possible. Such analyses are complicated when data verification involves subjective study endpoints or requires clinical expertise to determine each subject's status with respect to the study endpoint. Therefore, procedures are needed to obtain adjudicated data for interim analyses in an efficient manner. In the past, methods for handling such data included using locally reported results as surrogate endpoints, adjusting analysis methods for unadjudicated data, or simply performing the adjudication as rapidly as possible. These methods all have inadequacies that make their sole usage suboptimal.

PURPOSE

For a study of prophylaxis for invasive candidiasis, adjudication of both study eligibility criteria and clinical endpoints prior to two interim analyses was required. Because the study was expected to enroll at a moderate rate and the sponsor required adjudicated endpoints to be used for interim analyses, an efficient process for adjudication was required.

METHODS

We created a web-based endpoint adjudication system (WebEAS) that allows for expedited review by the endpoint adjudication committee (EAC). This system automatically identifies when a subject's data are complete, creates a subject profile from the study data, and assigns EAC reviewers. The reviewers use the WebEAS to review the subject profile and submit their completed review form. The WebEAS then compares the reviews, assigns an additional review as a tiebreaker if needed, and stores the adjudicated data.

RESULTS

The study for which this system was originally built was administratively closed after 10 months with only 38 subjects enrolled. The adjudication process was finalized and the WebEAS system activated prior to study closure. Some website accessibility issues presented initially. However, once these issues were resolved, the reviewers found the system user-friendly and easy to navigate.

LIMITATIONS

Web-based data adjudication depends upon expeditious data collection and verification. Further, ability to use web-based technologies, in addition to clinical expertise, must be considered in selecting EAC members.

CONCLUSION

The automated nature of this system makes it a practical mechanism for ensuring timely endpoint adjudication. The authors believe a similar approach could be useful for handling endpoint adjudication for future clinical trials.

Effects of the end point adjudication process on the results of the Perindopril Protection Against Recurrent Stroke Study (PROGRESS)

BACKGROUND AND PURPOSE

End point adjudication committees (EPAC) are widely used in large-scale clinical trials to ensure the robustness of diagnosis for end points.

METHODS

The Perindopril Protection Against Recurrent Stroke Study (PROGRESS) was a double-blind randomized trial of blood pressure lowering in 6105 participants with pre-existing cerebrovascular disease. Separate estimates of the effects of randomized treatment were determined using Cox regression models that were based on the unadjudicated events initially reported by the investigator and on the final events assigned by the EPAC.

RESULTS

There were 992 strokes initially reported by the investigators and 894 (90%) retained these diagnoses after adjudication by the EPAC. The hazard ratios (95% CIs) for the effect of randomized treatment on stroke were 0.74 (0.64 to 0.85) based on the investigator diagnoses and 0.72 (0.62 to 0.83) based on the EPAC diagnoses (P homogeneity=0.7). For each stroke subtype reported, the corresponding numbers of diagnoses (investigators/EPAC) were ischemic (593/565), hemorrhagic (124/111), and unknown (124/93) with no impact of the EPAC review on the estimates of treatment effects (all P homogeneity >0.3). There was likewise no detectable effect of reclassification of diagnoses for the effect estimates calculated for myocardial infarction or the main causes of death (all P homogeneity >0.5).

CONCLUSIONS

The EPAC process had no discernible impact on the trial conclusions. Very large trials powered to detect effects on stroke subtypes might obtain real scientific gain from an EPAC, but in the case of PROGRESS, the value of the EPAC was in the reassurance it provided.

Inadequate planning and reporting of adjudication committees in clinical trials: recommendation proposal

OBJECTIVES

Adjudication committees (ACs) are recommended in randomized controlled trials (RCTs) to standardize the assessment of events. We aimed to assess the reporting and functioning of ACs (synonyms: clinical event committees, endpoint committees) in clinical trials.

STUDY DESIGN AND SETTING

We searched five high-impact-factor medical journals for reports of RCTs with clinical event endpoints published between January 1, 2004 and December 31, 2005.

RESULTS

ACs were reported in 33.4% of the 314 reports of RCTs. ACs were reported in 29.6% of trials with low risk of misclassification (i.e., "hard" main outcome), in 47.5% of trials with medium risk of misclassification (i.e., subjective main outcome and intervention delivered in a blinded fashion) and in 31% of trials with high risk of misclassification (i.e., subjective main outcome without intervention delivered in a blinded fashion). Selected cases to be adjudicated consisted largely of events identified by site investigators (93.3%). Data provided to the AC were reported for 47.4% of ACs.

CONCLUSION

Reporting of ACs is not fitted to the risk of biased misclassification. Important aspects of the functioning of ACs are insufficiently reported or raise methodological issues. We propose some recommendations for planning and reporting ACs in clinical trials.

Determination of the underlying cause of death in three multicenter international HIV clinical trials

PURPOSE

Describe processes and challenges for an Endpoint Review Committee (ERC) in determining and adjudicating underlying causes of death in HIV clinical trials.

METHOD

Three randomized HIV trials (two evaluating interleukin-2 and one treatment interruption) enrolled 11,593 persons from 36 countries during 1999-2008. Three ERC members independently reviewed each death report and supporting source documentation to assign underlying cause of death; differences of opinion were adjudicated.

RESULTS

Of 453 deaths reported through January 14, 2008, underlying causes were as follows: 10% AIDS-defining diseases, 21% non-AIDS malignancies, 9% cardiac diseases, 9% liver disease, 8% non-AIDS-defining infections, 5% suicides, 5% other traumatic events/accidents, 4% drug overdoses/acute intoxications, 11% other causes, and 18% unknown. Major reasons for unknown classification were inadequate clinical information or supporting documentation to determine cause of death. Half (51%) of deaths reviewed by the ERC required follow-up adjudication; consensus was eventually always reached.

CONCLUSION

ERCs can successfully provide blinded, independent, and systematic determinations of underlying cause of death in HIV clinical trials. Committees should include those familiar with AIDS and non-AIDS-defining diseases and have processes for adjudicating differences of opinion. Training for local investigators and procedure manuals should emphasize obtaining maximum possible documentation and follow-up information on all trial deaths.

Do we need to adjudicate major clinical events?

PURPOSE

The use of centralized systems to adjudicate clinical events is common in large clinical trials, in spite of relatively little published literature concerning the rationale and justification. The purpose of this manuscript is to review the reasons for central adjudication and to discuss whether trials could be simplified by limiting or streamlining the adjudication process.

METHODS

We reviewed the literature concerning central adjudication and documented the experience of adjudication in several clinical trials. Since definitions for nonfatal events are generally heterogeneous and subjective, one reason for a central process of adjudication is to assist in assuring systematic application of the definition used in the trial. In open-label trials, assuring that the adjudication is done blinded to treatment assignment may provide protection against differential misclassification. Regulatory authorities, including the FDA, derive confidence in the validity of results when central adjudication is performed. The clinical community has become accustomed to a certain amount of adjudication and may criticize trials that lack adjudication.

LIMITATIONS

It is difficult to document the value of adjudication in trials that have reported adjudicated and nonadjudicated event rates and related treatment effects. Making rationale decisions about when and how to adjudicate is hampered by the lack of published study of when and how central adjudication is helpful to improve the quality and validity of trials and at what cost.

CONCLUSIONS

Adjudication has not been shown to improve the ability to determine treatment effects. Thus, adjudication may be overly complex and overused in many large simple trials. The appropriate role of central adjudication - which trials, which outcomes, what methods - deserves scrutiny and further study.