Adjudicating outcomes: fundamentals

Accuracy and interrater agreement of death event adjudications by physician trainees: validation of the ISTH definition of pulmonary embolism-related death in an autopsy cohort

BACKGROUND

We previously determined good agreement and high specificity of the International Society on Thrombosis and Haemostasis (ISTH) definition of pulmonary embolism (PE)-related death among an expert central adjudication committee (CAC). CACs are often composed of experts in the corresponding research field. Involving physician trainees in CACs would allow investigators to divide the workload and foster trainees' research experience.

OBJECTIVE

To evaluate the accuracy of the ISTH definition of PE-related death for PE- versus non-PE-related deaths as confirmed by autopsy and its interrater agreement among physician trainees.

METHODS

This retrospective autopsy cohort included all patients with PE-related deaths between January 2010 and July 2019 as well as patients who died in 2018 from a cause other than PE at the New York-Presbyterian Hospital. Based on premortem clinical summaries, two physician trainees independently determined the cause of death using the ISTH definition of PE-related death. We calculated the sensitivity and specificity of the ISTH definition to identify autopsy-confirmed PE-related death and its interrater agreement.

RESULTS

Overall, 126 death events were adjudicated (median age, 68 years; 60 [48%] women), of which 29 (23%) were due to PE, as confirmed by autopsy. Sensitivity and specificity of the ISTH definition for autopsy-confirmed PE-related death was 48% (95% CI, 29-67) and 100% (95% CI, 96-100), respectively. Interrater reliability for PE-related death was good (percentage agreement, 93%; 95% CI, 87-96, Cohen's Kappa, 0.67; 95% CI, 44-85).

CONCLUSION

Our findings are consistent with our previous validation study. They further support the use of the ISTH definition of PE-related death and revealed high agreement between adjudicators with varied experience.

Central Adjudication Committee and Clinical Site Investigator Agreement on Outcomes in the PARITY Trial: A Secondary Analysis of the PARITY Trial Data

BACKGROUND

The detection of a surgical site infection (SSI) in patients with metal implants requires a high degree of clinical acumen. The inherent subjectivity of SSI diagnosis poses a challenge in the design of surgical trials because this subjectivity raises concern for outcome assessment bias. Central Adjudication Committees (CACs) are often utilized to minimize the variability in outcome assessment. Little research has been done to determine the reliability of outcome assessment in trials utilizing a CAC. In the present study, we determined the agreement between the study CAC and the clinical site investigators for the primary and secondary outcome assessments.

METHODS

The Prophylactic Antibiotic Regimens in Tumor Surgery (PARITY) trial was a multicenter, blinded, parallel 2-arm, randomized controlled trial that aimed to determine the effect of a 5-day versus 1-day postoperative prophylactic antibiotic regimen on the rate of SSI in patients undergoing surgical excision of tumors in the femur or tibia. The blinded PARITY CAC adjudicated all primary and secondary outcomes identified during the 1-year study follow-up. In the present secondary analysis, the Cohen kappa statistic was utilized to determine the level of agreement.

RESULTS

The primary outcome of SSI diagnosis demonstrated a substantial level of agreement between the CAC and the site investigators (0.699; 95% confidence interval [CI], 0.595 to 0.803]). Categorization of the SSI (i.e., superficial, deep, or organ space) showed moderate agreement (0.470; 95% CI, 0.382 to 0.558). Secondary outcomes such as the types of reoperations and the indication for reoperation typically showed substantial to almost perfect agreement, whereas antibiotic-related complications showed fair agreement (0.241; 95% CI, 0.000 to 0.474).

CONCLUSIONS

Although there was a substantial level of agreement between the PARITY CAC and site investigators on the diagnosis of an SSI, as well as typically at least substantial agreement on the causes and types of reoperations, there was less agreement regarding the type of SSI and the occurrence of an antibiotic-related complication. Therefore, the CAC appears to have provided value when adjudicating the depth of infection and when determining the causality of medical complications associated with antibiotics.

LEVEL OF EVIDENCE

Prognostic Level II. See Instructions for Authors for a complete description of levels of evidence.

Assessment of the Impact of an Endpoint Committee in the Ocular Hypertension Treatment Study

PURPOSE

To assess the impact of a masked Endpoint Committee on estimates of the incidence of primary open-angle glaucoma (POAG) treatment efficacy and statistical power of the Ocular Hypertension Treatment Study-Phase 1, 1994-2002 (OHTS-1).

DESIGN

Retrospective interrater reliability analysis of endpoint attribution by the Endpoint Committee.

METHODS

After study closeout, we recalculated estimates of endpoint incidence, treatment efficacy, and statistical power using all-cause endpoints and POAG endpoints. To avoid bias, only the first endpoint per participant is included in this report.

RESULTS

The Endpoint Committee reviewed 267 first endpoints from 1636 participants. The Endpoint Committee attributed 58% (155 of 267) of the endpoints to POAG. The incidence of all-cause endpoints vs POAG endpoints was 19.5% and 13.2%, respectively, in the observation group and 13.1% and 5.8%, respectively, in the medication group. Treatment effect for all-cause endpoints was a 33% reduction in risk (relative risk = 0.67, 95% confidence interval [CI] of 0.54-0.84) and a 56% reduction in risk for POAG endpoints (relative risk = 0.44, 95% CI of 0.31-0.61). Post hoc statistical power for detecting treatment effect was 0.94 for all-cause endpoints and 0.99 for POAG endpoints.

CONCLUSION

Endpoint Committee adjudication of endpoints improved POAG incidence estimates, increased statistical power, and increased calculated treatment effect by 23%. An Endpoint Committee should be considered in therapeutic trials when common ocular and systemic comorbidities, other than the target condition, could compromise study results.

The inter-rater reliability of the diagnosis of surgical site infection in the context of a clinical trial

OBJECTIVES

The diagnosis of surgical site infection following endoprosthetic reconstruction for bone tumours is frequently a subjective diagnosis. Large clinical trials use blinded Central Adjudication Committees (CACs) to minimise the variability and bias associated with assessing a clinical outcome. The aim of this study was to determine the level of inter-rater and intra-rater agreement in the diagnosis of surgical site infection in the context of a clinical trial.

MATERIALS AND METHODS

The Prophylactic Antibiotic Regimens in Tumour Surgery (PARITY) trial CAC adjudicated 29 non-PARITY cases of lower extremity endoprosthetic reconstruction. The CAC members classified each case according to the Centers for Disease Control (CDC) criteria for surgical site infection (superficial, deep, or organ space). Combinatorial analysis was used to calculate the smallest CAC panel size required to maximise agreement. A final meeting was held to establish a consensus.

RESULTS

Full or near consensus was reached in 20 of the 29 cases. The Fleiss kappa value was calculated as 0.44 (95% confidence interval (CI) 0.35 to 0.53), or moderate agreement. The greatest statistical agreement was observed in the outcome of no infection, 0.61 (95% CI 0.49 to 0.72, substantial agreement). Panelists reached a full consensus in 12 of 29 cases and near consensus in five of 29 cases when CDC criteria were used (superficial, deep or organ space). A stable maximum Fleiss kappa of 0.46 (95% CI 0.50 to 0.35) at CAC sizes greater than three members was obtained.

CONCLUSIONS

There is substantial agreement among the members of the PARITY CAC regarding the presence or absence of surgical site infection. Agreement on the level of infection, however, is more challenging. Additional clinical information routinely collected by the prospective PARITY trial may improve the discriminatory capacity of the CAC in the parent study for the diagnosis of infection.Cite this article: J. Nuttall, N. Evaniew, P. Thornley, A. Griffin, B. Deheshi, T. O'Shea, J. Wunder, P. Ferguson, R. L. Randall, R. Turcotte, P. Schneider, P. McKay, M. Bhandari, M. Ghert. The inter-rater reliability of the diagnosis of surgical site infection in the context of a clinical trial. Bone Joint Res 2016;5:347-352. DOI: 10.1302/2046-3758.58.BJR-2016-0036.R1.

Comparison of central adjudication of outcomes and onsite outcome assessment on treatment effect estimates

BACKGROUND

Assessment of events by adjudication committees (ACs) is recommended in multicentre randomised controlled trials (RCTs). However, its usefulness has been questioned.

OBJECTIVES

The aim of this systematic review was to compare 1) treatment effect estimates of subjective clinical events assessed by onsite assessors versus by AC, and 2) treatment effect estimates according to the blinding status of the onsite assessor as well as the process used to select events to adjudicate.

SEARCH METHODS

We searched Cochrane Central Register of Controlled Trials (CENTRAL), PubMed, EMBASE, PsycINFO, CINAHL and Google Scholar (25 August 2015 as the last updated search date), using a combination of terms to retrieve RCTs with commonly used terms to describe ACs.

SELECTION CRITERIA

We included all reports of RCTs and the published RCTs included in reviews and meta-analyses that reported the same subjective outcome event assessed by both an onsite assessor and an AC.

DATA COLLECTION AND ANALYSIS

We extracted the odds ratio (OR) from onsite assessment and the corresponding OR from AC assessment and calculated the ratio of the odds ratios (ROR). A ratio of odds ratios < 1 indicated that onsite assessors generated larger effect estimates in favour of the experimental treatment than ACs.

MAIN RESULTS

Data from 47 RCTs (275,078 patients) were used in the meta-analysis. We excluded 11 RCTs because of incomplete outcome data to calculate the OR for onsite and AC assessments. On average, there was no difference in treatment effect estimates from onsite assessors and AC (combined ROR: 1.00, 95% confidence interval (CI) 0.97 to 1.04; I(2) = 0%, 47 RCTs). The combined ROR was 1.00 (95% CI 0.96 to 1.04; I(2) = 0%, 35 RCTs) when onsite assessors were blinded; 0.76 (95% CI 0.48 to 1.12, I(2) = 0%, two RCTs) when AC assessed events identified independently from unblinded onsite assessors; and 1.11 (95% CI 0.96 to 1.27, I(2) = 0%, 10 RCTs) when AC assessed events identified by unblinded onsite assessors. However, there was a statistically significant interaction between these subgroups (P = 0.03) AUTHORS' CONCLUSIONS: On average, treatment effect estimates for subjective outcome events assessed by onsite assessors did not differ from those assessed by ACs. Results of subgroup analysis showed an interaction according to the blinded status of onsite assessors and the process used to submit data to AC. These results suggest that the use of ACs might be most important when onsite assessors are not blinded and the risk of misclassification is high. Furthermore, research is needed to explore the impact of the different procedures used to select events to adjudicate.

Architecture design of a generic centralized adjudication module integrated in a web-based clinical trial management system

BACKGROUND

Centralized outcome adjudication has been used widely in multicenter clinical trials in order to prevent potential biases and to reduce variations in important safety and efficacy outcome assessments. Adjudication procedures could vary significantly among different studies. In practice, the coordination of outcome adjudication procedures in many multicenter clinical trials remains as a manual process with low efficiency and high risk of delay. Motivated by the demands from two large clinical trial networks, a generic outcome adjudication module has been developed by the network's data management center within a homegrown clinical trial management system. In this article, the system design strategy and database structure are presented.

METHODS

A generic database model was created to transfer different adjudication procedures into a unified set of sequential adjudication steps. Each adjudication step was defined by one activate condition, one lock condition, one to five categorical data items to capture adjudication results, and one free text field for general comments. Based on this model, a generic outcome adjudication user interface and a generic data processing program were developed within a homegrown clinical trial management system to provide automated coordination of outcome adjudication.

RESULTS

By the end of 2014, this generic outcome adjudication module had been implemented in 10 multicenter trials. A total of 29 adjudication procedures were defined with the number of adjudication steps varying from 1 to 7. The implementation of a new adjudication procedure in this generic module took an experienced programmer 1 or 2 days. A total of 7336 outcome events had been adjudicated and 16,235 adjudication step activities had been recorded. In a multicenter trial, 1144 safety outcome event submissions went through a three-step adjudication procedure and reported a median of 3.95 days from safety event case report form submission to adjudication completion. In another trial, 277 clinical outcome events were adjudicated by a six-step procedure and took a median of 23.84 days from outcome event case report form submission to adjudication procedure completion.

CONCLUSION

A generic outcome adjudication module integrated in the clinical trial management system made the automated coordination of efficacy and safety outcome adjudication a reality.

Prophylactic antibiotic regimens in tumour surgery (PARITY): a pilot multicentre randomised controlled trial

Objective

Clinical studies of patients with bone sarcomas have been challenged by insufficient numbers at individual centres to draw valid conclusions. Our objective was to assess the feasibility of conducting a definitive multi-centre randomised controlled trial (RCT) to determine whether a five-day regimen of post-operative antibiotics, in comparison to a 24-hour regimen, decreases surgical site infections in patients undergoing endoprosthetic reconstruction for lower extremity primary bone tumours.

Methods

We performed a pilot international multi-centre RCT. We used central randomisation to conceal treatment allocation and sham antibiotics to blind participants, surgeons, and data collectors. We determined feasibility by measuring patient enrolment, completeness of follow-up, and protocol deviations for the antibiotic regimens.

Results

We screened 96 patients and enrolled 60 participants (44 men and 16 women) across 21 sites from four countries over 24 months (mean 2.13 participants per site per year, standard deviation 2.14). One participant was lost to follow-up and one withdrew consent. Complete data were obtained for 98% of eligible patients at two weeks, 83% at six months, and 73% at one year (the remainder with partial data or pending queries). In total, 18 participants missed at least one dose of antibiotics or placebo post-operatively, but 93% of all post-operative doses were administered per protocol.

Conclusions

It is feasible to conduct a definitive multi-centre RCT of post-operative antibiotic regimens in patients with bone sarcomas, but further expansion of our collaborative network will be critical. We have demonstrated an ability to coordinate in multiple countries, enrol participants, maintain protocol adherence, and minimise losses to follow-up.

Cite this article: Bone Joint Res;4:154–162