Wisdom of the expert crowd prediction of response for 3 neurology randomized trials

The impact of feedback training on prediction of cancer clinical trial results

INTRODUCTION

Funders must make difficult decisions about which squared treatments to prioritize for randomized trials. Earlier research suggests that experts have no ability to predict which treatments will vindicate their promise. We tested whether a brief training module could improve experts' trial predictions.

METHODS

We randomized a sample of breast cancer and hematology-oncology experts to the presence or absence of a feedback training module where experts predicted outcomes for five recently completed randomized controlled trials and received feedback on accuracy. Experts then predicted primary outcome attainment for a sample of ongoing randomized controlled trials. Prediction skill was assessed by Brier scores, which measure the average deviation between their predictions and actual outcomes. Secondary outcomes were discrimination (ability to distinguish between positive and non-positive trials) and calibration (higher predictions reflecting higher probability of trials being positive).

RESULTS

A total of 148 experts (46 for breast cancer, 54 for leukemia, and 48 for lymphoma) were randomized between May and December 2017 and included in the analysis (1217 forecasts for 25 trials). Feedback did not improve prediction skill (mean Brier score for control: 0.22, 95% confidence interval = 0.20-0.24 vs feedback arm: 0.21, 95% confidence interval = 0.20-0.23; p = 0.51). Control and feedback arms showed similar discrimination (area under the curve = 0.70 vs 0.73, p = 0.24) and calibration (calibration index = 0.01 vs 0.01, p = 0.81). However, experts in both arms offered predictions that were significantly more accurate than uninformative forecasts of 50% (Brier score = 0.25).

DISCUSSION

A short training module did not improve predictions for cancer trial results. However, expert communities showed unexpected ability to anticipate positive trials.Pre-registration record: https://aspredicted.org/4ka6r.pdf.

Principal investigators over-optimistically forecast scientific and operational outcomes for clinical trials

Objective

To assess the accuracy of principal investigators’ (PIs) predictions about three events for their own clinical trials: positivity on trial primary outcomes, successful recruitment and timely trial completion.

Study design and setting

A short, electronic survey was used to elicit subjective probabilities within seven months of trial registration. When trial results became available, prediction skill was calculated using Brier scores (BS) and compared against uninformative prediction (i.e. predicting 50% all of the time).

Results

740 PIs returned surveys (16.7% response rate). Predictions on all three events tended to exceed observed event frequency. Averaged PI skill did not surpass uninformative predictions (e.g., BS = 0.25) for primary outcomes (BS = 0.25, 95% CI 0.20, 0.30) and were significantly worse for recruitment and timeline predictions (BS 0.38, 95% CI 0.33, 0.42; BS = 0.52, 95% CI 0.50, 0.55, respectively). PIs showed poor calibration for primary outcome, recruitment, and timelines (calibration index = 0.064, 0.150 and 0.406, respectively), modest discrimination in primary outcome predictions (AUC = 0.76, 95% CI 0.65, 0.85) but minimal discrimination in the other two outcomes (AUC = 0.64, 95% CI 0.57, 0.70; and 0.55, 95% CI 0.47, 0.62, respectively).

Conclusion

PIs showed overconfidence in favorable outcomes and exhibited limited skill in predicting scientific or operational outcomes for their own trials. They nevertheless showed modest ability to discriminate between positive and non-positive trial outcomes. Low survey response rates may limit generalizability.

OUP accepted manuscript

Background

Many randomized control trials (RCTs) evaluating programmed death receptor-1 (PD-1)/programmed death ligand-1 (PD-L1) targeting monoclonal antibodies (mAbs) have been completed or are in progress. We examined hypothesized hazard ratios (HHRs) and observed hazard ratios (OHRs) from published RCTs evaluating these mAbs.

Methods

Publications of RCTs evaluating at least one PD-1/PD-L1 targeting mAbs approved by the US Food and Drug Administration were identified through PubMed searches. The primary reports of RCTs were retrieved. Two investigators extracted HHR, OHR for the primary endpoint among other data elements independently. The differences (∆HR) in HHR and OHR were analyzed statistically. A separate search was conducted for secondary reports after longer follow-ups, the updated OHR was extracted.

Results

Forty-nine RCTs enrolling 36 867 patients were included. The mean HHR and OHR were 0.672 and 0.738 respectively. The mean ∆HR was 0.067 (range: –0.300 to 0.895; 95% confidence interval (CI), 0.003-0.130). HHR was met or exceeded in 22 (45%) RCTs. OHR was ≥ 1.0 in 6 RCTs (12%). PD-L1 expression was not associated with the magnitude of effect. Of 18 RCTs with follow-up reports, the magnitude of benefit decreased in 8 RCTs with extended follow-ups.

Conclusion

The majority of published RCTs evaluating PD-1/PD-L1 targeting mAbs did not achieve their hypothesized magnitude of benefit. The optimism bias requires attention from the cancer clinical research community given the number of these agents in development and the intense interest in evaluating these agents in a variety of disease settings.