Exploring the relationship between the causal-inference and meta-analytic paradigms for the evaluation of surrogate endpoints

A survey of methodologies on causal inference methods in meta-analyses of randomized controlled trials

BACKGROUND

Meta-analyses of randomized controlled trials (RCTs) have been considered as the highest level of evidence in the pyramid of the evidence-based medicine. However, the causal interpretation of such results is seldom studied.

METHODS

We systematically searched for methodologies pertaining to the implementation of a causally explicit framework for meta-analysis of randomized controlled trials and discussed the interpretation and scientific relevance of such causal estimands. We performed a systematic search in four databases to identify relevant methodologies, supplemented with hand-search. We included methodologies that described causality under counterfactuals and potential outcomes framework.

RESULTS

We only identified three efforts explicitly describing a causal framework on meta-analysis of RCTs. Two approaches required individual participant data, while for the last one, only summary data were required. All three approaches presented a sufficient framework under which a meta-analytical estimate is identifiable and estimable. However, several conceptual limitations remain, mainly in regard to the data generation process under which the selected RCTs rise.

CONCLUSIONS

We undertook a review of methodologies on causal inference methods in meta-analyses. Although all identified methodologies provide valid causal estimates, there are limitations in the assumptions regarding the data generation process and sampling of the potential RCTs to be included in the meta-analysis which pose challenges to the interpretation and scientific relevance of the identified causal effects. Despite both causal inference and meta-analysis being extensively studied in the literature, limited effort exists of combining those two frameworks.

The individual-level surrogate threshold effect in a causal-inference setting with normally distributed endpoints

In the meta-analytic surrogate evaluation framework, the trial-level coefficient of determination R trial 2 quantifies the strength of the association between the expected causal treatment effects on the surrogate (S) and the true (T) endpoints. Burzykowski and Buyse supplemented this metric of surrogacy with the surrogate threshold effect (STE), which is defined as the minimum value of the causal treatment effect on S for which the predicted causal treatment effect on T exceeds zero. The STE supplements R trial 2 with a more direct clinically interpretable metric of surrogacy. Alonso et al. proposed to evaluate surrogacy based on the strength of the association between the individual (rather than expected) causal treatment effects on S and T. In the current paper, the individual-level surrogate threshold effect (ISTE) is introduced in the setting where S and T are normally distributed variables. ISTE is defined as the minimum value of the individual causal treatment effect on S for which the lower limit of the prediction interval around the individual causal treatment effect on T exceeds zero. The newly proposed methodology is applied in a case study, and it is illustrated that ISTE has an appealing clinical interpretation. The R package surrogate implements the methodology and a web appendix (supporting information) that details how the analyses can be conducted in practice is provided.

Validity of Surrogate Endpoints and Their Impact on Coverage Recommendations: A Retrospective Analysis across International Health Technology Assessment Agencies

BACKGROUND

Surrogate endpoints (i.e., intermediate endpoints intended to predict for patient-centered outcomes) are increasingly common. However, little is known about how surrogate evidence is handled in the context of health technology assessment (HTA).

OBJECTIVES

1) To map methodologies for the validation of surrogate endpoints and 2) to determine their impact on acceptability of surrogates and coverage decisions made by HTA agencies.

METHODS

We sought HTA reports where evaluation relied on a surrogate from 8 HTA agencies. We extracted data on the methods applied for surrogate validation. We assessed the level of agreement between agencies and fitted mixed-effects logistic regression models to test the impact of validation approaches on the agency's acceptability of the surrogate endpoint and their coverage recommendation.

RESULTS

Of the 124 included reports, 61 (49%) discussed the level of evidence to support the relationship between the surrogate and the patient-centered endpoint, 27 (22%) reported a correlation coefficient/association measure, and 40 (32%) quantified the expected effect on the patient-centered outcome. Overall, the surrogate endpoint was deemed acceptable in 49 (40%) reports (k-coefficient 0.10, P = 0.004). Any consideration of the level of evidence was associated with accepting the surrogate endpoint as valid (odds ratio [OR], 4.60; 95% confidence interval [CI], 1.60-13.18, P = 0.005). However, we did not find strong evidence of an association between accepting the surrogate endpoint and agency coverage recommendation (OR, 0.71; 95% CI, 0.23-2.20; P = 0.55).

CONCLUSIONS

Handling of surrogate endpoint evidence in reports varied greatly across HTA agencies, with inconsistent consideration of the level of evidence and statistical validation. Our findings call for careful reconsideration of the issue of surrogacy and the need for harmonization of practices across international HTA agencies.

On the relationship between association and surrogacy when both the surrogate and true endpoint are binary outcomes

The relationship between association and surrogacy has been the focus of much debate in the surrogate marker literature. Recently, the individual causal association (ICA) has been introduced as a metric of surrogacy in the causal inference framework, when both the surrogate and the true endpoint are normally distributed and when both are binary. Earlier work on the normal case has demonstrated that, although the ICA and the adjusted association are related metrics, their relationship strongly depends on unidentifiable parameters and, consequently, the association between both endpoints conveys little information on the validity of the surrogate. In addition, in the normal setting, the magnitude of the ICA does not depend on the mean of the outcomes. The latter implies that identifiable parameters such as mean responses and treatment effects provide no information on the validity of the surrogate. In the present work it is shown that this is fundamentally different in the binary case. We demonstrate that the observed association between the outcomes as well as the success rates in both treatment groups are quite predictive for the ICA. It is shown that finding a good surrogate will be more likely when the association between the endpoints is large, there are sizeable treatment effects and the success rates for both endpoints are similar in both treatment groups. These results are demonstrated using extensive simulations and illustrated on a case study in multi-drug resistant tuberculosis.

Lessons learned from IDeAl - 33 recommendations from the IDeAl-net about design and analysis of small population clinical trials

BACKGROUND

IDeAl (Integrated designs and analysis of small population clinical trials) is an EU funded project developing new statistical design and analysis methodologies for clinical trials in small population groups. Here we provide an overview of IDeAl findings and give recommendations to applied researchers.

METHOD

The description of the findings is broken down by the nine scientific IDeAl work packages and summarizes results from the project's more than 60 publications to date in peer reviewed journals. In addition, we applied text mining to evaluate the publications and the IDeAl work packages' output in relation to the design and analysis terms derived from in the IRDiRC task force report on small population clinical trials.

RESULTS

The results are summarized, describing the developments from an applied viewpoint. The main result presented here are 33 practical recommendations drawn from the work, giving researchers a comprehensive guidance to the improved methodology. In particular, the findings will help design and analyse efficient clinical trials in rare diseases with limited number of patients available. We developed a network representation relating the hot topics developed by the IRDiRC task force on small population clinical trials to IDeAl's work as well as relating important methodologies by IDeAl's definition necessary to consider in design and analysis of small-population clinical trials. These network representation establish a new perspective on design and analysis of small-population clinical trials.

CONCLUSION

IDeAl has provided a huge number of options to refine the statistical methodology for small-population clinical trials from various perspectives. A total of 33 recommendations developed and related to the work packages help the researcher to design small population clinical trial. The route to improvements is displayed in IDeAl-network representing important statistical methodological skills necessary to design and analysis of small-population clinical trials. The methods are ready for use.