The use of surrogate endpoints in regulating medicines for cardio-renal disease: opinions of stakeholders

A framework for the definition and interpretation of the use of surrogate endpoints in interventional trials

Background

Interventional trials that evaluate treatment effects using surrogate endpoints have become increasingly common. This paper describes four linked empirical studies and the development of a framework for defining, interpreting and reporting surrogate endpoints in trials.

Methods

As part of developing the CONSORT (Consolidated Standards of Reporting Trials) and SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) extensions for randomised trials reporting surrogate endpoints, we undertook a scoping review, e-Delphi study, consensus meeting, and a web survey to examine current definitions and stakeholder (including clinicians, trial investigators, patients and public partners, journal editors, and health technology experts) interpretations of surrogate endpoints as primary outcome measures in trials.

Findings

Current surrogate endpoint definitional frameworks are inconsistent and unclear. Surrogate endpoints are used in trials as a substitute of the treatment effects of an intervention on the target outcome(s) of ultimate interest, events measuring how patients feel, function, or survive. Traditionally the consideration of surrogate endpoints in trials has focused on biomarkers (e.g., HDL cholesterol, blood pressure, tumour response), especially in the medical product regulatory setting. Nevertheless, the concept of surrogacy in trials is potentially broader. Intermediate outcomes that include a measure of function or symptoms (e.g., angina frequency, exercise tolerance) can also be used as substitute for target outcomes (e.g., all-cause mortality)-thereby acting as surrogate endpoints. However, we found a lack of consensus among stakeholders on accepting and interpreting intermediate outcomes in trials as surrogate endpoints or target outcomes. In our assessment, patients and health technology assessment experts appeared more likely to consider intermediate outcomes to be surrogate endpoints than clinicians and regulators.

Interpretation

There is an urgent need for better understanding and reporting on the use of surrogate endpoints, especially in the setting of interventional trials. We provide a framework for the definition of surrogate endpoints (biomarkers and intermediate outcomes) and target outcomes in trials to improve future reporting and aid stakeholders' interpretation and use of trial surrogate endpoint evidence.

Funding

SPIRIT-SURROGATE/CONSORT-SURROGATE project is Medical Research Council Better Research Better Health (MR/V038400/1) funded.

Current Resources for Evidence-Based Practice, July 2020

An extensive review of new resources to support the provision of evidence-based care for women and infants. The current column includes a discussion of whether it is ethical not to offer doula care to all women, and commentaries on reviews focused on folic acid and autism spectrum disorder, and timing of influenza vaccination during pregnancy.

Utility of the 5-Minute Apgar Score as a Research Endpoint

Although Apgar scores are commonly used as proxy outcomes, little evidence exists in support of the most common cutpoints (<7, <4). We used 2 data sets to explore this issue: one contained planned community births from across the United States (n = 52,877; 2012-2016), and the other contained hospital births from California (n = 428,877; 2010). We treated 5-minute Apgars as clinical "tests," compared against 18 known outcomes; we calculated sensitivity, specificity, positive and negative predictive values, and the area under the receiver operating characteristic curve for each. We used 3 different criteria to determine optimal cutpoints. Results were very consistent across data sets, outcomes, and all subgroups: The cutpoint that maximizes the trade-off between sensitivity and specificity is universally <9. However, extremely low positive predictive values for all outcomes at <9 indicate more misclassification than is acceptable for research. The areas under the receiver operating characteristic curves (which treat Apgars as quasicontinuous) were generally indicative of adequate discrimination between infants destined to experience poor outcomes and those not; comparing median Apgars between groups might be an analytical alternative to dichotomizing. Nonetheless, because Apgar scores are not clearly on any causal pathway of interest, we discourage researchers from using them unless the motivation for doing so is clear.

Challenges to hemoglobin A1c as a therapeutic target for type 2 diabetes mellitus

Glycated hemoglobin (HbA1c) is widely accepted as the most reliable measure of long-term glycemia. However, there is disagreement among professional medical societies on a proper glycemic target for long-term benefits in type 2 diabetes (T2D). The use of some glucose-lowering drugs was associated with heart failure despite substantial lowering of HbA1c. The failure of intensive glycemic control to reduce cardiovascular risk in some trials again brought into question the usefulness of HbA1c as a therapeutic target in T2D. In large cardiovascular outcome trials, some newer glucose-lowering drugs were associated with higher risks of heart failure or amputation despite comparable glycemic control between the test and placebo groups. Here, we provide evidence that variation in hemoglobin glycation between individuals is responsible for these inconsistencies. We suggest that further research be conducted in this area and that the findings be applied to clinical trials and practice.

The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal

BACKGROUND

The National Institute for Health and Care Excellence (NICE) commissioned a 'mock technology appraisal' to assess whether changes to its methods and processes are needed. This report presents the findings of independent research commissioned to inform this appraisal and the deliberations of a panel convened by NICE to evaluate the mock appraisal.

METHODS

Our research included reviews to identify issues, analysis methods and conceptual differences and the relevance of alternative decision frameworks, alongside the development of an exemplar case study of chimeric antigen receptor (CAR) T-cell therapy for treating acute lymphoblastic leukaemia.

RESULTS

An assessment of previous evaluations of regenerative medicines found that, although there were a number of evidential challenges, none was unique to regenerative medicines or was beyond the scope of existing methods used to conceptualise decision uncertainty. Regarding the clinical evidence for regenerative medicines, the issues were those associated with a limited evidence base but were not unique to regenerative medicines: small non-randomised studies, high variation in response and the intervention subject to continuing development. The relative treatment effects generated from single-arm trials are likely to be optimistic unless it is certain that the historical data have accurately estimated the efficacy of the control agent. Pivotal trials may use surrogate end points, which, on average, overestimate treatment effects. To reduce overall uncertainty, multivariate meta-analysis of all available data should be considered. Incorporating indirectly relevant but more reliable (more mature) data into the analysis can also be considered; such data may become available as a result of the evolving regulatory pathways being developed by the European Medicines Agency. For the exemplar case of CAR T-cell therapy, target product profiles (TPPs) were developed, which considered the 'curative' and 'bridging to stem-cell transplantation' treatment approaches separately. Within each TPP, three 'hypothetical' evidence sets (minimum, intermediate and mature) were generated to simulate the impact of alternative levels of precision and maturity in the clinical evidence. Subsequent assessments of cost-effectiveness were undertaken, employing the existing NICE reference case alongside additional analyses suggested within alternative frameworks. The additional exploratory analyses were undertaken to demonstrate how assessments of cost-effectiveness and uncertainty could be impacted by alternative managed entry agreements (MEAs), including price discounts, performance-related schemes and technology leasing. The panel deliberated on the range of TPPs, evidence sets and MEAs, commenting on the likely recommendations for each scenario. The panel discussed the challenges associated with the exemplar and regenerative medicines more broadly, focusing on the need for a robust quantification of the level of uncertainty in the cost-effective estimates and the potential value of MEAs in limiting the exposure of the NHS to high upfront costs and loss associated with a wrong decision.

CONCLUSIONS

It is to be expected that there will be a significant level of uncertainty in determining the clinical effectiveness of regenerative medicines and their long-term costs and benefits, but the existing methods available to estimate the implications of this uncertainty are sufficient. The use of risk sharing and MEAs between the NHS and manufacturers of regenerative medicines should be investigated further.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Unwarranted claims of drug efficacy in pharmaceutical sales visits: are drugs approved on the basis of surrogate outcomes promoted appropriately?

AIMS

This study compares physicians' recall of the claims of benefits on cardiovascular disease and diabetes made by pharmaceutical sales representatives for drugs approved on the basis of a surrogate outcome, i.e., an off-label claim, compared with those approved on the basis of a serious morbidity or mortality (clinical) outcome.

METHODS

Physicians in primary care practices in Montreal, Vancouver, Sacramento and Toulouse, who saw sales representatives as part of their usual practice and served a non-referral population, were contacted in blocks of 25 from a randomized list of all physicians practising in the relevant metropolitan area. We compared how frequently physicians reported that sales reps made claims of serious morbidity or mortality (clinically meaningful) benefits for drugs approved on the basis of surrogate outcomes vs. drugs approved on the basis of clinical outcomes.

RESULTS

There were 448 promotions for 58 unique brand name cardiovascular and diabetes drugs. Claims of clinically meaningful benefit were reported in 156 (45%) of the 347 promotions for surrogate outcome drugs, constituting unwarranted efficacy claims, i.e., off-label promotion. Claims of clinical benefit were reported in 72 of the 101 promotions (71%) for drugs approved on the basis of clinical outcomes, adjusted OR = 0.3 (95% CI 0.2, 0.6), P < 0.001.

CONCLUSIONS

Claims of efficacy made in sales visit promotions for drugs approved only on the basis of surrogate outcomes extended beyond the regulator-approved efficacy information for the product in almost half of promotions. Unapproved claims of drug efficacy constitute a form of off-label promotion and merit greater attention from regulators.

Time to Review the Role of Surrogate End Points in Health Policy: State of the Art and the Way Forward

The efficacy of medicines, medical devices, and other health technologies should be proved in trials that assess final patient-relevant outcomes such as survival or morbidity. Market access and coverage decisions are, however, often based on surrogate end points, biomarkers, or intermediate end points, which aim to substitute and predict patient-relevant outcomes that are unavailable because of methodological, financial, or practical constraints. We provide a summary of the present use of surrogate end points in health care policy, discussing the case for and against their adoption and reviewing validation methods. We introduce a three-step framework for policymakers to handle surrogates, which involves establishing the level of evidence, assessing the strength of the association, and quantifying relations between surrogates and final outcomes. Although the use of surrogates can be problematic, they can, when selected and validated appropriately, offer important opportunities for more efficient clinical trials and faster access to new health technologies that benefit patients and health care systems.

The Use of Surrogate and Patient-Relevant Endpoints in Outcomes-Based Market Access Agreements : Current Debate

The high cost of novel treatments is the major driver of negative or restricted reimbursement decisions by healthcare payers in many countries. Costly drugs can be subject to Market Access Agreements (MAAs), which are financial (Commercial Agreements [CAs]) or outcomes-based (Payment for Performance Agreements [P4Ps] or Coverage with Evidence Development agreements [CEDs]). Outcomes in outcomes-based MAAs are assessed through changes in surrogate endpoints (SEPs) or patient-relevant endpoints (PEPs). In May 2015, we reviewed published and grey literature on MAAs between manufacturers and large, institutionalised payers from all geographical areas, and classified the schemes into CAs, P4Ps and CEDs, as well as by therapeutic area and country. Outcomes-based MAAs were further categorized by the endpoint used. Overall, we identified 143 MAAs, 56 (39.2 %) of which were pure CAs, 53 (37.1 %) were CEDs, and 34 (23.8 %) were P4Ps. Among the CEDs, 49 were PEP CEDs and four were SEP CEDs; of the 34 P4Ps, 29 were SEP P4Ps for 30 drugs, and five were PEP P4Ps for at least six drugs; and among 87 outcomes-based MAAs (CEDs + P4Ps), PEP CEDs were the most common (56.3 %), followed by SEP P4Ps (34.1 %). The high proportion of SEPs used in P4Ps contrasts with the high proportion of PEPs used in CEDs. CEDs employ PEPs and it appears that they are used to reduce uncertainty about a drug's clinical outcomes and/or real-life use, and thus allow payers to align a product's value with price. We argue that P4Ps do not reduce uncertainty about real-life effectiveness and can only constitute an outcome guarantee for payers if they are based on PEPs or validated SEPs.

The Validity of Left Ventricular Mass as a Surrogate End Point for All-Cause and Cardiovascular Mortality Outcomes in People With CKD: A Systematic Review and Meta-analysis

BACKGROUND

Left ventricular mass (LVM) is a widely used surrogate end point in randomized trials involving people with chronic kidney disease (CKD) because treatment-induced LVM reductions are assumed to lower cardiovascular risk. The aim of this study was to assess the validity of LVM as a surrogate end point for all-cause and cardiovascular mortality in CKD.

STUDY DESIGN

Systematic review and meta-analysis.

SETTING & POPULATION

Participants with any stages of CKD.

SELECTION CRITERIA FOR STUDIES

Randomized controlled trials with 3 or more months' follow-up that reported LVM data.

INTERVENTION

Any pharmacologic or nonpharmacologic intervention.

OUTCOMES

The surrogate outcome of interest was LVM change from baseline to last measurement, and clinical outcomes of interest were all-cause and cardiovascular mortality. Standardized mean differences (SMDs) of LVM change and relative risk for mortality were estimated using pairwise random-effects meta-analysis. Correlations between surrogate and clinical outcomes were summarized across all interventions combined using bivariate random-effects Bayesian models, and 95% credible intervals were computed.

RESULTS

73 trials (6,732 participants) covering 25 intervention classes were included in the meta-analysis. Overall, risk of bias was uncertain or high. Only 3 interventions reduced LVM: erythropoiesis-stimulating agents (9 trials; SMD, -0.13; 95% CI, -0.23 to -0.03), renin-angiotensin-aldosterone system inhibitors (13 trials; SMD, -0.28; 95% CI, -0.45 to -0.12), and isosorbide mononitrate (2 trials; SMD, -0.43; 95% CI, -0.72 to -0.14). All interventions had uncertain effects on all-cause and cardiovascular mortality. There were weak and imprecise associations between the effects of interventions on LVM change and all-cause (32 trials; 5,044 participants; correlation coefficient, 0.28; 95% credible interval, -0.13 to 0.59) and cardiovascular mortality (13 trials; 2,327 participants; correlation coefficient, 0.30; 95% credible interval, -0.54 to 0.76).

LIMITATIONS

Limited long-term data, suboptimal quality of included studies.

CONCLUSIONS

There was no clear and consistent association between intervention-induced LVM change and mortality. Evidence for LVM as a valid surrogate end point in CKD is currently lacking.