The averted infections ratio: a novel measure of effectiveness of experimental HIV pre-exposure prophylaxis agents

Study design approaches for future active-controlled HIV prevention trials

Objectives

Vigorous discussions are ongoing about future efficacy trial designs of candidate human immunodeficiency virus (HIV) prevention interventions. The study design challenges of HIV prevention interventions are considerable given rapid evolution of the prevention landscape and evidence of multiple modalities of highly effective products; future trials will likely be 'active-controlled', i.e., not include a placebo arm. Thus, novel design approaches are needed to accurately assess new interventions against these highly effective active controls.

Methods

To discuss active control design challenges and identify solutions, an initial virtual workshop series was hosted and supported by the International AIDS Enterprise (October 2020-March 2021). Subsequent symposia discussions continue to advance these efforts. As the non-inferiority design is an important conceptual reference design for guiding active control trials, we adopt several of its principles in our proposed design approaches.

Results

We discuss six potential study design approaches for formally evaluating absolute prevention efficacy given data from an active-controlled HIV prevention trial including using data from: 1) a registrational cohort, 2) recency assays, 3) an external trial placebo arm, 4) a biomarker of HIV incidence/exposure, 5) an anti-retroviral drug concentration as a mediator of prevention efficacy, and 6) immune biomarkers as a mediator of prevention efficacy.

Conclusions

Our understanding of these proposed novel approaches to future trial designs remains incomplete and there are many future statistical research needs. Yet, each of these approaches, within the context of an active-controlled trial, have the potential to yield reliable evidence of efficacy for future biomedical interventions.

Interpretation of active-control randomised trials: the case for a new analytical perspective involving averted events

Active-control trials, where an experimental treatment is compared with an established treatment, are performed when the inclusion of a placebo control group is deemed to be unethical. For time-to-event outcomes, the primary estimand is usually the rate ratio, or the closely-related hazard ratio, comparing the experimental group with the control group. In this article we describe major problems in the interpretation of this estimand, using examples from COVID-19 vaccine and HIV pre-exposure prophylaxis trials. In particular, when the control treatment is highly effective, the rate ratio may indicate that the experimental treatment is clearly statistically inferior even when it is worthwhile from a public health perspective. We argue that it is crucially important to consider averted events as well as observed events in the interpretation of active-control trials. An alternative metric that incorporates this information, the averted events ratio, is proposed and exemplified. Its interpretation is simple and conceptually appealing, namely the proportion of events that would be averted by using the experimental treatment rather than the control treatment. The averted events ratio cannot be directly estimated from the active-control trial, and requires an additional assumption about either: (a) the incidence that would have been observed in a hypothetical placebo arm (the counterfactual incidence) or (b) the efficacy of the control treatment (relative to no treatment) that pertained in the active-control trial. Although estimation of these parameters is not straightforward, this must be attempted in order to draw rational inferences. To date, this method has been applied only within HIV prevention research, but has wider applicability to treatment trials and other disease areas.

Incorporating oral PrEP into standard prevention services for South African women: a nested interrupted time-series study

BACKGROUND

As oral pre-exposure prophylaxis (PrEP) becomes the standard of prevention globally, its potential effect on HIV incidence in clinical trials of new prevention interventions is unknown, particularly for trials among women. In a trial measuring HIV incidence in African women, oral PrEP was incorporated into the standard of prevention in the trial's last year. We assessed the effect of on-site access to PrEP on HIV incidence in this natural experiment.

METHODS

We did a nested interrupted time-series study using data from the ECHO trial. At 12 sites in four countries (Eswatini, Kenya, South Africa, and Zambia), women (aged 16-35 years) were randomly assigned to receive one of three contraceptives between Dec 14, 2015, and Sept 12, 2017, and followed up quarterly for up to 18 months to determine the effect of contraceptive method on HIV acquisition. Women were eligible if they wanted long-acting contraception, were medically qualified to receive study contraceptives, and had not used any of the study contraceptives in the past 6 months. The present analyses are limited to nine South African sites where on-site access to oral PrEP was implemented between March 13 and June 12, 2018. Using an interrupted time-series design, we compared HIV incidence before versus after PrEP access, limited to quarterly study visits at which on-site PrEP access was available to at least some participants and, in a sensitivity analysis, to the 180 days before and after access. The outcome was incident HIV infection, detected using two rapid HIV tests done in parallel for each participant at every scheduled follow-up visit. This study is registered on ClinicalTrials.gov, NCT02550067.

FINDINGS

2124 women were followed up after on-site PrEP access began, of whom 543 (26%) reported PrEP use. A total of 12 HIV seroconversions were observed in 556 person-years (incidence 2·16%) after on-site PrEP access, compared with 133 HIV seroconversions in 2860 person-years (4·65%) before PrEP access (adjusted incidence rate ratio [IRR] 0·45, 95% CI 0·25-0·82, p=0·0085). Similar results were also observed when limiting the analysis to 180 days before versus after PrEP access. A total of 46 HIV seroconversions were observed in 919 person-years within 180 days before PrEP access, compared with 11 seroconversions in 481 person-years in the 180 days following PrEP access (incidence 5·00 vs 2·29 per 100 person-years; IRR 0·43, 95% CI 0·22-0·88, p=0·012).

INTERPRETATION

On-site access to PrEP as part of standard of prevention in a clinical trial among women in South Africa was associated with halving HIV incidence, when approximately a quarter of women started PrEP. Providing access to on-site PrEP could decrease incidence in HIV prevention trials. These data are also among the first to show in any setting that access to PrEP is associated with decreased HIV acquisition among South African women.

FUNDING

Bill & Melinda Gates Foundation, United States Agency for International Development, President's Emergency Plan for AIDS Relief, the Swedish International Development Cooperation Agency, South African Medical Research Council, and United Nations Population Fund.

Statistical approaches to accelerate the development of long-acting antiretrovirals for HIV pre-exposure prophylaxis

PURPOSE OF REVIEW

This review considers statistical issues in the design and analysis of the studies used to develop long-acting formulations of antiretrovirals for pre-exposure prophylaxis (PrEP).

RECENT FINDING

An abundant pipeline of products is maturing. Accelerating their evaluation as clinical products requires abandonment of noninferiority standards. Randomized trials should be based on the comparison of principled but innovative estimates of background HIV risk and enrich enrollment for those who do not desire current PrEP products. At every stage of testing, innovative analyses can be applied to help inform and accelerate later studies.

SUMMARY

The development of new long-acting PrEP regimens can be accelerated by innovations in design, ingenuity in synthesizing data sources, and application of causal inference methods.

Using the adherence-efficacy relationship of emtricitabine and tenofovir disoproxil fumarate to calculate background hiv incidence: a secondary analysis of a randomized, controlled trial

INTRODUCTION

Randomized trials of new agents for HIV pre-exposure prophylaxis (PrEP) compare against emtricitabine and tenofovir disoproxil fumarate (F/TDF), without a placebo group. We used the well-characterized adherence-efficacy relationship for F/TDF to back-calculate the (non-PrEP) counterfactual background HIV incidence (bHIV) in a randomized trial of a novel PrEP agent and estimate comparative efficacy (to counterfactual bHIV).

METHODS

The DISCOVER trial (ClinicalTrials.gov: NCT02842086) randomized 5387 men who have sex with men (MSM) and transgender women who have sex with men and demonstrated non-inferiority of emtricitabine and tenofovir alafenamide (F/TAF) to F/TDF (HIV incidence rate ratio [IRR] 0·47, 95% CI: 0·19 to 1.15). Tenofovir diphosphate (TFV-DP) levels in dried blood spots (DBS) were assessed for all diagnosed with HIV and in a random 10% of the cohort. We used a Bayesian model with a diffuse prior distribution, derived from established data relating tenofovir diphosphate levels to HIV prevention efficacy. This prior, combined with the F/TDF seroconversion rate and tenofovir diphosphate levels in DISCOVER, yielded Bayesian inferences on the counterfactual bHIV.

RESULTS

There were six versus 11 postbaseline HIV infections (0.14 vs. 0.25/100 person-years [PY]) on F/TAF and F/TDF respectively. Of the 11 on F/TDF, 10 had low, none had medium and one had high tenofovir diphosphate levels; among HIV-negative controls, 5% of the person-time years had low, 9% had medium and 86% had high TFV-DP levels. A non-informative prior distribution for counterfactual bHIV, combined with the prior for TFV-DP level-efficacy relationship, yielded a posterior counterfactual bHIV of 3·4 infections/100 PY (0.80 Bayesian credible interval [CrI] 1·9 to 5·9), which suggests a median HIV efficacy of 96% (0.95 CrI [88% to 99%]) for F/TAF and 93% (0.95 CrI [87% to 96%]) for F/TDF compared to bHIV.

CONCLUSIONS

Based on the established connection of drug concentrations to PrEP prevention efficacy, a Bayesian framework can be used to estimate a synthetic non-PrEP control group in randomized, active-controlled PrEP trials that include a F/TDF-comparator group.

Use of propensity score matching to create counterfactual group to assess potential HIV prevention interventions

The design of HIV prevention trials in the context of effective HIV preventive methods is a challenge. Alternate designs, including using non-randomised 'observational control arms' have been proposed. We used HIV simulated vaccine efficacy trials (SiVETs) to show pitfalls that may arise from using such observational controls and suggest how to conduct the analysis in the face of the pitfalls. Two SiVETs were nested within previously established observational cohorts of fisherfolk (FF) and female sex workers (FSW) in Uganda. SiVET participants received a licensed Hepatitis B vaccine in a schedule (0, 1 and 6 months) similar to that for a possible HIV vaccine efficacy trial. All participants received HIV counselling and testing every quarter for one year to assess HIV incidence rate ratio (IRR) between SiVET and non-SiVET (observational data). Propensity scores, conditional on baseline characteristics were calculated for SiVET participation and matched between SiVET and non-SiVET in the period before and during the SiVET study. We compared IRR before and after propensity score matching (PSM). In total, 3989 participants were enrolled into observational cohorts prior to SiVET, (1575 FF prior to Jul 2012 and 2414 FSW prior to Aug 2014). SiVET enrolled 572 participants (Jul 2012 to Apr 2014 in FF and Aug 2014 to Apr 2017 in FSW), with 953 non-SiVET participants observed in the SiVET concurrent period and 2928 from the pre-SiVET period (before Jul 2012 in FF or before Apr 2014 in FSW). Imbalances in baseline characteristics were observed between SiVET and non-SiVET participants in both periods before PSM. Similarly, HIV incidence was lower in SiVET than non-SiVET; SiVET-concurrent period, IRR = 0.59, 95% CI 0.31-0.68, p = 0.033 and pre-SiVET period, IRR = 0.77, 95% CI 0.43-1.29, p = 0.161. After PSM, participants baseline characteristics were comparable and there were minimal differences in HIV incidence between SiVET and non-SiVET participants. The process of screening for eligibility for efficacy trial selects participants with baseline characteristics different from the source population, confounding any observed differences in HIV incidence. Propensity score matching can be a useful tool to adjust the imbalance in the measured participants' baseline characteristics creating a counterfactual group to estimate the effect of interventions on HIV incidence.

Confidence limits for the averted infections ratio estimated via the counterfactual placebo incidence rate

Objectives

The averted infections ratio (AIR) is a novel measure for quantifying the preservation-of-effect in active-control non-inferiority clinical trials with a time-to-event outcome. In the main formulation, the AIR requires an estimate of the counterfactual placebo incidence rate. We describe two approaches for calculating confidence limits for the AIR given a point estimate of this parameter, a closed-form solution based on a Taylor series expansion (delta method) and an iterative method based on the profile-likelihood.

Methods

For each approach, exact coverage probabilities for the lower and upper confidence limits were computed over a grid of values of (1) the true value of the AIR (2) the expected number of counterfactual events (3) the effectiveness of the active-control treatment.

Results

Focussing on the lower confidence limit, which determines whether non-inferiority can be declared, the coverage achieved by the delta method is either less than or greater than the nominal coverage, depending on the true value of the AIR. In contrast, the coverage achieved by the profile-likelihood method is consistently accurate.

Conclusions

The profile-likelihood method is preferred because of better coverage properties, but the simpler delta method is valid when the experimental treatment is no less effective than the control treatment. A complementary Bayesian approach, which can be applied when the counterfactual incidence rate can be represented as a prior distribution, is also outlined.

Primary HIV-1 infection in users of pre-exposure prophylaxis

Pre-exposure prophylaxis (PrEP) has proven to be a highly effective and safe way to prevent HIV infection. Seroconversion and primary HIV infection are exceptional if adherence to PrEP is good. However, primary HIV infection while using PrEP can occur, albeit rarely, and HIV drug resistance might develop. Furthermore, the scope of PrEP is expected to expand, and clinicians might face potential seroconversions and primary HIV infection in patients starting or taking PrEP. The characteristics of primary HIV infection in users of PrEP are poorly described. PrEP users present a lower viral load peak during primary HIV infection and, frequently, fewer symptoms than individuals not exposed to PrEP. Additionally, PrEP prolongs the stages of seroconversion, thus potentially complicating diagnosis of primary HIV infection. Drug resistance is rare, occurring mostly when PrEP is initiated in undiagnosed patients who are at an extremely early stage of infection, in whom detection of HIV-RNA was not used to rule out HIV infection. Therefore, careful exclusion of primary HIV infection before starting PrEP is crucial. In patients presenting with primary HIV infection while on PrEP, a drug with a high genetic barrier (or even two) should be added to tenofovir disoproxil fumarate-emtricitabine until test results for resistance are available.

A Bayesian averted infection framework for PrEP trials with low numbers of HIV infections: application to the results of the DISCOVER trial

Trials of candidate agents for HIV pre-exposure prophylaxis (PrEP) might randomly assign participants to be given a new PrEP agent or oral coformulated tenofovir disoproxil fumarate plus emtricitabine. This design presents unique challenges in interpretation. First, with two active arms, HIV incidence might be low. Second, the effectiveness of tenofovir disoproxil fumarate plus emtricitabine varies across populations; thus, similar HIV incidence between groups could be consistent with a wide range of effectiveness for the new PrEP. We propose a two-part approach to trial results. First, we use Bayesian methods to incorporate assumptions about the background incidence of HIV in the trial in the absence of PrEP, possibly augmented by external data. On the basis of the estimated background incidence, we estimate and compare the number of averted (or prevented) HIV infections in each of the two trial groups, calculating the averted infections ratio. We apply these methods to a completed trial of tenofovir alafenamide plus emtricitabine for PrEP. Our framework shows that leveraging external information to estimate averted infections and the averted infections ratio enhances the efficiency and interpretation of active-controlled PrEP trials.

Mathematical modeling to reveal breakthrough mechanisms in the HIV Antibody Mediated Prevention (AMP) trials

The ongoing Antibody Mediated Prevention (AMP) trials will uncover whether passive infusion of the broadly neutralizing antibody (bNAb) VRC01 can protect against HIV acquisition. Previous statistical simulations indicate these trials may be partially protective. In that case, it will be crucial to identify the mechanism of breakthrough infections. To that end, we developed a mathematical modeling framework to simulate the AMP trials and infer the breakthrough mechanisms using measurable trial outcomes. This framework combines viral dynamics with antibody pharmacokinetics and pharmacodynamics, and will be generally applicable to forthcoming bNAb prevention trials. We fit our model to human viral load data (RV217). Then, we incorporated VRC01 neutralization using serum pharmacokinetics (HVTN 104) and in vitro pharmacodynamics (LANL CATNAP database). We systematically explored trial outcomes by reducing in vivo potency and varying the distribution of sensitivity to VRC01 in circulating strains. We found trial outcomes could be used in a clinical trial regression model (CTRM) to reveal whether partially protective trials were caused by large fractions of VRC01-resistant (IC50>50 μg/mL) circulating strains or rather a global reduction in VRC01 potency against all strains. The former mechanism suggests the need to enhance neutralizing antibody breadth; the latter suggests the need to enhance VRC01 delivery and/or in vivo binding. We will apply the clinical trial regression model to data from the completed trials to help optimize future approaches for passive delivery of anti-HIV neutralizing antibodies.

Infusions of broadly neutralizing antibodies are currently being tested as a novel HIV prevention modality. To help interpret the results of these antibody mediated prevention (AMP) studies we developed a mathematical modeling framework. The approach combines antibody potency and drug levels with models of HIV viral dynamics, which will be generally applicable to future studies. Through simulating these clinical trials, we found trial outcomes can be used in combination to infer whether breakthrough infections are caused by large fractions of antibody-resistant circulating strains or some reduction in potency against all strains. This distinction helps to focus future trials on enhancing neutralizing antibody breadth or antibody delivery and/or in vivo binding.

Handling an uncertain control group event risk in non-inferiority trials: non-inferiority frontiers and the power-stabilising transformation

Background

Non-inferiority trials are increasingly used to evaluate new treatments that are expected to have secondary advantages over standard of care, but similar efficacy on the primary outcome. When designing a non-inferiority trial with a binary primary outcome, the choice of effect measure for the non-inferiority margin (e.g. risk ratio or risk difference) has an important effect on sample size calculations; furthermore, if the control event risk observed is markedly different from that assumed, the trial can quickly lose power or the results become difficult to interpret.

Methods

We propose a new way of designing non-inferiority trials to overcome the issues raised by unexpected control event risks. Our proposal involves using clinical judgement to specify a ‘non-inferiority frontier’, i.e. a curve defining the most appropriate non-inferiority margin for each possible value of control event risk. Existing trials implicitly use frontiers defined by a fixed risk ratio or a fixed risk difference. We discuss their limitations and propose a fixed arcsine difference frontier, using the power-stabilising transformation for binary outcomes, which may better represent clinical judgement. We propose and compare three ways of designing a trial using this frontier: testing and reporting on the arcsine scale; testing on the arcsine scale but reporting on the risk difference or risk ratio scale; and modifying the margin on the risk difference or risk ratio scale after observing the control event risk according to the power-stabilising frontier.

Results

Testing and reporting on the arcsine scale leads to results which are challenging to interpret clinically. For small values of control event risk, testing on the arcsine scale and reporting results on the risk difference scale produces confidence intervals at a higher level than the nominal one or non-inferiority margins that are slightly smaller than those back-calculated from the power-stabilising frontier alone. However, working on the arcsine scale generally requires a larger sample size compared to the risk difference scale. Therefore, working on the risk difference scale, modifying the margin after observing the control event risk, might be preferable, as it requires a smaller sample size. However, this approach tends to slightly inflate type I error rate; a solution is to use a slightly lower significance level for testing, although this modestly reduces power. When working on the risk ratio scale instead, the same approach based on the modification of the margin leads to power levels above the nominal one, maintaining type I error under control.

Conclusions

Our proposed methods of designing non-inferiority trials using power-stabilising non-inferiority frontiers make trial design more resilient to unexpected values of the control event risk, at the only cost of requiring somewhat larger sample sizes when the goal is to report results on the risk difference scale.

Novel metric for evaluating pre-exposure prophylaxis programme effectiveness in real-world settings

Although large-scale provision of HIV pre-exposure prophylaxis (PrEP) is gaining momentum, no systematic method to evaluate or compare the effectiveness of different scale-up strategies in real-world settings exists. To date, estimating the effectiveness of PrEP has relied on clinical trials or mathematical models. We propose a novel and pragmatic metric to evaluate and compare programme effectiveness using routine implementation data. Using South African and Zambian PrEP guidelines, we provide two examples of how to consistently measure PrEP-programme effectiveness with routinely collected data. PrEP effectiveness should account for HIV seroconversion, the variable risk of HIV infection (seasons of risk) estimated with routine risk assessment at each clinic visit (when available), and the persistence of PrEP use. Three criteria should be met in order to be considered a successful outcome: first, a person who initiates PrEP must not seroconvert; second, there should be no more than one period at high risk of HIV infection during the follow-up period when not taking PrEP; and finally, an individual must continue to attend health-care visits or discontinue prophylaxis in consultation with a health-care provider within a specified follow-up period. The number of PrEP successes could then be compared with the total number of people initiating PrEP to establish a success ratio. This outcome is a useful and easily interpretable metric to monitor effectiveness of PrEP programmes with routinely collected clinical data and can be used in cost-effectiveness analyses. These measurements allow for comparisons of scale-up strategies for PrEP programmes and, if widely adopted, will allow comparative studies of different approaches for PrEP service delivery.

Mosaic effectiveness: measuring the impact of novel PrEP methods

Various ongoing trials seek to evaluate long-acting pre-exposure prophylaxis (PrEP) agents by showing that they are non-inferior to daily oral tenofovir disoproxil fumarate and emtricitabine. Trials comparing oral PrEP to new methods examine effectiveness in a setting where only one or the other is provided; however, a new product will probably be delivered in a context where oral PrEP is also available. The effectiveness of a new PrEP product is best measured by its potential effect in a context that also includes oral tenofovir disoproxil fumarate and emtricitabine as an option. We offer an alternative standard for long-acting products-a measure of the effectiveness of the new product in addition to oral tenofovir disoproxil fumarate and emtricitabine as compared with oral PrEP alone. We term this measure mosaic effectiveness. We illustrate scenarios where a novel product can fail to show non-inferiority but show substantial mosaic effectiveness, thus implying the public health value of the novel product even if it is less effective than oral PrEP. Regulatory standards should consider mosaic effectiveness, not just comparative effectiveness. We assert that measurements that combine rigor with public health relevance can accelerate progress against the HIV epidemic.

The Connection between the Averted Infections Ratio and the Rate Ratio in Active-control Trials of Pre-exposure Prophylaxis Agents

The design and analysis of active-control trials to evaluate experimental HIV pre-exposure prophylaxis (PrEP) agents pose serious statistical challenges. We recently proposed a new outcome measure, the averted infections ratio (AIR) – the proportion of infections that would be averted by using the experimental agent rather than the control agent (compared to no intervention). The main aim of the current paper is to examine the mathematical connection between AIR and the HIV incidence rate ratio, the standard outcome measure. We also consider the sample size implications of the choice of primary outcome measure and explore the connection between effectiveness and efficacy under a simplified model of adherence.

Taking stock of the present and looking ahead: envisioning challenges in the design of future HIV prevention efficacy trials

Despite the recent success of antiretrovirals for HIV prevention, additional, more effective, or more acceptable biomedical interventions will ultimately be needed to end the HIV epidemic. Designing clinical trials to evaluate the efficacy of new products that reduce HIV infection risk is challenging because of the existence of highly effective interventions to prevent HIV. However, the implementation of these interventions is uneven, and the fact that multiple HIV prevention efficacy trials are currently evaluating new products means the field confronts uncertainty in the emerging standard of prevention. In this Viewpoint, we take stock of the current state of HIV prevention, and subsequently discuss the key challenges in designing future trials to evaluate the next generation of HIV prevention products. We also highlight gaps in the knowledge base that need to be addressed to advance the design of research. Future trials are tenable, even in the context of existing and effective interventions, and should involve careful statistical approaches and multidisciplinary collaborative design.

Advancing Novel PrEP Products - Alternatives to Non-Inferiority

With the scale-up of HIV pre-exposure prophylaxis (PrEP) with tenofovir (TDF) with or without emtricitabine (FTC), we have entered an era of highly effective HIV prevention with a growing pipeline of potential products to be studied. These studies are likely to be randomized trials with an oral TDF/FTC control arm. These studies require comparison of incident infections and can be time and resource intensive. Conventional approaches for design and analysis active controlled trial can lead to very large sample sizes. We demonstrate the important of assumptions about background infections for interpreting trial results and suggest alternative criteria for demonstrating the efficacy and effectiveness of potential PrEP agents.

Ongoing Vaccine and Monoclonal Antibody HIV Prevention Efficacy Trials and Considerations for Sequel Efficacy Trial Designs

Four randomized placebo-controlled efficacy trials of a candidate vaccine or passively infused monoclonal antibody for prevention of HIV-1 infection are underway (HVTN 702 in South African men and women; HVTN 705 in sub-Saharan African women; HVTN 703/HPTN 081 in sub-Saharan African women; HVTN 704/HPTN 085 in U.S., Peruvian, Brazilian, and Swiss men or transgender persons who have sex with men). Several challenges are posed to the optimal design of the sequel efficacy trials, including: (1) how to account for the evolving mosaic of effective prevention interventions that may be part of the trial design or standard of prevention; (2) how to define viable and optimal sequel trial designs depending on the primary efficacy results and secondary "correlates of protection" results of each of the ongoing trials; and (3) how to define the primary objective of sequel efficacy trials if HIV-1 incidence is expected to be very low in all study arms such that a standard trial design has a steep opportunity cost. After summarizing the ongoing trials, I discuss statistical science considerations for sequel efficacy trial designs, both generally and specifically to each trial listed above. One conclusion is that the results of "correlates of protection" analyses, which ascertain how different host immunological markers and HIV-1 viral features impact HIV-1 risk and prevention efficacy, have an important influence on sequel trial design. This influence is especially relevant for the monoclonal antibody trials because of the focused pre-trial hypothesis that potency and coverage of serum neutralization constitutes a surrogate endpoint for HIV-1 infection. Another conclusion is that while assessing prevention efficacy against a counterfactual placebo group is fraught with risks for bias, such analysis is nonetheless important and study designs coupled with analysis methods should be developed to optimize such inferences. I draw a parallel with non-inferiority designs, which are fraught with risks given the necessity of making unverifiable assumptions for interpreting results, but nevertheless have been accepted when a superiority design is not possible and a rigorous/conservative non-inferiority margin is used. In a similar way, counterfactual placebo group efficacy analysis should use rigorous/conservative inference techniques that formally build in a rigorous/conservative margin to potential biases that could occur due to departures from unverifiable assumptions. Because reliability of this approach would require new techniques for verifying that the study cohort experienced substantial exposure to HIV-1, currently it may be appropriate as a secondary objective but not as a primary objective.