US Food and Drug Administration Approvals of Drugs and Devices Based on Nonrandomized Clinical Trials: A Systematic Review and Meta-analysis

The causal relationship between physical activity, sedentary time and idiopathic pulmonary fibrosis risk: a Mendelian randomization study

BACKGROUND

Several observational studies have found that physical inactivity and sedentary time are associated with idiopathic pulmonary fibrosis (IPF) risk. However, the causality between them still requires further investigation. Therefore, our study aimed to investigate the causal effect of physical activity (PA) and sedentary time on the risk of IPF via two-sample Mendelian randomization (MR) analysis.

METHODS

Multiple genome-wide association study (GWAS) data involving individuals of European ancestry were analyzed. The datasets encompassed published UK Biobank data (91,105-377,234 participants) and IPF data (2018 cases and 373,064 controls) from FinnGen Biobank. The inverse variance weighting (IVW) method was the primary approach for our analysis. Sensitivity analyses were implemented with Cochran's Q test, MR-Egger regression, MR-PRESSO global test, and leave-one-out analysis.

RESULTS

Genetically predicted self-reported PA was associated with lower IPF risk [OR = 0.27; 95% CI 0.09-0.82; P = 0.02]. No causal effects of accelerometry-based PA or sedentary time on the risk of IPF were observed.

CONCLUSIONS

Our findings supported a protective relationship between self-reported PA and the risk for IPF. The results suggested that enhancing PA may be an effective preventive strategy for IPF.

How proven is a 'proven intervention'? Ethics of placebo controls in light of conditional approval programs

This article discusses the difficulties of establishing whether there exists a proven therapeutic intervention when regenerative experimental treatments are made accessible to patients under conditional approval programs (outside clinical trials). Conditional approvals are often made on the basis of less robust efficacy evidence than otherwise required for the registration of new treatments. Lower quality of evidence affects the ethical justification of using a placebo-control design. The absence of a proven intervention is important in evaluating whether it is ethically justifiable to use such a design in a clinical trial and is present in major ethical guidelines. The main argument in this paper is that conditionally approved therapies, if referred to as 'proven interventions', would make placebo-control design ethically unjustifiable. Conducting rigorous clinical trials after conditional approvals is crucial to establish the efficacy of therapeutic approaches under such approvals. Hindrances to running such trials and generating further efficacy evidence are brought to attention.

Increasing FDA-accelerated approval of single-arm trials in oncology (1992 to 2020)

OBJECTIVES

We aimed to map the characteristics of single-arm trials (SAT), report the Food and Drug Administration (FDA) transparency in presenting historical control, and to assess the confirmatory randomized controlled trials (RCTs).

STUDY DESIGN AND SETTING

This metaresearch included a review of all oncology indication approved using SAT by FDA-AA (FDA-Accelerated Approval) from 1992 to 2020. Two independent reviewers identified SAT, extracted data from FDA full medical reviews for historical controls reported and MEDLINE for searching for confirmatory RCT published.

RESULTS

Of 254 FDA-AA approvals, 119 (47%) were approved for oncologic indications using SAT. Fifty-four drugs for 72 oncology indications were for leukemia, lymphoma, lung cancer, urothelial cancer, multiple myeloma, and thyroid cancer. Overall, 37 (52%) treatments were converted into regular approval. Of these, 17 (46%) were based on confirmatory RCTs using overall survival (OS) as an outcome. Five indications were withdrawn from the market. Most trials outcomes were blindly assessed by independent research committees. Median trial sample size was 105 patients (min:8 to max:532). The FDA did not fully specify historical control selection in 75% of cases.

CONCLUSION

The granting of FDA-AAs based on SAT in oncology is increasing with more target drugs approved over time. Transparency in historical control reporting is necessary.

Real-world data in drug development strategies for orphan drugs: tafasitamab in B cell lymphoma, a case study for approval based on a single-arm combination trial

Tafasitamab (TAF) plus lenalidomide (LEN) is a novel treatment option for patients with relapsed/refractory diffuse large B cell lymphoma (rrDLBCL) who are not eligible for autologous stem cell transplantation. The initial US/EU approvals for TAF represent precedents because this is the first time that approval of a novel combination therapy was granted based on a pivotal single-arm trial (SAT). Matching real world-data (RWD) helped to disentangle the contribution of individual agents. In this review, we present the TAF development strategy, the prospective incorporation of RWD within the clinical development plan, the corresponding regulatory hurdles of this strategy, and the prior regulatory actions for other cancer drugs that previously incorporated RWD and propensity score matching in EU and US regulatory submissions. We also outline how RWD could further advance and impact orphan drug development.

IDENTIFICATION OF THRESHOLD FOR LARGE (DRAMATIC) EFFECTS THAT WOULD OBVIATE RANDOMIZED TRIALS IS NOT POSSIBLE

OBJECTIVE

To analyze distribution of "dramatic", large treatment effects.

STUDY DESIGN & SETTING

Pareto distribution modeling of previously reported cohorts of 3,486 randomized trials (RCTs) that enrolled 1,532,459 patients and 730 non-randomized studies (NRS) enrolling 1,650,658 patients.

RESULTS

We calculated the Pareto α parameter, which determines the tail of the distribution for various starting points of distribution [odds ratio_min (OR_min)]. In default analysis using all data at OR_min ≥1, Pareto distribution fit well to the treatment effects of RCTs favoring the new treatments (P = 0.21, Kolmogorov-Smirnov test) with best α = 2.32. For NRS, Pareto fit for OR_min ≥2 with best α = 1.91. For RCTs, theoretical 99th percentile OR was 32.7. The actual 99th percentile OR was 25; which converted into relative risk (RR) = 7.1. The maximum observed effect size was OR = 121 (RR = 11.45). For NRS, theoretical 99th percentile was OR = 315. The actual 99th percentile OR was 294 (RR = 13). The maximum observed effect size was OR = 1473 (RR = 66).

CONCLUSIONS

The effects sizes observed in RCTs and NRS considerably overlap. Large effects are rare and there is no clear threshold for dramatic effects that would obviate future RCTs.

U.S. Food and Drug Administration Reasoning in Approval Decisions When Efficacy Evidence Is Borderline, 2013-2018

BACKGROUND

The U.S. Food and Drug Administration (FDA) has substantial flexibility in its approval criteria in the context of life-threatening disease and unmet therapeutic need.

OBJECTIVE

To understand the FDA's evidentiary standards when flexible criteria are employed.

DESIGN

Case series.

SETTING

Applications submitted between 2013 and 2018 that went through multiple review cycles because the evidence for clinical efficacy was initially deemed insufficient.

MEASUREMENTS

Information was obtained from the approval package (available on Drugs@FDA), including advisory committee minutes, FDA reviews, and complete response letters.

RESULTS

Of 912 applications reviewed, 117 went through multiple review cycles; only 22 of these faced additional review primarily because of issues related to clinical efficacy. Concerns about the end point, the clinical meaningfulness of the observed effect, and inconsistent results were common bases for initial rejection. In 7 of the 22 cases, the approval did not require new evidence but rather new interpretations of the original evidence. No FDA decisions cited reasoning used in previous decisions.

LIMITATION

The conclusions rely on the authors' interpretation of the FDA statements and on a series of "close calls."

CONCLUSION

The FDA has no mechanism to find or tradition to cite similar cases when weighing evidence for approvals, resulting in standalone, bespoke decisions. These decisions show highly variable criteria for "substantial evidence" when flexible evidential criteria are used, highlighted by the recent approval of aducanumab. A precedential tradition and suitable information system are required for the FDA to improve institutional memory and build upon past decisions. These would increase the FDA's decisional transparency, consistency, and predictability, which are critical to preserving the FDA's most valuable asset, the public's trust.

PRIMARY FUNDING SOURCE

U.S. Food and Drug Administration.

Feasibility of Randomized Controlled Trials for Cancer Drugs Approved by the Food and Drug Administration Based on Single-Arm Studies

Background

The US Food and Drug Administration (FDA) introduced an Accelerated Approval (AA) pathway to expedite patient access to new drugs. AA accepts less rigorous trial designs, including single-arm studies (SAS), owing to perceived lack of feasibility of timely randomized controlled trials (RCTs).

Methods

We designed hypothetical RCTs with endpoints of overall response rate (ORR), progression-free survival (PFS), and overall survival (OS) for FDA approvals based on SAS for solid tumors during 2010-2019. Existing standards of care served as controls. RCTs were designed to detect a difference with power of 0.80, α-error of 5% (2-sided), and 1:1 randomization. Accrual duration was estimated based on participation by less than 5% of eligible patients derived from cancer-specific incidence and mortality rates in the United States.

Results

Of 172 (18.0%) approvals during the study period, 31 (18.0%) were based on SAS. Median sample size was 104 (range = 23-411), and 77.4% were AA. All studies reported ORR, 55% reported duration of response, 19.4% reported PFS, and 22.5% reported OS. Median sample sizes needed to conduct RCTs with endpoints of ORR, PFS, and OS were 206, 130, and 396, respectively. It would have been theoretically possible to conduct RCTs within duration comparable with that required by SAS for 84.6%, 94.1%, and 80.0% of approvals with endpoints of ORR, PFS, and OS, respectively.

Conclusion

An overwhelming majority of FDA approvals based on SAS should be feasible as RCTs within a reasonable time frame. Given the collateral harms to patients and to scientific rigor, drug approval based on SAS should only be permitted under exceptional circumstances.

When are randomized trials unnecessary? A signal detection theory approach to approving new treatments based on non-randomized studies

RATIONALE, AIMS AND OBJECTIVES

New therapies are increasingly approved by regulatory agencies such as the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) based on testing in non-randomized clinical trials. These treatments have typically displayed "dramatic effects" (ie, effects that are considered large enough to obviate the combined effects of biases and random errors that may affect the study results). The agencies, however, have not identified how large these effects should be to avoid the need for further testing in randomized controlled trials (RCTs). We investigated the effect size that would circumvent the need for further RCTs testing by the regulatory agencies. We hypothesized that the approval of therapeutic interventions by regulators is based on heuristic decision making whose accuracy can be best characterized by the application of signal detection theory (SDT).

METHODS

We merged the EMA and FDA database of approvals based on non-RCT comparisons. We excluded duplicate entries between the two databases. We included a total of 134 approvals of drugs and devices based on non-RCTs. We integrated Weber-Fechner law of psychophysics and recognition heuristics within SDT to provide descriptive explanations of the decisions made by the FDA and EMA to approve new treatments based on non-randomized studies without requiring further testing in RCTs.

RESULTS

Our findings suggest that when the difference between novel treatments and the historical control is at least one logarithm (base 10) of magnitude, the veracity of testing in non-RCTs seems to be established.

CONCLUSION

Drug developers and practitioners alike can use the change in one logarithm of effect size as a benchmark to decide if further testing in RCTs should be pursued, or as a guide to interpreting the results reported in non-randomized studies. However, further research would be useful to better characterize the threshold of effect size above which testing in RCTs is not needed.

Validity of observational evidence on putative risk and protective factors: appraisal of 3744 meta-analyses on 57 topics

BACKGROUND

The validity of observational studies and their meta-analyses is contested. Here, we aimed to appraise thousands of meta-analyses of observational studies using a pre-specified set of quantitative criteria that assess the significance, amount, consistency, and bias of the evidence. We also aimed to compare results from meta-analyses of observational studies against meta-analyses of randomized controlled trials (RCTs) and Mendelian randomization (MR) studies.

METHODS

We retrieved from PubMed (last update, November 19, 2020) umbrella reviews including meta-analyses of observational studies assessing putative risk or protective factors, regardless of the nature of the exposure and health outcome. We extracted information on 7 quantitative criteria that reflect the level of statistical support, the amount of data, the consistency across different studies, and hints pointing to potential bias. These criteria were level of statistical significance (pre-categorized according to 10^-6, 0.001, and 0.05 p-value thresholds), sample size, statistical significance for the largest study, 95% prediction intervals, between-study heterogeneity, and the results of tests for small study effects and for excess significance.

RESULTS

3744 associations (in 57 umbrella reviews) assessed by a median number of 7 (interquartile range 4 to 11) observational studies were eligible. Most associations were statistically significant at P < 0.05 (61.1%, 2289/3744). Only 2.6% of associations had P < 10^-6, ≥1000 cases (or ≥20,000 participants for continuous factors), P < 0.05 in the largest study, 95% prediction interval excluding the null, and no large between-study heterogeneity, small study effects, or excess significance. Across the 57 topics, large heterogeneity was observed in the proportion of associations fulfilling various quantitative criteria. The quantitative criteria were mostly independent from one another. Across 62 associations assessed in both RCTs and in observational studies, 37.1% had effect estimates in opposite directions and 43.5% had effect estimates differing beyond chance in the two designs. Across 94 comparisons assessed in both MR and observational studies, such discrepancies occurred in 30.8% and 54.7%, respectively.

CONCLUSIONS

Acknowledging that no gold-standard exists to judge whether an observational association is genuine, statistically significant results are common in observational studies, but they are rarely convincing or corroborated by randomized evidence.

Research synthesis of information theory measures of uncertainty: Meta-analysis of entropy and mutual information of diagnostic tests

RATIONALE, AIMS, AND OBJECTIVES

Assessing the performance of diagnostic tests requires evaluation of the amount of diagnostic uncertainty a test reduces. Statistical measures, such as sensitivity and specificity, currently dominating the evidence-based medicine (EBM) and related fields, cannot explicitly measure this reduction in diagnostic uncertainty. Mutual information (MI), an information theory statistic, explicitly quantifies diagnostic uncertainty by measuring information gain before vs after diagnostic testing. In this paper, we propose the use of MI as a single measure to express diagnostic test performance and demonstrate how it can be used in the meta-analysis of diagnostic test studies.

METHODS

We use two case studies from the literature to demonstrate the applicability of MI meta-analysis in assessing diagnostic performance. Meta-analysis of studies evaluating (a) ultrasonography (US) to detect endometrial cancer and (b) magnetic resonance angiography to detect arterial stenosis.

RESULTS

The results of MI meta-analyses are comparable to those of traditional statistical measures' meta-analyses. However, the results of MI are easier to understand as it relates directly to the extent of uncertainty a diagnostic test can reduce. For example, the US test, diagnosing endometrial cancer, is 40% specific and 94% sensitive. The combination of these values is difficult to interpret and may lead to inappropriate assessment (eg, one could favour the test due to its high sensitivity, ignoring its low specificity). In terms of MI, however, a single metric shows that the test reduces diagnostic uncertainty by 10%, which many users may consider small under most circumstances.

CONCLUSIONS

We have demonstrated the suitability of MI in assessing the performance of diagnostic tests, which can facilitate easier interpretation of the true utility of diagnostic tests. Similarly, to the guidance for interpretation of effect size of treatment interventions, we also propose the guidelines for interpretation of the utility of diagnostic tests based on the magnitude of reduction in diagnostic uncertainty.

Comparison of FDA accelerated vs regular pathway approvals for lung cancer treatments between 2006 and 2018

Regulatory agencies around the world have been using flexible requirements for approval of new drugs, especially for cancer drugs. The US Food and Drug Administration (FDA) is mostly the first agency to approve new drugs worldwide, mainly due to the faster terms of the accelerated pathway and breakthrough therapy designation. Surrogate endpoints and preliminary data (e.g. single-arm and phase 2 studies) are used for these new approvals, however larger effect sizes are expected. We aim to compare FDA Accelerated vs Regular Pathway approvals and Breakthrough therapy designations (BTD) for lung cancer treatments between 2006 and 2018 regarding study design, sample size, outcome measures and effect size. We assessed the FDA database to collect data from studies that formed the basis of approvals of new drugs or indications for lung cancer spanning from 2006 to 2018. We found that accelerated pathway approvals are based on significantly more single-arm studies with small sample sizes and surrogate primary endpoints. However, effect size was not different between the pathways. A large proportion of studies used to support regular pathway approvals also showed these characteristics that are related to low quality and uncertain evidence. Compared to other approvals, BTD were more frequently based on single-arm studies. There was no significant difference in use of surrogate endpoints or sample size. 44% of BTD were based on studies demonstrating large effect sizes, proportionally more than approvals not receiving this designation. In conclusion, based on the indicators of evidence quality we extracted, criteria’s for granting accelerated approval and breakthrough therapy designation seen not clear. Faster approvals are in the majority full of uncertainties which should be viewed with caution and the patient have to be communicated to allow shared decision making. Post-marketing validation is essential.

Laser interstitial thermal therapy (LITT) vs. bevacizumab for radiation necrosis in previously irradiated brain metastases

PURPOSE

Both laser interstitial thermal therapy (LITT) and bevacizumab have been used successfully to treat radiation necrosis (RN) after radiation for brain metastases. Our purpose is to compare pre-treatment patient characteristics and outcomes between the two treatment options.

METHODS

Single-institution retrospective chart review identified brain metastasis patients who developed RN between 2011 and 2018. Pre-treatment factors and treatment responses were compared between those treated with LITT versus bevacizumab.

RESULTS

Twenty-five patients underwent LITT and 13 patients were treated with bevacizumab. The LITT cohort had a longer overall survival (median 24.8 vs. 15.2 months for bevacizumab, p = 0.003) and trended to have a longer time to local recurrence (median 12.1 months vs. 2.0 for bevacizumab), although the latter failed to achieve statistical significance (p = 0.091). LITT resulted in an initial increase in lesional volume compared to bevacizumab (p < 0.001). However, this trend reversed in the long term follow-up, with LITT resulting in a median volume decrease at 1 year post-treatment of - 64.7% (range - 96.0% to +  > 100%), while bevacizumab patients saw a median volume increase of +  > 100% (range - 63.0% to +  > 100%), p = 0.010.

CONCLUSIONS

Our study suggests that patients undergoing LITT for RN have longer overall survival and better long-term lesional volume reduction than those treated with bevacizumab. However, it remains unclear whether our findings are due only to a difference in efficacy of the treatments or the implications of selection bias.

Generating comparative evidence on new drugs and devices after approval

Certain limitations of evidence available on drugs and devices at the time of market approval often persist in the post-marketing period. Often, post-marketing research landscape is fragmented. When regulatory agencies require pharmaceutical and device manufacturers to conduct studies in the post-marketing period, these studies might remain incomplete many years after approval. Even when completed, many post-marketing studies lack meaningful active comparators, have observational designs, and might not collect patient-relevant outcomes. Regulators, in collaboration with the industry and patients, ought to ensure that the key questions unanswered at the time of drug and device approval are resolved in a timely fashion during the post-marketing phase. We propose a set of seven key guiding principles that we believe will provide the necessary incentives for pharmaceutical and device manufacturers to generate comparative data in the post-marketing period. First, regulators (for drugs and devices), notified bodies (for devices in Europe), health technology assessment organisations, and payers should develop customised evidence generation plans, ensuring that future post-approval studies address any limitations of the data available at the time of market entry impacting the benefit-risk profiles of drugs and devices. Second, post-marketing studies should be designed hierarchically: priority should be given to efforts aimed at evaluating a product's net clinical benefit in randomised trials compared with current known effective therapy, whenever possible, to address common decisional dilemmas. Third, post-marketing studies should incorporate active comparators as appropriate. Fourth, use of non-randomised studies for the evaluation of clinical benefit in the post-marketing period should be limited to instances when the magnitude of effect is deemed to be large or when it is possible to reasonably infer the comparative benefits or risks in settings, in which doing a randomised trial is not feasible. Fifth, efficiency of randomised trials should be improved by streamlining patient recruitment and data collection through innovative design elements. Sixth, governments should directly support and facilitate the production of comparative post-marketing data by investing in the development of collaborative research networks and data systems that reduce the complexity, cost, and waste of rigorous post-marketing research efforts. Last, financial incentives and penalties should be developed or more actively reinforced.

Generating comparative evidence on new drugs and devices before approval

Fewer than half of new drugs have data on their comparative benefits and harms against existing treatment options at the time of regulatory approval in Europe and the USA. Even when active-comparator trials exist, they might not produce meaningful data to inform decisions in clinical practice and health policy. The uncertainty associated with the paucity of well designed active-comparator trials has been compounded by legal and regulatory changes in Europe and the USA that have created a complex mix of expedited programmes aimed at facilitating faster access to new drugs. Comparative evidence generation is even sparser for medical devices. Some have argued that the current process for regulatory approval needs to generate more evidence that is useful for patients, clinicians, and payers in health-care systems. We propose a set of five key principles relevant to the European Medicines Agency, European medical device regulatory agencies, US Food and Drug Administration, as well as payers, that we believe will provide the necessary incentives for pharmaceutical and device companies to generate comparative data on drugs and devices and assure timely availability of evidence that is useful for decision making. First, labelling should routinely inform patients and clinicians whether comparative data exist on new products. Second, regulators should be more selective in their use of programmes that facilitate drug and device approvals on the basis of incomplete benefit and harm data. Third, regulators should encourage the conduct of randomised trials with active comparators. Fourth, regulators should use prospectively designed network meta-analyses based on existing and future randomised trials. Last, payers should use their policy levers and negotiating power to incentivise the generation of comparative evidence on new and existing drugs and devices, for example, by explicitly considering proven added benefit in pricing and payment decisions.