Comparing the efficacy, exposure, and cost of clinical trial analysis methods

Demonstration of Group-Level and Individual-Level Efficacy Using Time-to-Event Designs for Clinical Trials of Antiseizure Medications

BACKGROUND AND OBJECTIVES

Participants with treatment-resistant epilepsy who are randomized to add-on placebo and remain in a trial for the typical 3 to 5-month maintenance period may be at increased risk of adverse outcomes. A novel trial design has been suggested, time to prerandomization monthly seizure count (T-PSC), which would limit participants' time on ineffective therapy. We reanalyzed 11 completed trials to determine whether the primary efficacy conclusions at T-PSC matched each of the original, longer trials.

METHODS

A total of 11 double-blind, placebo-controlled trials of levetiracetam, brivaracetam, lacosamide, topiramate, and lamotrigine for either focal-onset or generalized-onset epilepsy were selected. We evaluated the group-level and individual-level efficacy of treatments including the median percent reduction (MPR) in seizure frequency and 50% responder rate (50RR) at T-PSC, time to second seizure, and time to first seizure compared with the full-length trial.

RESULTS

The primary efficacy conclusions of 10 of the 11 trials would have been the same with a T-PSC design compared with the traditional design (the exception of lamotrigine had a very high initial placebo response). As a proportion of the full-length effect size, 90% of the MPR and 85% of the 50RR were seen at T-PSC (95% CI 73%-113% and 65%-110%, respectively). Using the T-PSC design, the time on blinded treatment was at least 312 participant-years shorter (40% of total duration) and 142,000 seizures occurred during this time (60% of total seizures). By contrast, the time to first or second seizure designs reproduced group-level effect size, but the primary efficacy conclusions of each trial and individual-level efficacy correspondence were fair to poor.

DISCUSSION

These results support the use of this trial design for new epilepsy medication trials because this reanalysis of 11 randomized controlled trials demonstrated that observation until T-PSC was sufficient to demonstrate efficacy while potentially improving participant safety by reducing the time of exposure to placebo and inadequate treatment. Despite analysis of 11 trials including 3,619 participants, we did not observe a significant reduction in the group-level effect size, which is directly related to statistical power. The next step is to evaluate whether T-PSC is sufficient to evaluate safety as measured by adverse events.

Time to prerandomization seizure count design sufficiently assessed the safety and tolerability of perampanel for the treatment of primary generalized tonic-clonic seizures

OBJECTIVE

Static assignment of participants in randomized clinical trials to placebo or ineffective treatment confers risk from continued seizures. An alternative trial design of time to exceed prerandomization monthly seizure count (T-PSC) has replicated the efficacy conclusions of traditionally designed trials, with shorter exposure to placebo and ineffective treatment. Trials aim to evaluate efficacy as well as safety and tolerability; therefore, we evaluated whether this T-PSC design also could replicate the trial's safety and tolerability conclusions.

METHODS

We retrospectively applied the T-PSC design to analyze treatment-emergent adverse events (TEAEs) from a blinded, placebo-controlled trial of perampanel for primary generalized tonic-clonic seizures (NCT01393743). The safety analysis set consisted of 81 and 82 participants randomized to perampanel and placebo arms, respectively. We evaluated the incidences of TEAEs, treatment-related TEAEs, serious TEAEs, and TEAEs of special interest that occurred before T-PSC relative to those observed during the full-length trial.

RESULTS

Of the 67 and 59 participants who experienced TEAEs in the perampanel and placebo arms during full-length trial, 66 (99%) and 54 (92%) participants experienced TEAEs with onset occurring before T-PSC, respectively. When limited to treatment-related TEAEs, 55 of 56 (98%) and 32 of 37 (86%) participants reported treatment-related TEAEs that occurred before T-PSC in the perampanel and placebo arms, respectively. There were more TEAEs after T-PSC with placebo as compared to perampanel (Fisher exact odds ratio = 8.6, p = .035), which resulted in overestimation of the difference in TEAE rate. There was a numerical reduction in serious TEAEs (3/13 occurred after T-PSC, one in placebo and two in perampanel).

SIGNIFICANCE

Almost all TEAEs occurred before T-PSC. More treatment-related TEAEs occurred after T-PSC for participants randomized to placebo than perampanel, which may be due to either a shorter T-PSC or delayed time to TEAE for placebo.

Challenges and directions in epilepsy diagnostics and therapeutics: Proceedings of the 17th Epilepsy Therapies and Diagnostics Development conference

Substantial efforts are underway toward optimizing the diagnosis, monitoring, and treatment of seizures and epilepsy. We describe preclinical programs in place for screening investigational therapeutic candidates in animal models, with particular attention to identifying and eliminating drugs that might paradoxically aggravate seizure burden. After preclinical development, we discuss challenges and solutions in the design and regulatory logistics of clinical trial execution, and efforts to develop disease biomarkers and interventions that may be not only seizure-suppressing, but also disease-modifying. As disease-modifying treatments are designed, there is clear recognition that, although seizures represent one critical therapeutic target, targeting nonseizure outcomes like cognitive development or functional outcomes requires changes to traditional designs. This reflects our increasing understanding that epilepsy is a disease with profound impact on quality of life for the patient and caregivers due to both seizures themselves and other nonseizure factors. This review examines selected key challenges and future directions in epilepsy diagnostics and therapeutics, from drug discovery to translational application.

Minimum clinical utility standards for wearable seizure detectors: A simulation study

OBJECTIVE

Epilepsy management employs self-reported seizure diaries, despite evidence of seizure underreporting. Wearable and implantable seizure detection devices are now becoming more widely available. There are no clear guidelines about what levels of accuracy are sufficient. This study aimed to simulate clinical use cases and identify the necessary level of accuracy for each.

METHODS

Using a realistic seizure simulator (CHOCOLATES), a ground truth was produced, which was then sampled to generate signals from simulated seizure detectors of various capabilities. Five use cases were evaluated: (1) randomized clinical trials (RCTs), (2) medication adjustment in clinic, (3) injury prevention, (4) sudden unexpected death in epilepsy (SUDEP) prevention, and (5) treatment of seizure clusters. We considered sensitivity (0%-100%), false alarm rate (FAR; 0-2/day), and device type (external wearable vs. implant) in each scenario.

RESULTS

The RCT case was efficient for a wide range of wearable parameters, though implantable devices were preferred. Lower accuracy wearables resulted in subtle changes in the distribution of patients enrolled in RCTs, and therefore higher sensitivity and lower FAR values were preferred. In the clinic case, a wide range of sensitivity, FAR, and device type yielded similar results. For injury prevention, SUDEP prevention, and seizure cluster treatment, each scenario required high sensitivity and yet was minimally influenced by FAR.

SIGNIFICANCE

The choice of use case is paramount in determining acceptable accuracy levels for a wearable seizure detection device. We offer simulation results for determining and verifying utility for specific use case and specific wearable parameters.

Inductive reasoning with large language models: a simulated randomized controlled trial for epilepsy

Importance

The analysis of electronic medical records at scale to learn from clinical experience is currently very challenging. The integration of artificial intelligence (AI), specifically foundational large language models (LLMs), into an analysis pipeline may overcome some of the current limitations of modest input sizes, inaccuracies, biases, and incomplete knowledge bases.

Objective

To explore the effectiveness of using an LLM for generating realistic clinical data and other LLMs for summarizing and synthesizing information in a model system, simulating a randomized clinical trial (RCT) in epilepsy to demonstrate the potential of inductive reasoning via medical chart review.

Design

An LLM-generated simulated RCT based on a RCT for treatment with an antiseizure medication, cenobamate, including a placebo arm and a full-strength drug arm, evaluated by an LLM-based pipeline versus a human reader.

Setting

Simulation based on realistic seizure diaries, treatment effects, reported symptoms and clinical notes generated by LLMs with multiple different neurologist writing styles.

Participants

Simulated cohort of 240 patients, divided 1:1 into placebo and drug arms.

Intervention

Utilization of LLMs for the generation of clinical notes and for the synthesis of data from these notes, aiming to evaluate the efficacy and safety of cenobamate in seizure control either with a human evaluator or AI-pipeline.

Measures

The AI and human analysis focused on identifying the number of seizures, symptom reports, and treatment efficacy, with statistical analysis comparing the 50%-responder rate and median percentage change between the placebo and drug arms, as well as side effect rates in each arm.

Results

AI closely mirrored human analysis, demonstrating the drug's efficacy with marginal differences (<3%) in identifying both drug efficacy and reported symptoms.

Conclusions and Relevance

This study showcases the potential of LLMs accurately simulate and analyze clinical trials. Significantly, it highlights the ability of LLMs to reconstruct essential trial elements, identify treatment effects, and recognize reported symptoms, within a realistic clinical framework. The findings underscore the relevance of LLMs in future clinical research, offering a scalable, efficient alternative to traditional data mining methods without the need for specialized medical language training.

Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist

PURPOSE OF REVIEW

Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice.

RECENT FINDINGS

ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.

Quantifying and controlling the impact of regression to the mean on randomized controlled trials in epilepsy

OBJECTIVE

Randomized controlled trials (RCTs) in epilepsy for drug treatments are plagued by high costs. One potential remedy is to reduce placebo response via better control over regression to the mean (RTM). Here, RTM represents an initial observed seizure rate higher than the long-term average, which gradually settles closer to the average, resulting in apparent response to treatment. This study used simulation to clarify the relationship between eligibility criteria and RTM.

METHODS

Using a statistically realistic seizure diary simulator, the impact of RTM on placebo response and trial efficacy was explored by varying eligibility criteria for a traditional treatment phase II/III RCT for drug-resistant epilepsy.

RESULTS

When the baseline period was included in the eligibility criteria, increasingly larger fractions of RTM were observed (25%-47% vs. 23%-25%). Higher fractions of RTM corresponded with higher expected placebo responses (50% responder rate [RR50]: 2%-9% vs. 0%-8%) and lower statistical efficacy (RR50: 47%-67% vs. 47%-81%). The exclusion of baseline from eligibility criteria was shown to decrease the number of patients needed by roughly 30%.

SIGNIFICANCE

The manipulation of eligibility criteria for RCTs has a predictable and important impact on RTM, and therefore on placebo response; the difference between drug and placebo was more easily detected. This in turn impacts trial efficacy and therefore cost. This study found dramatic improvements in efficacy and cost when baseline was not included in eligibility.

Increasing challenges to trial recruitment and conduct over time

OBJECTIVE

This study was undertaken to evaluate how the challenges in the recruitment and retention of participants in clinical trials for focal onset epilepsy have changed over time.

METHODS

In this systematic analysis of randomized clinical trials of adjunct antiseizure medications for medication-resistant focal onset epilepsy, we evaluated how the numbers of participants, sites, and countries have changed since the first such trial in 1990. We also evaluated the proportion of participants who completed each trial phase and their reasons for early trial exit. We analyzed these trends using mixed effects generalized linear models accounting for the influence of the number of trial sites and trial-specific variability.

RESULTS

The number of participants per site has steadily decreased over decades, with recent trials recruiting fewer than five participants per site (reduction by .16 participants/site/year, p < .0001). Fewer participants also progressed from recruitment to randomization over time (odds ratio = .94/year, p = .014). Concurrently, there has been an increase in the placebo response over time (increase in median percent reduction of .4%/year, p = .02; odds ratio of increase in 50% responder rate of 1.03/year, p = .02), which was not directly associated with the number of sites per trial (p > .20).

SIGNIFICANCE

This historical analysis highlights the increasing challenges with participant recruitment and retention, as well as increasing placebo response. It serves as a call to action to change clinical trial design to address these challenges.

Time-to-event clinical trial designs: Existing evidence and remaining concerns

Well-designed placebo-controlled clinical trials are critical to the development of novel treatments for epilepsy, but their design has not changed for decades. Patients, clinicians, regulators, and innovators all have concerns that recruiting for trials is challenging, in part, due to the static design of maintaining participants for long periods on add-on placebo when there are an increasing number of options for therapy. A traditional trial maintains participants on blinded treatment for a static period (e.g., 12 weeks of maintenance), during which participants on placebo have an elevated risk of sudden unexpected death in epilepsy compared to patients on an active treatment. Time-to-event trials observe participants on blinded treatment until a key event occurs (e.g., post-randomization seizure count matches pre-randomization monthly seizure count). In this article, we review the evidence for these designs based on re-analysis of prior trials, one published trial that used a time-to-second seizure design, and experience from an ongoing blinded trial. We also discuss remaining concerns regarding time-to-event trials. We conclude that, despite potential limitations, time-to-event trials are a potential promising mechanism to make trials more patient friendly and reduce placebo exposure, which are urgent needs to improve safety and increase recruitment to trials.

Flexible realistic simulation of seizure occurrence recapitulating statistical properties of seizure diaries

OBJECTIVE

A realistic seizure diary simulator is currently unavailable for many research needs, including clinical trial analysis and evaluation of seizure detection and seizure-forecasting tools. In recent years, important statistical features of seizure diaries have been characterized. These include (1) heterogeneity of individual seizure frequencies, (2) the relation between average seizure rate and standard deviation, (3) multiple risk cycles, (4) seizure clusters, and (5) limitations on inter-seizure intervals. The present study unifies these features into a single model.

METHODS

Our approach, Cyclic Heterogeneous Overdispersed Clustered Open-source L-relationship Adjustable Temporally limited E-diary Simulator (CHOCOLATES) is based on a hierarchical model centered on a gamma Poisson generator with several modifiers. This model accounts for the aforementioned statistical properties. The model was validated by simulating 10 000 randomized clinical trials (RCTs) of medication to compare with 23 historical RCTs. Metrics of 50% responder rate (RR50) and median percent change (MPC) were evaluated. We also used CHOCOLATES as input to a seizure-forecasting tool to test the flexibility of the model. We examined the area under the receiver-operating characteristic (ROC) curve (AUC) for test data with and without cycles and clusters.

RESULTS

The model recapitulated typical findings in 23 historical RCTs without the necessity of introducing an additional "placebo effect." The model produced the following RR50 values: placebo: 17 ± 4%; drug 38 ± 5%; and the following MPC values: placebo: 13 ± 6%; drug 40 ± 4%. These values are similar to historical data: for RR50: placebo, 21 ± 10%, drug: 43 ± 13%; and for MPC: placebo: 17 ± 10%, drug: 41 ± 11%. The seizure forecasts achieved an AUC of 0.68 with cycles and clusters, whereas without them the AUC was 0.51.

SIGNIFICANCE

CHOCOLATES represents the most realistic seizure occurrence simulator to date, based on observations from thousands of patients in different contexts. This tool is open source and flexible, and can be used for many applications, including clinical trial simulation and testing of seizure-forecasting tools.

Time to exceed pre-randomization monthly seizure count for perampanel in participants with primary generalized tonic-clonic seizures: A potential clinical end point

OBJECTIVE

To evaluate the exploratory time to exceed pre-randomization seizure count (T-PSC) in the determination of efficacy of adjunctive perampanel in participants with primary generalized tonic-clonic (PGTC) seizures in generalized-onset epilepsy.

METHODS

In this multicenter, double-blind study (ClinicalTrials.gov identifier: NCT01393743), participants ≥12 years of age with treatment-resistant idiopathic generalized epilepsy were randomized to receive placebo or adjunctive perampanel (≤8 mg/day) across a 17-week double-blind treatment phase (4-week titration; 13-week maintenance). We evaluated the pre-planned exploratory end point of the T-PSC using a Kaplan-Meier analysis. We also re-evaluated the correspondence of the primary end points of median percent seizure frequency change (MPC) and 50% responder rate (50RR) calculated at T-PSC and at the end of the trial.

RESULTS

The exploratory end point of median T-PSC on placebo was 43 days and >120 days on perampanel (log-rank p < .001). The primary end points calculated at T-PSC did not differ significantly from the end points at the end of the trial (MPC -31% vs -42% at T-PSC; 50RR 32% vs 51% at T-PSC). After T-PSC was reached, participants had a median (interquartile range) of 5 (3-13) additional seizures on placebo and 5 (2-10) on perampanel.

SIGNIFICANCE

The exploratory end point of T-PSC demonstrated the effectiveness of perampanel despite a shorter duration of monitoring. The seizures that occurred after T-PSC did not influence the conclusions of the trial; therefore, T-PSC may be a viable alternative to traditional trial end points that reduces the risk to participants.

Can machine learning improve randomized clinical trial analysis?

PURPOSE

Recently a realistic simulator of patient seizure diaries was developed that can reproduce effects seen in randomized clinical trials (RCTs). RCTs suffer from high costs and statistical inefficiencies. Using realistic simulation and machine learning this study aimed to identify a more statistically efficient outcome metric.

METHODS

Five candidate deep learning architectures with 54 permutations of hyperparameters were compared to the traditional standard, median percent change (MPC). Each were also tested for type 1 error. All models had similar outcomes, with appropriate low levels of type 1 error.

RESULTS

The simplest model was equivalent to a logistic regression of a histogram of individual percentage changes in seizure rate, requiring 21-22% less patients to discriminate drug from placebo at 90% power. This model was referred to as LPC.

CONCLUSION

Future studies to validate LPC may enable faster, cheaper and more efficient clinical trials.