Flexible realistic simulation of seizure occurrence recapitulating statistical properties of seizure diaries

Demonstration of Group-Level and Individual-Level Efficacy Using Time-to-Event Designs for Clinical Trials of Antiseizure Medications

BACKGROUND AND OBJECTIVES

Participants with treatment-resistant epilepsy who are randomized to add-on placebo and remain in a trial for the typical 3 to 5-month maintenance period may be at increased risk of adverse outcomes. A novel trial design has been suggested, time to prerandomization monthly seizure count (T-PSC), which would limit participants' time on ineffective therapy. We reanalyzed 11 completed trials to determine whether the primary efficacy conclusions at T-PSC matched each of the original, longer trials.

METHODS

A total of 11 double-blind, placebo-controlled trials of levetiracetam, brivaracetam, lacosamide, topiramate, and lamotrigine for either focal-onset or generalized-onset epilepsy were selected. We evaluated the group-level and individual-level efficacy of treatments including the median percent reduction (MPR) in seizure frequency and 50% responder rate (50RR) at T-PSC, time to second seizure, and time to first seizure compared with the full-length trial.

RESULTS

The primary efficacy conclusions of 10 of the 11 trials would have been the same with a T-PSC design compared with the traditional design (the exception of lamotrigine had a very high initial placebo response). As a proportion of the full-length effect size, 90% of the MPR and 85% of the 50RR were seen at T-PSC (95% CI 73%-113% and 65%-110%, respectively). Using the T-PSC design, the time on blinded treatment was at least 312 participant-years shorter (40% of total duration) and 142,000 seizures occurred during this time (60% of total seizures). By contrast, the time to first or second seizure designs reproduced group-level effect size, but the primary efficacy conclusions of each trial and individual-level efficacy correspondence were fair to poor.

DISCUSSION

These results support the use of this trial design for new epilepsy medication trials because this reanalysis of 11 randomized controlled trials demonstrated that observation until T-PSC was sufficient to demonstrate efficacy while potentially improving participant safety by reducing the time of exposure to placebo and inadequate treatment. Despite analysis of 11 trials including 3,619 participants, we did not observe a significant reduction in the group-level effect size, which is directly related to statistical power. The next step is to evaluate whether T-PSC is sufficient to evaluate safety as measured by adverse events.

How accurate do self-reported seizures need to be for effective medication management in epilepsy?

Studies suggest that self-reported seizure diaries suffer from 50% under-reporting on average. It is unknown to what extent this impacts medication management. This study used simulation to predict the seizure outcomes of a large heterogeneous clinic population treated with a standardized algorithm based on self-reported seizures. Using CHOCOLATES, a state-of-the-art realistic seizure diary simulator, 100 000 patients were simulated over 10 years. A standard algorithm for medication management was employed at 3 month intervals for all patients. The impact on true seizure rates, expected seizure rates, and time-to-steady-dose were computed for self-reporting sensitivities 0%-100%. Time-to-steady-dose and medication use mostly did not depend on sensitivity. True seizure rate decreased minimally with increasing self-reporting in a non-linear fashion, with the largest decreases at low sensitivity rates (0%-10%). This study suggests that an extremely wide range of sensitivity will have similar seizure outcomes when patients are clinically treated using an algorithm similar to the one presented. Conversely, patients with sensitivity ≤10% would be expected to benefit (via lower seizure rates) from objective devices that provide even small improvements in seizure sensitivity.

Prospective validation of a seizure diary forecasting falls short

OBJECTIVE

Recently, a deep learning artificial intelligence (AI) model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm.

METHODS

We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median = 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSSs) compared random forecasts and simple moving average forecasts to the AI.

RESULTS

The AI had an area under the receiver operating characteristic curve of .82. At the group level, the AI outperformed random forecasting (BSS = .53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (nonverified) diaries (with presumed underreporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor-quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts.

SIGNIFICANCE

The previously developed AI forecasting tool did not outperform a very simple moving average forecasting in this prospective cohort, suggesting that the AI model should be replaced.

Minimum clinical utility standards for wearable seizure detectors: A simulation study

OBJECTIVE

Epilepsy management employs self-reported seizure diaries, despite evidence of seizure underreporting. Wearable and implantable seizure detection devices are now becoming more widely available. There are no clear guidelines about what levels of accuracy are sufficient. This study aimed to simulate clinical use cases and identify the necessary level of accuracy for each.

METHODS

Using a realistic seizure simulator (CHOCOLATES), a ground truth was produced, which was then sampled to generate signals from simulated seizure detectors of various capabilities. Five use cases were evaluated: (1) randomized clinical trials (RCTs), (2) medication adjustment in clinic, (3) injury prevention, (4) sudden unexpected death in epilepsy (SUDEP) prevention, and (5) treatment of seizure clusters. We considered sensitivity (0%-100%), false alarm rate (FAR; 0-2/day), and device type (external wearable vs. implant) in each scenario.

RESULTS

The RCT case was efficient for a wide range of wearable parameters, though implantable devices were preferred. Lower accuracy wearables resulted in subtle changes in the distribution of patients enrolled in RCTs, and therefore higher sensitivity and lower FAR values were preferred. In the clinic case, a wide range of sensitivity, FAR, and device type yielded similar results. For injury prevention, SUDEP prevention, and seizure cluster treatment, each scenario required high sensitivity and yet was minimally influenced by FAR.

SIGNIFICANCE

The choice of use case is paramount in determining acceptable accuracy levels for a wearable seizure detection device. We offer simulation results for determining and verifying utility for specific use case and specific wearable parameters.

Inductive reasoning with large language models: a simulated randomized controlled trial for epilepsy

Importance

The analysis of electronic medical records at scale to learn from clinical experience is currently very challenging. The integration of artificial intelligence (AI), specifically foundational large language models (LLMs), into an analysis pipeline may overcome some of the current limitations of modest input sizes, inaccuracies, biases, and incomplete knowledge bases.

Objective

To explore the effectiveness of using an LLM for generating realistic clinical data and other LLMs for summarizing and synthesizing information in a model system, simulating a randomized clinical trial (RCT) in epilepsy to demonstrate the potential of inductive reasoning via medical chart review.

Design

An LLM-generated simulated RCT based on a RCT for treatment with an antiseizure medication, cenobamate, including a placebo arm and a full-strength drug arm, evaluated by an LLM-based pipeline versus a human reader.

Setting

Simulation based on realistic seizure diaries, treatment effects, reported symptoms and clinical notes generated by LLMs with multiple different neurologist writing styles.

Participants

Simulated cohort of 240 patients, divided 1:1 into placebo and drug arms.

Intervention

Utilization of LLMs for the generation of clinical notes and for the synthesis of data from these notes, aiming to evaluate the efficacy and safety of cenobamate in seizure control either with a human evaluator or AI-pipeline.

Measures

The AI and human analysis focused on identifying the number of seizures, symptom reports, and treatment efficacy, with statistical analysis comparing the 50%-responder rate and median percentage change between the placebo and drug arms, as well as side effect rates in each arm.

Results

AI closely mirrored human analysis, demonstrating the drug's efficacy with marginal differences (<3%) in identifying both drug efficacy and reported symptoms.

Conclusions and Relevance

This study showcases the potential of LLMs accurately simulate and analyze clinical trials. Significantly, it highlights the ability of LLMs to reconstruct essential trial elements, identify treatment effects, and recognize reported symptoms, within a realistic clinical framework. The findings underscore the relevance of LLMs in future clinical research, offering a scalable, efficient alternative to traditional data mining methods without the need for specialized medical language training.

Prospective validation of a seizure diary forecasting falls short

OBJECTIVE

Recently, a deep learning AI model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm.

METHODS

We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSS) compared random forecasts and simple moving average forecasts to the AI.

RESULTS

The AI had an AUC of 0.82. At the group level, the AI outperformed random forecasting (BSS=0.53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (non-verified) diaries (with presumed under-reporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts.

SIGNIFICANCE

The previously developed AI forecasting tool did not outperform a very simple moving average forecasting this prospective cohort, suggesting that the AI model should be replaced.

Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist

PURPOSE OF REVIEW

Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice.

RECENT FINDINGS

ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.

Quantifying and controlling the impact of regression to the mean on randomized controlled trials in epilepsy

OBJECTIVE

Randomized controlled trials (RCTs) in epilepsy for drug treatments are plagued by high costs. One potential remedy is to reduce placebo response via better control over regression to the mean (RTM). Here, RTM represents an initial observed seizure rate higher than the long-term average, which gradually settles closer to the average, resulting in apparent response to treatment. This study used simulation to clarify the relationship between eligibility criteria and RTM.

METHODS

Using a statistically realistic seizure diary simulator, the impact of RTM on placebo response and trial efficacy was explored by varying eligibility criteria for a traditional treatment phase II/III RCT for drug-resistant epilepsy.

RESULTS

When the baseline period was included in the eligibility criteria, increasingly larger fractions of RTM were observed (25%-47% vs. 23%-25%). Higher fractions of RTM corresponded with higher expected placebo responses (50% responder rate [RR50]: 2%-9% vs. 0%-8%) and lower statistical efficacy (RR50: 47%-67% vs. 47%-81%). The exclusion of baseline from eligibility criteria was shown to decrease the number of patients needed by roughly 30%.

SIGNIFICANCE

The manipulation of eligibility criteria for RCTs has a predictable and important impact on RTM, and therefore on placebo response; the difference between drug and placebo was more easily detected. This in turn impacts trial efficacy and therefore cost. This study found dramatic improvements in efficacy and cost when baseline was not included in eligibility.

Increasing challenges to trial recruitment and conduct over time

OBJECTIVE

This study was undertaken to evaluate how the challenges in the recruitment and retention of participants in clinical trials for focal onset epilepsy have changed over time.

METHODS

In this systematic analysis of randomized clinical trials of adjunct antiseizure medications for medication-resistant focal onset epilepsy, we evaluated how the numbers of participants, sites, and countries have changed since the first such trial in 1990. We also evaluated the proportion of participants who completed each trial phase and their reasons for early trial exit. We analyzed these trends using mixed effects generalized linear models accounting for the influence of the number of trial sites and trial-specific variability.

RESULTS

The number of participants per site has steadily decreased over decades, with recent trials recruiting fewer than five participants per site (reduction by .16 participants/site/year, p < .0001). Fewer participants also progressed from recruitment to randomization over time (odds ratio = .94/year, p = .014). Concurrently, there has been an increase in the placebo response over time (increase in median percent reduction of .4%/year, p = .02; odds ratio of increase in 50% responder rate of 1.03/year, p = .02), which was not directly associated with the number of sites per trial (p > .20).

SIGNIFICANCE

This historical analysis highlights the increasing challenges with participant recruitment and retention, as well as increasing placebo response. It serves as a call to action to change clinical trial design to address these challenges.

Machine Learning in Clinical Trials: A Primer with Applications to Neurology

We reviewed foundational concepts in artificial intelligence (AI) and machine learning (ML) and discussed ways in which these methodologies may be employed to enhance progress in clinical trials and research, with particular attention to applications in the design, conduct, and interpretation of clinical trials for neurologic diseases. We discussed ways in which ML may help to accelerate the pace of subject recruitment, provide realistic simulation of medical interventions, and enhance remote trial administration via novel digital biomarkers and therapeutics. Lastly, we provide a brief overview of the technical, administrative, and regulatory challenges that must be addressed as ML achieves greater integration into clinical trial workflows.