Can machine learning improve randomized clinical trial analysis?

Minimum clinical utility standards for wearable seizure detectors: A simulation study

OBJECTIVE

Epilepsy management employs self-reported seizure diaries, despite evidence of seizure underreporting. Wearable and implantable seizure detection devices are now becoming more widely available. There are no clear guidelines about what levels of accuracy are sufficient. This study aimed to simulate clinical use cases and identify the necessary level of accuracy for each.

METHODS

Using a realistic seizure simulator (CHOCOLATES), a ground truth was produced, which was then sampled to generate signals from simulated seizure detectors of various capabilities. Five use cases were evaluated: (1) randomized clinical trials (RCTs), (2) medication adjustment in clinic, (3) injury prevention, (4) sudden unexpected death in epilepsy (SUDEP) prevention, and (5) treatment of seizure clusters. We considered sensitivity (0%-100%), false alarm rate (FAR; 0-2/day), and device type (external wearable vs. implant) in each scenario.

RESULTS

The RCT case was efficient for a wide range of wearable parameters, though implantable devices were preferred. Lower accuracy wearables resulted in subtle changes in the distribution of patients enrolled in RCTs, and therefore higher sensitivity and lower FAR values were preferred. In the clinic case, a wide range of sensitivity, FAR, and device type yielded similar results. For injury prevention, SUDEP prevention, and seizure cluster treatment, each scenario required high sensitivity and yet was minimally influenced by FAR.

SIGNIFICANCE

The choice of use case is paramount in determining acceptable accuracy levels for a wearable seizure detection device. We offer simulation results for determining and verifying utility for specific use case and specific wearable parameters.

Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist

PURPOSE OF REVIEW

Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice.

RECENT FINDINGS

ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.

Increasing challenges to trial recruitment and conduct over time

OBJECTIVE

This study was undertaken to evaluate how the challenges in the recruitment and retention of participants in clinical trials for focal onset epilepsy have changed over time.

METHODS

In this systematic analysis of randomized clinical trials of adjunct antiseizure medications for medication-resistant focal onset epilepsy, we evaluated how the numbers of participants, sites, and countries have changed since the first such trial in 1990. We also evaluated the proportion of participants who completed each trial phase and their reasons for early trial exit. We analyzed these trends using mixed effects generalized linear models accounting for the influence of the number of trial sites and trial-specific variability.

RESULTS

The number of participants per site has steadily decreased over decades, with recent trials recruiting fewer than five participants per site (reduction by .16 participants/site/year, p < .0001). Fewer participants also progressed from recruitment to randomization over time (odds ratio = .94/year, p = .014). Concurrently, there has been an increase in the placebo response over time (increase in median percent reduction of .4%/year, p = .02; odds ratio of increase in 50% responder rate of 1.03/year, p = .02), which was not directly associated with the number of sites per trial (p > .20).

SIGNIFICANCE

This historical analysis highlights the increasing challenges with participant recruitment and retention, as well as increasing placebo response. It serves as a call to action to change clinical trial design to address these challenges.

Machine Learning in Clinical Trials: A Primer with Applications to Neurology

We reviewed foundational concepts in artificial intelligence (AI) and machine learning (ML) and discussed ways in which these methodologies may be employed to enhance progress in clinical trials and research, with particular attention to applications in the design, conduct, and interpretation of clinical trials for neurologic diseases. We discussed ways in which ML may help to accelerate the pace of subject recruitment, provide realistic simulation of medical interventions, and enhance remote trial administration via novel digital biomarkers and therapeutics. Lastly, we provide a brief overview of the technical, administrative, and regulatory challenges that must be addressed as ML achieves greater integration into clinical trial workflows.

Prescriptive Method for Optimizing Cost of Data Collection and Annotation in Machine Learning of Clinical Ultrasound

Machine learning in medical ultrasound faces a major challenge: the prohibitive costs of producing and annotating clinical data. Optimizing the data collection and annotation will improve model training efficiency, reducing project cost and times. This paper prescribes a 2-phase method for cost optimization based on iterative accuracy/sample size predictions, and active learning for annotation optimization.

METHODS

Using public breast, fetal, and lung ultrasound datasets we can: Optimize data collection by statistically predicting accuracy for a desired dataset size; and optimize labeling efficiency using Active Learning, where predictions with lowest certainty were labelled manually using feedback. A practical case study on BUSI data was used to demonstrate the method prescribed in this work.

RESULTS

With small data subsets, ~10%, dataset size vs. final accuracy relations can be predicted with diminishing results after 50% usage. Manual annotation was reduced by ~10% using active learning to focus the annotation.

CONCLUSION

This led to cost reductions of 50%-66%, depending on requirements and initial cost model, on BUSI dataset with a negligible accuracy drop of 3.75% from theoretical maximums.Clinical Relevance- This work provides methodology to optimize dataset size and manual data labelling, this allows generation of cost-effective datasets, of interest to all, but particularly for financially limited trials and feasibility studies, Reducing the time burden on annotating clinicians.

Flexible realistic simulation of seizure occurrence recapitulating statistical properties of seizure diaries

OBJECTIVE

A realistic seizure diary simulator is currently unavailable for many research needs, including clinical trial analysis and evaluation of seizure detection and seizure-forecasting tools. In recent years, important statistical features of seizure diaries have been characterized. These include (1) heterogeneity of individual seizure frequencies, (2) the relation between average seizure rate and standard deviation, (3) multiple risk cycles, (4) seizure clusters, and (5) limitations on inter-seizure intervals. The present study unifies these features into a single model.

METHODS

Our approach, Cyclic Heterogeneous Overdispersed Clustered Open-source L-relationship Adjustable Temporally limited E-diary Simulator (CHOCOLATES) is based on a hierarchical model centered on a gamma Poisson generator with several modifiers. This model accounts for the aforementioned statistical properties. The model was validated by simulating 10 000 randomized clinical trials (RCTs) of medication to compare with 23 historical RCTs. Metrics of 50% responder rate (RR50) and median percent change (MPC) were evaluated. We also used CHOCOLATES as input to a seizure-forecasting tool to test the flexibility of the model. We examined the area under the receiver-operating characteristic (ROC) curve (AUC) for test data with and without cycles and clusters.

RESULTS

The model recapitulated typical findings in 23 historical RCTs without the necessity of introducing an additional "placebo effect." The model produced the following RR50 values: placebo: 17 ± 4%; drug 38 ± 5%; and the following MPC values: placebo: 13 ± 6%; drug 40 ± 4%. These values are similar to historical data: for RR50: placebo, 21 ± 10%, drug: 43 ± 13%; and for MPC: placebo: 17 ± 10%, drug: 41 ± 11%. The seizure forecasts achieved an AUC of 0.68 with cycles and clusters, whereas without them the AUC was 0.51.

SIGNIFICANCE

CHOCOLATES represents the most realistic seizure occurrence simulator to date, based on observations from thousands of patients in different contexts. This tool is open source and flexible, and can be used for many applications, including clinical trial simulation and testing of seizure-forecasting tools.

Time to exceed pre-randomization monthly seizure count for perampanel in participants with primary generalized tonic-clonic seizures: A potential clinical end point

OBJECTIVE

To evaluate the exploratory time to exceed pre-randomization seizure count (T-PSC) in the determination of efficacy of adjunctive perampanel in participants with primary generalized tonic-clonic (PGTC) seizures in generalized-onset epilepsy.

METHODS

In this multicenter, double-blind study (ClinicalTrials.gov identifier: NCT01393743), participants ≥12 years of age with treatment-resistant idiopathic generalized epilepsy were randomized to receive placebo or adjunctive perampanel (≤8 mg/day) across a 17-week double-blind treatment phase (4-week titration; 13-week maintenance). We evaluated the pre-planned exploratory end point of the T-PSC using a Kaplan-Meier analysis. We also re-evaluated the correspondence of the primary end points of median percent seizure frequency change (MPC) and 50% responder rate (50RR) calculated at T-PSC and at the end of the trial.

RESULTS

The exploratory end point of median T-PSC on placebo was 43 days and >120 days on perampanel (log-rank p < .001). The primary end points calculated at T-PSC did not differ significantly from the end points at the end of the trial (MPC -31% vs -42% at T-PSC; 50RR 32% vs 51% at T-PSC). After T-PSC was reached, participants had a median (interquartile range) of 5 (3-13) additional seizures on placebo and 5 (2-10) on perampanel.

SIGNIFICANCE

The exploratory end point of T-PSC demonstrated the effectiveness of perampanel despite a shorter duration of monitoring. The seizures that occurred after T-PSC did not influence the conclusions of the trial; therefore, T-PSC may be a viable alternative to traditional trial end points that reduces the risk to participants.