1
|
Kerr WT, Kok N, Reddy AS, McFarlane KN, Stern JM, Pennell PB, Stacey W, French J. Demonstration of Group-Level and Individual-Level Efficacy Using Time-to-Event Designs for Clinical Trials of Antiseizure Medications. Neurology 2024; 103:e209713. [PMID: 39052963 PMCID: PMC11271390 DOI: 10.1212/wnl.0000000000209713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 05/24/2024] [Indexed: 07/27/2024] Open
Abstract
BACKGROUND AND OBJECTIVES Participants with treatment-resistant epilepsy who are randomized to add-on placebo and remain in a trial for the typical 3 to 5-month maintenance period may be at increased risk of adverse outcomes. A novel trial design has been suggested, time to prerandomization monthly seizure count (T-PSC), which would limit participants' time on ineffective therapy. We reanalyzed 11 completed trials to determine whether the primary efficacy conclusions at T-PSC matched each of the original, longer trials. METHODS A total of 11 double-blind, placebo-controlled trials of levetiracetam, brivaracetam, lacosamide, topiramate, and lamotrigine for either focal-onset or generalized-onset epilepsy were selected. We evaluated the group-level and individual-level efficacy of treatments including the median percent reduction (MPR) in seizure frequency and 50% responder rate (50RR) at T-PSC, time to second seizure, and time to first seizure compared with the full-length trial. RESULTS The primary efficacy conclusions of 10 of the 11 trials would have been the same with a T-PSC design compared with the traditional design (the exception of lamotrigine had a very high initial placebo response). As a proportion of the full-length effect size, 90% of the MPR and 85% of the 50RR were seen at T-PSC (95% CI 73%-113% and 65%-110%, respectively). Using the T-PSC design, the time on blinded treatment was at least 312 participant-years shorter (40% of total duration) and 142,000 seizures occurred during this time (60% of total seizures). By contrast, the time to first or second seizure designs reproduced group-level effect size, but the primary efficacy conclusions of each trial and individual-level efficacy correspondence were fair to poor. DISCUSSION These results support the use of this trial design for new epilepsy medication trials because this reanalysis of 11 randomized controlled trials demonstrated that observation until T-PSC was sufficient to demonstrate efficacy while potentially improving participant safety by reducing the time of exposure to placebo and inadequate treatment. Despite analysis of 11 trials including 3,619 participants, we did not observe a significant reduction in the group-level effect size, which is directly related to statistical power. The next step is to evaluate whether T-PSC is sufficient to evaluate safety as measured by adverse events.
Collapse
Affiliation(s)
- Wesley T Kerr
- From the Department of Neurology (W.T.K., N.K., A.S.R., K.N.M., P.B.P.), and Department of Biomedical Informatics (W.T.K.), University of Pittsburgh, PA; Department of Neurology (W.T.K., N.K., A.S.R., W.S.), University of Michigan, Ann Arbor; Department of Neurology (W.T.K., J.M.S.), University of California, Los Angeles; Department of Biomedical Engineering (W.S.), University of Michigan, Ann Arbor; and Comprehensive Epilepsy Center (J.F.), New York University Grossman School of Medicine, New York
| | - Neo Kok
- From the Department of Neurology (W.T.K., N.K., A.S.R., K.N.M., P.B.P.), and Department of Biomedical Informatics (W.T.K.), University of Pittsburgh, PA; Department of Neurology (W.T.K., N.K., A.S.R., W.S.), University of Michigan, Ann Arbor; Department of Neurology (W.T.K., J.M.S.), University of California, Los Angeles; Department of Biomedical Engineering (W.S.), University of Michigan, Ann Arbor; and Comprehensive Epilepsy Center (J.F.), New York University Grossman School of Medicine, New York
| | - Advith S Reddy
- From the Department of Neurology (W.T.K., N.K., A.S.R., K.N.M., P.B.P.), and Department of Biomedical Informatics (W.T.K.), University of Pittsburgh, PA; Department of Neurology (W.T.K., N.K., A.S.R., W.S.), University of Michigan, Ann Arbor; Department of Neurology (W.T.K., J.M.S.), University of California, Los Angeles; Department of Biomedical Engineering (W.S.), University of Michigan, Ann Arbor; and Comprehensive Epilepsy Center (J.F.), New York University Grossman School of Medicine, New York
| | - Katherine N McFarlane
- From the Department of Neurology (W.T.K., N.K., A.S.R., K.N.M., P.B.P.), and Department of Biomedical Informatics (W.T.K.), University of Pittsburgh, PA; Department of Neurology (W.T.K., N.K., A.S.R., W.S.), University of Michigan, Ann Arbor; Department of Neurology (W.T.K., J.M.S.), University of California, Los Angeles; Department of Biomedical Engineering (W.S.), University of Michigan, Ann Arbor; and Comprehensive Epilepsy Center (J.F.), New York University Grossman School of Medicine, New York
| | - John M Stern
- From the Department of Neurology (W.T.K., N.K., A.S.R., K.N.M., P.B.P.), and Department of Biomedical Informatics (W.T.K.), University of Pittsburgh, PA; Department of Neurology (W.T.K., N.K., A.S.R., W.S.), University of Michigan, Ann Arbor; Department of Neurology (W.T.K., J.M.S.), University of California, Los Angeles; Department of Biomedical Engineering (W.S.), University of Michigan, Ann Arbor; and Comprehensive Epilepsy Center (J.F.), New York University Grossman School of Medicine, New York
| | - Page B Pennell
- From the Department of Neurology (W.T.K., N.K., A.S.R., K.N.M., P.B.P.), and Department of Biomedical Informatics (W.T.K.), University of Pittsburgh, PA; Department of Neurology (W.T.K., N.K., A.S.R., W.S.), University of Michigan, Ann Arbor; Department of Neurology (W.T.K., J.M.S.), University of California, Los Angeles; Department of Biomedical Engineering (W.S.), University of Michigan, Ann Arbor; and Comprehensive Epilepsy Center (J.F.), New York University Grossman School of Medicine, New York
| | - William Stacey
- From the Department of Neurology (W.T.K., N.K., A.S.R., K.N.M., P.B.P.), and Department of Biomedical Informatics (W.T.K.), University of Pittsburgh, PA; Department of Neurology (W.T.K., N.K., A.S.R., W.S.), University of Michigan, Ann Arbor; Department of Neurology (W.T.K., J.M.S.), University of California, Los Angeles; Department of Biomedical Engineering (W.S.), University of Michigan, Ann Arbor; and Comprehensive Epilepsy Center (J.F.), New York University Grossman School of Medicine, New York
| | - Jacqueline French
- From the Department of Neurology (W.T.K., N.K., A.S.R., K.N.M., P.B.P.), and Department of Biomedical Informatics (W.T.K.), University of Pittsburgh, PA; Department of Neurology (W.T.K., N.K., A.S.R., W.S.), University of Michigan, Ann Arbor; Department of Neurology (W.T.K., J.M.S.), University of California, Los Angeles; Department of Biomedical Engineering (W.S.), University of Michigan, Ann Arbor; and Comprehensive Epilepsy Center (J.F.), New York University Grossman School of Medicine, New York
| |
Collapse
|
2
|
Goldenholz D, Brinkmann BH, Westover MB. How accurate do self-reported seizures need to be for effective medication management in epilepsy? Epilepsia 2024; 65:e104-e112. [PMID: 38776216 PMCID: PMC11251847 DOI: 10.1111/epi.18019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2024] [Revised: 05/05/2024] [Accepted: 05/06/2024] [Indexed: 05/24/2024]
Abstract
Studies suggest that self-reported seizure diaries suffer from 50% under-reporting on average. It is unknown to what extent this impacts medication management. This study used simulation to predict the seizure outcomes of a large heterogeneous clinic population treated with a standardized algorithm based on self-reported seizures. Using CHOCOLATES, a state-of-the-art realistic seizure diary simulator, 100 000 patients were simulated over 10 years. A standard algorithm for medication management was employed at 3 month intervals for all patients. The impact on true seizure rates, expected seizure rates, and time-to-steady-dose were computed for self-reporting sensitivities 0%-100%. Time-to-steady-dose and medication use mostly did not depend on sensitivity. True seizure rate decreased minimally with increasing self-reporting in a non-linear fashion, with the largest decreases at low sensitivity rates (0%-10%). This study suggests that an extremely wide range of sensitivity will have similar seizure outcomes when patients are clinically treated using an algorithm similar to the one presented. Conversely, patients with sensitivity ≤10% would be expected to benefit (via lower seizure rates) from objective devices that provide even small improvements in seizure sensitivity.
Collapse
Affiliation(s)
- Daniel Goldenholz
- Department of Neurology, Harvard Medical School, Boston USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| | | | - M. Brandon Westover
- Department of Neurology, Harvard Medical School, Boston USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
- Department of Neurology, Massachusetts General Hospital, Boston USA
- McCace Center, Boston USA
| |
Collapse
|
3
|
Goldenholz DM, Eccleston C, Moss R, Westover MB. Prospective validation of a seizure diary forecasting falls short. Epilepsia 2024; 65:1730-1736. [PMID: 38606580 PMCID: PMC11166505 DOI: 10.1111/epi.17984] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2023] [Revised: 03/13/2024] [Accepted: 04/02/2024] [Indexed: 04/13/2024]
Abstract
OBJECTIVE Recently, a deep learning artificial intelligence (AI) model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm. METHODS We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median = 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSSs) compared random forecasts and simple moving average forecasts to the AI. RESULTS The AI had an area under the receiver operating characteristic curve of .82. At the group level, the AI outperformed random forecasting (BSS = .53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (nonverified) diaries (with presumed underreporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor-quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts. SIGNIFICANCE The previously developed AI forecasting tool did not outperform a very simple moving average forecasting in this prospective cohort, suggesting that the AI model should be replaced.
Collapse
Affiliation(s)
- Daniel M. Goldenholz
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA
- Department of Neurology, Harvard Medical School, Boston, Massachusetts, USA
| | - Celena Eccleston
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA
- Department of Neurology, Harvard Medical School, Boston, Massachusetts, USA
| | | | - M. Brandon Westover
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA
- Department of Neurology, Harvard Medical School, Boston, Massachusetts, USA
- Department of Neurology, Massachusetts General Hospital, Boston, Massachusetts, USA
- McCance Center for Brain Health, Boston, Massachusetts, USA
| |
Collapse
|
4
|
Goldenholz DM, Karoly PJ, Viana PF, Nurse E, Loddenkemper T, Schulze-Bonhage A, Vieluf S, Bruno E, Nasseri M, Richardson MP, Brinkmann BH, Westover MB. Minimum clinical utility standards for wearable seizure detectors: A simulation study. Epilepsia 2024; 65:1017-1028. [PMID: 38366862 PMCID: PMC11018505 DOI: 10.1111/epi.17917] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Revised: 01/11/2024] [Accepted: 02/01/2024] [Indexed: 02/18/2024]
Abstract
OBJECTIVE Epilepsy management employs self-reported seizure diaries, despite evidence of seizure underreporting. Wearable and implantable seizure detection devices are now becoming more widely available. There are no clear guidelines about what levels of accuracy are sufficient. This study aimed to simulate clinical use cases and identify the necessary level of accuracy for each. METHODS Using a realistic seizure simulator (CHOCOLATES), a ground truth was produced, which was then sampled to generate signals from simulated seizure detectors of various capabilities. Five use cases were evaluated: (1) randomized clinical trials (RCTs), (2) medication adjustment in clinic, (3) injury prevention, (4) sudden unexpected death in epilepsy (SUDEP) prevention, and (5) treatment of seizure clusters. We considered sensitivity (0%-100%), false alarm rate (FAR; 0-2/day), and device type (external wearable vs. implant) in each scenario. RESULTS The RCT case was efficient for a wide range of wearable parameters, though implantable devices were preferred. Lower accuracy wearables resulted in subtle changes in the distribution of patients enrolled in RCTs, and therefore higher sensitivity and lower FAR values were preferred. In the clinic case, a wide range of sensitivity, FAR, and device type yielded similar results. For injury prevention, SUDEP prevention, and seizure cluster treatment, each scenario required high sensitivity and yet was minimally influenced by FAR. SIGNIFICANCE The choice of use case is paramount in determining acceptable accuracy levels for a wearable seizure detection device. We offer simulation results for determining and verifying utility for specific use case and specific wearable parameters.
Collapse
Affiliation(s)
- Daniel M Goldenholz
- Department of Neurology, Harvard Medical School, Boston, Massachusetts, USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA
| | - Philippa J Karoly
- Department of Neurology, University of Melbourne, Melbourne, Victoria, Australia
| | - Pedro F Viana
- School of Neuroscience, Institute of Psychiatry, Psychology, and Neuroscience, King's College London, London, UK
- Faculty of Medicine, University of Lisbon, Lisbon, Portugal
| | - Ewan Nurse
- Seer Medical, Melbourne, Victoria, Australia
| | - Tobias Loddenkemper
- Department of Neurology, Harvard Medical School, Boston, Massachusetts, USA
- Department of Neurology, Boston Children's Hospital, Boston, Massachusetts, USA
| | - Andreas Schulze-Bonhage
- Epilepsy Center, University Medical Center Freiburg-University of Freiburg, Freiburg, Germany
| | - Solveig Vieluf
- Department of Neurology, Harvard Medical School, Boston, Massachusetts, USA
- Department of Neurology, Boston Children's Hospital, Boston, Massachusetts, USA
| | - Elisa Bruno
- School of Neuroscience, Institute of Psychiatry, Psychology, and Neuroscience, King's College London, London, UK
| | - Mona Nasseri
- Department of Neurology, Mayo Clinic, Rochester, Minnesota, USA
| | - Mark P Richardson
- School of Neuroscience, Institute of Psychiatry, Psychology, and Neuroscience, King's College London, London, UK
| | | | - M Brandon Westover
- Department of Neurology, Harvard Medical School, Boston, Massachusetts, USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA
- Department of Neurology, Massachusetts General Hospital, Boston, Massachusetts, USA
- McCace Center, Boston, Massachusetts, USA
| |
Collapse
|
5
|
Goldenholz DM, Goldenholz SR, Habib S, Westover MB. Inductive reasoning with large language models: a simulated randomized controlled trial for epilepsy. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.03.18.24304493. [PMID: 38562831 PMCID: PMC10984041 DOI: 10.1101/2024.03.18.24304493] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Importance The analysis of electronic medical records at scale to learn from clinical experience is currently very challenging. The integration of artificial intelligence (AI), specifically foundational large language models (LLMs), into an analysis pipeline may overcome some of the current limitations of modest input sizes, inaccuracies, biases, and incomplete knowledge bases. Objective To explore the effectiveness of using an LLM for generating realistic clinical data and other LLMs for summarizing and synthesizing information in a model system, simulating a randomized clinical trial (RCT) in epilepsy to demonstrate the potential of inductive reasoning via medical chart review. Design An LLM-generated simulated RCT based on a RCT for treatment with an antiseizure medication, cenobamate, including a placebo arm and a full-strength drug arm, evaluated by an LLM-based pipeline versus a human reader. Setting Simulation based on realistic seizure diaries, treatment effects, reported symptoms and clinical notes generated by LLMs with multiple different neurologist writing styles. Participants Simulated cohort of 240 patients, divided 1:1 into placebo and drug arms. Intervention Utilization of LLMs for the generation of clinical notes and for the synthesis of data from these notes, aiming to evaluate the efficacy and safety of cenobamate in seizure control either with a human evaluator or AI-pipeline. Measures The AI and human analysis focused on identifying the number of seizures, symptom reports, and treatment efficacy, with statistical analysis comparing the 50%-responder rate and median percentage change between the placebo and drug arms, as well as side effect rates in each arm. Results AI closely mirrored human analysis, demonstrating the drug's efficacy with marginal differences (<3%) in identifying both drug efficacy and reported symptoms. Conclusions and Relevance This study showcases the potential of LLMs accurately simulate and analyze clinical trials. Significantly, it highlights the ability of LLMs to reconstruct essential trial elements, identify treatment effects, and recognize reported symptoms, within a realistic clinical framework. The findings underscore the relevance of LLMs in future clinical research, offering a scalable, efficient alternative to traditional data mining methods without the need for specialized medical language training.
Collapse
Affiliation(s)
- Daniel M Goldenholz
- Department of Neurology, Harvard Medical School, Boston USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| | - Shira R Goldenholz
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| | - Sara Habib
- Department of Neurology, Harvard Medical School, Boston USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| | - M Brandon Westover
- Department of Neurology, Harvard Medical School, Boston USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston USA
| |
Collapse
|
6
|
Goldenholz DM, Eccleston C, Moss R, Westover MB. Prospective validation of a seizure diary forecasting falls short. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.01.11.24301175. [PMID: 38260666 PMCID: PMC10802655 DOI: 10.1101/2024.01.11.24301175] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/24/2024]
Abstract
OBJECTIVE Recently, a deep learning AI model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm. METHODS We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSS) compared random forecasts and simple moving average forecasts to the AI. RESULTS The AI had an AUC of 0.82. At the group level, the AI outperformed random forecasting (BSS=0.53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (non-verified) diaries (with presumed under-reporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts. SIGNIFICANCE The previously developed AI forecasting tool did not outperform a very simple moving average forecasting this prospective cohort, suggesting that the AI model should be replaced.
Collapse
Affiliation(s)
- Daniel M Goldenholz
- Dept. of Neurology, Beth Israel Deaconess Medical Center, Boston 02215 MA
- Dept. of Neurology, Harvard Medical School, Boston 02215 MA
| | - Celena Eccleston
- Dept. of Neurology, Beth Israel Deaconess Medical Center, Boston 02215 MA
- Dept. of Neurology, Harvard Medical School, Boston 02215 MA
| | | | - M Brandon Westover
- Dept. of Neurology, Beth Israel Deaconess Medical Center, Boston 02215 MA
- Dept. of Neurology, Harvard Medical School, Boston 02215 MA
- Dept. of Neurology, Massachusetts General Hospital, Boston 02114 MA
- McCance Center for Brain Health, Boston, 02114 MA
| |
Collapse
|
7
|
Kerr WT, McFarlane KN. Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist. Curr Neurol Neurosci Rep 2023; 23:869-879. [PMID: 38060133 DOI: 10.1007/s11910-023-01318-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/24/2023] [Indexed: 12/08/2023]
Abstract
PURPOSE OF REVIEW Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice. RECENT FINDINGS ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.
Collapse
Affiliation(s)
- Wesley T Kerr
- Department of Neurology, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA.
- Department of Biomedical Informatics, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA.
- Department of Neurology, Michigan Medicine, University of Michigan, Ann Arbor, MI, USA.
| | - Katherine N McFarlane
- Department of Neurology, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA
| |
Collapse
|
8
|
Goldenholz DM, Goldenholz EB, Kaptchuk TJ. Quantifying and controlling the impact of regression to the mean on randomized controlled trials in epilepsy. Epilepsia 2023; 64:2635-2643. [PMID: 37505116 PMCID: PMC10592227 DOI: 10.1111/epi.17730] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2023] [Revised: 07/24/2023] [Accepted: 07/25/2023] [Indexed: 07/29/2023]
Abstract
OBJECTIVE Randomized controlled trials (RCTs) in epilepsy for drug treatments are plagued by high costs. One potential remedy is to reduce placebo response via better control over regression to the mean (RTM). Here, RTM represents an initial observed seizure rate higher than the long-term average, which gradually settles closer to the average, resulting in apparent response to treatment. This study used simulation to clarify the relationship between eligibility criteria and RTM. METHODS Using a statistically realistic seizure diary simulator, the impact of RTM on placebo response and trial efficacy was explored by varying eligibility criteria for a traditional treatment phase II/III RCT for drug-resistant epilepsy. RESULTS When the baseline period was included in the eligibility criteria, increasingly larger fractions of RTM were observed (25%-47% vs. 23%-25%). Higher fractions of RTM corresponded with higher expected placebo responses (50% responder rate [RR50]: 2%-9% vs. 0%-8%) and lower statistical efficacy (RR50: 47%-67% vs. 47%-81%). The exclusion of baseline from eligibility criteria was shown to decrease the number of patients needed by roughly 30%. SIGNIFICANCE The manipulation of eligibility criteria for RCTs has a predictable and important impact on RTM, and therefore on placebo response; the difference between drug and placebo was more easily detected. This in turn impacts trial efficacy and therefore cost. This study found dramatic improvements in efficacy and cost when baseline was not included in eligibility.
Collapse
Affiliation(s)
| | | | - Ted J Kaptchuk
- Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA
| |
Collapse
|
9
|
Kerr WT, Reddy AS, Seo SH, Kok N, Stacey WC, Stern JM, Pennell PB, French JA. Increasing challenges to trial recruitment and conduct over time. Epilepsia 2023; 64:2625-2634. [PMID: 37440282 PMCID: PMC10592378 DOI: 10.1111/epi.17716] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 07/10/2023] [Accepted: 07/11/2023] [Indexed: 07/14/2023]
Abstract
OBJECTIVE This study was undertaken to evaluate how the challenges in the recruitment and retention of participants in clinical trials for focal onset epilepsy have changed over time. METHODS In this systematic analysis of randomized clinical trials of adjunct antiseizure medications for medication-resistant focal onset epilepsy, we evaluated how the numbers of participants, sites, and countries have changed since the first such trial in 1990. We also evaluated the proportion of participants who completed each trial phase and their reasons for early trial exit. We analyzed these trends using mixed effects generalized linear models accounting for the influence of the number of trial sites and trial-specific variability. RESULTS The number of participants per site has steadily decreased over decades, with recent trials recruiting fewer than five participants per site (reduction by .16 participants/site/year, p < .0001). Fewer participants also progressed from recruitment to randomization over time (odds ratio = .94/year, p = .014). Concurrently, there has been an increase in the placebo response over time (increase in median percent reduction of .4%/year, p = .02; odds ratio of increase in 50% responder rate of 1.03/year, p = .02), which was not directly associated with the number of sites per trial (p > .20). SIGNIFICANCE This historical analysis highlights the increasing challenges with participant recruitment and retention, as well as increasing placebo response. It serves as a call to action to change clinical trial design to address these challenges.
Collapse
Affiliation(s)
- Wesley T. Kerr
- Department of Neurology, University of Michigan, Ann Arbor, Michigan, USA
- Department of Neurology, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Advith S. Reddy
- Department of Neurology, University of Michigan, Ann Arbor, Michigan, USA
| | - Sung Hyun Seo
- Department of Neurology, University of Michigan, Ann Arbor, Michigan, USA
| | - Neo Kok
- Department of Neurology, University of Michigan, Ann Arbor, Michigan, USA
| | - William C. Stacey
- Department of Neurology, University of Michigan, Ann Arbor, Michigan, USA
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, Michigan, USA
| | - John M. Stern
- Department of Neurology, David Geffen School of Medicine at the University of California, Los Angeles, Los Angeles, California, USA
| | - Page B. Pennell
- Department of Neurology, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | | |
Collapse
|
10
|
Miller MI, Shih LC, Kolachalama VB. Machine Learning in Clinical Trials: A Primer with Applications to Neurology. Neurotherapeutics 2023; 20:1066-1080. [PMID: 37249836 PMCID: PMC10228463 DOI: 10.1007/s13311-023-01384-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/21/2023] [Indexed: 05/31/2023] Open
Abstract
We reviewed foundational concepts in artificial intelligence (AI) and machine learning (ML) and discussed ways in which these methodologies may be employed to enhance progress in clinical trials and research, with particular attention to applications in the design, conduct, and interpretation of clinical trials for neurologic diseases. We discussed ways in which ML may help to accelerate the pace of subject recruitment, provide realistic simulation of medical interventions, and enhance remote trial administration via novel digital biomarkers and therapeutics. Lastly, we provide a brief overview of the technical, administrative, and regulatory challenges that must be addressed as ML achieves greater integration into clinical trial workflows.
Collapse
Affiliation(s)
- Matthew I Miller
- Department of Medicine, Boston University Chobanian & Avedisian School of Medicine, 72 E. Concord Street, Evans 636, Boston, MA, 02118, USA
| | - Ludy C Shih
- Department of Neurology, Boston University Chobanian & Avedisian School of Medicine, Boston, MA, 02118, USA
| | - Vijaya B Kolachalama
- Department of Medicine, Boston University Chobanian & Avedisian School of Medicine, 72 E. Concord Street, Evans 636, Boston, MA, 02118, USA.
- Department of Computer Science and Faculty of Computing & Data Sciences, Boston University, Boston, MA, 02115, USA.
| |
Collapse
|