Does including machine learning predictions in ALS clinical trial analysis improve statistical power?

Predicting amyotrophic lateral sclerosis (ALS) progression with machine learning

OBJECTIVE

To predict ALS progression with varying observation and prediction window lengths, using machine learning (ML).

METHODS

We used demographic, clinical, and laboratory parameters from 5030 patients in the Pooled Resource Open-Access ALS Clinical Trials (PRO-ACT) database to model ALS disease progression as fast (at least 1.5 points decline in ALS Functional Rating Scale-Revised (ALSFRS-R) per month) or non-fast, using Extreme Gradient Boosting (XGBoost) and Bayesian Long Short Term Memory (BLSTM). XGBoost identified predictors of progression while BLSTM provided a confidence level for each prediction.

RESULTS

ML models achieved area under receiver-operating-characteristics curve (AUROC) of 0.570-0.748 and were non-inferior to clinician assessments. Performance was similar with observation lengths of a single visit, 3, 6, or 12 months and on a holdout validation dataset, but was better for longer prediction lengths. 21 important predictors were identified, with the top 3 being days since disease onset, past ALSFRS-R and forced vital capacity. Nonstandard predictors included phosphorus, chloride and albumin. BLSTM demonstrated higher performance for the samples about which it was most confident. Patient screening by models may reduce hypothetical Phase II/III clinical trial sizes by 18.3%.

CONCLUSION

Similar accuracies across ML models using different observation lengths suggest that a clinical trial observation period could be shortened to a single visit and clinical trial sizes reduced. Confidence levels provided by BLSTM gave additional information on the trustworthiness of predictions, which could aid decision-making. The identified predictors of ALS progression are potential biomarkers and therapeutic targets for further research.

Prescriptive Method for Optimizing Cost of Data Collection and Annotation in Machine Learning of Clinical Ultrasound

Machine learning in medical ultrasound faces a major challenge: the prohibitive costs of producing and annotating clinical data. Optimizing the data collection and annotation will improve model training efficiency, reducing project cost and times. This paper prescribes a 2-phase method for cost optimization based on iterative accuracy/sample size predictions, and active learning for annotation optimization.

METHODS

Using public breast, fetal, and lung ultrasound datasets we can: Optimize data collection by statistically predicting accuracy for a desired dataset size; and optimize labeling efficiency using Active Learning, where predictions with lowest certainty were labelled manually using feedback. A practical case study on BUSI data was used to demonstrate the method prescribed in this work.

RESULTS

With small data subsets, ~10%, dataset size vs. final accuracy relations can be predicted with diminishing results after 50% usage. Manual annotation was reduced by ~10% using active learning to focus the annotation.

CONCLUSION

This led to cost reductions of 50%-66%, depending on requirements and initial cost model, on BUSI dataset with a negligible accuracy drop of 3.75% from theoretical maximums.Clinical Relevance- This work provides methodology to optimize dataset size and manual data labelling, this allows generation of cost-effective datasets, of interest to all, but particularly for financially limited trials and feasibility studies, Reducing the time burden on annotating clinicians.

Dynamic effects of prognostic factors and individual survival prediction for amyotrophic lateral sclerosis disease

OBJECTIVE

Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disease affecting motor neurons, with broad heterogeneity in disease progression and survival in different patients. Therefore, an accurate prediction model will be crucial to implement timely interventions and prolong patient survival time.

METHODS

A total of 1260 ALS patients from the PRO-ACT database were included in the analysis. Their demographics, clinical variables, and death reports were included. We constructed an ALS dynamic Cox model through the landmarking approach. The predictive performance of the model at different landmark time points was evaluated by calculating the area under the curve (AUC) and Brier score.

RESULTS

Three baseline covariates and seven time-dependent covariates were selected to construct the ALS dynamic Cox model. For better prognostic analysis, this model identified dynamic effects of treatment, albumin, creatinine, calcium, hematocrit, and hemoglobin. Its prediction performance (at all landmark time points, AUC ≥ 0.70 and Brier score ≤ 0.12) was better than that of the traditional Cox model, and it predicted the dynamic 6-month survival probability according to the longitudinal information of individual patients.

INTERPRETATION

We developed an ALS dynamic Cox model with ALS longitudinal clinical trial datasets as the inputs. This model can not only capture the dynamic prognostic effect of both baseline and longitudinal covariates but also make individual survival predictions in real time, which are valuable for improving the prognosis of ALS patients and providing a reference for clinicians to make clinical decisions.

Artificial Intelligence Applied to clinical trials: opportunities and challenges

Background

Clinical Trials (CTs) remain the foundation of safe and effective drug development. Given the evolving data-driven and personalized medicine approach in healthcare, it is imperative for companies and regulators to utilize tailored Artificial Intelligence (AI) solutions that enable expeditious and streamlined clinical research. In this paper, we identified opportunities, challenges, and potential implications of AI in CTs.

Methods

Following an extensive search in relevant databases and websites, we gathered publications tackling the use of AI and Machine Learning (ML) in CTs from the past 5 years in the US and Europe, including Regulatory Authorities' documents.

Results

Documented applications of AI commonly concern the oncology field and are mostly being applied in the area of recruitment. Main opportunities discussed aim to create efficiencies across CT activities, including the ability to reduce sample sizes, improve enrollment and conduct faster, more optimized adaptive CTs. While AI is an area of enthusiastic development, the identified challenges are ethical in nature and relate to data availability, standards, and most importantly, lack of regulatory guidance hindering the acceptance of AI tools in drug development. However, future implications are significant and are anticipated to improve the probability of success, reduce trial burden and overall, speed up research and regulatory approval.

Conclusion

The use of AI in CTs is in its relative infancy; however, it is a fast-evolving field. As regulators provide more guidance on the acceptability of AI in specific areas, we anticipate the scope of use to broaden and the volume of implementation to increase rapidly.

A novel quantitative indicator for disease progression rate in amyotrophic lateral sclerosis

OBJECTIVE

The current study sought to develop a new indicator for disease progression rate in amyotrophic lateral sclerosis (ALS).

METHODS

We used a nonparametric method to score diverse patterns of decline in the percentage of predicted forced vital capacity (%FVC) in patients with ALS. This involved 6317 longitudinal %FVC data sets from 920 patients in the Pooled Resource Open-Access ALS Clinical Trials (PRO-ACT) database volunteered by PRO-ACT Consortium members. To assess the utility of the derived scores as a disease indicator, we examined changes over time, the association with prognosis, and correlation with the Risk Profile of the Treatment Research Initiative to Cure ALS (TRICALS). Our local cohort (n = 92) was used for external validation.

RESULTS

We derived scores ranging from 35 to 106 points to construct the FVC Decline Pattern scale (FVC-DiP). Individuals' FVC-DiP scores were determined from a single measurement of %FVC and disease duration at assessment. Although the %FVC declined over the disease course (p < 0.0001), the FVC-DiP remained relatively stable. Low FVC-DiP scores were associated with rapid disease progression. Using our cohort, we demonstrated an association between FVC-DiP and the survival prognosis, the stability of the FVC-DiP per individual, and a correlation between FVC-DiP scores and the TRICALS Risk Profile (r² = 0.904, p < 0.0001).

CONCLUSIONS

FVC-DiP scores reflected patterns of declining %FVC over the natural course of ALS and indicated the disease progression rate. The FVC-DiP may enable easy assessment of disease progression patterns and could be used for assessing treatment efficacy.

Considerations for Amyotrophic Lateral Sclerosis (ALS) Clinical Trial Design

Thoughtful clinical trial design is critical for efficient therapeutic development, particularly in the field of amyotrophic lateral sclerosis (ALS), where trials often aim to detect modest treatment effects among a population with heterogeneous disease progression. Appropriate outcome measure selection is necessary for trials to provide decisive and informative results. Investigators must consider the outcome measure's reliability, responsiveness to detect change when change has actually occurred, clinical relevance, and psychometric performance. ALS clinical trials can also be performed more efficiently by utilizing statistical enrichment techniques. Innovations in ALS prediction models allow for selection of participants with less heterogeneity in disease progression rates without requiring a lead-in period, or participants can be stratified according to predicted progression. Statistical enrichment can reduce the needed sample size and improve study power, but investigators must find a balance between optimizing statistical efficiency and retaining generalizability of study findings to the broader ALS population. Additional progress is still needed for biomarker development and validation to confirm target engagement in ALS treatment trials. Selection of an appropriate biofluid biomarker depends on the treatment mechanism of interest, and biomarker studies should be incorporated into early phase trials. Inclusion of patients with ALS as advisors and advocates can strengthen clinical trial design and study retention, but more engagement efforts are needed to improve diversity and equity in ALS research studies. Another challenge for ALS therapeutic development is identifying ways to respect patient autonomy and improve access to experimental treatment, something that is strongly desired by many patients with ALS and ALS advocacy organizations. Expanded access programs that run concurrently to well-designed and adequately powered randomized controlled trials may provide an opportunity to broaden access to promising therapeutics without compromising scientific integrity or rushing regulatory approval of therapies without adequate proof of efficacy.

Phenotypic categorisation of individual subjects with motor neuron disease based on radiological disease burden patterns: A machine-learning approach

Motor neuron disease is an umbrella term encompassing a multitude of clinically heterogeneous phenotypes. The early and accurate categorisation of patients is hugely important, as MND phenotypes are associated with markedly different prognoses, progression rates, care needs and benefit from divergent management strategies. The categorisation of patients shortly after symptom onset is challenging, and often lengthy clinical monitoring is needed to assign patients to the appropriate phenotypic subgroup. In this study, a multi-class machine-learning strategy was implemented to classify 300 patients based on their radiological profile into diagnostic labels along the UMN-LMN spectrum. A comprehensive panel of cortical thickness measures, subcortical grey matter variables, and white matter integrity metrics were evaluated in a multilayer perceptron (MLP) model. Additional exploratory analyses were also carried out using discriminant function analyses (DFA). Excellent classification accuracy was achieved for amyotrophic lateral sclerosis in the testing cohort (93.7%) using the MLP model, but poor diagnostic accuracy was detected for primary lateral sclerosis (43.8%) and poliomyelitis survivors (60%). Feature importance analyses highlighted the relevance of white matter diffusivity metrics and the evaluation of cerebellar indices, cingulate measures and thalamic radiation variables to discriminate MND phenotypes. Our data suggest that radiological data from single patients may be meaningfully interpreted if large training data sets are available and the provision of diagnostic probability outcomes may be clinically useful in patients with short symptom duration. The computational interpretation of multimodal radiology datasets herald viable diagnostic, prognostic and clinical trial applications.