Predicting drug approvals: The Novartis data science and artificial intelligence challenge

ChemAP: predicting drug approval with chemical structures before clinical trial phase by leveraging multi-modal embedding space and knowledge distillation

Recent studies showed that the likelihood of drug approval can be predicted with clinical data and structure information of drug using computational approaches. Predicting the likelihood of drug approval can be innovative and of high impact. However, models that leverage clinical data are applicable only in clinical stages, which is not very practical. Prioritizing drug candidates and early-stage decision-making in the de novo drug development process is crucial in pharmaceutical research to optimize resource allocation. For early-stage decision-making, we need a computational model that uses only chemical structures. This seemingly impossible task may utilize the predictive power with multi-modal features including clinical data. In this work, we introduce ChemAP (Chemical structure-based drug Approval Predictor), a novel deep learning scheme for drug approval prediction in the early-stage drug discovery phase. ChemAP aims to enhance the possibility of early-stage decision-making by enriching semantic knowledge to fill in the gap between multi-modal and single-modal chemical spaces through knowledge distillation techniques. This approach facilitates the effective construction of chemical space solely from chemical structure data, guided by multi-modal knowledge related to efficacy, such as clinical trials and patents of drugs. In this study, ChemAP achieved state-of-the-art performance, outperforming both traditional machine learning and deep learning models in drug approval prediction, with AUROC and AUPRC scores of 0.782 and 0.842 respectively on the drug approval benchmark dataset. Additionally, we demonstrated its generalizability by outperforming baseline models on a recent external dataset, which included drugs from the 2023 FDA-approved list and the 2024 clinical trial failure drug list, achieving AUROC and AUPRC scores of 0.694 and 0.851. These results demonstrate that ChemAP is an effective method in predicting drug approval only with chemical structure information of drug so that decision-making can be done at the early stages of drug development process. To the best of our knowledge, our work is the first of its kind to show that prediction of drug approval is possible only with structure information of drug by defining the chemical space of approved and unapproved drugs using deep learning technology.

Informatics and Artificial Intelligence-Guided Assessment of the Regulatory and Translational Research Landscape of First-in-Class Oncology Drugs in the United States, 2018-2022

PURPOSE

Cancer drug development remains a critical but challenging process that affects millions of patients and their families. Using biomedical informatics and artificial intelligence (AI) approaches, we assessed the regulatory and translational research landscape defining successful first-in-class drugs for patients with cancer.

METHODS

This is a retrospective observational study of all novel first-in-class drugs approved by the US Food and Drug Administration (FDA) from 2018 to 2022, stratified by cancer versus noncancer drugs. A biomedical informatics pipeline leveraging interoperability standards and ChatGPT performed integration and analysis of public databases provided by the FDA, National Institutes of Health, and WHO.

RESULTS

Between 2018 and 2022, the FDA approved a total of 247 novel drugs, of which 107 (43.3%) were first-in-class drugs involving a new biologic target. Of these first-in-class drugs, 30 (28%) treatments were indicated for patients with cancer, including 19 (63.3%) for solid tumors and the remaining 11 (36.7%) for hematologic cancers. A median of 68 publications of basic, clinical, and other relevant translational science preceded successful FDA approval of first-in-class cancer drugs, with oncology-related treatments involving fewer median years of target-based research than therapies not related to cancer (33 v 43 years; P < .05). Overall, 94.4% of first-in-class drugs had at least 25 years of target-related research papers, while 85.5% of first-in-class drugs had at least 10 years of translational research publications.

CONCLUSION

Novel first-in-class cancer treatments are defined by diverse clinical indications, personalized molecular targets, dependence on expedited regulatory pathways, and translational research metrics reflecting this complex landscape. Biomedical informatics and AI provide scalable, data-driven ways to assess and even address important challenges in the drug development pipeline.

Going beyond probability of success: Opportunities for statisticians to influence quantitative decision-making at the portfolio level

The pharmaceutical industry is plagued with long, costly development and high risk. Therefore, a company's effective management and optimisation of a portfolio of projects is critical for success. Project metrics such as the probability of success enable modelling of a company's pipeline accounting for the high uncertainty inherent within the industry. Making portfolio decisions inherently involves managing risk, and statisticians are ideally positioned to champion not only the derivation of metrics for individual projects, but also advocate decision-making at a broader portfolio level. This article aims to examine the existing different portfolio decision-making approaches and to suggest opportunities for statisticians to add value in terms of introducing probabilistic thinking, quantitative decision-making, and increasingly advanced methodologies.

Outcome differences by sex in oncology clinical trials

Identifying sex differences in outcomes and toxicity between males and females in oncology clinical trials is important and has also been mandated by National Institutes of Health policies. Here we analyze the Trialtrove database, finding that, strikingly, only 472/89,221 oncology clinical trials (0.5%) had curated post-treatment sex comparisons. Among 288 trials with comparisons of survival, outcome, or response, 16% report males having statistically significant better survival outcome or response, while 42% reported significantly better survival outcome or response for females. The strongest differences are in trials of EGFR inhibitors in lung cancer and rituximab in non-Hodgkin's lymphoma (both favoring females). Among 44 trials with side effect comparisons, more trials report significantly lesser side effects in males (N = 22) than in females (N = 13). Thus, while statistical comparisons between sexes in oncology trials are rarely reported, important differences in outcome and toxicity exist. These considerable outcome and toxicity differences highlight the need for reporting sex differences more thoroughly going forward.

Drug approval prediction based on the discrepancy in gene perturbation effects between cells and humans

BACKGROUND

Poor translation between in vitro and clinical studies due to the cells/humans discrepancy in drug target perturbation effects leads to safety failures in clinical trials, thus increasing drug development costs and reducing patients' life quality. Therefore, developing a predictive model for drug approval considering the cells/humans discrepancy is needed to reduce drug attrition rates in clinical trials.

METHODS

Our machine learning framework predicts drug approval in clinical trials based on the cells/humans discrepancy in drug target perturbation effects. To evaluate the discrepancy to predict drug approval (1404 approved and 1070 unapproved drugs), we analysed CRISPR-Cas9 knockout and loss-of-function mutation rate-based gene perturbation effects on cells and humans, respectively. To validate the risk of drug targets with the cells/humans discrepancy, we examined the targets of failed and withdrawn drugs due to safety problems.

FINDINGS

Drug approvals in clinical trials were correlated with the cells/humans discrepancy in gene perturbation effects. Genes tolerant to perturbation effects on cells but intolerant to those on humans were associated with failed drug targets. Furthermore, genes with the cells/humans discrepancy were related to drugs withdrawn due to severe side effects. Motivated by previous studies assessing drug safety through chemical properties, we improved drug approval prediction by integrating chemical information with the cells/humans discrepancy.

INTERPRETATION

The cells/humans discrepancy in gene perturbation effects facilitates drug approval prediction and explains drug safety failures in clinical trials.

FUNDING

S.K. received grants from the Korean National Research Foundation (2021R1A2B5B01001903 and 2020R1A6A1A03047902) and IITP (2019-0-01906, Artificial Intelligence Graduate School Program, POSTECH).

Artificial Intelligence and Machine Learning for Lead-to-Candidate Decision-Making and Beyond

The use of artificial intelligence (AI) and machine learning (ML) in pharmaceutical research and development has to date focused on research: target identification; docking-, fragment-, and motif-based generation of compound libraries; modeling of synthesis feasibility; rank-ordering likely hits according to structural and chemometric similarity to compounds having known activity and affinity to the target(s); optimizing a smaller library for synthesis and high-throughput screening; and combining evidence from screening to support hit-to-lead decisions. Applying AI/ML methods to lead optimization and lead-to-candidate (L2C) decision-making has shown slower progress, especially regarding predicting absorption, distribution, metabolism, excretion, and toxicology properties. The present review surveys reasons why this is so, reports progress that has occurred in recent years, and summarizes some of the issues that remain. Effective AI/ML tools to derisk L2C and later phases of development are important to accelerate the pharmaceutical development process, ameliorate escalating development costs, and achieve greater success rates.

Characterization of Oncology Clinical Trials Using Germline Genetic Data

IMPORTANCE

The recent successes of poly-ADP ribose polymerase (PARP) inhibitors and belzutifan support germline genetic data as an exciting, accessible source for biomarkers in cancer treatment. This study hypothesizes, however, that most oncology clinical trials using germline data largely prioritize BRCA1/2 as biomarkers and PARP inhibitors as therapy.

OBJECTIVE

To characterize past and ongoing oncology trials that use germline data.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective cross-sectional study of oncology trials used the Informa Trialtrove database to evaluate trial attributes. Trials using germline information (including the terms germline, hereditary, or inherited in the title, treatment plan, interventions, end points, objectives, results, or notes) and conducted globally between December 1, 1990, and April 4, 2022 (data freeze date), were included.

MAIN OUTCOMES AND MEASURES

Trials by cancer type, phase, participants, sponsor type, end points, outcomes, and locations were described. Associated biomarkers and mechanisms of action for studied therapeutic interventions were counted. How germline data in trial inclusion and exclusion criteria are associated with end points, outcomes, and enrollment were also examined.

RESULTS

A total of 887 of 84 297 (1.1%) oncology clinical trials in the Trialtrove database that use germline data were identified. Most trials were conducted in cancer types where PARP inhibitors are already approved. A total of 74.8% (672) of trials were performed in the phase 2 setting or above. Trials were primarily sponsored by industry (523 trials [59.0%]), academia (382 trials [43.1%]), and the government (274 trials [30.9%]), where trials may have multiple sponsor types. Among 343 trials using germline data with outcomes in Trialtrove, 180 (52.5%) reported meeting primary end points. Although BRCA1/2 are the most frequent biomarkers seen (BRCA1, 224 trials [25.3%]; BRCA2, 228 trials [25.7%]), trials also examine pharmacogenomic variants and germline mediators of somatic biomarkers. PARP inhibitors or immunotherapy were tested in 69.9% of trials; PARP inhibition was the most frequently studied mechanism (367 trials [41.4%]). An overwhelming number of trials using germline data were conducted in the US, Canada, and Europe vs other countries, mirroring disparities in cancer genetics data. Germline data in inclusion and exclusion criteria are associated with altered end point, outcomes, and enrollment compared with oncology trials with no germline data use. Examples of inclusion and exclusion criteria regarding germline data that may unintentionally exclude patients were identified.

CONCLUSIONS AND RELEVANCE

These findings suggest that for germline biomarkers to gain clinical relevance, trials must expand biomarkers, therapies, and populations under study.

Empowering the discovery of novel target-disease associations via machine learning approaches in the open targets platform

Background

The Open Targets (OT) Platform integrates a wide range of data sources on target-disease associations to facilitate identification of potential therapeutic drug targets to treat human diseases. However, due to the complexity that targets are usually functionally pleiotropic and efficacious for multiple indications, challenges in identifying novel target to indication associations remain. Specifically, persistent need exists for new methods for integration of novel target-disease association evidence and biological knowledge bases via advanced computational methods. These offer promise for increasing power for identification of the most promising target-disease pairs for therapeutic development. Here we introduce a novel approach by integrating additional target-disease features with machine learning models to further uncover druggable disease to target indications.

Results

We derived novel target-disease associations as supplemental features to OT platform-based associations using three data sources: (1) target tissue specificity from GTEx expression profiles; (2) target semantic similarities based on gene ontology; and (3) functional interactions among targets by embedding them from protein–protein interaction (PPI) networks. Machine learning models were applied to evaluate feature importance and performance benchmarks for predicting targets with known drug indications. The evaluation results show the newly integrated features demonstrate higher importance than current features in OT. In addition, these also show superior performance over association benchmarks and may support discovery of novel therapeutic indications for highly pursued targets.

Conclusion

Our newly generated features can be used to represent additional underlying biological relatedness among targets and diseases to further empower improved performance for predicting novel indications for drug targets through advanced machine learning models. The proposed methodology enables a powerful new approach for systematic evaluation of drug targets with novel indications.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-022-04753-4.

Identifying and Mitigating Potential Biases in Predicting Drug Approvals

INTRODUCTION

Machine learning models are increasingly applied to predict the drug development outcomes based on intermediary clinical trial results. A key challenge to this task is to address various forms of bias in the historical drug approval data.

OBJECTIVE

We aimed to identify and mitigate the bias in drug approval predictions and quantify the impacts of debiasing in terms of financial value and drug safety.

METHODS

We instantiated the Debiasing Variational Autoencoder, the state-of-the-art model for automated debiasing. We trained and evaluated the model on the Citeline dataset provided by Informa Pharma Intelligence to predict the final drug development outcome from phase II trial results.

RESULTS

The debiased Debiasing Variational Autoencoder model achieved better performance (measured by the [Formula: see text] score 0.48) in predicting the drug development outcomes than its un-debiased baseline ([Formula: see text] score 0.25). It had a much higher true-positive rate than baseline (60% vs 15%), while its true-negative rate was slightly lower (88% vs 99%). The Debiasing Variational Autoencoder distinguished between drugs developed by large pharmaceutical firms and those by small biotech companies. The model prediction is strongly influenced by multiple factors such as prior approval of the drug for another indication, whether the trial meets the positive/negative endpoints, and the year when the trial is completed. We estimate that the debiased model generates financial value for the drug developer in six major therapeutic areas, with a range of US$763-1,365 million.

CONCLUSIONS

Our analysis shows that debiasing improves the financial efficiency of late-stage drug development. From the pharmacovigilance perspective, the debiased model is more likely to identify drugs that are both safe and effective. Meanwhile, it may predict a higher probability of success for drugs with potential adverse effects (because of its lower true-negative rate), thus it must be used with caution to predict the development outcomes of drug candidates currently in the pipeline.

HINT: Hierarchical interaction network for clinical-trial-outcome predictions

Clinical trials are crucial for drug development but often face uncertain outcomes due to safety, efficacy, or patient-recruitment problems. We propose the Hierarchical Interaction Network (HINT) to predict clinical trial outcomes. First, HINT encodes multi-modal data (drug molecule, target disease, trial eligibility criteria) into embeddings. Then, HINT trains knowledge-embedding modules using drug pharmacokinetic and historical trial data. Finally, a hierarchical interaction graph connects all of the embeddings to capture their interactions and predict trial outcomes. HINT was trained and validated on 1,160 phase I trials, 4,449 phase II trials, and 3,436 phase III trials. It obtained 0.665, 0.620, and 0.847 F1 scores on separate test sets of 627 phase I, 1,653 phase II, and 1,140 phase III trials, respectively. HINT significantly outperforms the best baseline method on most metrics. The benchmark dataset and codes are released at https://github.com/futianfan/clinical-trial-outcome-prediction.

A New Comprehensive Approach to Assess the Probability of Success of Development Programs Before Pivotal Trials

The point at which clinical development programs transition from early phase to pivotal trials is a critical milestone. Substantial uncertainty about the outcome of pivotal trials may remain even after seeing positive early phase data, and companies may need to make difficult prioritization decisions for their portfolio. The probability of success (PoS) of a program, a single number expressed as a percentage reflecting the multitude of risks that may influence the final program outcome, is a key decision-making tool. Despite its importance, companies often rely on crude industry benchmarks that may be "adjusted" by experts based on undocumented criteria and which are typically misaligned with the definition of success used to drive commercial forecasts, leading to overly optimistic expected net present value calculations. We developed a new framework to assess the PoS of a program before pivotal trials begin. Our definition of success encompasses the successful outcome of pivotal trials, regulatory approval and meeting the requirements for market access as outlined in the target product profile. The proposed approach is organized in four steps and uses an innovative Bayesian approach to synthesize all relevant evidence. The new PoS framework is systematic and transparent. It will help organizations to make more informed decisions. In this paper, we outline the rationale and elaborate on the structure of the proposed framework, provide examples, and discuss the benefits and challenges associated with its adoption.