Reinforcement learning and Bayesian data assimilation for model-informed precision dosing in oncology

Personalized Dosing of Medicines for Children: A Primer on Pediatric Pharmacometrics for Clinicians

The widespread use of drugs for unapproved purposes remains common in children, primarily attributable to practical, ethical, and financial constraints associated with pediatric drug research. Pharmacometrics, the scientific discipline that involves the application of mathematical models to understand and quantify drug effects, holds promise in advancing pediatric pharmacotherapy by expediting drug development, extending applications, and personalizing dosing. In this review, we delineate the principles of pharmacometrics, and explore its clinical applications and prospects. The fundamental aspect of any pharmacometric analysis lies in the selection of appropriate methods for quantifying pharmacokinetics and pharmacodynamics. Population pharmacokinetic modeling is a data-driven method ('top-down' approach) to approximate population-level pharmacokinetic parameters, while identifying factors contributing to inter-individual variability. Model-informed precision dosing is increasingly used to leverage population pharmacokinetic models and patient data, to formulate individualized dosing recommendations. Physiologically based pharmacokinetic models integrate physicochemical drug properties with biological parameters ('bottom-up approach'), and is particularly valuable in situations with limited clinical data, such as early drug development, assessing drug-drug interactions, or adapting dosing for patients with specific comorbidities. The effective implementation of these complex models hinges on strong collaboration between clinicians and pharmacometricians, given the pivotal role of data availability. Promising advancements aimed at improving data availability encompass innovative techniques such as opportunistic sampling, minimally invasive sampling approaches, microdialysis, and in vitro investigations. Additionally, ongoing research efforts to enhance measurement instruments for evaluating pharmacodynamics responses, including biomarkers and clinical scoring systems, are expected to significantly bolster our capacity to understand drug effects in children.

Systematic literature review on reinforcement learning in non-communicable disease interventions

There is evidence that reducing modifiable risk factors and strengthening medical and health interventions can reduce early mortality and economic losses from non-communicable diseases (NCDs). Machine learning (ML) algorithms have been successfully applied to preventing and controlling NCDs. Reinforcement learning (RL) is the most promising of these approaches because of its ability to dynamically adapt interventions to NCD disease progression and its commitment to achieving long-term intervention goals. This paper reviews the preferred algorithms, data sources, design details, and obstacles to clinical application in existing studies to facilitate the early application of RL algorithms in clinical practice research for NCD interventions. We screened 40 relevant papers for quantitative and qualitative analysis using the PRISMA review flow diagram. The results show that researchers tend to use Deep Q-Network (DQN) and Actor-Critic as well as their improved or hybrid algorithms to train and validate RL models on retrospective datasets. Often, the patient's physical condition is the main defining parameter of the state space, while interventions are the main defining parameter of the action space. Mostly, changes in the patient's physical condition are used as a basis for immediate rewards to the agent. Various attempts have been made to address the challenges to clinical application, and several approaches have been proposed from existing research. However, as there is currently no universally accepted solution, the use of RL algorithms in clinical practice for NCD interventions necessitates more comprehensive responses to the issues addressed in this paper, which are safety, interpretability, training efficiency, and the technical aspect of exploitation and exploration in RL algorithms.

Reinforcement Learning and PK-PD Models Integration to Personalize the Adaptive Dosing Protocol of Erdafitinib in Patients with Metastatic Urothelial Carcinoma

The integration of pharmacokinetic-pharmacodynamic (PK-PD) modeling and simulations with artificial intelligence/machine learning algorithms is one of the most attractive areas of the pharmacometric research. These hybrid techniques are currently under investigation to perform several tasks, among which precision dosing. In this scenario, this paper presents and evaluates a new framework embedding PK-PD models into a reinforcement learning (RL) algorithm, Q-learning (QL), to personalize pharmacological treatment. Each patient is represented with a set of PK-PD parameters and has a personal QL agent which optimizes the individual treatment. In the training phase, leveraging PK-PD simulations, the QL agent assesses different actions, defined consistently with the clinical knowledge to consider only plausible dose-adjustments, in order to find the optimal rules. The proposed framework was evaluated to optimize the erdafitinib treatment in patients with metastatic urothelial carcinoma. This drug was approved by the US Food and Drug Administration (FDA) with a dose-adaptive protocol based on monitoring the levels of serum phosphate, which represent a biomarker of both treatment efficacy and toxicity. To evaluate the flexibility of the methodology, a heterogeneous virtual population of 141 patients was generated using an erdafitinib population PK (PopPK)-PD literature model. For each patient, treatment response was simulated by using both QL-optimized protocol and the clinical one. QL agents outperform the approved dose-adaptive rules, increasing more than 10% the efficacy and the safety of treatment at each end point. Results confirm the great potentialities of the integration of PopPK-PD models and RL algorithms to optimize precision dosing tasks.

Bridging the Worlds of Pharmacometrics and Machine Learning

Precision medicine requires individualized modeling of disease and drug dynamics, with machine learning-based computational techniques gaining increasing popularity. The complexity of either field, however, makes current pharmacological problems opaque to machine learning practitioners, and state-of-the-art machine learning methods inaccessible to pharmacometricians. To help bridge the two worlds, we provide an introduction to current problems and techniques in pharmacometrics that ranges from pharmacokinetic and pharmacodynamic modeling to pharmacometric simulations, model-informed precision dosing, and systems pharmacology, and review some of the machine learning approaches to address them. We hope this would facilitate collaboration between experts, with complementary strengths of principled pharmacometric modeling and flexibility of machine learning leading to synergistic effects in pharmacological applications.

Simulating clinical trials for model-informed precision dosing: using warfarin treatment as a use case

Treatment response variability across patients is a common phenomenon in clinical practice. For many drugs this inter-individual variability does not require much (if any) individualisation of dosing strategies. However, for some drugs, including chemotherapies and some monoclonal antibody treatments, individualisation of dosages are needed to avoid harmful adverse events. Model-informed precision dosing (MIPD) is an emerging approach to guide the individualisation of dosing regimens of otherwise difficult-to-administer drugs. Several MIPD approaches have been suggested to predict dosing strategies, including regression, reinforcement learning (RL) and pharmacokinetic and pharmacodynamic (PKPD) modelling. A unified framework to study the strengths and limitations of these approaches is missing. We develop a framework to simulate clinical MIPD trials, providing a cost and time efficient way to test different MIPD approaches. Central for our framework is a clinical trial model that emulates the complexities in clinical practice that challenge successful treatment individualisation. We demonstrate this framework using warfarin treatment as a use case and investigate three popular MIPD methods: 1. Neural network regression; 2. Deep RL; and 3. PKPD modelling. We find that the PKPD model individualises warfarin dosing regimens with the highest success rate and the highest efficiency: 75.1% of the individuals display INRs inside the therapeutic range at the end of the simulated trial; and the median time in the therapeutic range (TTR) is 74%. In comparison, the regression model and the deep RL model have success rates of 47.0% and 65.8%, and median TTRs of 45% and 68%. We also find that the MIPD models can attain different degrees of individualisation: the Regression model individualises dosing regimens up to variability explained by covariates; the Deep RL model and the PKPD model individualise dosing regimens accounting also for additional variation using monitoring data. However, the Deep RL model focusses on control of the treatment response, while the PKPD model uses the data also to further the individualisation of predictions.

Towards Model-Informed Precision Dosing of Voriconazole: Challenging Published Voriconazole Nonlinear Mixed-Effects Models with Real-World Clinical Data

BACKGROUND AND OBJECTIVES

Model-informed precision dosing (MIPD) frequently uses nonlinear mixed-effects (NLME) models to predict and optimize therapy outcomes based on patient characteristics and therapeutic drug monitoring data. MIPD is indicated for compounds with narrow therapeutic range and complex pharmacokinetics (PK), such as voriconazole, a broad-spectrum antifungal drug for prevention and treatment of invasive fungal infections. To provide guidance and recommendations for evidence-based application of MIPD for voriconazole, this work aimed to (i) externally evaluate and compare the predictive performance of a published so-called 'hybrid' model for MIPD (an aggregate model comprising features and prior information from six previously published NLME models) versus two 'standard' NLME models of voriconazole, and (ii) investigate strategies and illustrate the clinical impact of Bayesian forecasting for voriconazole.

METHODS

A workflow for external evaluation and application of MIPD for voriconazole was implemented. Published voriconazole NLME models were externally evaluated using a comprehensive in-house clinical database comprising nine voriconazole studies and prediction-/simulation-based diagnostics. The NLME models were applied using different Bayesian forecasting strategies to assess the influence of prior observations on model predictivity.

RESULTS

The overall best predictive performance was obtained using the aggregate model. However, all NLME models showed only modest predictive performance, suggesting that (i) important PK processes were not sufficiently implemented in the structural submodels, (ii) sources of interindividual variability were not entirely captured, and (iii) interoccasion variability was not adequately accounted for. Predictive performance substantially improved by including the most recent voriconazole observations in MIPD.

CONCLUSION

Our results highlight the potential clinical impact of MIPD for voriconazole and indicate the need for a comprehensive (pre-)clinical database as basis for model development and careful external model evaluation for compounds with complex PK before their successful use in MIPD.

Precision Oncology by Point-of-Care Therapeutic Drug Monitoring and Dosage Adjustment of Conventional Cytotoxic Chemotherapies: A Perspective

Therapeutic drug monitoring (TDM) of conventional cytotoxic chemotherapies is strongly supported yet poorly implemented in daily practice in hospitals. Analytical methods for the quantification of cytotoxic drugs are instead widely presented in the scientific literature, while the use of these therapeutics is expected to keep going for longer. There are two main issues hindering the implementation of TDM: turnaround time, which is incompatible with the dosage profiles of these drugs, and exposure surrogate marker, namely total area under the curve (AUC). Therefore, this perspective article aims to define the adjustment needed from current to efficient TDM practice for cytotoxics, namely point-of-care (POC) TDM. For real-time dose adjustment, which is required for chemotherapies, such POC TDM is only achievable with analytical methods that match the sensitivity and selectivity of current methods, such as chromatography, as well as model-informed precision dosing platforms to assist the oncologist with dose fine-tuning based on quantification results and targeted intervals.

Model-informed precision dosing of teicoplanin for the rapid achievement of the target area under the concentration-time curve: A simulation study

Teicoplanin, a glycopeptide antimicrobial, is recommended for therapeutic drug monitoring, but it remains unclear how to target the area under the concentration-time curve (AUC). This simulation study purposed to demonstrate the potential of the Bayesian forecasting approach for the rapid achievement of the target AUC for teicoplanin. We generated concordant and discordant virtual populations against a Japanese population pharmacokinetic model. The predictive performance of the Bayesian posterior AUC in limited sampling on the first day against the reference AUC was evaluated as an acceptable target AUC ratio within the range of 0.8-1.2. In the concordant population, the probability for the maximum a priori or Bayesian posterior AUC on the first day (AUC_0-24 ) was 61.3% or more than 77.0%, respectively. The Bayesian posterior AUC on the second day (AUC_24-48 ) was more than 75.1%. In the discordant population, the probability for the maximum a priori or Bayesian posterior AUC_0-24 was 15.5% or 11.7-80.7%, respectively. The probability for the maximum a priori or Bayesian posterior AUC_24-48 was 23.4%, 30.2-82.1%. The AUC at steady-state (AUC_SS ) was correlated with trough concentration at steady-state, with a coefficient of determination of 0.930; the coefficients on days 7 and 4 were 0.442 and 0.125, respectively. In conclusion, this study demonstrated that early sampling could improve the probability of AUC_0-24 and AUC_24-48 but did not adequately predict AUC_SS . Further studies are necessary to apply early sampling-based model-informed precision dosing in the clinical settings.

Translating Data Science Results into Precision Oncology Decisions: A Mini Review

While reviewing and discussing the potential of data science in oncology, we emphasize medical imaging and radiomics as the leading contextual frameworks to measure the impacts of Artificial Intelligence (AI) and Machine Learning (ML) developments. We envision some domains and research directions in which radiomics should become more significant in view of current barriers and limitations.

Model enhanced reinforcement learning to enable precision dosing: A theoretical case study with dosing of propofol

Extending the potential of precision dosing requires evaluating methodologies offering more flexibility and higher degree of personalization. Reinforcement learning (RL) holds promise in its ability to integrate multidimensional data in an adaptive process built toward efficient decision making centered on sustainable value creation. For general anesthesia in intensive care units, RL is applied and automatically adjusts dosing through monitoring of patient's consciousness. We further explore the problem of optimal control of anesthesia with propofol by combining RL with state-of-the-art tools used to inform dosing in drug development. In particular, we used pharmacokinetic-pharmacodynamic (PK-PD) modeling as a simulation engine to generate experience from dosing scenarios, which cannot be tested experimentally. Through simulations, we show that, when learning from retrospective trial data, more than 100 patients are needed to reach an accuracy within the range of what is achieved with a standard dosing solution. However, embedding a model of drug effect within the RL algorithm improves accuracy by reducing errors to target by 90% through learning to take dosing actions maximizing long-term benefit. Data residual variability impacts accuracy while the algorithm efficiently coped with up to 50% interindividual variability in the PK and 25% in the PD model's parameters. We illustrate how extending the state definition of the RL agent with meaningful variables is key to achieve high accuracy of optimal dosing policy. These results suggest that RL constitutes an attractive approach for precision dosing when rich data are available or when complemented with synthetic data from model-based tools used in model-informed drug development.

Model-Based Prediction of Irinotecan-Induced Grade 4 Neutropenia in Advanced Cancer Patients: Influence of Demographic and Clinical Factors

Severe neutropenia is the major dose-liming toxicity of irinotecan-based chemotherapy. The objective was to assess to what extent a population pharmacokinetic/pharmacodynamic model including patient-specific demographic/clinical characteristics, individual pharmacokinetics, and absolute neutrophil counts (ANCs) can predict irinotecan-induced grade 4 neutropenia. A semimechanistic population pharmacokinetic/pharmacodynamic model was developed to describe neutrophil response over time in 197 patients with cancer receiving irinotecan. For covariate analysis, sex, race, age, pretreatment total bilirubin, and body surface area were evaluated to identify significant covariates on system-related parameters (mean transit time (MTT) and ɣ) and sensitivity to neutropenia effects of irinotecan and SN-38 (SLOPE). The model-based simulation was performed to assess the contribution of the identified covariates, individual pharmacokinetics, and baseline ANC alone or with incremental addition of weekly ANC up to 3 weeks on predicting irinotecan-induced grade 4 neutropenia. The time course of neutrophil response was described using the model assuming that irinotecan and SN-38 have toxic effects on bone marrow proliferating cells. Sex and pretreatment total bilirubin explained 10.5% of interindividual variability in MTT. No covariates were identified for SLOPE and γ. Incorporating sex and pretreatment total bilirubin (area under the receiver operating characteristic curve (AUC-ROC): 50%, 95% CI 50-50%) or with the addition of individual pharmacokinetics (AUC-ROC: 62%, 95% CI 53-71%) in the model did not result in accurate prediction of grade 4 neutropenia. However, incorporating ANC only at baseline and week 1 in the model achieved a good prediction (AUC-ROC: 78%, 95% CI 69-88%). These results demonstrate the potential applicability of a model-based approach to predict irinotecan-induced neutropenia, which ultimately allows for personalized intervention to maximize treatment outcomes.

Multi‐model averaging improves the performance of model‐guided infliximab dosing in patients with inflammatory bowel diseases

Infliximab dosage de‐escalation without prior knowledge of drug concentrations may put patients at risk for underexposure and trigger the loss of response. A single‐model approach for model‐informed precision dosing during infliximab maintenance therapy has proven its clinical benefit in patients with inflammatory bowel diseases. We evaluated the predictive performances of two multi‐model approaches, a model selection algorithm and a model averaging algorithm, using 18 published population pharmacokinetic models of infliximab for guiding dosage de‐escalation. Data of 54 patients with Crohn’s disease and ulcerative colitis who underwent infliximab dosage de‐escalation after an earlier escalation were used. A priori prediction (based solely on covariate data) and maximum a posteriori prediction (based on covariate data and trough concentrations) were compared using accuracy and precision metrics and the classification accuracy at the trough concentration target of 5.0 mg/L. A priori prediction was inaccurate and imprecise, with the lowest classification accuracies irrespective of the approach (median 59%, interquartile range 59%–63%). Using the maximum a posteriori prediction, the model averaging algorithm had systematically better predictive performance than the model selection algorithm or the single‐model approach with any model, regardless of the number of concentration data. Only a single trough concentration (preferably at the point of care) sufficed for accurate and precise prediction. Predictive performance of both single‐ and multi‐model approaches was robust to the lack of covariate data. Model averaging using four models demonstrated similar predictive performance with a five‐fold shorter computation time. This model averaging algorithm was implemented in the TDMx software tool to guide infliximab dosage de‐escalation in the forthcoming prospective MODIFI study (NCT04982172).

Model Informed Precision Dosing Tool Forecasts Trough Infliximab and Associates with Disease Status and Tumor Necrosis Factor-Alpha Levels of Inflammatory Bowel Diseases

Background: Substantial inter-and intra-individual variability of Infliximab (IFX) pharmacokinetics necessitates tailored dosing approaches. Here, we evaluated the performances of a Model Informed Precision Dosing (MIPD) Tool in forecasting trough Infliximab (IFX) levels in association with disease status and circulating TNF-α in patients with Inflammatory Bowel Diseases (IBD). Methods: Consented patients undergoing every 8-week maintenance therapy with IFX were enrolled. Midcycle specimens were collected, IFX, antibodies to IFX, albumin were determined and analyzed with weight using nonlinear mixed effect models coupled with Bayesian data assimilation to forecast trough levels. Accuracy of forecasted as compared to observed trough IFX levels were evaluated using Demings’s regression. Association between IFX levels, CRP-based clinical remission and TNF-α levels were analyzed using logistic regression and linear mixed effect models. Results: In 41 patients receiving IFX (median dose = 5.3 mg/Kg), median IFX levels decreased from 13.0 to 3.9 µg/mL from mid to end of cycle time points, respectively. Midcycle IFX levels forecasted trough with Deming’s slope = 0.90 and R2 = 0.87. Observed end cycle and forecasted trough levels above 5 µg/mL associated with CRP-based clinical remission (OR = 7.2 CI95%: 1.7−30.2; OR = 21.0 CI95%: 3.4−127.9, respectively) (p < 0.01). Median TNF-α levels increased from 4.6 to 8.0 pg/mL from mid to end of cycle time points, respectively (p < 0.01). CRP and TNF-α levels associated independently and additively to decreased IFX levels (p < 0.01). Conclusions: These data establish the value of our MIPD tool in forecasting trough IFX levels in patients with IBD. Serum TNF-α and CRP are reflective of inflammatory burden which impacts exposure.

Reinforcement learning as an innovative model-based approach: Examples from precision dosing, digital health and computational psychiatry

Model-based approaches are instrumental for successful drug development and use. Anchored within pharmacological principles, through mathematical modeling they contribute to the quantification of drug response variability and enables precision dosing. Reinforcement learning (RL)-a set of computational methods addressing optimization problems as a continuous learning process-shows relevance for precision dosing with high flexibility for dosing rule adaptation and for coping with high dimensional efficacy and/or safety markers, constituting a relevant approach to take advantage of data from digital health technologies. RL can also support contributions to the successful development of digital health applications, recognized as key players of the future healthcare systems, in particular for reducing the burden of non-communicable diseases to society. RL is also pivotal in computational psychiatry-a way to characterize mental dysfunctions in terms of aberrant brain computations-and represents an innovative modeling approach forpsychiatric indications such as depression or substance abuse disorders for which digital therapeutics are foreseen as promising modalities.

A continued learning approach for model-informed precision dosing: updating models in clinical practice

Model-informed precision dosing (MIPD) is a quantitative dosing framework that combines prior knowledge on the drug-disease-patient system with patient data from therapeutic drug/biomarker monitoring (TDM) to support individualized dosing in ongoing treatment. Structural models and prior parameter distributions used in MIPD approaches typically build on prior clinical trials that involve only a limited number of patients selected according to some exclusion/inclusion criteria. Compared to the prior clinical trial population, the patient population in clinical practice can be expected to include also altered behavior and/or increased interindividual variability, the extent of which, however, is typically unknown. Here, we address the question of how to adapt and refine models on the level of the model parameters to better reflect this real-world diversity. We propose an approach for continued learning across patients during MIPD using a sequential hierarchical Bayesian framework. The approach builds on two stages to separate the update of the individual patient parameters from updating the population parameters. Consequently, it enables continued learning across hospitals or study centers, since only summary patient data (on the level of model parameters) need to be shared, but no individual TDM data. We illustrate this continued learning approach with neutrophil-guided dosing of paclitaxel. The present study constitutes an important step towards building confidence in MIPD and eventually establishing MIPD increasingly in everyday therapeutic use.

Reinforcement Learning for Precision Oncology

Precision oncology is grounded in the increasing understanding of genetic and molecular mechanisms that underly malignant disease and offer different treatment pathways for the individual patient. The growing complexity of medical data has led to the implementation of machine learning techniques that are vastly applied for risk assessment and outcome prediction using either supervised or unsupervised learning. Still largely overlooked is reinforcement learning (RL) that addresses sequential tasks by exploring the underlying dynamics of an environment and shaping it by taking actions in order to maximize cumulative rewards over time, thereby achieving optimal long-term outcomes. Recent breakthroughs in RL demonstrated remarkable results in gameplay and autonomous driving, often achieving human-like or even superhuman performance. While this type of machine learning holds the potential to become a helpful decision support tool, it comes with a set of distinctive challenges that need to be addressed to ensure applicability, validity and safety. In this review, we highlight recent advances of RL focusing on studies in oncology and point out current challenges and pitfalls that need to be accounted for in future studies in order to successfully develop RL-based decision support systems for precision oncology.

A hybrid machine learning/pharmacokinetic approach outperforms maximum a posteriori Bayesian estimation by selectively flattening model priors

Model‐informed precision dosing (MIPD) approaches typically apply maximum a posteriori (MAP) Bayesian estimation to determine individual pharmacokinetic (PK) parameters with the goal of optimizing future dosing regimens. This process combines knowledge about the individual, in the form of drug levels or pharmacodynamic biomarkers, with prior knowledge of the drug PK in the general population. Use of “flattened priors” (FPs), in which the weight of the model priors is reduced relative to observations about the patient, has been previously proposed to estimate individual PK parameters in instances where the patient is poorly described by the PK model. However, little is known about the predictive performance of FPs and when to apply FPs in MIPD. Here, FP is evaluated in a data set of 4679 adult patients treated with vancomycin. Depending on the PK model, prediction error could be reduced by applying FPs in 42–55% of PK parameter estimations. Machine learning (ML) models could identify instances where FPs would outperform MAPs with a specificity of 81–86%, reducing overall root mean squared error (RMSE) of PK model predictions by 12–22% (0.5–1.2 mg/L) relative to MAP alone. The factors most indicative of the use of FPs were past prediction residuals and bias in past PK predictions. A more clinically practical minimal model was developed using only these two features, reducing RMSE by 5–18% (0.20–0.93 mg/L) relative to MAP. This hybrid ML/PK approach advances the precision dosing toolkit by leveraging the power of ML while maintaining the mechanistic insight and interpretability of PK models.