Nonlinear adaptive control of competitive release and chemotherapeutic resistance

Reinforcement learning informs optimal treatment strategies to limit antibiotic resistance

Antimicrobial resistance was estimated to be associated with 4.95 million deaths worldwide in 2019. It is possible to frame the antimicrobial resistance problem as a feedback-control problem. If we could optimize this feedback-control problem and translate our findings to the clinic, we could slow, prevent, or reverse the development of high-level drug resistance. Prior work on this topic has relied on systems where the exact dynamics and parameters were known a priori. In this study, we extend this work using a reinforcement learning (RL) approach capable of learning effective drug cycling policies in a system defined by empirically measured fitness landscapes. Crucially, we show that it is possible to learn effective drug cycling policies despite the problems of noisy, limited, or delayed measurement. Given access to a panel of 15 [Formula: see text]-lactam antibiotics with which to treat the simulated Escherichia coli population, we demonstrate that RL agents outperform two naive treatment paradigms at minimizing the population fitness over time. We also show that RL agents approach the performance of the optimal drug cycling policy. Even when stochastic noise is introduced to the measurements of population fitness, we show that RL agents are capable of maintaining evolving populations at lower growth rates compared to controls. We further tested our approach in arbitrary fitness landscapes of up to 1,024 genotypes. We show that minimization of population fitness using drug cycles is not limited by increasing genome size. Our work represents a proof-of-concept for using AI to control complex evolutionary processes.

HSP90a promotes the resistance to oxaliplatin in HCC through regulating IDH1-induced cell competition

Though burgeoning research manifests that cell competition, an essential selection and quality control mechanism for maintaining tissue or organ growth and homeostasis in multicellular organisms, is closely related to tumorigenesis and development, the mechanism of cell competition associated with tumor drug resistance remains elusive. In the study, we uncovered that oxaliplatin-resistant hepatocellular carcinoma (HCC) cells exhibit a pronounced competitive advantage against their sensitive counterparts, which is related to lipid takeover of resistant cells from sensitive cells. Of note, such lipid takeover is dependent on the existence of isocitrate dehydrogenase 1 (IDH1) in resistant HCC cells. Mechanistically, IDH1 activity is regulated by heat shock protein 90 alpha (HSP90α) through binding with each other, which orchestrates the expressions of lipid metabolic enzymes and lipid accumulation in resistant HCC cells. Our results suggest that HCC cell competition-driven chemoresistance can be regulated by HSP90α/IDH1-mediated lipid metabolism, which may serve as a promising target for overcoming drug resistance in HCC.

Reinforcement Learning informs optimal treatment strategies to limit antibiotic resistance

Antimicrobial resistance was estimated to be associated with 4.95 million deaths worldwide in 2019. It is possible to frame the antimicrobial resistance problem as a feedback-control problem. If we could optimize this feedback-control problem and translate our findings to the clinic, we could slow, prevent or reverse the development of high-level drug resistance. Prior work on this topic has relied on systems where the exact dynamics and parameters were known a priori. In this study, we extend this work using a reinforcement learning (RL) approach capable of learning effective drug cycling policies in a system defined by empirically measured fitness landscapes. Crucially, we show that is possible to learn effective drug cycling policies despite the problems of noisy, limited, or delayed measurement. Given access to a panel of 15 β-lactam antibiotics with which to treat the simulated E. coli population, we demonstrate that RL agents outperform two naive treatment paradigms at minimizing the population fitness over time. We also show that RL agents approach the performance of the optimal drug cycling policy. Even when stochastic noise is introduced to the measurements of population fitness, we show that RL agents are capable of maintaining evolving populations at lower growth rates compared to controls. We further tested our approach in arbitrary fitness landscapes of up to 1024 genotypes. We show that minimization of population fitness using drug cycles is not limited by increasing genome size. Our work represents a proof-of-concept for using AI to control complex evolutionary processes.

Stochastic competitive release and adaptive chemotherapy

We develop a finite-cell model of tumor natural selection dynamics to investigate the stochastic fluctuations associated with multiple rounds of adaptive chemotherapy. The adaptive cycles are designed to avoid chemoresistance in the tumor by managing the ecological mechanism of competitive release of a resistant subpopulation. Our model is based on a three-component evolutionary game played among healthy (H), sensitive (S), and resistant (R) populations of N cells, with a chemotherapy control parameter, C(t), which we use to dynamically impose selection pressure on the sensitive subpopulation to slow tumor growth and manage competitive release of the resistant population. The adaptive chemoschedule is designed based on the deterministic (N→∞) adjusted replicator dynamical system, then implemented using the finite-cell stochastic frequency dependent Moran process model (N=10K-50K) to ascertain the cumulative effect of the stochastic fluctuations on the efficacy of the adaptive schedules over multiple rounds. We quantify the stochastic fixation probability regions of the R and S populations in the HSR trilinear phase plane as a function of the control parameter C∈[0,1], showing that the size of the R region increases with increasing C. We then implement an adaptive time-dependent schedule C(t) for the stochastic model and quantify the variances (using principal component coordinates) associated with the evolutionary cycles over multiple rounds of adaptive therapy. The variances increase subquadratically through several rounds before the evolutionary cycle begins to break down. Despite this, we show the stochastic adaptive schedules are more effective at delaying resistance than standard maximum tolerated dose and low-dose metronomic schedules. The simplified low-dimensional model provides some insights on how well multiple rounds of adaptive therapies are likely to perform over a range of tumor sizes (i.e., different values of N) if the goal is to maintain a sustained balance among competing subpopulations of cells to avoid chemoresistance via competitive release in a stochastic environment.

A survey of open questions in adaptive therapy: Bridging mathematics and clinical translation

Adaptive therapy is a dynamic cancer treatment protocol that updates (or 'adapts') treatment decisions in anticipation of evolving tumor dynamics. This broad term encompasses many possible dynamic treatment protocols of patient-specific dose modulation or dose timing. Adaptive therapy maintains high levels of tumor burden to benefit from the competitive suppression of treatment-sensitive subpopulations on treatment-resistant subpopulations. This evolution-based approach to cancer treatment has been integrated into several ongoing or planned clinical trials, including treatment of metastatic castrate resistant prostate cancer, ovarian cancer, and BRAF-mutant melanoma. In the previous few decades, experimental and clinical investigation of adaptive therapy has progressed synergistically with mathematical and computational modeling. In this work, we discuss 11 open questions in cancer adaptive therapy mathematical modeling. The questions are split into three sections: (1) integrating the appropriate components into mathematical models (2) design and validation of dosing protocols, and (3) challenges and opportunities in clinical translation.

COVID-19 vaccine incentive scheduling using an optimally controlled reinforcement learning model

We model Covid-19 vaccine uptake as a reinforcement learning dynamic between two populations: the vaccine adopters, and the vaccine hesitant. Using data available from the Center for Disease Control (CDC), we estimate the payoff matrix governing the interaction between these two groups over time and show they are playing a Hawk-Dove evolutionary game with an internal evolutionarily stable Nash equilibrium (the asymptotic percentage of vaccinated in the population). We then ask whether vaccine adoption can be improved by implementing dynamic incentive schedules that reward/punish the vaccine hesitant, and if so, what schedules are optimal and how effective are they likely to be? When is the optimal time to start an incentive program, how large should the incentives be, and is there a point of diminishing returns? By using a tailored replicator dynamic reinforcement learning model together with optimal control theory, we show that well designed and timed incentive programs can improve vaccine uptake by shifting the Nash equilibrium upward in large populations, but only so much, and incentive sizes above a certain threshold show diminishing returns.

Mechanism underlying circRNA dysregulation in the TME of digestive system cancer

Circular RNAs (circRNAs) are a new series of noncoding RNAs (ncRNAs) that have been reported to be expressed in eukaryotic cells and have a variety of biological functions in the regulation of cancer pathogenesis and progression. The TME, as a microscopic ecological environment, consists of a variety of cells, including tumor cells, immune cells and other normal cells, ECM and a large number of signaling molecules. The crosstalk between circRNAs and the TME plays a complicated role in affecting the malignant behaviors of digestive system cancers. Herein, we summarize the mechanisms underlying aberrant circRNA expression in the TME of the digestive system cancers, including immune surveillance, angiogenesis, EMT, and ECM remodelling. The regulation of the TME by circRNA is expected to be a new therapeutic method.

Evolution-based mathematical models significantly prolong response to abiraterone in metastatic castrate-resistant prostate cancer and identify strategies to further improve outcomes

Background

Abiraterone acetate is an effective treatment for metastatic castrate-resistant prostate cancer (mCRPC), but evolution of resistance inevitably leads to progression. We present a pilot study in which abiraterone dosing is guided by evolution-informed mathematical models to delay onset of resistance.

Methods

In the study cohort, abiraterone was stopped when PSA was <50% of pretreatment value and resumed when PSA returned to baseline. Results are compared to a contemporaneous cohort who had >50% PSA decline after initial abiraterone administration and met trial eligibility requirements but chose standard of care (SOC) dosing.

Results

17 subjects were enrolled in the adaptive therapy group and 16 in the SOC group. All SOC subjects have progressed, but four patients in the study cohort remain stably cycling (range 53-70 months). The study cohort had significantly improved median time to progression (TTP; 33.5 months; p<0.001) and median overall survival (OS; 58.5 months; hazard ratio, 0.41, 95% confidence interval (CI), 0.20-0.83, p<0.001) compared to 14.3 and 31.3 months in the SOC cohort. On average, study subjects received no abiraterone during 46% of time on trial. Longitudinal trial data demonstrated the competition coefficient ratio (αRS/αSR) of sensitive and resistant populations, a critical factor in intratumoral evolution, was two- to threefold higher than pre-trial estimates. Computer simulations of intratumoral evolutionary dynamics in the four long-term survivors found that, due to the larger value for αRS/αSR, cycled therapy significantly decreased the resistant population. Simulations in subjects who progressed predicted further increases in OS could be achieved with prompt abiraterone withdrawal after achieving 50% PSA reduction.

Conclusions

Incorporation of evolution-based mathematical models into abiraterone monotherapy for mCRPC significantly increases TTP and OS. Computer simulations with updated parameters from longitudinal trial data can estimate intratumoral evolutionary dynamics in each subject and identify strategies to improve outcomes.

Funding

Moffitt internal grants and NIH/NCI U54CA143970-05 (Physical Science Oncology Network).

Optimal dynamic incentive scheduling for Hawk-Dove evolutionary games

The Hawk-Dove evolutionary game offers a paradigm of the trade-offs associated with aggressive and passive behaviors. When two (or more) populations of players compete, their success or failure is measured by their frequency in the population, and the system is governed by the replicator dynamics. We develop a time-dependent optimal-adaptive control theory for this dynamical system in which the entries of the payoff matrix are dynamically altered to produce control schedules that minimize and maximize the aggressive population through a finite-time cycle. These schedules provide upper and lower bounds on the outcomes for all possible strategies since they represent two extremizers of the cost function. We then adaptively extend the optimal control schedules over multiple cycles to produce absolute maximizers and minimizers for the system.

Drug-induced resistance evolution necessitates less aggressive treatment

Increasing body of experimental evidence suggests that anticancer and antimicrobial therapies may themselves promote the acquisition of drug resistance by increasing mutability. The successful control of evolving populations requires that such biological costs of control are identified, quantified and included to the evolutionarily informed treatment protocol. Here we identify, characterise and exploit a trade-off between decreasing the target population size and generating a surplus of treatment-induced rescue mutations. We show that the probability of cure is maximized at an intermediate dosage, below the drug concentration yielding maximal population decay, suggesting that treatment outcomes may in some cases be substantially improved by less aggressive treatment strategies. We also provide a general analytical relationship that implicitly links growth rate, pharmacodynamics and dose-dependent mutation rate to an optimal control law. Our results highlight the important, but often neglected, role of fundamental eco-evolutionary costs of control. These costs can often lead to situations, where decreasing the cumulative drug dosage may be preferable even when the objective of the treatment is elimination, and not containment. Taken together, our results thus add to the ongoing criticism of the standard practice of administering aggressive, high-dose therapies and motivate further experimental and clinical investigation of the mutagenicity and other hidden collateral costs of therapies.

Are Adaptive Chemotherapy Schedules Robust? A Three-Strategy Stochastic Evolutionary Game Theory Model

We investigate the robustness of adaptive chemotherapy schedules over repeated cycles and a wide range of tumor sizes. Using a non-stationary stochastic three-component fitness-dependent Moran process model (to track frequencies), we quantify the variance of the response to treatment associated with multidrug adaptive schedules that are designed to mitigate chemotherapeutic resistance in an idealized (well-mixed) setting. The finite cell (N tumor cells) stochastic process consists of populations of chemosensitive cells, chemoresistant cells to drug 1, and chemoresistant cells to drug 2, and the drug interactions can be synergistic, additive, or antagonistic. Tumor growth rates in this model are proportional to the average fitness of the tumor as measured by the three populations of cancer cells compared to a background microenvironment average value. An adaptive chemoschedule is determined by using the N→∞ limit of the finite-cell process (i.e., the adjusted replicator equations) which is constructed by finding closed treatment response loops (which we call evolutionary cycles) in the three component phase-space. The schedules that give rise to these cycles are designed to manage chemoresistance by avoiding competitive release of the resistant cell populations. To address the question of how these cycles perform in practice over large patient populations with tumors across a range of sizes, we consider the variances associated with the approximate stochastic cycles for finite N, repeating the idealized adaptive schedule over multiple periods. For finite cell populations, the distributions remain approximately multi-Gaussian in the principal component coordinates through the first three cycles, with variances increasing exponentially with each cycle. As the number of cycles increases, the multi-Gaussian nature of the distribution breaks down due to the fact that one of the three sub-populations typically saturates the tumor (competitive release) resulting in treatment failure. This suggests that to design an effective and repeatable adaptive chemoschedule in practice will require a highly accurate tumor model and accurate measurements of the sub-population frequencies or the errors will quickly (exponentially) degrade its effectiveness, particularly when the drug interactions are synergistic. Possible ways to extend the efficacy of the stochastic cycles in light of the computational simulations are discussed.

Role of synergy and antagonism in designing multidrug adaptive chemotherapy schedules

Chemotherapeutic resistance via the mechanism of competitive release of resistant tumor cell subpopulations is a major problem associated with cancer treatments and one of the main causes of tumor recurrence. Often, chemoresistance is mitigated by using multidrug schedules (two or more combination therapies) that can act synergistically, additively, or antagonistically on the heterogeneous population of cells as they evolve. In this paper, we develop a three-component evolutionary game theory model to design two-drug adaptive schedules that mitigate chemoresistance and delay tumor recurrence in an evolving collection of tumor cells with two resistant subpopulations and one chemosensitive population that has a higher baseline fitness but is not resistant to either drug. Using the nonlinear replicator dynamical system with a payoff matrix of Prisoner's Dilemma (PD) type (enforcing a cost to resistance), we investigate the nonlinear dynamics of this three-component system along with an additional tumor growth model whose growth rate is a function of the fitness landscape of the tumor cell populations. A key parameter determines whether the two drugs interact synergistically, additively, or antagonistically. We show that antagonistic drug interactions generally result in slower rates of adaptation of the resistant cells than synergistic ones, making them more effective in combating the evolution of resistance. We then design evolutionary cycles (closed loops) in the three-component phase space by shaping the fitness landscape of the cell populations (i.e., altering the evolutionary stable states of the game) using appropriately designed time-dependent schedules (adaptive therapy), altering the dosages and timing of the two drugs. We describe two key bifurcations associated with our drug interaction parameter which help explain why antagonistic interactions are more effective at controlling competitive release of the resistant population than synergistic interactions in the context of an evolving tumor.

Maximizing cooperation in the prisoner's dilemma evolutionary game via optimal control

The prisoner's dilemma (PD) game offers a simple paradigm of competition between two players who can either cooperate or defect. Since defection is a strict Nash equilibrium, it is an asymptotically stable state of the replicator dynamical system that uses the PD payoff matrix to define the fitness landscape of two interacting evolving populations. The dilemma arises from the fact that the average payoff of this asymptotically stable state is suboptimal. Coaxing the players to cooperate would result in a higher payoff for both. Here we develop an optimal control theory for the prisoner's dilemma evolutionary game in order to maximize cooperation (minimize the defector population) over a given cycle time T, subject to constraints. Our two time-dependent controllers are applied to the off-diagonal elements of the payoff matrix in a bang-bang sequence that dynamically changes the game being played by dynamically adjusting the payoffs, with optimal timing that depends on the initial population distributions. Over multiple cycles nT (n>1), the method is adaptive as it uses the defector population at the end of the nth cycle to calculate the optimal schedule over the n+1st cycle. The control method, based on Pontryagin's maximum principle, can be viewed as determining the optimal way to dynamically alter incentives and penalties in order to maximize the probability of cooperation in settings that track dynamic changes in the frequency of strategists, with potential applications in evolutionary biology, economics, theoretical ecology, social sciences, reinforcement learning, and other fields where the replicator system is used.

The Evolution and Ecology of Resistance in Cancer Therapy

Despite the continuous deployment of new treatment strategies and agents over many decades, most disseminated cancers remain fatal. Cancer cells, through their access to the vast information of the human genome, have a remarkable capacity to deploy adaptive strategies for even the most effective treatments. We note there are two critical steps in the clinical manifestation of treatment resistance. The first, which is widely investigated, requires molecular machinery necessary to eliminate the cytotoxic effect of the treatment. However, the emergence of a resistant phenotype is not in itself clinically significant. That is, resistant cells affect patient outcomes only when they succeed in the second step of resistance by proliferating into a sufficiently large population to allow tumor progression and treatment failure. Importantly, proliferation of the resistant phenotype is by no means certain and, in fact, depends on complex Darwinian dynamics governed by the costs and benefits of the resistance mechanisms in the context of the local environment and competing populations. Attempts to target the molecular machinery of resistance have had little clinical success largely because of the diversity within the human genome-therapeutic interruption of one mechanism simply results in its replacement by an alternative. Here we explore evolutionarily informed strategies (adaptive, double-bind, and extinction therapies) for overcoming treatment resistance that seek to understand and exploit the critical evolutionary dynamics that govern proliferation of the resistant phenotypes. In general, this approach has demonstrated that, while emergence of resistance mechanisms in cancer cells to every current therapy is inevitable, proliferation of the resistant phenotypes is not and can be delayed and even prevented with sufficient understanding of the underlying eco-evolutionary dynamics.

Antibiotics can be used to contain drug-resistant bacteria by maintaining sufficiently large sensitive populations

Standard infectious disease practice calls for aggressive drug treatment that rapidly eliminates the pathogen population before resistance can emerge. When resistance is absent, this elimination strategy can lead to complete cure. However, when resistance is already present, removing drug-sensitive cells as quickly as possible removes competitive barriers that may slow the growth of resistant cells. In contrast to the elimination strategy, a containment strategy aims to maintain the maximum tolerable number of pathogens, exploiting competitive suppression to achieve chronic control. Here, we combine in vitro experiments in computer-controlled bioreactors with mathematical modeling to investigate whether containment strategies can delay failure of antibiotic treatment regimens. To do so, we measured the "escape time" required for drug-resistant Escherichia coli populations to eclipse a threshold density maintained by adaptive antibiotic dosing. Populations containing only resistant cells rapidly escape the threshold density, but we found that matched resistant populations that also contain the maximum possible number of sensitive cells could be contained for significantly longer. The increase in escape time occurs only when the threshold density-the acceptable bacterial burden-is sufficiently high, an effect that mathematical models attribute to increased competition. The findings provide decisive experimental confirmation that maintaining the maximum number of sensitive cells can be used to contain resistance when the size of the population is sufficiently large.

Towards Multidrug Adaptive Therapy

A new ecologically inspired paradigm in cancer treatment known as "adaptive therapy" capitalizes on competitive interactions between drug-sensitive and drug-resistant subclones. The goal of adaptive therapy is to maintain a controllable stable tumor burden by allowing a significant population of treatment-sensitive cells to survive. These, in turn, suppress proliferation of the less-fit resistant populations. However, there remain several open challenges in designing adaptive therapies, particularly in extending these therapeutic concepts to multiple treatments. We present a cancer treatment case study (metastatic castrate-resistant prostate cancer) as a point of departure to illustrate three novel concepts to aid the design of multidrug adaptive therapies. First, frequency-dependent "cycles" of tumor evolution can trap tumor evolution in a periodic, controllable loop. Second, the availability and selection of treatments may limit the evolutionary "absorbing region" reachable by the tumor. Third, the velocity of evolution significantly influences the optimal timing of drug sequences. These three conceptual advances provide a path forward for multidrug adaptive therapy. SIGNIFICANCE: Driving tumor evolution into periodic, repeatable treatment cycles provides a path forward for multidrug adaptive therapy.