Comparison of some control strategies for three-compartment PK/PD models

Hierarchical Deep Reinforcement Learning-Based Propofol Infusion Assistant Framework in Anesthesia

This article aims to provide a hierarchical reinforcement learning (RL)-based solution to the automated drug infusion field. The learning policy is divided into the tasks of: 1) learning trajectory generative model and 2) planning policy model. The proposed deep infusion assistant policy gradient (DIAPG) model draws inspiration from adversarial autoencoders (AAEs) and learns latent representations of hypnotic depth trajectories. Given the trajectories drawn from the generative model, the planning policy infers a dose of propofol for stable sedation of a patient under total intravenous anesthesia (TIVA) using propofol and remifentanil. Through extensive evaluation, the DIAPG model can effectively stabilize bispectral index (BIS) and effect site concentration given a potentially time-varying target sequence. The proposed DIAPG shows an increased performance of 530% and 15% when a human expert and a standard reinforcement algorithm are used to infuse drugs, respectively.

Experiment design for nonparametric models based on minimizing Bayes Risk: application to voriconazole¹

An experimental design approach is presented for individualized therapy in the special case where the prior information is specified by a nonparametric (NP) population model. Here, a NP model refers to a discrete probability model characterized by a finite set of support points and their associated weights. An important question arises as to how to best design experiments for this type of model. Many experimental design methods are based on Fisher information or other approaches originally developed for parametric models. While such approaches have been used with some success across various applications, it is interesting to note that they largely fail to address the fundamentally discrete nature of the NP model. Specifically, the problem of identifying an individual from a NP prior is more naturally treated as a problem of classification, i.e., to find a support point that best matches the patient's behavior. This paper studies the discrete nature of the NP experiment design problem from a classification point of view. Several new insights are provided including the use of Bayes Risk as an information measure, and new alternative methods for experiment design. One particular method, denoted as MMopt (multiple-model optimal), will be examined in detail and shown to require minimal computation while having distinct advantages compared to existing approaches. Several simulated examples, including a case study involving oral voriconazole in children, are given to demonstrate the usefulness of MMopt in pharmacokinetics applications.

SEDATION OF SIMULATED ICU PATIENTS USING REINFORCEMENT LEARNING BASED CONTROL

The Intensive Care Unit (ICU) is a challenging environment to both patient and caregiver. Continued shortages in staffing increase risk to patients. To evaluate the use of intelligent systems in the improvement of patient care, an intelligent agent was developed to regulate ICU patient sedation. A temporal differencing form of reinforcement learning was used to train the agent in the administration of intravenous propofol in simulated ICU patients. The agent utilized a well-studied pharmacokinetic model to calculate the distribution of drug within the patient. Pharmacodynamics were then estimated for the drug effect. A derivative of the electroencephalograms, the bispectral index, served as the system control variable. The agent demonstrated satisfactory control of the simulated patient's consciousness level in static and dynamic setpoint conditions. The agent demonstrated superior stability and responsiveness when compared to a well-tuned PID controller, the control method of choice in closed-loop sedation control literature.

Individualization of pharmacological anemia management using reinforcement learning

Effective management of anemia due to renal failure poses many challenges to physicians. Individual response to treatment varies across patient populations and, due to the prolonged character of the therapy, changes over time. In this work, a Reinforcement Learning-based approach is proposed as an alternative method for individualization of drug administration in the treatment of renal anemia. Q-learning, an off-policy approximate dynamic programming method, is applied to determine the proper dosing strategy in real time. Simulations compare the proposed methodology with the currently used dosing protocol. Presented results illustrate the ability of the proposed method to achieve the therapeutic goal for individuals with different response characteristics and its potential to become an alternative to currently used techniques.

Propofol-sufentanil anesthesia for thyroid surgery: optimal concentrations for hemodynamic and electroencephalogram stability, and recovery features

Hypnotics and opioids interact synergistically to block responses to surgery and different dose combinations may be used to provide adequate anesthesia. In this study, we sought to determine the optimal concentrations of propofol and sufentanil, given by target-controlled infusions, to ensure hemodynamic stability, adequate hypnosis (assessed by electroencephalogram bispectral index), and fast recovery for a moderately painful operation. Forty-five patients, ASA physical status I or II, undergoing thyroidectomy, were randomly assigned to a sufentanil target concentration (STC) that was maintained throughout surgery (0.1, 0.2, or 0.3 ng/mL). The propofol target concentration was adjusted to keep mean arterial blood pressure within 30% of a reference value, and bispectral index between 40 and 60. Adequate anesthesia was obtained in all groups. Hypertension and clinically dangerous movements were more frequent with the small STC, and hypotension requiring treatment was more frequent with the large STC. Propofol target concentration during surgery decreased significantly with increasing STC (median at thyroid removal 5.0, 4.0, and 2.5 microg/mL, respectively) as well as the propofol consumption (740, 668, 474 mg/h). The 0.3 ng/mL STC significantly delayed the return of spontaneous breathing.

IMPLICATIONS

Given as a target-controlled infusion for thyroid surgery, sufentanil 0.3 ng/mL for intubation and 0.2 ng/mL during surgery, combined with propofol 4 microg/mL (corresponding to a maintenance infusion rate of approximately 7-10 mg. kg(-1). h(-1)), is recommended to ensure both optimal intraoperative stability and fast recovery.

A regulatory perspective on pharmacokinetic/pharmacodynamic modelling

We present an integrated summary from statistical and pharmacological perspectives of pharmacokinetic/pharmacodynamic (PK/PD) modelling and its use in drug development and regulation for guiding appropriate dosing. An overview of the technical aspects of PK/PD modelling describes how structural models are constructed and refined using pharmacokinetic and pharmacodynamic principles and how random effects models are used to account for individual differences in desired (and undesired) responses due to patient characteristics. Lastly, we describe applications of PK/PD modelling for the purposes of drug labelling, for resolving a safety concern, and for improving therapeutic monitoring of anaesthetic depth during surgery.