Deep Q-learning for the selection of optimal isocratic scouting runs in liquid chromatography

Deep reinforcement learning in chemistry: A review

Reinforcement learning (RL) has been applied to various domains in computational chemistry and has found wide-spread success. In this review, we first motivate the application of RL to chemistry and list some broad application domains, for example, molecule generation, geometry optimization, and retrosynthetic pathway search. We set up some of the formalism associated with reinforcement learning that should help the reader translate their chemistry problems into a form where RL can be used to solve them. We then discuss the solution formulations and algorithms proposed in recent literature for these problems, the advantages of one over the other, together with the necessary details of the RL algorithms they employ. This article should help the reader understand the state of RL applications in chemistry, learn about some relevant actively-researched open problems, gain insight into how RL can be used to approach them and hopefully inspire innovative RL applications in Chemistry.

Deep reinforcement learning for the direct optimization of gradient separations in liquid chromatography

While Reinforcement Learning (RL) has already proven successful in performing complex tasks, such as controlling large-scale epidemics, mitigating influenza and playing computer games beyond expert level, it is currently largely unexplored in the field of separation sciences. This paper therefore aims to introduce RL, specifically proximal policy optimization (PPO), in liquid chromatography, and evaluate whether it can be trained to optimize separations directly, based solely on the outcome of a single generic separation as input, and a reward signal based on the resolution between peak pairs (taking a value between [-1,1]). More specifically, PPO algorithms or agents were trained to select linear (1-segment) or multi-segment (2-, 3-, or 16-segment) gradients in 1 experiment, based on the outcome of an initial, generic linear gradient (ϕstart=0.3, ϕend=1.0, and tg=20min), to improve separations. The size of the mixtures to be separated varied between 10 and 20 components. Furthermore, two agents, selecting 16-segment gradients, were trained to perform this optimization using either 2 or 3 experiments, in sequence, to investigate whether the agents could improve separations further, based on previous outcomes. Results showed that the PPO agent can improve separations given the outcome of one generic scouting run as input, by selecting ϕ-programs tailored to the mixture under consideration. Allowing agents more freedom in selecting multi-segment gradients increased the reward from 0.891 to 0.908 on average; and allowing the agents to perform an additional experiment increased the reward from 0.908 to 0.918 on average. Finally, the agent outperformed random experiments as well as standard experiments (ϕstart=0.0, ϕend=1.0, and tg=20min) significantly; as random experiments resulted in average rewards between 0.220 and 0.283, and standard experiments resulted in average rewards of 0.840. In conclusion, while there is room for improvement, the results demonstrate the potential of RL in chromatography and present an interesting future direction for the automated optimization of separations.

Automated method development in high-pressure liquid chromatography

Method development in liquid chromatography is a crucial step in the optimization of analytical separations for various applications. However, it is often a challenging endeavour due to its time-consuming, resource intensive and costly nature, which is further hampered by its complexity requiring highly skilled and experienced scientists. This review presents an examination of the methods that are required for a completely automated method development procedure in liquid chromatography, aimed at taking the human out of the decision loop. Some of the presented approaches have recently witnessed an important increase in interest as they offer the promise to facilitate, streamline and speed up the method development process. The review first discusses the mathematical description of the separation problem by means of multi-criteria optimization functions. Two different strategies to resolve this optimization are then presented; an experimental and a model-based approach. Additionally, methods for automated peak detection and peak tracking are reviewed, which, upon integration in an instrument, allow for a completely closed-loop method development process. For each of these approaches, various currently applied methods are presented, recent trends and approaches discussed, short-comings pointed out, and future prospects highlighted.

A perspective on the use of deep deterministic policy gradient reinforcement learning for retention time modeling in reversed-phase liquid chromatography

Artificial intelligence and machine learning techniques are increasingly used for different tasks related to method development in liquid chromatography. In this study, the possibilities of a reinforcement learning algorithm, more specifically a deep deterministic policy gradient algorithm, are evaluated for the selection of scouting runs for retention time modeling. As a theoretical exercise, it is investigated whether such an algorithm can be trained to select scouting runs for any compound of interest allowing to retrieve its correct retention parameters for the three-parameter Neue-Kuss retention model. It is observed that three scouting runs are generally sufficient to retrieve the retention parameters with an accuracy (mean relative percentage error MRPE) of 1 % or less. When given the opportunity to select additional scouting runs, this does not lead to a significantly improved accuracy. It is also observed that the agent tends to give preference to isocratic scouting runs for retention time modeling, and is only motivated towards selecting gradient scouting runs when penalized (strongly) for large analysis/gradient times. This seems to reinforce the general power and usefulness of isocratic scouting runs for retention time modeling. Finally, the best results (lowest MRPE) are obtained when the agent manages to retrieve retention time data for % ACN at elution of the compound under consideration that spread the entire relevant range of ACN (5 % ACN to 95 % ACN) as well as possible, i.e., resulting in retention data at a low, intermediate and high % ACN. Based on the obtained results, we believe reinforcement learning holds great potential to automate and rationalize method development in liquid chromatography in the future.

Machine learning in bioprocess development: from promise to practice

Fostered by novel analytical techniques, digitalization, and automation, modern bioprocess development provides large amounts of heterogeneous experimental data, containing valuable process information. In this context, data-driven methods like machine learning (ML) approaches have great potential to rationally explore large design spaces while exploiting experimental facilities most efficiently. Herein we demonstrate how ML methods have been applied so far in bioprocess development, especially in strain engineering and selection, bioprocess optimization, scale-up, monitoring, and control of bioprocesses. For each topic, we will highlight successful application cases, current challenges, and point out domains that can potentially benefit from technology transfer and further progress in the field of ML.

Prediction of the performance of pre-packed purification columns through machine learning

Pre-packed columns have been increasingly used in process development and biomanufacturing thanks to their ease of use and consistency. Traditionally, packing quality is predicted through rate models, which require extensive calibration efforts through independent experiments to determine relevant mass transfer and kinetic rate constants. Here we propose machine learning as a complementary predictive tool for column performance. A machine learning algorithm, extreme gradient boosting, was applied to a large data set of packing quality (plate height and asymmetry) for pre-packed columns as a function of quantitative parameters (column length, column diameter, and particle size) and qualitative attributes (backbone and functional mode). The machine learning model offered excellent predictive capabilities for the plate height and the asymmetry (90 and 93%, respectively), with packing quality strongly influenced by backbone (∼70% relative importance) and functional mode (∼15% relative importance), well above all other quantitative column parameters. The results highlight the ability of machine learning to provide reliable predictions of column performance from simple, generic parameters, including strategic qualitative parameters such as backbone and functionality, usually excluded from quantitative considerations. Our results will guide further efforts in column optimization, for example, by focusing on improvements of backbone and functional mode to obtain optimized packings.

Accuracy of retention model parameters obtained from retention data in liquid chromatography

In liquid chromatography (LC), it is often very useful to have an accurate model of the retention factor, k, over a wide range of isocratic elution conditions. In principle, the parameters of a retention model can be obtained by fitting either isocratic or gradient retention factor data. However, in spite of many of our own attempts to accurately predict isocratic k values using retention models trained with gradient retention data, this has not worked in our hands. In the present study we have used synthetic isocratic and gradient retention data for small molecules under reversed-phase LC conditions. This allows us to discover challenges associated with predicting isocratic k's without the confounding influences of experimental issues that are difficult to model or eliminate. The results indicate that it is not currently possible to consistently predict isocratic retention factors for small molecules with accuracies better than 10%, even when using synthetic gradient retention data. Two distinct challenges in fitting gradient retention data were identified: 1) a lack of 'uniqueness' in the parameters; and 2) an inability to find the global optimum fit in a complex fitting landscape. Working with experimental data where measurement noise is unavoidable will only make the accuracy worse. This article is protected by copyright. All rights reserved.

Bayesian optimization of comprehensive two-dimensional liquid chromatography separations

Comprehensive two-dimensional liquid chromatography (LC×LC), is a powerful, emerging separation technique in analytical chemistry. However, as many instrumental parameters need to be tuned, the technique is troubled by lengthy method development. To speed up this process, we applied a Bayesian optimization algorithm. The algorithm can optimize LC×LC method parameters by maximizing a novel chromatographic response function based on the concept of connected components of a graph. The algorithm was benchmarked against a grid search (11,664 experiments) and a random search algorithm on the optimization of eight gradient parameters for four different samples of 50 compounds. The worst-case performance of the algorithm was investigated by repeating the optimization loop for 100 experiments with random starting experiments and seeds. Given an optimization budget of 100 experiments, the Bayesian optimization algorithm generally outperformed the random search and often improved upon the grid search. Moreover, the Bayesian optimization algorithm offered a considerably more sample-efficient alternative to grid searches, as it found similar optima to the grid search in far fewer experiments (a factor of 16-100 times less). This could likely be further improved by a more informed choice of the initialization experiments, which could be provided by the analyst's experience or smarter selection procedures. The algorithm allows for expansion to other method parameters (e.g., temperature, flow rate, etc.) and unlocks closed-loop automated method development.

Testing experimental designs in liquid chromatography (II): Influence of the design geometry on the prediction performance of retention models

In liquid chromatography, the reliability of predictions carried out with retention models depends critically on the quality of the training experimental design. The search of the best design is more complex when gradient runs are used instead of isocratic experiments. In Part I of this work (JCA 1624 (2020) 461180), a general methodology based on the error propagation theory was developed and validated for assessing the quality of training designs involving gradients. The treatment relates the mathematical properties of a retention model with the geometry of the training designs and their subsequent predictions. In that work, only five usual designs were considered. Part II investigates in detail the effects on predictions when the features of the training design (number and distribution of the experiments, initial and final modifier content, gradient slope(s), and location of gradient nodes and pulses) are varied systematically. Several groups of related designs containing one or more isocratic steps, linear or multi-linear gradients, or mixed isocratic/gradient runs, among others (in total 38 designs) were evaluated. Box and whiskers and triple plots of expected relative uncertainties were used to evidence the differences in prediction performance. The purpose was to give recommendations to construct designs with good prediction performance. The best designs sample (considering all runs) concentrations as diverse as possible, at any gradient time.

Comparison of the Fitting Performance of Retention Models and Elution Strength Behaviour in Hydrophilic-Interaction and Reversed-Phase Liquid Chromatography

Hydrophilic interaction liquid chromatography (HILIC) is able to separate from polar to highly polar solutes, using similar eluents to those in the reversed-phase mode (RPLC) and a polar stationary phase, where water is adsorbed onto its surface. It is widely accepted that multiple modes of interaction take place in the HILIC environment, which can be far more complex than the interactions in an RPLC column. The behaviour in HILIC should be adequately modelled to predict the retention with optimisation purposes and improve the understanding on retention mechanisms, as is the case for RPLC. In this work, the prediction performance of several retention models is studied for seven HILIC columns (underivatised silica, and silica containing diol, amino and sulfobetaine functional groups, together with three columns recently manufactured with neutral, anionic, and cationic character), using uracil and six polar nucleosides (adenosine, cytidine, guanosine, thymidine, uridine, and xanthosine) as probe compounds. The results in HILIC are compared with those that were offered by the elution of several polar sulphonamides and diuretics analysed with two C18 columns (Chromolith Speed ROD and Zorbax Eclipse XDB). It is shown that eight retention models, which only consider partitioning or both partitioning and adsorption, give similar good accuracy in predictions for both HILIC and RPLC columns. However, the study on the elution strength behaviour, at varying mobile phase composition, reveals similarities (or differences) between RPLC and HILIC columns of diverse nature. The particular behaviour for the HILIC and RPLC columns was also revealed when the retention, in both modes, was fitted to a model that describes the change in the elution strength with the modifier concentration.