Reinforcement learning for batch bioprocess optimization

Machine learning: an advancement in biochemical engineering

One of the most remarkable techniques recently introduced into the field of bioprocess engineering is machine learning. Bioprocess engineering has drawn much attention due to its vast application in different domains like biopharmaceuticals, fossil fuel alternatives, environmental remediation, and food and beverage industry, etc. However, due to their unpredictable mechanisms, they are very often challenging to optimize. Furthermore, biological systems are extremely complicated; hence, machine learning algorithms could potentially be utilized to improve and build new biotechnological processes. Gaining insight into the fundamental mathematical understanding of commonly used machine learning algorithms, including Support Vector Machine, Principal Component Analysis, Partial Least Squares and Reinforcement Learning, the present study aims to discuss various case studies related to the application of machine learning in bioprocess engineering. Recent advancements as well as challenges posed in this area along with their potential solutions are also presented.

Multiobjective Optimization of Papermaking Wastewater Treatment Processes under Economic, Energy, and Environmental Goals

Due to the heterogeneity of recycled paper materials and the production conditions, pollutants in papermaking wastewater fluctuate sharply over time. Quality control of the papermaking wastewater treatment process (PWTP) is challenging and costly. As regulations are also growing about the environmental effects of the PWTP on greenhouse gas (GHG) emission, energy consumption, etc., the PWTP formulates a complex multiobjective optimization problem. This research established a multiagent deep reinforcement learning framework to simultaneously optimize process cost, energy consumption, and GHG emission in the PWTP, subjected to the effluent quality, to realize economic, energy, and environmental (3E) goals. The biological treatment process of wastewater in paper mills was simulated using benchmark simulation model no. 1 (BSM1). The data generated based on the BSM manual was utilized for model training, and real data acquired from a local papermaking factory was used to estimate the model performance. The results show that the proposed method outperforms conventional techniques in identifying the best control strategies for multiple targets.

Machine learning in bioprocess development: from promise to practice

Fostered by novel analytical techniques, digitalization, and automation, modern bioprocess development provides large amounts of heterogeneous experimental data, containing valuable process information. In this context, data-driven methods like machine learning (ML) approaches have great potential to rationally explore large design spaces while exploiting experimental facilities most efficiently. Herein we demonstrate how ML methods have been applied so far in bioprocess development, especially in strain engineering and selection, bioprocess optimization, scale-up, monitoring, and control of bioprocesses. For each topic, we will highlight successful application cases, current challenges, and point out domains that can potentially benefit from technology transfer and further progress in the field of ML.

Advances in bioreactor control for production of biotherapeutic products

Advanced control strategies are well established in chemical, pharmaceutical, and food processing industries. Over the past decade, the application of these strategies is being explored for control of bioreactors for manufacturing of biotherapeutics. Most of the industrial bioreactor control strategies apply classical control techniques, with the control system designed for the facility at hand. However, with the recent progress in sensors, machinery, and industrial internet of things, and advancements in deeper understanding of the biological processes, coupled with the requirement of flexible production, the need to develop a robust and advanced process control system that can ease process intensification has emerged. This has further fuelled the development of advanced monitoring approaches, modeling techniques, process analytical technologies, and soft sensors. It is seen that proper application of these concepts can significantly improve bioreactor process performance, productivity, and reproducibility. This review is on the recent advancements in bioreactor control and its related aspects along with the associated challenges. This study also offers an insight into the future prospects for development of control strategies that can be designed for industrial-scale production of biotherapeutic products.

Opportunities and challenges of machine learning in bioprocesses: Categorization from different perspectives and future direction

Recent advances in machine learning (ML) have revolutionized an extensive range of research and industry fields by successfully addressing intricate problems that cannot be resolved with conventional approaches. However, low interpretability and incompatibility make it challenging to apply ML to complicated bioprocesses, which rely on the delicate metabolic interplay among living cells. This overview attempts to delineate ML applications to bioprocess from different perspectives, and their inherent limitations (i.e., uncertainties in prediction) were then discussed with unique attempts to supplement the ML models. A clear classification can be made depending on the purpose of the ML (supervised vs unsupervised) per application, as well as on their system boundaries (engineered vs natural). Although a limited number of hybrid approaches with meaningful outcomes (e.g., improved accuracy) are available, there is still a need to further enhance the interpretability, compatibility, and user-friendliness of ML models.

Machine learning in fermentative biohydrogen production: Advantages, challenges, and applications

Hydrogen can be produced in an environmentally friendly manner through biological processes using a variety of organic waste and biomass as feedstock. However, the complexity of biological processes limits their predictability and reliability, which hinders the scale-up and dissemination. This article reviews contemporary research and perspectives on the application of machine learning in biohydrogen production technology. Several machine learning algorithems have recently been implemented for modeling the nonlinear and complex relationships among operational and performance parameters in biohydrogen production as well as predicting the process performance and microbial population dynamics. Reinforced machine learning methods exhibited precise state prediction and retrieved the underlying kinetics effectively. Machine-learning based prediction was also improved by using microbial sequencing data as input parameters. Further research on machine learning could be instrumental in designing a process control tool to maintain reliable hydrogen production performance and identify connection between the process performance and the microbial population.

Accelerating reinforcement learning with case-based model-assisted experience augmentation for process control

In the context of intelligent manufacturing in the process industry, traditional model-based optimization control methods cannot adapt to the situation of drastic changes in working conditions or operating modes. Reinforcement learning (RL) directly achieves the control objective by interacting with the environment, and has significant advantages in the presence of uncertainty since it does not require an explicit model of the operating plant. However, most RL algorithms fail to retain transfer learning capabilities in the presence of mode variation, which becomes a practical obstacle to industrial process control applications. To address these issues, we design a framework that uses local data augmentation to improve the training efficiency and transfer learning (adaptability) performance. Therefore, this paper proposes a novel RL control algorithm, CBR-MA-DDPG, organically integrating case-based reasoning (CBR), model-assisted (MA) experience augmentation, and deep deterministic policy gradient (DDPG). When the operating mode changes, CBR-MA-DDPG can quickly adapt to the varying environment and achieve the desired control performance within several training episodes. Experimental analyses on a continuous stirred tank reactor (CSTR) and an organic Rankine cycle (ORC) demonstrate the superiority of the proposed method in terms of both adaptability and control performance/robustness. The results show that the control performance of the CBR-MA-DDPG agent outperforms the conventional PI and MPC control schemes, and that it has higher training efficiency than the state-of-the-art DDPG, TD3, and PPO algorithms in transfer learning scenarios with mode shift situations.

Deep reinforcement learning for automated search of model parameters: photo-fenton wastewater disinfection case study

Numerical optimization solves problems that are analytically intractable at the cost of arriving at a sufficiently good but rarely optimal solution. To maximize the result, optimization algorithms are run with the guidance and supervision of a human, usually an expert in the problem. Recent advances in deep reinforcement learning motivate interest in an artificial agent capable of learning to do the expert’s task. Specifically, we present a proximal policy optimization agent that learns to optimize in a real case study such as the modeling of the photo-fenton disinfection process, which involves a number of parameters that have to be adjusted to minimize the error of the model with respect to the experimental data collected in several trials. The expert spends an average of 4 h to find a suitable set of parameters. On the other hand, the agent we present does not require a human expert to guide or validate the optimization procedure and achieves similar results in $$2.5\times$$ 2.5 × less time.

Strain design optimization using reinforcement learning

Engineered microbial cells present a sustainable alternative to fossil-based synthesis of chemicals and fuels. Cellular synthesis routes are readily assembled and introduced into microbial strains using state-of-the-art synthetic biology tools. However, the optimization of the strains required to reach industrially feasible production levels is far less efficient. It typically relies on trial-and-error leading into high uncertainty in total duration and cost. New techniques that can cope with the complexity and limited mechanistic knowledge of the cellular regulation are called for guiding the strain optimization.

In this paper, we put forward a multi-agent reinforcement learning (MARL) approach that learns from experiments to tune the metabolic enzyme levels so that the production is improved. Our method is model-free and does not assume prior knowledge of the microbe’s metabolic network or its regulation. The multi-agent approach is well-suited to make use of parallel experiments such as multi-well plates commonly used for screening microbial strains.

We demonstrate the method’s capabilities using the genome-scale kinetic model of Escherichia coli, k-ecoli457, as a surrogate for an in vivo cell behaviour in cultivation experiments. We investigate the method’s performance relevant for practical applicability in strain engineering i.e. the speed of convergence towards the optimum response, noise tolerance, and the statistical stability of the solutions found. We further evaluate the proposed MARL approach in improving L-tryptophan production by yeast Saccharomyces cerevisiae, using publicly available experimental data on the performance of a combinatorial strain library.

Overall, our results show that multi-agent reinforcement learning is a promising approach for guiding the strain optimization beyond mechanistic knowledge, with the goal of faster and more reliably obtaining industrially attractive production levels.

Machine learning approaches for biomolecular, biophysical, and biomaterials research

A fluent conversation with a virtual assistant, person-tailored news feeds, and deep-fake images created within seconds-all those things that have been unthinkable for a long time are now a part of our everyday lives. What these examples have in common is that they are realized by different means of machine learning (ML), a technology that has fundamentally changed many aspects of the modern world. The possibility to process enormous amount of data in multi-hierarchical, digital constructs has paved the way not only for creating intelligent systems but also for obtaining surprising new insight into many scientific problems. However, in the different areas of biosciences, which typically rely heavily on the collection of time-consuming experimental data, applying ML methods is a bit more challenging: Here, difficulties can arise from small datasets and the inherent, broad variability, and complexity associated with studying biological objects and phenomena. In this Review, we give an overview of commonly used ML algorithms (which are often referred to as "machines") and learning strategies as well as their applications in different bio-disciplines such as molecular biology, drug development, biophysics, and biomaterials science. We highlight how selected research questions from those fields were successfully translated into machine readable formats, discuss typical problems that can arise in this context, and provide an overview of how to resolve those encountered difficulties.

Industrial data science – a review of machine learning applications for chemical and process industries

Understand and optimize industrial processes via machine learning and chemical engineering principles.

Optimal control towards sustainable wastewater treatment plants based on multi-agent reinforcement learning

Wastewater treatment plants (WWTPs) are designed to eliminate pollutants and alleviate environmental pollution resulting from human activities. However, the construction and operation of WWTPs consume resources, emit greenhouse gases (GHGs) and produce residual sludge, thus require further optimization. WWTPs are complex to control and optimize because of high non-linearity and variation. This study used a novel technique, multi-agent deep reinforcement learning (MADRL), to simultaneously optimize dissolved oxygen (DO) and chemical dosage in a WWTP. The reward function was specially designed from life cycle perspective to achieve sustainable optimization. Five scenarios were considered: baseline, three different effluent quality and cost-oriented scenarios. The result shows that optimization based on LCA has lower environmental impacts compared to baseline scenario, as cost, energy consumption and greenhouse gas emissions reduce to 0.890 CNY/m³-ww, 0.530 kWh/m³-ww, 2.491 kg CO²-eq/m³-ww respectively. The cost-oriented control strategy exhibits comparable overall performance to the LCA-driven strategy since it sacrifices environmental benefits but has lower cost as 0.873 CNY/m³-ww. It is worth mentioning that the retrofitting of WWTPs based on resources should be implemented with the consideration of impact transfer. Specifically, LCA-SW scenario decreases 10 kg PO⁴-eq in eutrophication potential compared to the baseline within 10 days, while significantly increases other indicators. The major contributors of each indicator are identified for future study and improvement. Last, the authors discussed that novel dynamic control strategies required advanced sensors or a large amount of data, so the selection of control strategies should also consider economic and ecological conditions. In a nutshell, there are still limitations of this work and future studies are required.

Generic and specific recurrent neural network models: Applications for large and small scale biopharmaceutical upstream processes

The calculation of temporally varying upstream process outcomes is a challenging task. Over the last years, several parametric, semi-parametric as well as non-parametric approaches were developed to provide reliable estimates for key process parameters. We present generic and product-specific recurrent neural network (RNN) models for the computation and study of growth and metabolite-related upstream process parameters as well as their temporal evolution. Our approach can be used for the control and study of single product-specific large-scale manufacturing runs as well as generic small-scale evaluations for combined processes and products at development stage. The computational results for the product titer as well as various major upstream outcomes in addition to relevant process parameters show a high degree of accuracy when compared to experimental data and, accordingly, a reasonable predictive capability of the RNN models. The calculated values for the root-mean squared errors of prediction are significantly smaller than the experimental standard deviation for the considered process run ensembles, which highlights the broad applicability of our approach. As a specific benefit for platform processes, the generic RNN model is also used to simulate process outcomes for different temperatures in good agreement with experimental results. The high level of accuracy and the straightforward usage of the approach without sophisticated parameterization and recalibration procedures highlight the benefits of the RNN models, which can be regarded as promising alternatives to existing parametric and semi-parametric methods.

In silico, in vitro, and in vivo machine learning in synthetic biology and metabolic engineering

Among the main learning methods reviewed in this study and used in synthetic biology and metabolic engineering are supervised learning, reinforcement and active learning, and in vitro or in vivo learning. In the context of biosynthesis, supervised machine learning is being exploited to predict biological sequence activities, predict structures and engineer sequences, and optimize culture conditions. Active and reinforcement learning methods use training sets acquired through an iterative process generally involving experimental measurements. They are applied to design, engineer, and optimize metabolic pathways and bioprocesses. The nascent but promising developments with in vitro and in vivo learning comprise molecular circuits performing simple tasks such as pattern recognition and classification.

Paradigm Shift: The Promise of Deep Learning in Molecular Systems Engineering and Design

The application of deep learning to a diverse array of research problems has accelerated progress across many fields, bringing conventional paradigms to a new intelligent era. Just as the roles of instrumentation in the old chemical revolutions, we reinforce the necessity for integrating deep learning in molecular systems engineering and design as a transformative catalyst towards the next chemical revolution. To meet such research needs, we summarize advances and progress across several key elements of molecular systems: molecular representation, property estimation, representation learning, and synthesis planning. We further spotlight recent advances and promising directions for several deep learning architectures, methods, and optimization platforms. Our perspective is of interest to both computational and experimental researchers as it aims to chart a path forward for cross-disciplinary collaborations on synthesizing knowledge from available chemical data and guiding experimental efforts.

Hybrid Process Models in Electrochemical Syntheses under Deep Uncertainty

Chemical process engineering and machine learning are merging rapidly, and hybrid process models have shown promising results in process analysis and process design. However, uncertainties in first-principles process models have an adverse effect on extrapolations and inferences based on hybrid process models. Parameter sensitivities are an essential tool to understand better the underlying uncertainty propagation and hybrid system identification challenges. Still, standard parameter sensitivity concepts may fail to address comprehensive parameter uncertainty problems, i.e., deep uncertainty with aleatoric and epistemic contributions. This work shows a highly effective and reproducible sampling strategy to calculate simulation uncertainties and global parameter sensitivities for hybrid process models under deep uncertainty. We demonstrate the workflow with two electrochemical synthesis simulation studies, including the synthesis of furfuryl alcohol and 4-aminophenol. Compared with Monte Carlo reference simulations, the CPU-time was significantly reduced. The general findings of the hybrid model sensitivity studies under deep uncertainty are twofold. First, epistemic uncertainty has a significant effect on uncertainty analysis. Second, the predicted parameter sensitivities of the hybrid process models add value to the interpretation and analysis of the hybrid models themselves but are not suitable for predicting the real process/full first-principles process model’s sensitivities.

Potential of Integrating Model-Based Design of Experiments Approaches and Process Analytical Technologies for Bioprocess Scale-Down

Typically, bioprocesses on an industrial scale are dynamic systems with a certain degree of variability, system inhomogeneities, and even population heterogeneities. Therefore, the scaling of such processes from laboratory to industrial scale and vice versa is not a trivial task. Traditional scale-down methodologies consider several technical parameters, so that systems on the laboratory scale tend to qualitatively reflect large-scale effects, but not the dynamic situation in an industrial bioreactor over the entire process, from the perspective of a cell. Supported by the enormous increase in computing power, the latest scientific focus is on the application of dynamic models, in combination with computational fluid dynamics to quantitatively describe cell behavior. These models allow the description of possible cellular lifelines which in turn can be used to derive a regime analysis for scale-down experiments. However, the approaches described so far, which were for a very few process examples, are very labor- and time-intensive and cannot be validated easily. In parallel, alternatives have been developed based on the description of the industrial process with hybrid process models, which describe a process mechanistically as far as possible in order to determine the essential process parameters with their respective variances. On-line analytical methods allow the characterization of population heterogeneity directly in the process. This detailed information from the industrial process can be used in laboratory screening systems to select relevant conditions in which the cell and process related parameters reflect the situation in the industrial scale. In our opinion, these technologies, which are available in research for modeling biological systems, in combination with process analytical techniques are so far developed that they can be implemented in industrial routines for faster development of new processes and optimization of existing ones.

Comparing Reinforcement Learning Methods for Real-Time Optimization of a Chemical Process

One popular method for optimizing systems, referred to as ANN-PSO, uses an artificial neural network (ANN) to approximate the system and an optimization method like particle swarm optimization (PSO) to select inputs. However, with reinforcement learning developments, it is important to compare ANN-PSO to newer algorithms, like Proximal Policy Optimization (PPO). To investigate ANN-PSO’s and PPO’s performance and applicability, we compare their methodologies, apply them on steady-state economic optimization of a chemical process, and compare their results to a conventional first principles modeling with nonlinear programming (FP-NLP). Our results show that ANN-PSO and PPO achieve profits nearly as high as FP-NLP, but PPO achieves slightly higher profits compared to ANN-PSO. We also find PPO has the fastest computational times, 10 and 10,000 times faster than FP-NLP and ANN-PSO, respectively. However, PPO requires more training data than ANN-PSO to converge to an optimal policy. This case study suggests PPO has better performance as it achieves higher profits and faster online computational times. ANN-PSO shows better applicability with its capability to train on historical operational data and higher training efficiency.

Microalgae with artificial intelligence: A digitalized perspective on genetics, systems and products

With recent advances in novel gene-editing tools such as RNAi, ZFNs, TALENs, and CRISPR-Cas9, the possibility of altering microalgae toward designed properties for various application is becoming a reality. Alteration of microalgae genomes can modify metabolic pathways to give elevated yields in lipids, biomass, and other components. The potential of such genetically optimized microalgae can give a "domino effect" in further providing optimization leverages down the supply chain, in aspects such as cultivation, processing, system design, process integration, and revolutionary products. However, the current level of understanding the functional information of various microalgae gene sequences is still primitive and insufficient as microalgae genome sequences are long and complex. From this perspective, this work proposes to link up this knowledge gap between microalgae genetic information and optimized bioproducts using Artificial Intelligence (AI). With the recent acceleration of AI research, large and complex data from microalgae research can be properly analyzed by combining the cutting-edge of both fields. In this work, the most suitable class of AI algorithms (such as active learning, semi-supervised learning, and meta-learning) are discussed for different cases of microalgae applications. This work concisely reviews the current state of the research milestones and highlight some of the state-of-art that has been carried out, providing insightful future pathways. The utilization of AI algorithms in microalgae cultivation, system optimization, and other aspects of the supply chain is also discussed. This work opens the pathway to a digitalized future for microalgae research and applications.