An influent responsive control strategy with machine learning: Q-learning based optimization method for a biological phosphorus removal system

Unified control of diverse actions in a wastewater treatment activated sludge system using reinforcement learning for multi-objective optimization

The operation of modern wastewater treatment facilities is a balancing act in which a multitude of variables are controlled to achieve a wide range of objectives, many of which are conflicting. This is especially true within secondary activated sludge systems, where significant research and industry effort has been devoted to advance control optimization strategies, both domain-driven and data-driven. Among data-driven control strategies, reinforcement learning (RL) stands out for its ability to achieve better than human performance in complex environments. While RL has been applied to activated sludge process optimization in existing literature, these applications are typically limited in scope, and never for the control of more than three actions. Expanding the scope of RL control has the potential to increase the optimization potential while concurrently reducing the number of control systems that must be tuned and maintained by operations staff. This study examined several facets of the implementation of multi-action, multi-objective RL agents, namely how many actions a single agent could successfully control and what extent of environment data was necessary to train such agents. This study observed improved control optimization with increasing action scope, though control of waste activated sludge remains a challenge. Furthermore, agents were able to maintain a high level of performance under decreased observation scope, up to a point. When compared to baseline control of the Benchmark Simulation Model No. 1 (BSM1), an RL agent controlling seven individual actions improved the average BSM1 performance metric by 8.3 %, equivalent to an annual cost savings of $40,200 after accounting for the cost of additional sensors.

Optimizing wastewater treatment through artificial intelligence: recent advances and future prospects

Artificial intelligence (AI) is increasingly being applied to wastewater treatment to enhance efficiency, improve processes, and optimize resource utilization. This review focuses on objectives, advantages, outputs, and major findings of various AI models in the three key aspects: the prediction of removal efficiency for both organic and inorganic pollutants, real-time monitoring of essential water quality parameters (such as pH, COD, BOD, turbidity, TDS, and conductivity), and fault detection in the processes and equipment integral to wastewater treatment. The prediction accuracy (R2 value) of AI technologies for pollutant removal has been reported to vary between 0.64 and 1.00. A critical aspect explored in this review is the cost-effectiveness of implementing AI systems in wastewater treatment. Numerous countries and municipalities are actively engaging in pilot projects and demonstrations to assess the feasibility and effectiveness of AI applications in wastewater treatment. Notably, the review highlights successful outcomes from these initiatives across diverse geographical contexts, showcasing the adaptability and positive impact of AI in revolutionizing wastewater treatment on a global scale. Further, insights on the ethical considerations and potential future directions for the use of AI in wastewater treatment plants have also been provided.

Real-time control of urban drainage systems using neuro-evolution

With climate change and urbanization, existing urban drainage systems are being stressed beyond their design capacity in many parts of the world. Real-time control (RTC) can improve the performance of these systems and reduce the need for system upgrades. However, developing optimal control policies for RTC is a challenging research area due to computational demands, high uncertainties and system dynamics. This study presents a new RTC method using neuro-evolution for controlling combined sewer overflow (CSO) in urban drainage systems. Neuro-evolution is an approach to neural network research by evolutionary algorithms. Neuro-evolution realizes RTC by training the control policy in advance, thus avoiding the online optimization process in the application period. The simulation results of the benchmark Astlingen network indicate that the trained control policy outperforms the equal filling degree strategy in terms of CSO volume reduction and robustness in the face of tank level uncertainty. The performance analysis of the typical CSO events shows that the control policy mainly makes positive contributions during 'small' CSO events rather than 'large' ones. In particular, the effectiveness of the control policy in 'small' CSO events is more prominent in the initial phase of the events compared with the final phase. This work stands to support a foundation for future studies in the control of urban water systems based on neuro-evolution.

Deep reinforcement learning challenges and opportunities for urban water systems

The efficient and sustainable supply and transport of water is a key component to any functioning civilisation making the role of urban water systems (UWS) inherently crucial to the wellbeing of its customers. However, managing water is not a simple task. Whether it is ageing infrastructure, transient flows, air cavities or low pressures; water can be lost as a result of many issues that face UWSs. The complexity of those networks grows with the high urbanisation trends and climate change making water companies and regulatory bodies in need of new solutions. So, it comes as no surprise that many researchers are invested in innovating within the water industry to ensure that the future of our water is safe. Deep reinforcement learning (DRL) has the potential to tackle complexities that used to be very challenging as it relies on deep neural networks for function approximation and representation. This technology has conquered many fields due to its impressive results and can effectively revolutionise UWS. In this article, we explain the background of DRL and the milestones of this field using a novel taxonomy of the DRL algorithms. This will be followed by with a novel review of DRL applications in the UWS which focus on water distribution networks and stormwater systems. The review will be concluded with critical insights on how DRL can benefit different aspects of urban water systems.

Exploring Optimization of Zeolites as Adsorbents for Rare Earth Elements in Continuous Flow by Machine Learning Techniques

Unsupervised machine learning (ML) techniques are applied to the characterization of the adsorption of rare earth elements (REEs) by zeolites in continuous flow. The successful application of principal component analysis (PCA) and K-Means algorithms from ML allowed for a wide range assessment of the adsorption results. This global approach permits the evaluation of the different stages of the sorption cycles and their optimization and improvement. The results from ML are also used for the definition of a regression model to estimate other REEs' recoveries based on the known values of the tested REEs. Overall, it was possible to remove more than 70% of all REEs from aqueous solutions during the adsorption assays and to recover over 80% of the REEs entrapped on the zeolites using an optimized desorption cycle.

Multiobjective Integrated Optimal Control for Nonlinear Systems

The multiobjective optimal control method optimizes the performance indexes of nonlinear systems to obtain setpoints, and designs a controller to track the setpoints. However, the stepwise optimal control method that independently analyzes the optimization process may obtain unfeasible and difficult to track setpoints, which will reduce the operation and control performance of the systems. To solve this problem, a multiobjective integrated optimal control (MIOC) strategy is proposed for nonlinear systems in this article. The main contributions of MIOC are threefold. First, in the framework of multiobjective model predictive control, an integrated control structure with a comprehensive cost function and a collaborative optimization algorithm is designed to achieve the coordinate optimal control. Second, for the time inconformity of setpoints and control laws caused by the characteristic of tracking control, the different prediction horizons are designed for the comprehensive cost function. Then, the collaborative optimization algorithm is proposed for the comprehensive cost function to achieve the integrated solution of setpoints and control laws to enhance the operation and control performance of nonlinear systems. Third, the stability and control performance analysis of MIOC is provided. Finally, the proposed MIOC method is applied for a nonlinear system to demonstrate its effectiveness.

A predictive model for determining the nitrite concentration in the effluent of an anammox reactor using ensemble regression tree algorithm

Anaerobic ammonium oxidation (anammox) is a cost-effective biological nitrogen removal method for treating wastewater. Nitrite has strong negative effect on microbial activity of anammox bacteria, while the conventional equitment available for determining nitrite on-line is challenging due to high price. By knowing the concentration of nitrite in the effluent, its concentration in the reactor can be controlled accordingly. To investigate this, an ensemble regression tree algorithm was used to establish the predictive model proposed in the current work. Moreover, the Bayesian algorithm was adopted to systematically optimize various parameters of machine learning algorithms. The predicted concentrations of nitrite were in good agreement with the observed values, and the coefficient of determination (R2) and root mean squared error (RMSE) values reached 0.91 and 4.81, respectively. Furthermore, the model established by the ensemble regression tree algorithm was compared with models established by commonly used machine learning algorithms. Finally, the established models were applied to another anammox reactor, and the predicted results of ensemble regression tree model were found to be in good agreement with the experimental values with R2 and RMSE values of 0.84 and 6.34, respectively.

Effective adsorption of nutrients from simulated domestic sewage by modified maifanite

Modified maifanite (MMF) was prepared by the synthesized method with sulfuric acid treatment and high-temperature calcination and evaluated as an effective adsorption material to remove the nutrient salt in waste watery. Compared with the raw maifanite (RMF), the MMF exhibited a higher adsorption capacity and higher removal efficiency. The results showed that the adsorption rates of total phosphorus (TP), total nitrogen (TN), ammonia nitrogen (NH₃-N), nitrate-nitrogen (NO_x-N), and chemical oxygen demand (COD) by MMF were 86.7%, 44.9%, 29.1%, 19.8%, and 11.9%, respectively, and compared to RMF, the average adsorption capacity of these nutrients by MMF increased by 20.5 mg/kg, 126.2 mg/kg, 61.9 mg/kg, 117.18 mg/kg, and 86.9 mg/kg, respectively. MMF maintained the basic structure and composition of maifanite, while having a rougher and looser surface, more irregular pores, wider gaps, and more active materials such as oxidizing Fe. This study suggests that MMF can be further applied to treat domestic sewage and eutrophic water.

Steelmaking Process Optimised through a Decision Support System Aided by Self-Learning Machine Learning

This paper presents the application of a reinforcement learning (RL) algorithm, concretely Q-Learning, as the core of a decision support system (DSS) for a steelmaking subprocess, the Composition Adjustment by Sealed Argon-bubbling with Oxygen Blowing (CAS-OB) from the SSAB Raahe steel plant. Since many CAS-OB actions are selected based on operator experience, this research aims to develop a DSS to assist the operator in taking the proper decisions during the process, especially less experienced operators. The DSS is intended to supports the operators in real-time during the process to facilitate their work and optimise the process, improving material and energy efficiency, thus increasing the operation’s sustainability. The objective is that the algorithm learns the process based only on raw data from the CAS-OB historical database, and on rewards set according to the objectives. Finally, the DSS was tested and validated by a developer engineer from the CAS-OB steelmaking plant. The results show that the algorithm successfully learns the process, recommending the same actions as those taken by the operator 69.23% of the time. The algorithm also suggests a better option in 30.76% of the remaining cases. Thanks to the DSS, the heat rejection due to wrong composition is reduced by 4%, and temperature accuracy is increased to 83.33%. These improvements resulted in an estimated reduction of 2% in CO2 emissions, 0.5% in energy consumption and 1.5% in costs. Additionally, actions taken based on the operator’s experience are incorporated into the DSS knowledge, facilitating the integration of operators with lower experience in the process.

Optimal control towards sustainable wastewater treatment plants based on multi-agent reinforcement learning

Wastewater treatment plants (WWTPs) are designed to eliminate pollutants and alleviate environmental pollution resulting from human activities. However, the construction and operation of WWTPs consume resources, emit greenhouse gases (GHGs) and produce residual sludge, thus require further optimization. WWTPs are complex to control and optimize because of high non-linearity and variation. This study used a novel technique, multi-agent deep reinforcement learning (MADRL), to simultaneously optimize dissolved oxygen (DO) and chemical dosage in a WWTP. The reward function was specially designed from life cycle perspective to achieve sustainable optimization. Five scenarios were considered: baseline, three different effluent quality and cost-oriented scenarios. The result shows that optimization based on LCA has lower environmental impacts compared to baseline scenario, as cost, energy consumption and greenhouse gas emissions reduce to 0.890 CNY/m³-ww, 0.530 kWh/m³-ww, 2.491 kg CO²-eq/m³-ww respectively. The cost-oriented control strategy exhibits comparable overall performance to the LCA-driven strategy since it sacrifices environmental benefits but has lower cost as 0.873 CNY/m³-ww. It is worth mentioning that the retrofitting of WWTPs based on resources should be implemented with the consideration of impact transfer. Specifically, LCA-SW scenario decreases 10 kg PO⁴-eq in eutrophication potential compared to the baseline within 10 days, while significantly increases other indicators. The major contributors of each indicator are identified for future study and improvement. Last, the authors discussed that novel dynamic control strategies required advanced sensors or a large amount of data, so the selection of control strategies should also consider economic and ecological conditions. In a nutshell, there are still limitations of this work and future studies are required.

Identification of Genome Sequences of Polyphosphate-Accumulating Organisms by Machine Learning

In the field of sewage treatment, the identification of polyphosphate-accumulating organisms (PAOs) usually relies on biological experiments. However, biological experiments are not only complicated and time-consuming, but also costly. In recent years, machine learning has been widely used in many fields, but it is seldom used in the water treatment. The present work presented a high accuracy support vector machine (SVM) algorithm to realize the rapid identification and prediction of PAOs. We obtained 6,318 genome sequences of microorganisms from the publicly available microbial genome database for comparative analysis (MBGD). Minimap2 was used to compare the genomes of the obtained microorganisms in pairs, and read the overlap. The SVM model was established using the similarity of the genome sequences. In this SVM model, the average accuracy is 0.9628 ± 0.019 with 10-fold cross-validation. By predicting 2,652 microorganisms, 22 potential PAOs were obtained. Through the analysis of the predicted potential PAOs, most of them could be indirectly verified their phosphorus removal characteristics from previous reports. The SVM model we built shows high prediction accuracy and good stability.

Activated sludge and other aerobic suspended culture processes

This paper provides an overview of activated sludge related to suspended growth processes for the year 2019. The review encompasses process modeling of activated sludge, microbiology of activated sludge, process kinetics and mechanism, nitrogen and phosphorus control, design, and operation in the activated sludge field. The fate and effect of xenobiotics in activated sludge, including trace organic contaminant and heavy metal xenobiotics, which had influence on the growth of suspended sludge, are covered in this review. Compared to past reviews, many topics show increase in activity in 2019. These include, biokinetics process of aerobic granular sludge formation, pyrolysis kinetic mechanism of granular sludge. These topics are referred to formation and disintegration of granular sludge. Other sections include activated sludge settling model, toxicity resistant microbial community, nitritation-anammox processes for nitrogen removal, and respirometry used in the operation of real wastewater treatment plant are especially highlighted in this review. PRACTITIONER POINTS: Biokinetics process of aerobic granular sludge formation Toxicity resistant microbial community in activated sludge Nitritation-anammox processes for nitrogen removal in activated sludge.

An improved ASM-GDA approach to evaluate the production kinetics of loosely bound and tightly bound extracellular polymeric substances in biological phosphorus removal process

This study established an extended activated sludge model no. 2 (ASM2) for providing a new recognition of the contributions of both loosely-bound EPS (LB-EPS) and tightly-bound EPS (TB-EPS) into phosphorus (P) removal by incorporating their formation and degradation processes during the anaerobic-aerobic cycle. For determining the best-fit values for the new model parameters (k _h,TB-EPS, k _h,LB-EPS, f _PP,TB-EPS, and f _PP,LB-EPS) in this extended ASM2, a novel and convenient gradient descent algorithm (GDA) based ASM (ASM-GDA) method was developed. Sensitivity analysis of f _PP,TB-EPS, f _PP,LB-EPS, k _h,TB-EPS, and k _h,LB-EPS on the model target outputs of S _PO4 , X _TB-EPS, X _LB-EPS, and X _PP proved the accuracy of the chosen parameters. Eight batch experiments conducted under different influential chemical oxygen demand (COD) and P conditions were quantitatively and qualitatively analyzed. Respectively, 9.37-9.64% and 4.17-4.29% of P removal by TB-EPS and LB-EPS were achieved. Self-Organizing Map (SOM) has shown its high performance for visualization and abstraction for exhibiting the high correlations of the influential COD/P concentrations and the P% removal by TB-EPS (and LB-EPS). Comprehensive analyses of the influences of influential COD and P concentration on the biological phosphorus removal process help us in successfully establishing the mechanism kinetics of production and degradation of P in a dynamic P biological-treatment model.

Modeling the performance of Single-stage Nitrogen removal using Anammox and Partial nitritation (SNAP) process with backpropagation neural network and response surface methodology

Two novel feedforward backpropagation Artificial Neural Networks (ANN)-based-models (8:N_H:1 and 7:N_H:1) combined with Box-Behnken design of experiments methodology was proposed and developed to model NH₄⁺ and Total Nitrogen (TN) removal within an upflow-sludge-bed (USB) reactor treating nitrogen-rich wastewater via Single-stage Nitrogen removal using Anammox and Partial nitritation (SNAP) process. ANN were developed by optimizing network architecture parameters via response surface methodology. Based on the goodness-of-fit standards, the proposed three-layered NH₄⁺ and TN removal ANN-based-models trained with Levenberg-Marquardt-algorithm demonstrated high-performance as computations exhibited smaller deviations-(±2.1%) as well as satisfactory coefficient of determination (R²), fractional variance-(FV), and index of agreement-(IA) ranging 0.989-0.997, 0.003-0.031 and 0.993-0.998, respectively. The computational results affirmed that the ANN architecture which was optimized with response surface methodology enhanced the efficiency of the ANN-based-models. Furthermore, the overall performance of the developed ANN-based models revealed that modeling intricate biological systems (such as SNAP) using ANN-based models with the view to improve removal efficiencies, establish process control strategies and optimize performance is highly feasible. Microbial community analysis conducted with 16S rRNA high-throughput approach revealed that Candidatus Kuenenia was the most pronounced genera which accounted for 13.11% followed by Nitrosomonas-(6.23%) and Proteocatella-(3.1%), an indication that nitrogen removal pathway within the USB was mainly via partial-nitritation/anammox process.