Optimal control towards sustainable wastewater treatment plants based on multi-agent reinforcement learning

Artificial Intelligence in Aquatic Biodiversity Research: A PRISMA-Based Systematic Review

Freshwater ecosystems are increasingly threatened by climate change and anthropogenic activities, necessitating innovative and scalable monitoring solutions. Artificial intelligence (AI) has emerged as a transformative tool in aquatic biodiversity research, enabling automated species identification, predictive habitat modeling, and conservation planning. This systematic review follows the PRISMA framework to analyze AI applications in freshwater biodiversity studies. Using a structured literature search across Scopus, Web of Science, and Google Scholar, we identified 312 relevant studies published between 2010 and 2024. This review categorizes AI applications into species identification, habitat assessment, ecological risk evaluation, and conservation strategies. A risk of bias assessment was conducted using QUADAS-2 and RoB 2 frameworks, highlighting methodological challenges, such as measurement bias and inconsistencies in the model validation. The citation trends demonstrate exponential growth in AI-driven biodiversity research, with leading contributions from China, the United States, and India. Despite the growing use of AI in this field, this review also reveals several persistent challenges, including limited data availability, regional imbalances, and concerns related to model generalizability and transparency. Our findings underscore AI's potential in revolutionizing biodiversity monitoring but also emphasize the need for standardized methodologies, improved data integration, and interdisciplinary collaboration to enhance ecological insights and conservation efforts.

Bayesian Optimization-Enhanced Reinforcement learning for Self-adaptive and multi-objective control of wastewater treatment

Controllers of wastewater treatment plants (WWTPs) often struggle to maintain optimal performance due to dynamic influent characteristics and the need to balance multiple operational objectives. In this study, Reinforcement Learning (RL) algorithms across different activated sludge process configurations was tested, and a novel approach that integrates RL with Bayesian Optimization (BO) to enhance the control of critical operational parameters in activated sludge processes was developed. This study extended the application of advanced machine learning techniques to complex WWTP control problems, moving beyond simplified benchmarks. The integration of BO with RL avoided sub-optimal performance and accelerated convergence to optimal control policies in controlling the A2O process, resulting in a significant 46% reduction in operational costs and a 12% decrease in energy consumption while maintaining compliance with effluent discharge standards. This approach offers a practical pathway for WWTPs to enhance treatment efficiency, reduce operational costs, and contribute to sustainable wastewater management practices.

Unified control of diverse actions in a wastewater treatment activated sludge system using reinforcement learning for multi-objective optimization

The operation of modern wastewater treatment facilities is a balancing act in which a multitude of variables are controlled to achieve a wide range of objectives, many of which are conflicting. This is especially true within secondary activated sludge systems, where significant research and industry effort has been devoted to advance control optimization strategies, both domain-driven and data-driven. Among data-driven control strategies, reinforcement learning (RL) stands out for its ability to achieve better than human performance in complex environments. While RL has been applied to activated sludge process optimization in existing literature, these applications are typically limited in scope, and never for the control of more than three actions. Expanding the scope of RL control has the potential to increase the optimization potential while concurrently reducing the number of control systems that must be tuned and maintained by operations staff. This study examined several facets of the implementation of multi-action, multi-objective RL agents, namely how many actions a single agent could successfully control and what extent of environment data was necessary to train such agents. This study observed improved control optimization with increasing action scope, though control of waste activated sludge remains a challenge. Furthermore, agents were able to maintain a high level of performance under decreased observation scope, up to a point. When compared to baseline control of the Benchmark Simulation Model No. 1 (BSM1), an RL agent controlling seven individual actions improved the average BSM1 performance metric by 8.3 %, equivalent to an annual cost savings of $40,200 after accounting for the cost of additional sensors.

Assessment and optimization of wastewater treatment plant in terms of effluent quality, energy footprint, and greenhouse gas emissions: An integrated modeling approach

Wastewater treatment plants (WWTPs) can benefit from utilizing digital technologies to reduce greenhouse gas (GHG) emissions and to comply with effluent quality standards. In this study, the GHG emissions and electricity consumption of a WWTP were evaluated via computer simulation by varying the dissolved oxygen (DO), mixed liquor recirculation (MLR), and return activated sludge (RAS) parameters. Three different measures, namely, effluent water quality, GHG emissions, and energy consumption, were combined as water-energy-carbon coupling index (WECCI) to compare the effects of the parameters on WWTPs, and the optimal operating condition was determined. The initial conditions of the A2O process were set to 4.0 mg/L of DO, 100 % MLR, and 90.7 % RAS. Eighty scenarios with various DO, MLR, and RAS were simulated under steady-state condition to optimize the biological treatment process. The optimal operating conditions were found to be 1.5 mg/L of DO, 190 % MLR, and 90.9 % RAS, which had the highest WECCI of 2.40 when compared to the WECCI of the initial condition (1.07). This optimal condition simultaneously reduced GHG emissions by 1348 kg CO2-eq/d and energy consumption by 11.64 MWh/d. This implies that controlling DO, MLR, and RAS through sensors, valves, and pumps offers a promising approach to operating WWTPs with reduced electricity consumption and GHG emissions while attaining effluent quality standards. Additionally, the nitrous oxide stripping rate exhibited linear relationships with the effluent total ammonia and nitrite concentrations in the aerobic reactor, suggesting that monitoring dissolved nitrogen compounds in the effluent and reactor could be a viable strategy to control MLR and DO in the biological reactor. The digital-based assessment and optimization tools developed in this study are expected to hold promise for application in broader environmental management efforts.

Multiobjective Optimization of Papermaking Wastewater Treatment Processes under Economic, Energy, and Environmental Goals

Due to the heterogeneity of recycled paper materials and the production conditions, pollutants in papermaking wastewater fluctuate sharply over time. Quality control of the papermaking wastewater treatment process (PWTP) is challenging and costly. As regulations are also growing about the environmental effects of the PWTP on greenhouse gas (GHG) emission, energy consumption, etc., the PWTP formulates a complex multiobjective optimization problem. This research established a multiagent deep reinforcement learning framework to simultaneously optimize process cost, energy consumption, and GHG emission in the PWTP, subjected to the effluent quality, to realize economic, energy, and environmental (3E) goals. The biological treatment process of wastewater in paper mills was simulated using benchmark simulation model no. 1 (BSM1). The data generated based on the BSM manual was utilized for model training, and real data acquired from a local papermaking factory was used to estimate the model performance. The results show that the proposed method outperforms conventional techniques in identifying the best control strategies for multiple targets.

Towards stable and efficient nitrogen removal in wastewater treatment processes via an adaptive neural network based sliding mode controller

Advanced controllers often offer an innovative solution to proper quality control in wastewater treatment processes (WWTPs). However, nonlinearity and uncertain disturbances usually make the conventional control strategies inadequate or impossible for the stable operations of WWTPs. To guarantee the stability of ammonia nitrogen concentration ( S N H ) control in WWTPs, a direct adaptive neural networks-based sliding mode control (ANNSMC) strategy has been proposed in this article. A sliding mode controller is designed and implemented with the help of an adaptive Neural Network (ANN), named Radial Basis Function Neural Network (RBFNN), which can approach the desired control law accurately. Also, the stability of a system installed with the ANNSMC is analyzed by using the Lyapunov theorem, which ensures system robustness and adaptability. Additionally, to deal with high energy consumption and low treatment efficiency problems in the wastewater denitrification processes, this paper proposes a dual-loop denitrification control strategy and validates it in the Benchmark Simulation Model No.2 (BSM2) platform. The strategy can strengthen the denitrification efficiency by collaborating the S N H with nitrate nitrogen ( S N O ) concentration in the WWTPs properly. The experimental results demonstrate that the proposed strategy can obtain remarkable stability and robustness, reducing energy consumption effectively compared with other standard and advanced control strategies.

Deep reinforcement learning challenges and opportunities for urban water systems

The efficient and sustainable supply and transport of water is a key component to any functioning civilisation making the role of urban water systems (UWS) inherently crucial to the wellbeing of its customers. However, managing water is not a simple task. Whether it is ageing infrastructure, transient flows, air cavities or low pressures; water can be lost as a result of many issues that face UWSs. The complexity of those networks grows with the high urbanisation trends and climate change making water companies and regulatory bodies in need of new solutions. So, it comes as no surprise that many researchers are invested in innovating within the water industry to ensure that the future of our water is safe. Deep reinforcement learning (DRL) has the potential to tackle complexities that used to be very challenging as it relies on deep neural networks for function approximation and representation. This technology has conquered many fields due to its impressive results and can effectively revolutionise UWS. In this article, we explain the background of DRL and the milestones of this field using a novel taxonomy of the DRL algorithms. This will be followed by with a novel review of DRL applications in the UWS which focus on water distribution networks and stormwater systems. The review will be concluded with critical insights on how DRL can benefit different aspects of urban water systems.

Carbon capture, utilization and sequestration systems design and operation optimization: Assessment and perspectives of artificial intelligence opportunities

Carbon capture, utilization, and sequestration (CCUS) is a promising solution to decarbonize the energy and industrial sectors to mitigate climate change. An integrated assessment of technological options is required for the effective deployment of CCUS large-scale infrastructure between CO2 production and utilization/sequestration nodes. However, developing cost-effective strategies from engineering and operation perspectives to implement CCUS is challenging. This is due to the diversity of upstream emitting processes located in different geographical areas, available downstream utilization technologies, storage sites capacity/location, and current/future energy/emissions/economic conditions. This paper identifies the need to achieve a robust hybrid assessment tool for CCUS modeling, simulation, and optimization based mainly on artificial intelligence (AI) combined with mechanistic methods. Thus, a critical literature review is conducted to assess CCUS technologies and their related process modeling/simulation/optimization techniques, while evaluating the needs for improvements or new developments to reduce overall CCUS systems design and operation costs. These techniques include first principles- based and data-driven ones, i.e. AI and related machine learning (ML) methods. Besides, the paper gives an overview on the role of life cycle assessment (LCA) to evaluate CCUS systems where the combined LCA-AI approach is assessed. Other advanced methods based on the AI/ML capabilities/algorithms can be developed to optimize the whole CCUS value chain. Interpretable ML combined with explainable AI can accelerate optimum materials selection by giving strong rules which accelerates the design of capture/utilization plants afterwards. Besides, deep reinforcement learning (DRL) coupled with process simulations will accelerate process design/operation optimization through considering simultaneous optimization of equipment sizing and operating conditions. Moreover, generative deep learning (GDL) is a key solution to optimum capture/utilization materials design/discovery. The developed AI methods can be generalizable where the extracted knowledge can be transferred to future works to help cutting the costs of CCUS value chain.

Indicator based multi-criteria decision support systems for wastewater treatment plants

Wastewater treatment plant decision makers face stricter regulations regarding human health protection, environmental preservation, and emissions reduction, meaning they must improve process sustainability and circularity, whilst maintaining economic performance. This creates complex multi-objective problems when operating and selecting technologies to meet these demands, resulting in the development of many decision support systems for the water sector. European Commission publications highlight their ambition for greater levels of sustainability, circularity, and environmental and human health protection, which decision support system implementation should align with to be successful in this region. Following the review of 57 wastewater treatment plant decision support systems, the main function of multi-criteria decision-making tools are technology selection and the optimisation of process operation. A large contrast regarding their aims is found, as process optimisation tools clearly define their goals and indicators used, whilst technology selection procedures often use vague language making it difficult for decision makers to connect selected indicators and resultant outcomes. Several recommendations are made to improve decision support system usage, such as more rigorous indicator selection protocols including participatory selection approaches and expansion of indicators sets, as well as more structured investigation of results including the use of sensitivity or uncertainty analysis, and error quantification.

Deep learning in water protection of resources, environment, and ecology: achievement and challenges

The breathtaking economic development put a heavy toll on ecology, especially on water pollution. Efficient water resource management has a long-term influence on the sustainable development of the economy and society. Economic development and ecology preservation are tangled together, and the growth of one is not possible without the other. Deep learning (DL) is ubiquitous in autonomous driving, medical imaging, speech recognition, etc. The spectacular success of deep learning comes from its power of richer representation of data. In view of the bright prospects of DL, this review comprehensively focuses on the development of DL applications in water resources management, water environment protection, and water ecology. First, the concept and modeling steps of DL are briefly introduced, including data preparation, algorithm selection, and model evaluation. Finally, the advantages and disadvantages of commonly used algorithms are analyzed according to their structures and mechanisms, and recommendations on the selection of DL algorithms for different studies, as well as prospects for the application and development of DL in water science are proposed. This review provides references for solving a wider range of water-related problems and brings further insights into the intelligent development of water science.

Deep-reinforcement-learning-based water diversion strategy

Water diversion is a common strategy to enhance water quality in eutrophic lakes by increasing available water resources and accelerating nutrient circulation. Its effectiveness depends on changes in the source water and lake conditions. However, the challenge of optimizing water diversion remains because it is difficult to simultaneously improve lake water quality and minimize the amount of diverted water. Here, we propose a new approach called dynamic water diversion optimization (DWDO), which combines a comprehensive water quality model with a deep reinforcement learning algorithm. We applied DWDO to a region of Lake Dianchi, the largest eutrophic freshwater lake in China and validated it. Our results demonstrate that DWDO significantly reduced total nitrogen and total phosphorus concentrations in the lake by 7% and 6%, respectively, compared to previous operations. Additionally, annual water diversion decreased by an impressive 75%. Through interpretable machine learning, we identified the impact of meteorological indicators and the water quality of both the source water and the lake on optimal water diversion. We found that a single input variable could either increase or decrease water diversion, depending on its specific value, while multiple factors collectively influenced real-time adjustment of water diversion. Moreover, using well-designed hyperparameters, DWDO proved robust under different uncertainties in model parameters. The training time of the model is theoretically shorter than traditional simulation-optimization algorithms, highlighting its potential to support more effective decision-making in water quality management.

Systematic Performance Evaluation of Reinforcement Learning Algorithms Applied to Wastewater Treatment Control Optimization

Treatment of wastewater using activated sludge relies on several complex, nonlinear processes. While activated sludge systems can provide high levels of treatment, including nutrient removal, operating these systems is often challenging and energy intensive. Significant research investment has been made in recent years into improving control optimization of such systems, through both domain knowledge and, more recently, machine learning. This study leverages a novel interface between a common process modeling software and a Python reinforcement learning environment to evaluate four common reinforcement learning algorithms for their ability to minimize treatment energy use while maintaining effluent compliance within the Benchmark Simulation Model No. 1 (BSM1) simulation. Three of the algorithms tested, deep Q-learning, proximal policy optimization, and synchronous advantage actor critic, generally performed poorly over the scenarios tested in this study. In contrast, the twin delayed deep deterministic policy gradient (TD3) algorithm consistently produced a high level of control optimization while maintaining the treatment requirements. Under the best selection of state observation features, TD3 control optimization reduced aeration and pumping energy requirements by 14.3% compared to the BSM1 benchmark control, outperforming the advanced domain-based control strategy of ammonia-based aeration control, although future work is necessary to improve robustness of RL implementation.

Critical analysis on the transformation and upgrading strategy of Chinese municipal wastewater treatment plants: Towards sustainable water remediation and zero carbon emissions

In the light of circular economy aspects, processing of large-scale municipal wastewater treatment plants (WWTPs) needs reconsideration to limit the overuse of energy, implement of non-green technologies and emit abundant greenhouse gas. Along with the huge increase in the worldwide population and agro-industrial activities, global environmental organizations have issued several recent roles to boost scientific and industrial communities towards sustainable development. Over recent years, China has imposed national and regional standards to control and manage the discharged liquid and solid waste, as well as to achieve carbon peaking and carbon neutrality. The aim of this report is to analyze the current state of Chinese WWTPs routing and related issues such as climate change and air pollution. The used strategies in Chinese WWTPs and upgrading trends were critically discussed. Several points were addressed including the performance, environmental impact, and energy demand of bio-enhanced technologies, including hydrolytic acidification pretreatment, efficient (toxic) strain treatment, and anaerobic ammonia oxidation denitrification technology, as well as advanced treatment technologies composed of physical and chemical treatment technologies, biological treatment technology and combined treatment technology. Discussion and critical analysis based on the current data and national policies were provided and employed to develop the future development trend of municipal WWTPs in China from the construction of sustainable and "Zero carbon" WWTPs.

Visual analysis of greenhouse gas emissions from sewage treatment plants based on CiteSpace: from the perspective of bibliometrics

With the global reduction actions of greenhouse gas (GHG) emissions, environmental facilities, including sewage treatment plants (STPs), need to reduce pollutants while minimizing GHG emissions. Therefore, more and more publications revealed the formation mechanism of GHGs in STPs and committed to finding better reduction schemes. From the perspective of bibliometrics, this study used CiteSpace to conduct quantitative and visual analysis based on 1,543 publications retrieved from Web of Science between 2000 and 2021 around the world. We have systematically evaluated the structure, development trend, hot spots, and research frontier in the field of GHG emissions from STPs and compared with the contents of top journals to verify the scientificity of the analysis. The results show that the number of publications has increased year by year, and the networks of authors and institutions show a strong correlation. Among them, the clusters of nitrous oxide, anaerobic digestion, and life cycle assessment (LCA) started earlier and received extensive attention, which derived other clusters in the research process. With the development of the field, researchers have gradually changed from single water treatment facilities to multi-carriers that can realize energy regeneration and utilization simultaneously. Accordingly, the GHG reduction of STPs through energy regeneration and resource recovery has become a hot point and frontier direction, which also challenges the breakthroughs in relevant technologies. Furthermore, it provides scientific support for the formulation of relevant incentive policies and economic subsidy systems, so as to alleviate the pressure of global warming and realize the sustainable development of STPs concurrently.

Towards coordinated and robust real-time control: a decentralized approach for combined sewer overflow and urban flooding reduction based on multi-agent reinforcement learning

The real-time control (RTC) of urban drainage systems can make full use of the capabilities of existing infrastructures to mitigate combined sewer overflow (CSO) and urban flooding. Despite the benefits of RTC, it may encounter potential risks and failures, which need further consideration to enhance its robustness. Besides failures of hardware components such as sensors and actuators, the RTC performance is also sensitive to communication failures between the devices that are spatially distributed in a catchment-scale system. This paper proposes a decentralized control strategy based on multi-agent reinforcement learning to enhance communication robustness and coordinate the decentralized control agents through centralized training. To investigate different control structures, a centralized and a fully decentralized strategy are also developed based on reinforcement learning (RL) for comparison. A benchmark drainage model and a real-world drainage model are formulated as two cases, and the control agents are trained to control the orifices or pumps for CSO or flooding mitigation in each case. The three RL strategies reduce the CSO volume by 5.62-9.30% compared with a static baseline in historical rainfalls of the benchmark case and reduce the CSO and flooding volume by 14.39-21.36% compared with currently-used rule-based control in synthetic rainfalls of the real-world case. Benefitting from centralized training, the decentralized agents can achieve similar performance to the centralized agent. The decentralized control also enhances the communication robustness with smaller performance loss than the centralized control when observation communication fails, and provides a robust backup at the local level to limit the uncertainties when action commands from the centralized agent are lost. The results and findings indicate that multi-agent RL contributes to a coordinated and robust solution for RTC of urban drainage systems.

Automatic control and optimal operation for greenhouse gas mitigation in sustainable wastewater treatment plants: A review

In order to promote low-carbon sustainable operational management of the wastewater treatment plants (WWTPs), automatic control and optimal operation technologies, which devote to improving effluent quality, operational costs and greenhouse gas (GHG) emissions, have flourished in recent years. There is no consensus on the design procedure for optimal control/operation of sustainable WWTPs. In this review, we summarize recent researches on developing control and optimization strategies for GHG mitigation in WWTPs. Faced with the fact that direct carbon dioxide (CO₂) emissions (considered biological origin) are generally not included in the carbon footprint of WWTPs, direct emissions (nitrous oxide (N₂O), methane (CH₄)) and indirect emissions are paid much attention. Firstly, the plant-wide models with GHG dynamic simulation, which are employed to design and evaluate the automatic control schemes as well as representative studies on identifying key factors affecting GHG emissions or comprehensive performance are outlined. Then, both traditional and advanced control methods commonly used in GHG mitigation are reviewed in detail, followed by the multi-objective optimization practices of control/operational parameters. Based on the mentioned control and (or) optimization strategies, a novel design framework for the optimal control/operation of sustainable WWTPs is proposed. The findings and design framework proposed in the paper will provide guidance for GHG mitigation and sustainable operation in WWTPs. It is foreseeable that more accurate and appropriate plant-wide models together with flexible control methods and intelligent optimization strategies will be developed to satisfy the upgrading requirements of WWTPs in the future.

The role of deep learning in urban water management: A critical review

Deep learning techniques and algorithms are emerging as a disruptive technology with the potential to transform global economies, environments and societies. They have been applied to planning and management problems of urban water systems in general, however, there is lack of a systematic review of the current state of deep learning applications and an examination of potential directions where deep learning can contribute to solving urban water challenges. Here we provide such a review, covering water demand forecasting, leakage and contamination detection, sewer defect assessment, wastewater system state prediction, asset monitoring and urban flooding. We find that the application of deep learning techniques is still at an early stage as most studies used benchmark networks, synthetic data, laboratory or pilot systems to test the performance of deep learning methods with no practical adoption reported. Leakage detection is perhaps at the forefront of receiving practical implementation into day-to-day operation and management of urban water systems, compared with other problems reviewed. Five research challenges, i.e., data privacy, algorithmic development, explainability and trustworthiness, multi-agent systems and digital twins, are identified as key areas to advance the application and implementation of deep learning in urban water management. Future research and application of deep learning systems are expected to drive urban water systems towards high intelligence and autonomy. We hope this review will inspire research and development that can harness the power of deep learning to help achieve sustainable water management and digitalise the water sector across the world.

Integrated supervisory and override control strategies for effective biological phosphorus removal and reduced operational costs in wastewater treatment processes

A novel control strategy is developed for a municipal wastewater treatment plant (WWTP) consisting of anaerobic-anoxic-aerobic reactors. The idea is to generate more organic matter with a reduction of nitrate concentration in the anoxic section so that more biological phosphorus (P) removal happens. For this, the Supervisory and Override Control Approach (SOPCA) is designed based on the benchmark simulation model (BSM1-P) and is evaluated by considering dynamic influent. In the supervisory layer, proportional integral (PI) and fuzzy controllers are designed. Additionally, dissolved oxygen (S_o) control loops in the aerobic reactors are designed. PI controller is designed for control of nitrate levels in the anoxic reactors and is integrated with override control and supervisory layer. It is found that the novel SOPCA approach gave better nutrient removal with slightly higher operating costs when S_o control is not put in place. With three S_o control loops in place, the WWTP showed better effluent quality and lower cost. Here, the improved removal efficiency of 28.5% and 20.5% are obtained when Fuzzy and PI control schemes respectively are used in the supervisory layer. Therefore, the application of SOPCA is recommended for a better P removal rate.