Automatic design of hyper-heuristic based on reinforcement learning

A review of reinforcement learning based hyper-heuristics

The reinforcement learning based hyper-heuristics (RL-HH) is a popular trend in the field of optimization. RL-HH combines the global search ability of hyper-heuristics (HH) with the learning ability of reinforcement learning (RL). This synergy allows the agent to dynamically adjust its own strategy, leading to a gradual optimization of the solution. Existing researches have shown the effectiveness of RL-HH in solving complex real-world problems. However, a comprehensive introduction and summary of the RL-HH field is still blank. This research reviews currently existing RL-HHs and presents a general framework for RL-HHs. This article categorizes the type of algorithms into two categories: value-based reinforcement learning hyper-heuristics and policy-based reinforcement learning hyper-heuristics. Typical algorithms in each category are summarized and described in detail. Finally, the shortcomings in existing researches on RL-HH and future research directions are discussed.

Challenging the Limits of Binarization: A New Scheme Selection Policy Using Reinforcement Learning Techniques for Binary Combinatorial Problem Solving

In this study, we introduce an innovative policy in the field of reinforcement learning, specifically designed as an action selection mechanism, and applied herein as a selector for binarization schemes. These schemes enable continuous metaheuristics to be applied to binary problems, thereby paving new paths in combinatorial optimization. To evaluate its efficacy, we implemented this policy within our BSS framework, which integrates a variety of reinforcement learning and metaheuristic techniques. Upon resolving 45 instances of the Set Covering Problem, our results demonstrate that reinforcement learning can play a crucial role in enhancing the binarization techniques employed. This policy not only significantly outperformed traditional methods in terms of precision and efficiency, but also proved to be extensible and adaptable to other techniques and similar problems. The approach proposed in this article is capable of significantly surpassing traditional methods in precision and efficiency, which could have important implications for a wide range of real-world applications. This study underscores the philosophy behind our approach: utilizing reinforcement learning not as an end in itself, but as a powerful tool for solving binary combinatorial problems, emphasizing its practical applicability and potential to transform the way we address complex challenges across various fields.

A Hyperheuristic With Q-Learning for the Multiobjective Energy-Efficient Distributed Blocking Flow Shop Scheduling Problem

Carbon peaking and carbon neutrality, which are the significant national strategy for sustainable development, have attracted considerable attention from production enterprises. In this study, the energy consumption is considered in the distributed blocking flow shop scheduling problem (DBFSP). A hyperheuristic with Q -learning (HHQL) is presented to address the energy-efficient DBFSP (EEDBFSP). Q -learning is employed to select an appropriate low-level heuristic (LLH) from a predesigned LLH set according to historical information fed back by LLH. An initialization method, which considers both total tardiness (TTD) and total energy consumption (TEC), is proposed to construct the initial population. The ε -greedy strategy is introduced to utilize the learned knowledge while retaining a certain degree of exploration in the process of selecting LLH. The acceleration operation of the job on the critical path is designed to optimize TTD. The deceleration operation of the job on the noncritical path is designed to optimize TEC. The statistical and computational experimentation in an extensive benchmark testified that the HHQL outperforms the other comparison algorithm regarding efficiency and significance in solving EEDBFSP.

A novel intelligent hyper-heuristic algorithm for solving optimization problems

In recent years, solving combinatorial optimization problems involves more complications, high dimensions, and multi-objective considerations. Combining the advantages of other evolutionary algorithms to enhance the performance of a unique evolutionary algorithm and form a new hybrid heuristic algorithm has become a way to strengthen the performance of the algorithm effectively. However, the intelligent hybrid heuristic algorithm destroys the integrity, universality, and robustness of the original algorithm to a certain extent and increases its time complexity. This paper implements a new idea “ML to choose heuristics” (a heuristic algorithm combined with machine learning technology) which uses the Q-learning method to learn different strategies in genetic algorithm. Moreover, a selection-based hyper-heuristic algorithm is obtained that can guide the algorithm to make decisions at different time nodes to select appropriate strategies. The algorithm is the hybrid strategy using Q-learning on StudGA (HSQ-StudGA). The experimental results show that among the 14 standard test functions, the evolutionary algorithm guided by Q-learning can effectively improve the quality of arithmetic solution. Under the premise of not changing the evolutionary structure of the algorithm, the hyper-heuristic algorithm represents a new method to solve combinatorial optimization problems.

A Novel Learning-Based Binarization Scheme Selector for Swarm Algorithms Solving Combinatorial Problems

Currently, industry is undergoing an exponential increase in binary-based combinatorial problems. In this regard, metaheuristics have been a common trend in the field in order to design approaches to successfully solve them. Thus, a well-known strategy includes the employment of continuous swarm-based algorithms transformed to perform in binary environments. In this work, we propose a hybrid approach that contains discrete smartly adapted population-based strategies to efficiently tackle binary-based problems. The proposed approach employs a reinforcement learning technique, known as SARSA (State–Action–Reward–State–Action), in order to utilize knowledge based on the run time. In order to test the viability and competitiveness of our proposal, we compare discrete state-of-the-art algorithms smartly assisted by SARSA. Finally, we illustrate interesting results where the proposed hybrid outperforms other approaches, thus, providing a novel option to tackle these types of problems in industry.

A Hyper Heuristic Algorithm Based Genetic Programming for Steel Production Scheduling of Cyber-Physical System-ORIENTED

Intelligent manufacturing is the trend of the steel industry. A cyber-physical system oriented steel production scheduling system framework is proposed. To make up for the difficulty of dynamic scheduling of steel production in a complex environment and provide an idea for developing steel production to intelligent manufacturing. The dynamic steel production scheduling model characteristics are studied, and an ontology-based steel cyber-physical system production scheduling knowledge model and its ontology attribute knowledge representation method are proposed. For the dynamic scheduling, the heuristic scheduling rules were established. With the method, a hyper-heuristic algorithm based on genetic programming is presented. The learning-based high-level selection strategy method was adopted to manage the low-level heuristic. An automatic scheduling rule generation framework based on genetic programming is designed to manage and generate excellent heuristic rules and solve scheduling problems based on different production disturbances. Finally, the performance of the algorithm is verified by a simulation case.

Q-Learnheuristics: Towards Data-Driven Balanced Metaheuristics

One of the central issues that must be resolved for a metaheuristic optimization process to work well is the dilemma of the balance between exploration and exploitation. The metaheuristics (MH) that achieved this balance can be called balanced MH, where a Q-Learning (QL) integration framework was proposed for the selection of metaheuristic operators conducive to this balance, particularly the selection of binarization schemes when a continuous metaheuristic solves binary combinatorial problems. In this work the use of this framework is extended to other recent metaheuristics, demonstrating that the integration of QL in the selection of operators improves the exploration-exploitation balance. Specifically, the Whale Optimization Algorithm and the Sine-Cosine Algorithm are tested by solving the Set Covering Problem, showing statistical improvements in this balance and in the quality of the solutions.

A Preliminary Study on Score-Based Hyper-heuristics for Solving the Bin Packing Problem

The bin packing problem is a widespread combinatorial problem. It aims at packing a set of items by using as few bins as possible. Among the many available solving methods, approximation ones such as heuristics have become popular due to their reduced cost and generally acceptable solutions. A further step in this regard is given by hyper-heuristics, which literature usually defines as “high-level heuristics to choose heuristics”. Hyper-heuristics choose one suitable heuristic from a set of available ones, to solve a particular portion of an instance. As the search progresses, heuristics can be exchanged, adapting the solution process to the current problem state under exploration. In this work, we describe how to generate and use hyper-heuristics that keep a record of the scores achieved by individual heuristics on previously solved bin packing problem instances in the form of rules. Then, hyper-heuristics manage those scores to estimate the performance of such heuristics on unseen instances. In this way, the previous actions of the hyper-heuristics determine which heuristic to use on future unseen cases. The experiments conducted under different scenarios yield promising results where some of the hyper-heuristics produced outperform isolated heuristics.