Search

Deep Reinforcement Learning (DRL) recently emerged as a possibility to control complex systems without the need to model them mathematically. In contrast to classical controllers, DRL alleviates the need for constant parameter tuning, tedious design of control laws, and re-identification procedures in the event of performance degradation. However, the application of DRL algorithms remains fairly modest, and they have not yet established a significant position in process industries. One major obstacle has been their sample inefficiency when facing tasks featuring large state-action spaces. In this work, we show that it is possible to use DRL for plant-wide control by decentralizing and coordinating reinforcement learning. Namely, we express the global policy as a collection of local policies. Every local policy receives local observations and is responsible for controlling a different region of the environment. To enable coordination among local policies, we present a mechanism based on message passing. Messages are encoded by a shared communication channel, which is equipped with a model-based stream to capture the dynamics of the system and enable effective pre-training. The proposed method is evaluated on a set of robotic tasks and a large-scale vinyl acetate monomer (VAM) plant. Experimental results highlight that the proposed model exhibits drastic improvements over baselines in terms of mean scores and sample efficiency.

We present a novel framework for controlling non-steady situations in chemical plants to address the behavioural gaps between the simulator for constructing the reinforcement learning-based controller and the real plant considered for deploying the framework. In the field of reinforcement learning, the performance deterioration problem owing to such gaps are referred to as simulation-to-reality gaps (Sim-to-Real gaps). These gaps are triggered by multiple factors, including modelling errors on the simulators, incorrect state identifications, and unpredicted disturbances on the real situations. We focus on these issues and divided the objective of performing optimal control under gapped situations into three tasks, namely, (1) identifying the model parameters and current state, (2) optimizing the operation procedures, and (3) letting the real situations close to the simulated and predicted situations by adjusting the control inputs. Each task is assigned to a reinforcement learning agent and trained individually. After the training, the agents are integrated and collaborate on the original objective. We present the evaluation of our method in an actual chemical distillation plant, which demonstrates that our system successfully narrows down the gaps due to the emulated disturbance of a weather change (heavy rain) as well as the modelling errors and achieves the desired states.

In cylindrical traverse grinding, a slender workpiece is bent by the normal grinding force due to its low stiffness. Consequently, a form error is generated by the elastic deformation of the workpiece during the grinding process. To prevent such deformation of the workpiece, a steady rest is generally used. However, when the contact point between the workpiece and the steady rest is ground, the pushing force of the steady rest is lost, and the elastic deformation of the workpiece is increased immediately. As a result, a step shape error is generated at the contact point. To improve the use of the steady rest, the elastic deformation of the workpiece during the grinding process is simulated using the beam model considering the steady rest in this study. From the analysis results, it was determined that placing the steady rest outside the grinding part can reduce the elastic deformation of the workpiece. In addition, the traverse speed was controlled to maintain a constant elastic deformation. Through grinding experiments, it was confirmed that the form accuracy of the slender workpiece was improved by setting the steady rest outside the grinding part and adjusting the traverse speed.

In cylindrical grinding, the size accuracy of a workpiece decreases due to the thermal deformation of the ground workpiece. To improve the size accuracy, it is necessary to consider the thermal deformation of the workpiece during grinding process. In this study, we have developed an intelligent grinding system that can consider the thermal deformation of the workpiece. In this system, the thermal deformation of the workpiece during grinding process is simulated by measuring tangential grinding force. The net stock removal, that is the stock removal measured after the workpiece has been cooled, is estimated by adding the measured stock removal and the simulated thermal deformation of the workpiece. To achieve the grinding process with high accuracy, the grinding wheel is retracted when the net stock removal is reached to the grinding allowance. Using this intelligent grinding system, grinding experiments were carried out. In these experiments, the size error was reduced less than 0.3μm in radius even though the thermal deformation was more than 2μm in radius. With grinding experiments, it was confirmed that the developed grinding system could improve the size accuracy successfully.

Sample inefficiency of deep reinforcement learning methods is a major obstacle for their use in real-world tasks as they naturally feature sparse rewards. In fact, this from-scratch approach is often impractical in environments where extreme negative outcomes are possible. Recent advances in imitation learning have improved sample efficiency by leveraging expert demonstrations. Most work along this line of research employs neural network-based approaches to recover an expert cost function. However, the complexity and lack of transparency make neural networks difficult to trust and deploy in the real world. In contrast, we present a method for extracting interpretable symbolic reward functions from expert data, which offers several advantages. First, the learned reward function can be parsed by a human to understand, verify and predict the behavior of the agent. Second, the reward function can be improved and modified by an expert. Finally, the structure of the reward function can be leveraged to extract explanations that encode richer domain knowledge than standard scalar rewards. To this end, we use an autoregressive recurrent neural network that generates hierarchical symbolic rewards represented by simple symbolic trees. The recurrent neural network is trained via risk-seeking policy gradients. We test our method in MuJoCo environments as well as a chemical plant simulator. We show that the discovered rewards can significantly accelerate the training process and achieve similar or better performance than neural network-based algorithms. [ABSTRACT FROM AUTHOR]