Search

Advances in artificial intelligence (AI) have led to its application in many areas of everyday life. In the context of control engineering, reinforcement learning (RL) represents a particularly promising approach as it is centred around the idea of allowing an agent to freely interact with its environment to find an optimal strategy. One of the challenges professionals face when training and deploying RL agents is that the latter often have to run on dedicated embedded devices. This could be to integrate them into an existing toolchain or to satisfy certain performance criteria like real-time constraints. Conventional RL libraries, however, cannot be easily utilised in conjunction with that kind of hardware. In this paper, we present a framework named LExCI, the Learning and Experiencing Cycle Interface, which bridges this gap and provides end-users with a free and open-source tool for training agents on embedded systems using the open-source library RLlib. Its operability is demonstrated with two state-of-the-art RL-algorithms and a rapid control prototyping system., Comment: The code, models, and data used for this work are available in a separate branch of LExCI's GitHub repository (https://github.com/mechatronics-RWTH/lexci-2/tree/lexci_paper). This paper has been submitted to Applied Intelligence (https://link.springer.com/journal/10489). 2024-06-27: Updated the footnote on the title page so that it provides information about the paper's Version of Record

The process of developing control functions for embedded systems is resource-, time-, and data-intensive, often resulting in sub-optimal cost and solutions approaches. Reinforcement Learning (RL) has great potential for autonomously training agents to perform complex control tasks with minimal human intervention. Due to costly data generation and safety constraints, however, its application is mostly limited to purely simulated domains. To use RL effectively in embedded system function development, the generated agents must be able to handle real-world applications. In this context, this work focuses on accelerating the training process of RL agents by combining Transfer Learning (TL) and X-in-the-Loop (XiL) simulation. For the use case of transient exhaust gas re-circulation control for an internal combustion engine, use of a computationally cheap Model-in-the-Loop (MiL) simulation is made to select a suitable algorithm, fine-tune hyperparameters, and finally train candidate agents for the transfer. These pre-trained RL agents are then fine-tuned in a Hardware-in-the-Loop (HiL) system via TL. The transfer revealed the need for adjusting the reward parameters when advancing to real hardware. Further, the comparison between a purely HiL-trained and a transferred agent showed a reduction of training time by a factor of 5.9. The results emphasize the necessity to train RL agents with real hardware, and demonstrate that the maturity of the transferred policies affects both training time and performance, highlighting the strong synergies between TL and XiL simulation.

Model Predictive Control (MPC) provides an optimal control solution based on a cost function while allowing for the implementation of process constraints. As a model-based optimal control technique, the performance of MPC strongly depends on the model used where a trade-off between model computation time and prediction performance exists. One solution is the integration of MPC with a machine learning (ML) based process model which are quick to evaluate online. This work presents the experimental implementation of a deep neural network (DNN) based nonlinear MPC for Homogeneous Charge Compression Ignition (HCCI) combustion control. The DNN model consists of a Long Short-Term Memory (LSTM) network surrounded by fully connected layers which was trained using experimental engine data and showed acceptable prediction performance with under 5% error for all outputs. Using this model, the MPC is designed to track the Indicated Mean Effective Pressure (IMEP) and combustion phasing trajectories, while minimizing several parameters. Using the acados software package to enable the real-time implementation of the MPC on an ARM Cortex A72, the optimization calculations are completed within 1.4 ms. The external A72 processor is integrated with the prototyping engine controller using a UDP connection allowing for rapid experimental deployment of the NMPC. The IMEP trajectory following of the developed controller was excellent, with a root-mean-square error of 0.133 bar, in addition to observing process constraints., Comment: Submitted to 2024 American Control Conference (ACC), July 8-12, 2024 in Toronto, Canada. ACC is the annual conference of the American Automatic Control Council (AACC), the U.S. national member organization of the International Federation for Automatic Control (IFAC)

The high thermal efficiency and reliability of the compression-ignition engine makes it the first choice for many applications. For this to continue, a reduction of the pollutant emissions is needed. One solution is the use of machine learning (ML) and model predictive control (MPC) to minimize emissions and fuel consumption, without adding substantial computational cost to the engine controller. ML is developed in this paper for both modeling engine performance and emissions and for imitating the behaviour of an Linear Parameter Varying (LPV) MPC. Using a support vector machine-based linear parameter varying model of the engine performance and emissions, a model predictive controller is implemented for a 4.5 Cummins diesel engine. This online optimized MPC solution offers advantages in minimizing the \nox~emissions and fuel consumption compared to the baseline feedforward production controller. To reduce the computational cost of this MPC, a deep learning scheme is designed to mimic the behavior of the developed controller. The performance in reducing NOx emissions at a constant load by the imitative controller is similar to that of the online optimized MPC compared to the Cummins production controller. In addition, the imitative controller requires 50 times less computation time compared to that of the online MPC optimization., Comment: Submitted to Advances in Automotive Control - 10th AAC 2022

Machine learning (ML) and a nonlinear model predictive controller (NMPC) are used in this paper to minimize the emissions and fuel consumption of a compression ignition engine. In this work machine learning is applied in two methods. In the first application, ML is used to identify a model for implementation in model predictive control optimization problems. In the second application, ML is used as a replacement of the NMPC where the ML controller learns the optimal control action by imitating or mimicking the behavior of the model predictive controller. In this study, a deep recurrent neural network including long-short term memory (LSTM) layers are used to model the emissions and performance of an industrial 4.5 liter 4-cylinder Cummins diesel engine. This model is then used for model predictive controller implementation. Then, a deep learning scheme is deployed to clone the behavior of the developed controller. In the LSTM integration, a novel scheme is used by augmenting hidden and cell states of the network in an NMPC optimization problem. The developed LSTM-NMPC and the imitative NMPC are compared with the Cummins calibrated Engine Control Unit (ECU) model in an experimentally validated engine simulation platform. Results show a significant reduction in Nitrogen Oxides (\nox) emissions and a slight decrease in the injected fuel quantity while maintaining the same load. In addition, the imitative NMPC has a similar performance as the NMPC but with a two orders of magnitude reduction of the computation time., Comment: Submitted to Control engineering Practice (Submission date: March 9, 2022) Revised version (Submission date: June 18, 2022) Accepted on July 30, 2022