Search

This paper presents findings on machine learning agent behavior prediction in a board game application developed by a group of students. The goal of this research is to create a model facilitating collaboration between a user and an AI to play together in the board game using a Human-in-the-Loop architecture. By injecting explainability, the aim is to enhance communication and understanding between the user and the AI agent. Featuring a competitive Artificial Intelligence (AI) based on the Proximal Policy Optimization model, this research explores methods to make AI decisions transparent for enhanced player understanding. Two predictive models, a Decision Tree (DT) and a Deep Learning (DL) classifier, were developed and compared. The results show that the DT model is effective for short-term predictions but limited in broader applications, while the DL classifier shows potential for long-term prediction without requiring direct access to the game's AI. This study contributes to understanding human-AI interaction in gaming and offers insights into AI decision-making processes. [ABSTRACT FROM AUTHOR]

[Display omitted] • This paper proposes a novel approach to train generators without any training data or shaped rewards by learning at multiple sizes. • This paper presents an application of the multi-size training approach to train generative flow networks for controllable 2D tile-based game level generation at various sizes. • This paper modifies diversity sampling for compatibility with generative flow networks, and to expand the generators' expressive ranges. A level generator is a tool that generates game levels from noise. Training a generator without a dataset suffers from feedback sparsity, since it is unlikely to generate a playable level via random exploration. A common solution is shaped rewards, which guides the generator to achieve subgoals towards level playability, but they consume effort to design and require game-specific domain knowledge. This paper proposes a novel approach to train generators without datasets or shaped rewards by learning at multiple level sizes starting from small sizes and up to the desired sizes. The denser feedback at small sizes negates the need for shaped rewards. Additionally, the generators learn to build levels at various sizes, including sizes they were not trained for. We apply our approach to train recurrent autoregressive generative flow networks (GFlowNets) for controllable level generation. We also adapt diversity sampling to be compatible with GFlowNets. The results show that our generators create diverse playable levels at various sizes for Sokoban, Zelda, and Danger Dave. When compared with controllable reinforcement learning level generators for Sokoban, the results show that our generators achieve better controllability and competitive diversity, while being 9 × faster at training and level generation. [ABSTRACT FROM AUTHOR]

Stocktaking refers to the examination or counting over of materials, supplies, goods or products inside a stockroom, store, warehouse. There are many challenges concerning this important business function, which are even more demanding inside an unstructured large warehouse. Huge variance in size, volume, shape of the stock-keeping units that are stored in bulk without packaging, leads to many emerging problems that necessitate significant research endeavors so as to develop solutions that can automate this process. Nowadays, there are many approaches related to computer vision-based stocktaking in unstructured warehouses with products stored in bulk, which employ several existing technologies namely Deep Learning, Optical Character Recognition, and 3D model retrieval. The paper at hand aims at presenting the i-Count Air concept which outlines the utilization of Unmanned Aerial Vehicles, endowed with cognitive hardware sensors, active vision and an innovative perception and stocktaking software, that will push the boundaries of robotic perception and help automate the inventory counting function in a Warehouse 5.0 environment. [ABSTRACT FROM AUTHOR]

The following research article constitutes a comprehensive review of state-of-the-art methodologies and problems addressed by current academic literature in the field of deploying autonomous Unmanned Aerial Vehicles (UAVs) for disaster management. The work compares several papers to identify the current trend of the field. It also presents an in-depth analysis of possible future research directions in the domain. [ABSTRACT FROM AUTHOR]

This paper proposes a novel Reinforcement Learning (RL) approach for sim-to-real policy transfer of Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTOL-UAV). The proposed approach is designed for VTOL-UAV landing on offshore docking stations in maritime operations. VTOL-UAVs in maritime operations encounter limitations in their operational range, primarily stemming from constraints imposed by their battery capacity. The concept of autonomous landing on a charging platform presents an intriguing prospect for mitigating these limitations by facilitating battery charging and data transfer. However, current Deep Reinforcement Learning (DRL) methods exhibit drawbacks, including lengthy training times, and modest success rates. In this paper, we tackle these concerns comprehensively by decomposing the landing procedure into a sequence of more manageable but analogous tasks in terms of an approach phase and a landing phase. The proposed architecture utilizes a model-based control scheme for the approach phase, where the VTOL-UAV is approaching the offshore docking station. In the Landing phase, DRL agents were trained offline to learn the optimal policy to dock on the offshore station. The Joint North Sea Wave Project (JONSWAP) spectrum model has been employed to create a wave model for each episode, enhancing policy generalization for sim2real transfer. A set of DRL algorithms have been tested through numerical simulations including value-based agents and policy-based agents such as Deep Q Networks (DQN) and Proximal Policy Optimization (PPO) respectively. The numerical experiments show that the PPO agent can learn complicated and efficient policies to land in uncertain environments, which in turn enhances the likelihood of successful sim-to-real transfer. • Novel deep reinforcement learning for VTOL-UAV docking. • Successful VTOL-UAV landing on uncertain offshore docking stations. • Generalized policy for sim-to-real transfer introduced to increase the success rates in landing operations. • DRL agents learn using a virtual environment subjected to a randomized disturbance describing the hydrodynamics waves. • A set of policy-based and value-based DRL agents are tested against randomized disturbance. [ABSTRACT FROM AUTHOR]

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Two participants with competing objectives are considered in one-to-one orbital pursuit-evasion problem, one of which attempts to satisfy some certain terminal conditions with a minimum cost through its own control while the other player attempts to maximize the specified payoff and avoids entering this terminal range. This paper treats the problem of minimax time to a given final distance, within which the interception between pursuer and evader is defined to occur. To achieve the successful interception with an evader that adopts any unknown maneuvers, a near-optimal interception strategy for the orbital pursuit-evasion is presented in this paper. The pursuit-evasion problem is formulated as a differential game first, and a closed-form barrier solution is provided next. A near-optimal guidance law using deep learning is proposed to intercept the evader inside the capture zone. For the games that start outside the barrier, a learning algorithm for the capture zone embedding strategy is presented based on deep reinforcement learning to help the game state cross the barrier surfaces. The simulation cases of intercepting the evaders that adopt different maneuvering strategies are provided, and the results demonstrate the effectiveness and onboard feasibility of the proposed strategy in solving the orbital interception problem. • The closed-form barrier solution for a planar orbital pursuit-evasion game is provided. • A deep neural network is trained on a dataset consisting of plenty of barrier trajectories generated through the closed-form barrier formula. • The near-optimal guidance law inside capture zone is proposed using trained neural network. • A capture zone embedding strategy is presented based on deep reinforcement learning. • The closed loop interception strategy is proposed combining the strategies outside and inside capture zone. [ABSTRACT FROM AUTHOR]

Electric airplanes have attracted a great deal of attentions because of their strategic commercial values and low emissions. One of the essentials for efficient control and management of hybrid electric propulsion aircraft is a well-designed energy management strategy (EMS). A new EMS approach based on deep learning is introduced in this study to realize thrust distribution and power distribution, consequently lowering aircraft fuel consumption. A model of hybrid electric propulsion power system including turbine engines and lithium-ion battery pack is constructed to act as an application environment. An intelligent energy management strategy based on deep reinforcement learning (DRL) is then proposed in this paper. Two DRL agents based on twin delayed deep deterministic policy gradient (TD3) are combined in the hierarchical framework. The multi-agents are trained to handle the energy management problem in this paper. In a finite period of time, the training process is accomplished. After training, the simulation result of the test shows that this DRL-based EMS reduces the equivalent fuel consumption by 2.2% compared with that using the existing method, and preserves 99.9% of the battery state of health (SOH). The battery pack's state of charge (SOC) regresses precisely to the reference value at the end. Particularly, the proposed EMS is able to work properly for different flight profiles and various initial SOCs. The results of this paper provide preliminary supports for the feasibility and superiority of deep reinforcement learning applied to the energy management strategy of series hybrid electric aircraft. [ABSTRACT FROM AUTHOR]

Automated industries lead to high quality production, lower manufacturing cost and better utilization of human resources. Robotic manipulator arms have major role in the automation process. However, for complex manipulation tasks, hard coding efficient and safe trajectories is challenging and time consuming. Machine learning methods have the potential to learn such controllers based on expert demonstrations. Despite promising advances, better approaches must be developed to improve safety, reliability, and efficiency of ML methods in both training and deployment phases. This survey aims to review cutting edge technologies and recent trends on ML methods applied to real-world manipulation tasks. After reviewing the related background on ML, the rest of the paper is devoted to ML applications in different domains such as industry, healthcare, agriculture, space, military, and search and rescue. The paper is closed with important research directions for future works. • Recent trends in applying ML to real-world manipulation tasks are reviewed. • Different application domains utilizing manipulators are considered. • Challenges of using reinforcement learning for robot control are highlighted. • Possible methods to enforce robot safety during training RL agents are investigated. [ABSTRACT FROM AUTHOR]

• Develop a generic framework for maintenance scheduling of multi-component systems with environment modeling, state representation and policy optimization. • Propose Guided-Probabilistic Reinforcement Learning for sampling-efficient maintenance scheduling by incorporating domain expert knowledge. • Develop Bayesian formulations to address data sparsity for environmental reward modeling, performance degradation, and exploration of action spaces. In recent years, multi-agent deep reinforcement learning has progressed rapidly as reflected by its increasing adoptions in industrial applications. This paper proposes a Guided Probabilistic Reinforcement Learning (Guided-PRL) model to tackle maintenance scheduling of multi-component systems in the presence of uncertainty with the goal of minimizing the overall life-cycle cost. The proposed Guided-PRL is deeply rooted in the Actor-Critic (AC) scheme. Since traditional AC falls short in sampling efficiency and suffers from getting stuck in local minima in the context of multi-agent reinforcement learning, it is thus challenging for the actor network to converge to a solution of desirable quality even when the critic network is properly configured. To address these issues, we develop a generic framework to facilitate effective training of the actor network, and the framework consists of environmental reward modeling, degradation formulation, state representation, and policy optimization. The convergence speed of the actor network is significantly improved with a guided sampling scheme for environment exploration by exploiting rules-based domain expert policies. To handle data scarcity, the environmental modeling and policy optimization are approximated with Bayesian models for effective uncertainty quantification. The Guided-PRL model is evaluated using the simulations of a 12-component system as well as GE90 and CFM56 engines. Compared with four alternative deep reinforcement learning schemes, the Guided-PRL lowers life-cycle cost by 34.92 % to 88.07 %. In comparison with rules-based expert policies, the Guided-PRL decreases the life-cycle cost by 23.26 % to 51.36 %. [ABSTRACT FROM AUTHOR]

The users of cloud span to several types of tasks for various purposes, such as users who need to accomplish tasks that utilize cloud based on as Infrastructure as a Service. These tasks are usually of high latency. There are some tasks that require immediate response. That is, ultra-low latency tasks like IoT device requirements. It is not feasible to always depend on a far cloud datacenter. Unlike traditional cloud computing, it is possible to place edge and fog nodes can be placed close to the IoT devices provides noticeable reduction in latency. The emerging fog computing technology is characterized by ultra-low latency response, which benefits several time-sensitive services and applications. Nodes in fog are deployed in less centralized. In a fog layer, the computing equipment dedicates parts of its limited resources to process IoT application tasks. Therefore, efficient utilization of computing resources is essential and requires an optimal and intelligent strategy for task scheduling. This paper focuses on scheduling IoT tasks in a fog-based environment with the aim to minimizing energy, cost, and service delay. Towards this end, a deep reinforcement learning based algorithm named Clipped Double Deep Q-learning using target networks and experience replay techniques is proposed. To ensure there is no lag in using resources optimally, a parallel queueing is utilized. One of the important factors in cloud (and fog) computing research is to address the problem of long waiting time of the task in the virtual machine queue. This paper proposes a dual queue method. [ABSTRACT FROM AUTHOR]

With the rapid development of deep learning and structures of neural networks, parameter settings of trained neural networks become new products from knowledge. Since supervised learning is used in 85% of machine learning to train the neural networks, well-labeled training data becomes new knowledge capital. In order to stimulate the knowledge sharing and protect the Intelligent Property (IP) right and privacy of knowledge capital at the same time, we need a framework to conduct training of neural networks with certain encrypted data and exchange training results with certain rewards. Federated learning provides a framework to train the networks without disclose the privacy and a block-chain based transaction system can organize a group of participants under certain consensus and reward mechanism. Their combination is an effective solution to manage knowledge capital, called knowledge capital bank in this paper. This paper defines the economic meaning of knowledge capital and all behaviors of related participants. Then we propose a block-chain based federated learning algorithm for secured knowledge sharing with IP and privacy protection. Finally, a complete design for a use case of this algorithm called Knowledge capital bank is demonstrated. [ABSTRACT FROM AUTHOR]

[Display omitted] • A physics-informed fast voltage control method of distribution network is proposed. • It develops a novel deep reinforcement learning algorithm with graph knowledge. • The proposed algorithm uses multiple agents with continuous action space. With the recent rapid uptake of photovoltaic (PV) resources, the overvoltage condition during low load periods or intermittent renewable generation is one of the most critical challenges for electricity distribution networks. Traditional model-based local or centralized control methods that rely on proper system parameters are difficult to mitigate rapid changes in power systems. Utilizing model-free/data-driven multiagent deep reinforcement learning (MADRL) has been recognized as an effective solution to active voltage control. However, existing MADRL-based control approaches trained solely on data are agnostic to the underlying real-world physics principles. Therefore, this paper aims to incorporate physical knowledge of regional distribution networks into MADRL's decision-making. The main contributions of this paper are summarized as follows. First, a novel physics-informed MADRL -based distributed voltage control method is proposed, which is still under a centralized training and distributed execution framework and only requires local measurements. Second, graph neural networks are employed to help MADRL agents learn graph knowledge (node features and topological information). Further, transformer is introduced to extract discriminative representations and ensure agents' cooperative control. Third, we adopt a physics-guided architecture of neural networks in the actor network to stabilize the training process and improve sample efficiency. Finally, simulation results based upon modified IEEE 33-bus and 141-bus networks validate the proposed method's effectiveness, robustness and computation efficiency. [ABSTRACT FROM AUTHOR]

Maintenance planning of networked multi-asset systems is a complex problem due to the inherent individual and collective asset constraints and dynamics as well as the size of the system and interdependencies among assets. Although multi-asset systems have been studied numerous times in the past decades, maintenance planning implications of the system's network characteristics have been barely analysed. Likewise, solutions that consider the network perspective suffer from scalability issues as a network-wide observability is assumed. This paper proposes a network maintenance planning approach based on the decomposition of the multi-asset network into fixed-size localised subnetworks. The overall network maintenance plan is produced by aggregating the subnetwork maintenance plans, which are computed independently via a multi-agent deep reinforcement learning (MARL) algorithm. The results are evaluated against a network-wide approach as well as the commonly-used individual approach. The paper also introduces a systematic approach to integrate the MARL resulting policy in a multi-asset agent-based model. Simulation results of several random asset networks and a large nationwide network infrastructure show that, although a network-wide approach outperforms, on average, other approaches considered, the localised subnetworks approach, provides an acceptable alternative in networks with small-world properties, without the need of a network-wide view. • A Network-wide maintenance approach outperforms individual approaches in networked assets with the topologies analysed. • In analysed networks with small-world properties, the Localised approach shows performance close to the network-wide approach. • A systematic integration of MARL and multi-asset agent-based modelling enables evaluation of maintenance approaches. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel graph-based semi-supervised learning approach for traffic light management in multiple intersections. Specifically, the basic premise behind our paper is that if we know some of the occupied roads and predict which roads will be congested, we can dynamically change traffic lights at the intersections that are connected to the roads anticipated to be congested. Comparative performance evaluations show that the proposed approach can produce comparable average vehicle waiting time and reduce the training/learning time of learning adequate traffic light configurations for all intersections within a few seconds, while a deep learning-based approach can be trained in a few days for learning similar light configurations. • With this work we aimed to optimize traffic signal systems by dynamically setting the system to behave as per predicted traffic flow. • Vehicle flow is predicted using Graph-based semi-supervised and active learning on edge flows. • The systems were created and tested using Simulation Of Urban Mobility (SUMO). • Vehicle data and routes were also generated using this application. • The proposed approach was compared against other methods, such as an occupancy based method, a scoring-based method and reinforcement learning. [ABSTRACT FROM AUTHOR]

This paper investigates visual navigation and control of a cooperative unmanned surface vehicle (USV)-unmanned aerial vehicle (UAV) system for marine search and rescue. First, a deep learning-based visual detection architecture is developed to extract positional information from the images taken by the UAV. With specially designed convolutional layers and spatial softmax layers, the visual positioning accuracy and computational efficiency are improved. Next, a reinforcement learning-based USV control strategy is proposed, which could learn a motion control policy with an enhanced ability to reject wave disturbances. The simulation experiment results show that the proposed visual navigation architecture can provide stable and accurate position and heading angle estimation in different weather and lighting conditions. The trained control policy also demonstrates satisfactory USV control ability under wave disturbances. • Cooperative USV–UAV systems for marine search and rescue. • Convolutional neural network with spatial softmax layers for visual navigation. • Reinforcement learning based USV control with disturbance rejection ability. [ABSTRACT FROM AUTHOR]

In this paper, we propose the strategy-following distributed attentional actor architecture after conditional attention (sfDA6-X) for multi-agent deep reinforcement learning (MADRL). The architecture is designed to provide controllability of coordinated behaviors in MADRL through the use of a saliency vector that captures conditions from the environment and rough, high-level instructions given by external experts such as the system designers or users. We introduce destination channels, which enable agents to be instructed to change their behaviors only by specifying the regions where individual agents should work. To validate the effectiveness of sfDA6-X, we conducted experiments in the object collection game and analyzed how agents change their coordinated and cooperative behaviors based on the given instructions. Our findings suggest that our approach provides a preliminary but promising solution for controlling coordinated behaviors in MADRL via arbitrary instructions represented in destination channels. [ABSTRACT FROM AUTHOR]

More recently, the advancement of deep learning techniques enables reinforcement learning (RL) to handle high-dimensional decision-making problems. It brings increasing attention in transport areas. Some work has applied them to solve the intractable traffic signal control (TSC) problem and achieved promising results. However, very few research comprehensively investigates the impacts of key design elements, including state definitions, reward functions and deep reinforcement learning (DRL) methods, on TSC performance. To fill this research gap, this paper first selects commonly used design elements from existing literature. Then, we compare their learning stability and control performance at an isolated intersection under different scenarios via simulation experiments. The experimental results show that the quantitative state (e.g., the number of vehicles on lanes) and image-like state (e.g., vehicle position and speed) have no significant impact on the performance under different traffic demands. However, the impact of various reward functions on performance is vivid, especially under high traffic demand. Also, high-resolution vehicular network data may not be possible to improve control performance versus ordinary camera data. In addition, the value-based DRL algorithms outperform the policy-based algorithm and traditional TSC control methods. The findings of this research would provide insights and guidance for transport engineers to design an efficient DRL-based TSC system in a real traffic environment. [ABSTRACT FROM AUTHOR]

• Urban flow prediction from spatiotemporal data is systematically reviewed. • Some existing urban flow prediction methods based on machine learning are overviewed. • The difficulties and some ideas of urban flow prediction are discussed briefly. Urban spatiotemporal flow prediction is of great importance to traffic management, land use, public safety. This prediction task is affected by several complex and dynamic factors, such as patterns of human activities, weather, events, and holidays. Datasets evaluated the flow come from various sources in different domains, e.g. mobile phone data, taxi trajectories data, metro/bus swiping data, bike-sharing data. To summarize these methodologies of urban flow prediction, in this paper, we first introduced four main factors affecting urban flow. Second, in order to further analyze urban flow, we partitioned the preparation process of multi-source spatiotemporal data related with urban flow into three groups. Third, we chose the spatiotemporal dynamic data as a case study for the urban flow prediction task. Fourth, we analyzed and compared some representative flow prediction methods in detail, classifying them into five categories: statistics-based, traditional machine learning-based, deep learning-based, reinforcement learning-based, and transfer learning-based methods. Finally, we showed open challenges of urban flow prediction and discussed many recent research works on urban flow prediction. This paper will facilitate researchers to find suitable methods and public datasets for addressing urban spatiotemporal flow forecast problems. [ABSTRACT FROM AUTHOR]

• State of the art on continual / lifelong learning and its implications for robotics. • Proposal of a framework to present Continual Learning algorithms. • Summary of existing metrics for continual learning. • Presenting robotics applications for continual learning, current challenges and opportunities. Continual learning (CL) is a particular machine learning paradigm where the data distribution and learning objective change through time, or where all the training data and objective criteria are never available at once. The evolution of the learning process is modeled by a sequence of learning experiences where the goal is to be able to learn new skills all along the sequence without forgetting what has been previously learned. CL can be seen as an online learning where knowledge fusion needs to take place in order to learn from streams of data presented sequentially in time. Continual learning also aims at the same time at optimizing the memory, the computation power and the speed during the learning process. An important challenge for machine learning is not necessarily finding solutions that work in the real world but rather finding stable algorithms that can learn in real world. Hence, the ideal approach would be tackling the real world in a embodied platform: an autonomous agent. Continual learning would then be effective in an autonomous agent or robot, which would learn autonomously through time about the external world, and incrementally develop a set of complex skills and knowledge.Robotic agents have to learn to adapt and interact with their environment using a continuous stream of observations. Some recent approaches aim at tackling continual learning for robotics, but most recent papers on continual learning only experiment approaches in simulation or with static datasets. Unfortunately, the evaluation of those algorithms does not provide insights on whether their solutions may help continual learning in the context of robotics. This paper aims at reviewing the existing state of the art of continual learning, summarizing existing benchmarks and metrics, and proposing a framework for presenting and evaluating both robotics and non robotics approaches in a way that makes transfer between both fields easier. We put light on continual learning in the context of robotics to create connections between fields and normalize approaches. [ABSTRACT FROM AUTHOR]

This work develops a deep reinforcement learning based approach for Six Degree-of-Freedom (DOF) planetary powered descent and landing. Future Mars missions will require advanced guidance, navigation, and control algorithms for the powered descent phase to target specific surface locations and achieve pinpoint accuracy (landing error ellipse <5 m radius). This requires both a navigation system capable of estimating the lander's state in real-time and a guidance and control system that can map the estimated lander state to a commanded thrust for each lander engine. In this paper, we present a novel integrated guidance and control algorithm designed by applying the principles of reinforcement learning theory. The latter is used to learn a policy mapping the lander's estimated state directly to a commanded thrust for each engine, resulting in accurate and almost fuel-optimal trajectories over a realistic deployment ellipse. Specifically, we use proximal policy optimization, a policy gradient method, to learn the policy. Another contribution of this paper is the use of different discount rates for terminal and shaping rewards, which significantly enhances optimization performance. We present simulation results demonstrating the guidance and control system's performance in a 6-DOF simulation environment and demonstrate robustness to noise and system parameter uncertainty. [ABSTRACT FROM AUTHOR]

For deep reinforcement learning (DRL) system, it is difficult to design a reward function for complex tasks, so this paper proposes a framework of behavior fusion for the actor–critic architecture, which learns the policy based on an advantage function that consists of two value functions. Firstly, the proposed method decomposes a complex task into several sub-tasks, and merges the trained policies for those sub-tasks into a unified policy for the complex task, instead of designing a new reward function and training for the policy. Each sub-task is trained individually by an actor–critic algorithm using a simple reward function. These pre-trained sub-tasks are building blocks that are used to rapidly assemble a rapid prototype of a complicated task. Secondly, the proposed method integrates modules in the calculation of the policy gradient by calculating the accumulated returns to reduce variation. Thirdly, two alternative methods to acquire integrated returns for the complicated task are also proposed. The Atari 2600 pong game and a wafer probe task are used to validate the performance of the proposed methods by comparison with the method using a gate network. • A behavior fusion framework is designed for an actor–critic method that uses a policy gradient. • This paper proposes two alternative methods with different ways to calculate the merged return for weighting the gradient in policy gradient. • This article is the first to study the problem of how to combine multiple pre-trained homomorphy models (ACs) into a more complex one based on the policy gradient method. [ABSTRACT FROM AUTHOR]

The air pollution these days is a serious environmental concern and it is not just a mere fact but a harsh reality which is creating problems for the mankind such as some serious health issues. In some parts of the world the air quality index have reached to an irrefutable level which demands for a solution now. Hence, the purpose of this paper is to tackle this problem at a large scale by providing with an artificially intelligent mobile air purifier for "outdoors". The main problem of DMAPS is to provide with purified air in the outdoors, such as a residential colony, in apartments, office complex, etc. since most of the times people travel outdoors and stay away from their home for a long time. This ability of providing with such a smart-sensing, self-driving machine is the advantage of the paper because other air purifiers exist but they are made for indoors, they do not have mobility and also do not possess AI capabilities. The method used in this air purifier is formulated by connecting three major functions. The first one being, 'the smart mobility' of the machine using the approach of deep reinforcement learning, the second is the air 'purification using the Arduino system' and the last is, 'the detection of humans' so that DMAPS can be around them and provide them with purified air, based on the YOLO algorithm. The whole system, works in a way that each and every function is very much connected and works simultaneously. In a nutshell, this air purifier, moves from one place (where there is low pollution content) to another place (where there is higher pollution content), purifies the environment and simultaneously have an objective to smartly detect humans to give out fresh air to them. Hence, in this way the whole methodology tries to fix the problem of air pollution by purifying the air outdoors to make it safe to inhale. [ABSTRACT FROM AUTHOR]

Recent advances in Intelligent Transport Systems (ITS) and Artificial Intelligence (AI) have stimulated and paved the way toward the widespread introduction of Autonomous Vehicles (AVs). This has opened new opportunities for smart roads, intelligent traffic safety, and traveler comfort. Autonomous Vehicles have become a highly popular research topic in recent years because of their significant capability to reduce road accidents and human injuries. This paper is an attempt to survey all recent AI based techniques used to deal with major functions in AVs, namely scene understanding, motion planning, decision making, vehicle control, social behavior, and communication. Our survey focuses solely on deep learning and reinforcement learning based approaches; it does not include conventional (shallow) shallow based techniques, a subject that has been extensively investigated in the past. Our survey builds a taxonomy of DL and RL algorithms that have been used so far to bring solutions to the four main issues in autonomous driving. Finally, this survey highlights the open challenges and points out possible future research directions. [ABSTRACT FROM AUTHOR]

Recent advancements in deep learning (DL) have posed a significant challenge for automatic speech recognition (ASR). ASR relies on extensive training datasets, including confidential ones, and demands substantial computational and storage resources. Enabling adaptive systems improves ASR performance in dynamic environments. DL techniques assume training and testing data originate from the same domain, which is not always true. Advanced DL techniques like deep transfer learning (DTL), federated learning (FL), and deep reinforcement learning (DRL) address these issues. DTL allows high-performance models using small yet related datasets, FL enables training on confidential data without dataset possession, and DRL optimizes decision-making in dynamic environments, reducing computation costs. This survey offers a comprehensive review of DTL, FL, and DRL-based ASR frameworks, aiming to provide insights into the latest developments and aid researchers and professionals in understanding the current challenges. Additionally, Transformers, which are advanced DL techniques heavily used in proposed ASR frameworks, are considered in this survey for their ability to capture extensive dependencies in the input ASR sequence. The paper starts by presenting the background of DTL, FL, DRL, and Transformers and then adopts a well-designed taxonomy to outline the state-of-the-art (SOTA) approaches. Subsequently, a critical analysis is conducted to identify the strengths and weaknesses of each framework. Additionally, a comparative study is presented to highlight the existing challenges, paving the way for future research opportunities. • Recent DL advancements challenge ASR, demanding adaptive systems. • DTL, FL, RL tackle ASR challenges: data domain differences, resource demands. • Survey reviews DTL, FL, RL-based ASR frameworks, aiding understanding. • Transformers, key in ASR, capture extensive dependencies in input sequences. • Analyzes strengths, weaknesses, and future research in ASR frameworks. [ABSTRACT FROM AUTHOR]

The integration of photovoltaics (PVs), regenerative braking (RB) techniques, and energy storage devices has become crucial to promote energy conservation and emission reduction for a sustainable future of urban rail traction networks (URTNs). This paper proposes a tri-level multi-time scale energy management framework for the economic and low-carbon operation of URTNs with PV–RB hybrid energy storage systems (HESSs) based on multi-agent deep reinforcement learning (MADRL). A two-stage stochastic scheduling approach is developed to minimize daily operation and carbon trading costs at the upper level and correct day-ahead scheduling deviations against multi-source uncertainties at the middle level. A MADRL-based real-time energy management strategy is established to optimize the PV–RB power flow and promote its utilization by coordinating distributed HESSs at the lower level. The HESS control problem is formulated as a decentralized partially observable Markov decision process and solved by a multi-agent control algorithm based on monotonic value function factorization. A Copula-based spatio-temporal dependency model is devised to characterize the PV, passenger flow, and traction load uncertainties and generate daily URTN operation scenarios for enhancing day-ahead and intraday decisions. Comparative studies demonstrate the effectiveness of the proposed framework in terms of a cost reduction by 11.98% and a PV–RB energy utilization improvement by 13.94%. • Proposal of multi-time scale energy management for sustainable urban rails. • Development of two-stage stochastic dispatch for minimizing overall daily cost. • Proposal of MADRL-based distributed HESS control for PV-RB power flow optimization. • Consideration of spatio-temporal correlations in PV outputs and passenger flows. • Performance of enhanced economic benefits and renewable energy utilization. [ABSTRACT FROM AUTHOR]

Due to the increasing high penetration of Photovoltaic (PV), it brings great challenge for voltage control issue of distribution network. To address this problem, this paper presents an improved collaborative Multi-Agent Reinforcement Learning (MARL) approach for proactive voltage control, aimed at mitigating voltage violations and minimizing power losses. The self-attention mechanism is embedded into the multi-agent soft actor-critic (MASAC) algorithm to enhance the collaboration of multi-agent system, which can well improve the learning efficiency to ensure the voltage safety of Distribution Network. In addition, the proposed learning approach is implemented on IEEE 33-bus, 141-bus and 322-bus systems, and the simulation results reveal that the proposed approach can control the voltage into safety domain as well as reduce power losses. [ABSTRACT FROM AUTHOR]

Wildfires have emerged as one of the most destructive natural disasters worldwide, causing catastrophic losses. These losses have underscored the urgent need to improve public knowledge and advance existing techniques in wildfire management. Recently, the use of Artificial Intelligence (AI) in wildfires, propelled by the integration of Unmanned Aerial Vehicles (UAVs) and deep learning models, has created an unprecedented momentum to implement and develop more effective wildfire management. Although existing survey papers have explored learning-based approaches in wildfire, drone use in disaster management, and wildfire risk assessment, a comprehensive review emphasizing the application of AI-enabled UAV systems and investigating the role of learning-based methods throughout the overall workflow of multi-stage wildfire management, including pre-fire (e.g., vision-based vegetation fuel measurement), active-fire (e.g., fire growth modeling), and post-fire tasks (e.g., evacuation planning) is notably lacking. This survey synthesizes and integrates state-of-the-science reviews and research at the nexus of wildfire observations and modeling, AI, and UAVs — topics at the forefront of advances in wildfire management, elucidating the role of AI in performing monitoring and actuation tasks from pre-fire, through the active-fire stage, to post-fire management. To this aim, we provide an extensive analysis of the existing remote sensing systems with a particular focus on the UAV advancements, device specifications, and sensor technologies relevant to wildfire management. We also examine the pre-fire and post-fire management approaches, including fuel monitoring, prevention strategies, as well as evacuation planning, damage assessment, and operation strategies. Additionally, we review and summarize a wide range of computer vision techniques in active-fire management, with an emphasis on Machine Learning (ML), Reinforcement Learning (RL), and Deep Learning (DL) algorithms for wildfire classification, segmentation, detection, and monitoring tasks. Ultimately, we underscore the substantial advancement in wildfire modeling through the integration of cutting-edge AI techniques and UAV-based data, providing novel insights and enhanced predictive capabilities to understand dynamic wildfire behavior. [Display omitted] • An in-depth review of AI-based state-of-the-art in multi-stage wildfire management. • A comprehensive analysis of employed UAV and sensor technologies for fire management. • An extensive review of recent ML, DL, and RL advancements in active-fire management. • Detailed exploration of existing strategies for pre-fire and post-fire management. • Highlights advanced wildfire modeling, focusing on integrating AI and UAV systems. [ABSTRACT FROM AUTHOR]

Operational forecasting often requires predicting collections of related, multivariate time series data that are high-dimensional in nature. This can be tackled by fitting a single function to all series (global approach) or assuming each series as a separate prediction problem and fitting one function to each (local approach) – the global–local trade-off. Deep learning models inspired by different data generation processes aim to combine the benefits of global and local approaches. Specifically, these frequently propose feeding the statistical expressiveness of classical local models into more complex global networks. Following recent trends in neural networks, the theoretical foundations of these hybrid models can also explain the surge of Transformer-based time series forecasting applications, which showcase potential benefits for the global and local equilibrium. Dynamic reinforcement learning (RL) models have also been explored to optimize the balance of global and local signals in general prediction problems, frequently through Q-learning algorithms. RL models can be proposed to dynamically adjust the influence of global versus local information to improve predictive performance at both scales. This paper conducted a concise literature review focused on these two research streams to optimize the balance between globality and locality in forecasting collections of time series. It focuses on their evolution across time and hints at opportunities to close some of the research gaps by intersecting both propositions. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) guidelines and achieved a selection of 143 publications since 2000. The main findings reveal that global models have achieved strong expressiveness in capturing the most complex structural patterns while still enabling probabilistic outcomes to be delivered through uncertainty estimates. On the other hand, RL based methods depict great benefits in mitigating the risks of generalization by imprinting contextual diversity when predicting each step ahead for each series. Within those, the adoption of other computational learning or evolutionary-based methods (e.g. Genetic Programming) to improve the parametrization of the learning policies is also highlighted as an area of future work yet to be uncovered. This review advances knowledge at the intersection of two distinct yet potentially complementary research areas, identifying opportunities to combine different methodological approaches in addressing the global–local trade-off in forecasting collections of time series. This is achieved by surfacing shared limitations in current research and presenting avenues for integrating distinct methodologies, namely by further developing the theoretical underpinnings of reinforcement learning techniques. With this work we seek to enhance understanding of the relevant research landscape and help inform future solutions by establishing a foundation for collaborative work. [Display omitted] • Review of global and local time series forecasting approaches including Deep Learning (DL) models combining global and local signals and Dynamic Reinforcement Learning (RL) methods optimizing this trade-off • Identification of opportunities to combine distinct forecasting methodologies like global DL models and RL based local modeling to address the global–local trade-off. • Search for ways to integrate RL techniques for time series forecasting by further developing their theoretical underpinnings based on limitations found in current research. • Main findings show global models achieve strong expressiveness capturing complex patterns while enabling probabilistic outcomes, and RL methods mitigate risks when predicting each step for each series [ABSTRACT FROM AUTHOR]

This paper addresses a crucial issue in the airline industry by tackling a dynamic pricing problem in the presence of patient customers, a scenario that has gained significance due to the revenue loss of airlines caused by customers' non-myopic decision-making. To effectively capture this non-myopic characteristic, we propose a Markov decision process (MDP) including a history of offered prices as a state variable. In contrast to previous studies, distributions of customers' properties are assumed to be unknown in advance for a more realistic representation of real-world scenarios. To deal with the new challenges of the problem, we propose utilizing a specific learning framework (i.e., deep exploration-based RL) that is unexplored in this domain. The numerical experiments demonstrate that its performance can be improved on the MDP we designed and show that it outperforms the benchmark algorithm. The structures of pricing policies generated from the bootstrapped deep Q-network algorithm imply that airlines should offer high and low prices alternately from the beginning of the sales period rather than increasing prices as time goes on. We also ascertain that more frequent consecutive high-priced periods can increase airlines' revenue in environments with higher customer patience levels. [ABSTRACT FROM AUTHOR]

This paper reviews exploration techniques in deep reinforcement learning. Exploration techniques are of primary importance when solving sparse reward problems. In sparse reward problems, the reward is rare, which means that the agent will not find the reward often by acting randomly. In such a scenario, it is challenging for reinforcement learning to learn rewards and actions association. Thus more sophisticated exploration methods need to be devised. This review provides a comprehensive overview of existing exploration approaches, which are categorised based on the key contributions as: reward novel states, reward diverse behaviours, goal-based methods, probabilistic methods, imitation-based methods, safe exploration and random-based methods. Then, unsolved challenges are discussed to provide valuable future research directions. Finally, the approaches of different categories are compared in terms of complexity, computational effort and overall performance. • Exploration in deep reinforcement learning is investigated comprehensively. • New classification method for exploratory approaches is designed. • Promising future research directions on exploration are discussed. [ABSTRACT FROM AUTHOR]

Today, more and more, it is necessary that most applications and documents developed in previous or current technologies to be accessible online on cloud-based infrastructures. That's why the migration of legacy systems including their hosts of documents to new technologies and online infrastructures, using modern Artificial Intelligence techniques, is absolutely necessary. With the advancement of Artificial Intelligence and Deep Learning with its multitude of applications, a new area of research is emerging – that of automated systems development and maintenance. The underlying work objective that led to this paper aims to research and develop truly intelligent systems able to analyze user interfaces from various sources and generate real and usable inferences ranging from architecture analysis to actual code generation. One key element of such systems is that of artificial scene detection and analysis based on deep learning computer vision systems. Computer vision models and particularly deep directed acyclic graphs based on convolutional modules are generally constructed and trained based on natural images datasets. Due to this fact, the models will develop during the training process natural image feature detectors apart from the base graph modules that will learn basic primitive features. In the current paper, we will present the base principles of a deep neural pipeline for computer vision applied to artificial scenes (scenes generated by user interfaces or similar). Finally, we will present the conclusions based on experimental development and benchmarking against state-of-the-art transfer-learning implemented deep vision models. [ABSTRACT FROM AUTHOR]

This paper reviews existing works on (deep) reinforcement learning considerations in electric power system control. The works are reviewed as they relate to electric power system operating states (normal, preventive, emergency, restorative) and control levels (local, household, microgrid, subsystem, wide-area). Due attention is paid to the control-related problems considerations (cyber-security, big data analysis, short-term load forecast, and composite load modelling). Observations from reviewed literature are drawn and perspectives discussed. In order to make the text compact and as easy as possible to read, the focus is only on the works published (or "in press") in journals and books while conference publications are not included. Exceptions are several work available in open repositories likely to become journal publications in near future. Hopefully this paper could serve as a good source of information for all those interested in solving similar problems. [ABSTRACT FROM AUTHOR]

Highlights • Multiple optimization targets are considered for a generalized energy Internet. • Power supply-demand balance is realized in the entire energy Internet system. • A model-free approach is applied with real-world power data. • Advanced deep reinforcement learning approach is applied. Abstract This paper investigates the energy management problem in the field of energy Internet (EI) with interdisciplinary techniques. The concept of EI has been proposed for a while. However, there still exist many fundamental and technical issues that have not been fully investigated. In this paper, a new energy regulation issue is considered based on the operational principles of EI. Multiple targets are considered along with constraints. Then, the practical energy management problem is formulated as a constrained optimal control problem. Notably, no explicit mathematical model for power of renewable power generation devices and loads is utilized. Due to the complexity of this problem, conventional methods appear to be inapplicable. To obtain the desired control scheme, a model-free deep reinforcement learning algorithm is applied. A practical solution is obtained, and the feasibility as well as the performance of the proposed method are evaluated with numerical simulations. [ABSTRACT FROM AUTHOR]

Highlights • Review of the application of reinforcement learning (RL) for demand response (DR) • DR is relevant for integrating renewable energy source into the smart grid. • Considering RL/DR from energy generation and storage, to supply and user satisfaction. • Typical algorithms and open research directions are discussed. • Most articles focus on single-agent systems in stationary environments. Abstract Buildings account for about 40% of the global energy consumption. Renewable energy resources are one possibility to mitigate the dependence of residential buildings on the electrical grid. However, their integration into the existing grid infrastructure must be done carefully to avoid instability, and guarantee availability and security of supply. Demand response, or demand-side management, improves grid stability by increasing demand flexibility, and shifts peak demand towards periods of peak renewable energy generation by providing consumers with economic incentives. This paper reviews the use of reinforcement learning, a machine learning algorithm, for demand response applications in the smart grid. Reinforcement learning has been utilized to control diverse energy systems such as electric vehicles, heating ventilation and air conditioning (HVAC) systems, smart appliances, or batteries. The future of demand response greatly depends on its ability to prevent consumer discomfort and integrate human feedback into the control loop. Reinforcement learning is a potentially model-free algorithm that can adapt to its environment, as well as to human preferences by directly integrating user feedback into its control logic. Our review shows that, although many papers consider human comfort and satisfaction, most of them focus on single-agent systems with demand-independent electricity prices and a stationary environment. However, when electricity prices are modelled as demand-dependent variables, there is a risk of shifting the peak demand rather than shaving it. We identify a need to further explore reinforcement learning to coordinate multi-agent systems that can participate in demand response programs under demand-dependent electricity prices. Finally, we discuss directions for future research, e.g., quantifying how RL could adapt to changing urban conditions such as building refurbishment and urban or population growth. [ABSTRACT FROM AUTHOR]

Graphical abstract Abstract In this paper, artificial Intelligence (AI) algorithm is used in the Rietveld refinement process instead of the human decision. The program, PowderBot, is developed based on Fullprof engine, which proves the effectiveness of AI in Rietveld refinement. In this program, the decision making in refinement process is modelled as a Markov decision process (MDP), and solved by a reinforcement learning algorithm. PowderBot is designed to be a self-learning system capable of conducting structure refinement without human intervention. The program has already been successfully applied to Rietveld refinements. We hope this paper will encourage more Rietveld programs become more intelligent by the help of AI algorithm. [ABSTRACT FROM AUTHOR]

Collecting samples from extraterrestrial body is of great significance for deep space scientific exploration. This paper proposes an autonomous grasping method to collect unknown natural samples from unstructured extraterrestrial surface in a safe, gentle and robust way. A deep reinforcement learning based end-to-end grasping pose estimation framework is designed. The proposed framework takes visual information as input and learns an appropriate grasping strategy. A feature extraction deep neural network and a reinforcement learning policy network are trained simultaneously so as to obtain light-weight networks for extraterrestrial detector. Meanwhile, in order to protect the sample from being damaged during grasping, collision detection is incorporated into the closed-loop during training. The grasping strategy is trained in simulation and is then transferred to real-world. Simulation and real-world experiments show that the learned policy can adapt to unseen irregular stones with less times of collision and at a high grasp success rate under single, scattered, and cluttered scenes. • Autonomous grasping method is proposed to collect unknown natural samples. • Deep reinforcement learning based end-to-end grasping pose estimation framework is designed.. • Light-weight networks is built for extraterrestrial detector. • Gentle grasping behavior is achieved for safety and reliability concern. • The learned policy is verified by simulation and real-world experiments. [ABSTRACT FROM AUTHOR]

UAV base station platform has become the current research hotspot of assisting ground base station for wireless coverage.At present, the most important issue is how to make path planning to provide the stable communication guarantee for multiple mobile users. In this article, we model the air-to-ground channel to describe the path loss between the UAV platform and the user and build a simulation environment for training based on the OpenAI-GYM architecture. In addition, this paper proposes a reinforcement learning algorithm based on intrinsic rewards, which uses the mean square error of the state prediction results to quantify the novelty of the state. Algorithms enable agents to efficiently carry out strategy iterations. Experiments results showed that our algorithm has a higher score and takes less time. [ABSTRACT FROM AUTHOR]

Reactive power control of PV inverters can be applied to mitigate the voltage increase caused by reverse power flow and voltage fluctuations caused by PV output fluctuations in the distribution system. This paper focuses on the Volt–Var control of PV smart inverters to minimize power losses. It proposes a multi-agent type cooperative voltage control framework to optimize the blind band and slope of the VVC. The proposed method utilizes the optimized VVC to eliminate voltage deviations using only local information. The optimization problem is formed as a Markov decision process and solved using a deep reinforcement learning algorithm called multi-agent soft-actor critic, which achieves fast control that fully accounts for uncertainty while achieving global optimization. Therefore, the proposed method can contribute to reducing power loss, which is a system-wide problem, with only local information. Case studies were conducted using a small and medium-sized distribution network model with IEEE 33 buses and 12 demand cases. The comparison results demonstrate that the proposed method is superior to other benchmark methods in terms of minimizing power losses and mitigating voltage variations, as the proposed method can reduce power losses by about 9% while achieving avoidance of voltage deviations. • Multi-agent reinforcement learning is needed when some agents are in the environment. • We utilize a multi-agent Soft Actor-Critic with SAC applied to multi-agent DDPG. • Agents with different time scales determine the slope and dead band of the VVC. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of distributed energy management in multi-area integrated energy systems (MA-IES) using a multi-agent deep reinforcement learning approach. The MA-IES consists of interconnected electric and thermal networks, incorporating renewable energy sources and heat conversion systems. The objective is to optimize the operation of the system while minimizing operational costs and maximizing renewable energy utilization. We propose a distributed energy management strategy that makes hierarchical decisions on intra-area heat energy and inter-area electric energy. The strategy is based on a multi-agent deep reinforcement learning framework, where each agent represents a component or unit in the MA-IES. We formulate the problem as a Markov Decision Process and employ Q-learning with experience replay and double networks to train the agents. The proposed strategy is evaluated using a simulation of a four-area MA-IES. The results demonstrate significant improvements in energy management compared to traditional methods, with higher renewable energy utilization and lower operational costs. Specifically, the strategy achieves 100% utilization of wind power, and decreases operational costs by 5.53%. Furthermore, it leverages the generalization capabilities of reinforcement learning to respond in real-time to uncertainties in demand and wind power output. The results highlight the advantages of the proposed strategy, making it a promising solution for optimizing the operation of multi-area integrated energy systems. • Hierarchical framework prioritizes heat-electric energy balance, facilitates joint optimization, enhances system performance. • MADRL-based approach manages dynamic loads, uncertainties, renewables in distributed energy systems. • Proposed method optimizes energy sources, demand, grid constraints in multi-area integration. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning (DRL) has achieved considerable success in games involving perfect and imperfect information, such as Go, Texas Hold'em, Stratego, and DouDiZhu. Nevertheless, training a state-of-the-art model for complex imperfect-information card games like DouDiZhu and Big2 remains resource and time-intensive. To address this challenge, this paper introduces two innovative methods: the Opponent Model and Optimized Deep Monte-Carlo (ODMC). These methods are designed to improve the training efficiency of Deep Monte-Carlo (DMC) for imperfect-information card games. The Opponent Model predicts hidden information, enhancing the agent's learning speed in DMC compared to the original training that only utilizes observed information as input features. In ODMC, the Minimum Combination Search (MCS) is a heuristic search algorithm based on dynamic programming. It calculates the minimum combination of actions in the current state, and ODMC uses MCS to filter suboptimal actions in each state. This reduces the action space considered by DMC, resulting in faster training that focuses on evaluating the most promising actions. The effectiveness of the proposed approach is evaluated by examining two complex card games with imperfect information: DouDiZhu and Big2. Ablation experiments are conducted to evaluate both the Opponent Model (D+OM and B+OM) and ODMC (D+ODMC and B+ODMC), along with their combined variants (D+OMODMC and B+OMODMC). Furthermore, D+OMODMC and B+OMODMC are compared with state-of-the-art DouDiZhu and Big2 artificial intelligence (AI) programs, respectively. The experimental results demonstrate that the proposed methods achieve comparable performance to the original DMC, but with only 25.5% of the training time on the same device. These findings are valuable for mitigating both the equipment requirements and training time in complex imperfect-information card games. • Propose a novel Opponent Model and ODMC to improve the learning efficiency of DMC. • Opponent Model predicts hidden information, enhancing learning. • ODMC uses MCS to filter suboptimal actions, speeding up training. • Proposed methods achieve comparable performance with reduced training time. [ABSTRACT FROM AUTHOR]

The financial market, as a complex nonlinear dynamic system frequently influenced by various factors, such as international investment capital, is very challenging to build trading strategies from the obtained market information. Deep Reinforcement Learning (DRL) combines the perceptual capability of deep learning and the control decision making capability of reinforcement learning to learn the mapping between financial market states and trading decisions by interacting with the environment. In this paper, an enhanced stock trading strategy, denominated efficient deep State-Action-Reward-State-Action (SARSA), is presented to tackle the algorithmic trading problem of determining optimal trading positions in the daily trading activities of the stock market. This algorithm is recognized for its properties of stable learning and convergence, attributes of critical significance within the financial domain, where stability assumes paramount importance, and excessive risk-taking should be averted. Furthermore, a novel deep network architecture called Bidirectional Long Short-Term Memory (BiLSTM)-Attention is introduced to address the challenge of accurately presenting the complex and volatile stock market. The BiLSTM-Attention greatly enhances the network's capacity to recognize key features and patterns in stock market data, allowing agents to focus on the most relevant aspects of the data. Evaluations on DJI, SP500, GE, IXIC datasets from January 1, 2008 to December 31, 2022 show that our efficient deep SARSA algorithm outperforms a wide range of traditional strategies (B&H, S&H, MR, TF) and DRL-based strategies (TDQN, DQN-Vanilla). For example, on the IXIC dataset, the efficient deep SARSA strategy achieves an attractive Cumulative Return (CR) of 582.17% and a Sharpe Ratio (ShR) of 1.86, outperforming all other methods. These experimental results prove the performance of our method in enhancing stock trading strategies. • Proposed the efficient deep SARSA model for algorithmic trading. • Proposed a network BiLSTM-Attention to extract key features in stock data. • The proposed efficient deep SARSA outperforms other baseline methods (TDQN etc.). [ABSTRACT FROM AUTHOR]

• Achieved automatic segmentation using "Deep Reinforcement Learning" algorithms. • Original architecture design that uses Deep Q-Network (DQN) agents for heart segmentation. • Original architecture design that uses Proximal Policy Optimization (PPO) agents for heart segmentation. • Both off-policy approaches using discrete actions and on-policy approaches using continuous actions achieve segmentation generalization. In the automated segmentation field, the most promising results were obtained using Deep learning (DL) architectures that employ a mix of "Fully convolutional neural networks" (FCN), "Generative adversarial networks" (GAN), or "Recurrent neural networks" (RNN). In our paper we aim to challenge the status-quo of using supervised learning algorithms by proposing and implementing different Deep Reinforcement learning (DRL) architectures that execute the same task – human organ automatic segmentation. We present a completely new approach that defines the segmentation as a Markov Decision Process (MDP) problem where an agent navigates in the environment (CTs) and learns a policy to delineate the contour of the human organ. For solving this problem, we utilized different architectures built on DRL agents. The first architecture employs the Deep Q-Network (DQN) algorithm and uses discrete actions to navigate through CTs while executing the automatic segmentation for the heart. The second architecture implements the Proximal Policy Optimization (PPO) algorithm and utilizes a continuous action space in order to execute the same segmentation process. We tested these two architectures in different setups and using varied reward systems and our results prove that DRL can be used successfully for automated medical segmentation. We also proved that both off-policy, discrete action algorithms or on-policy, continuous action algorithms converge to generalized results. [ABSTRACT FROM AUTHOR]

In the Internet of Things (IoT), a huge amount of valuable data is generated by various IoT applications. As the IoT technologies become more complex, the attack methods are more diversified and can cause serious damages. Thus, establishing a secure IoT network based on user trust evaluation to defend against security threats and ensure the reliability of data source of collected data have become urgent issues, in this paper, a Data Fusion and transfer learning empowered granular Trust Evaluation mechanism (DFTE) is proposed to address the above challenges. Specifically, to meet the granularity demands of trust evaluation, time–space empowered fine/coarse grained trust evaluation models are built utilizing deep transfer learning algorithms based on data fusion. Moreover, to prevent privacy leakage and task sabotage, a dynamic reward and punishment mechanism is developed to encourage honest users by dynamically adjusting the scale of reward or punishment and accurately evaluating users' trusts. The extensive experiments show that: (i) the proposed DFTE achieves high accuracy of trust evaluation under different granular demands through efficient data fusion; (ii) DFTE performs excellently in participation rate and data reliability. • To reduce the model training time, TL is applied for the unified trust evaluation. • To meet granular demands, two time–space based trust evaluation models are built. • To encourage honest users, a reward–punishment mechanism is developed using DRL. • Experiments show the proposed strategy performs excellently in trust evaluation. [ABSTRACT FROM AUTHOR]

• A co-simulation platform for Hybrid Electric Vehicles is developed for virtual assessment of Energy Management, performance, CO 2 & pollutant emissions. • CO 2 & pollutant emissions models are based on convolutional neural network for accurate transient state prediction. • A continuous deep reinforcement learning based EMS of HEV is proposed. • DDQL is trained with random generated driving cycles based on Markov Chain approach from real recordings. • Cumulative d error for predicted pollutants emissions is below 8.5% while HEV simulation is running up to 10 times faster than real time in a standard laptop. The Energy Management Strategy (EMS) in an HEV is the key for improving fuel economy and simultaneously reducing pollutant emissions. This paper presents a methodology for developing hybrid models that enable EMS testing as well as the evaluation of fuel consumption, CO 2 and pollutant emissions (CO, NO x and THC). In this context, pollutant emissions are hard to quantify with static models such as the well-known map-based approach which is mainly due to the pronounced impact of transient effects. The novelty of this paper primarily comes from the characterization of pollutant emissions through Convolutional Neural Networks (CNN), providing high accuracy for both, instantaneous and cumulative values. The input parameters are classical Internal Combustion Engine (ICE) measurements such as engine speed, air mass flow, torque and exhaust temperature. The proposed CNNs are reduced to a minimum for low complexity and fast computability. These models are developed with experimental data from chassis dyno testing of a conventional turbo-charged gasoline engine vehicle. The pollutant emission models are used in conjunction with physical models of the remaining powertrain allowing for real time simulations of the complete HEV vehicle. The Double Deep-Q learning algorithm is proposed for the EMS and compared to the Dynamic programming (DP) solution. The introduced methodology is developed in a co-simulation framework between MATLAB-Simulink and AMESIM. The resulting model runs between 8 and 10 times faster than real time in an off-the-shelf PC. This provides the capability for developing models suitable for HIL (hardware-in-the-loop) and SIL (software-in-the-loop) applications. The final error in predicted CO 2 remains below 2.5% while the final cumulative error for pollutants is below 8.5% in the case of CO and HC emissions. [ABSTRACT FROM AUTHOR]

The Resource Provisioning (RP) and Task Scheduling (TS) issues has become an attractive paradigms in cloud industry, this is due to the increasing demand for the services provided by virtual machines that are structured by physical servers owned by the data centers of cloud service providers (CSPs). In this paper, we propose a new model based on multi-agent system for the RP and TS reducing the cost of energy using Deep Reinforcement Learning DRL. A Quantile Regression Deep Q Network (QR-DQN) algorithm generates an appropriate policy and the optimal long-term decisions. A set of experiments show the efficiency of our proposed scheduling approach and the performance of our task allocation method.. [ABSTRACT FROM AUTHOR]

We have interested in a conscious decision-making system in an environment where multiple types of rewards and penalties exist. We know a method using a basis function and a method using a penalty avoidance list for this problem. Though the method using the penalty avoidance list is considered promising compared to the former, it has the problem that when all actions are registered in the avoidance list, the actions are selected at random. We propose a method for selecting actions using deep reinforcement learning in order to avoid such random selection as much as possible in this paper. The effectiveness of the proposed method is confirmed by numerical experiments. [ABSTRACT FROM AUTHOR]