Search

Predicting stock price movements remains a major challenge in time series analysis. Despite extensive research on various machine learning techniques, few models have consistently achieved success in automated stock trading. One of the main challenges in stock price forecasting is that the optimal model changes over time due to market dynamics. This paper aims to predict stock prices using automated reinforcement learning algorithms and to analyse their efficiency compared with conventional methods. We automate DQN models and their variants, known for their adaptability, by continuously retraining them using recent data to capture market dynamics. We demonstrate that our dynamic models improve the accuracy of predicting the directions of various DAX stocks from 50.00% to approximately 60.00%, compared with conventional methods. Additionally, we conclude that dynamic models should be updated in response to shifts rather than at fixed intervals. [ABSTRACT FROM AUTHOR]

Energy sources are critical in the industrial sector, particularly as population growth intensifies the pressure on industries to scale up production to meet increasing demands. Integrating renewable energy sources such as biomass, biogas, and photovoltaic systems in the palm oil production process can be considered a pivotal strategy for mitigating carbon emissions. However, load fluctuations due to mill size and insufficient supply of biogas and photovoltaic are significant challenges in energy management for the palm oil industry. Adopting renewable energy solutions is hindered by process disturbances and climatic variability caused by weather, ambient temperature, relative humidity, and solar radiation. These factors pose constraints on the implementation of renewable energy solutions. Therefore, this study aims to fill the gap in terms of renewable energy management within the palm oil industry by applying reinforcement learning to achieve effective energy management and reduce carbon emissions under five climatic scenarios. The result shows that reinforcement learning effectively combines renewable energy sources to generate electricity and steam while efficiently managing energy storage without exceeding predefined levels. Additionally, deploying a photovoltaic system contributes to energy savings from biomass and biogas of 2.9% and reduces average daily carbon dioxide emissions by 3137 kg. The proposed method has proved beneficial across all climate scenarios for energy savings and emission reduction. [ABSTRACT FROM AUTHOR]

In cloud computing, task scheduling is a critical process that involves efficiently allocating computing resources to fulfill diverse task requirements. To address issues such as unstable response times, extensive computations, and challenges in parameter adjustment faced by traditional task scheduling methods, an enhanced deep Q-learning cloud-task-scheduling algorithm was proposed. This algorithm utilizes deep reinforcement learning and introduces an improved strategy. The optimization of the objective function was achieved by defining the state space, action space, and reward function. The agent's exploration capability was enhanced through the utilization of a UCB exploration strategy and Boltzmann action exploration. Simulation experiments were conducted using Pycloudsim. The average instruction response time ratio and standard deviation of CPU utilization were compared to measure the advantages and disadvantages of the algorithm. The results indicate that the proposed algorithm surpasses the random, earliest, and RR algorithms in terms of the instruction-to-response time ratio and CPU utilization, demonstrating enhanced efficiency and performance in cloud-task scheduling. [ABSTRACT FROM AUTHOR]

As modern systems become more complex, their control strategy no longer relies only on measurement data from probes; it also requires information from mathematical models for non-measurable places. On the other hand, those mathematical models can lead to unbearable computation times due to their own complexity, making the control process non-viable. To overcome this problem, it is possible to implement any kind of surrogate model that enables the computation of such estimates within an acceptable time frame, which allows for making decisions. Using a Physics-Informed Neural Network as a surrogate model, it is possible to compute the temperature distribution at each time step, replacing the need for running direct numerical simulations. This approach enables the use of a Deep Reinforcement Learning algorithm to train a control strategy. On this work, we considered a one-dimensional heat conduction problem, in which temperature distribution feeds a control system. Such control system has the objective of reacing and maintaining constant temperature value at a specific location of the 1D problem by activating a heat source; the desired location somehow cannot be directly measured so, the PINN approach allows to estimate its temperature with a minimum computational workload. With this approach, the control training becomes much faster without the need of performing numerical simulations or laboratory measurements. [ABSTRACT FROM AUTHOR]

During the movement of a robotic arm, collisions can easily occur if the arm directly grasps at multiple tightly stacked objects, thereby leading to grasp failures or machine damage. Grasp success can be improved through the rearrangement or movement of objects to clear space for grasping. This paper presents a high-performance deep Q-learning framework that can help robotic arms to learn synchronized push and grasp tasks. In this framework, a grasp quality network is used for precisely identifying stable grasp positions on objects to expedite model convergence and solve the problem of sparse rewards caused during training because of grasp failures. Furthermore, a novel reward function is proposed for effectively evaluating whether a pushing action is effective. The proposed framework achieved grasp success rates of 92% and 89% in simulations and real-world experiments, respectively. Furthermore, only 200 training steps were required to achieve a grasp success rate of 80%, which indicates the suitability of the proposed framework for rapid deployment in industrial settings. [ABSTRACT FROM AUTHOR]

Recent work on decentralized computational trust models for open multi-agent systems has resulted in the development of CA, a biologically inspired model which focuses on the trustee's perspective. This new model addresses a serious unresolved problem in existing trust and reputation models, namely the inability to handle constantly changing behaviors and agents' continuous entry and exit from the system. In previous work, we compared CA to FIRE, a well-known trust and reputation model, and found that CA is superior when the trustor population changes, whereas FIRE is more resilient to the trustee population changes. Thus, in this paper, we investigate how the trustors can detect the presence of several dynamic factors in their environment and then decide which trust model to employ in order to maximize utility. We frame this problem as a machine learning problem in a partially observable environment, where the presence of several dynamic factors is not known to the trustor, and we describe how an adaptable trustor can rely on a few measurable features so as to assess the current state of the environment and then use Deep Q-Learning (DQL), in a single-agent reinforcement learning setting, to learn how to adapt to a changing environment. We ran a series of simulation experiments to compare the performance of the adaptable trustor with the performance of trustors using only one model (FIRE or CA) and we show that an adaptable agent is indeed capable of learning when to use each model and, thus, perform consistently in dynamic environments. [ABSTRACT FROM AUTHOR]

Path planning is a fundamental task for autonomous mobile robots (AMRs). Classic approaches provide an analytical solution by searching for the trajectory with the shortest distance; however, reinforcement learning (RL) techniques have been proven to be effective in solving these problems with the experiences gained by agents in real time. This study proposes a reward function that motivates an agent to select the shortest path with fewer turns. The solution to the RL technique is obtained via dynamic programming and Deep Q-Learning methods. In addition, a path-tracking control design is proposed based on the Lyapunov candidate function. The results indicate that RL algorithms show superior performance compared to classic A* algorithms. The number of turns is reduced by 50%, resulting in a decrease in the total distance ranging from 3.2% to 36%. [ABSTRACT FROM AUTHOR]

As modern systems become more complex, their control strategy no longer relies only on measurement data from probes; it also requires information from mathematical models for non-measurable places. On the other hand, those mathematical models can lead to unbearable computation times due to their own complexity, making the control process non-viable. To overcome this problem, it is possible to implement any kind of surrogate model that enables the computation of such estimates within an acceptable time frame, which allows for making decisions. Using a Physics-Informed Neural Network as a surrogate model, it is possible to compute the temperature distribution at each time step, replacing the need for running direct numerical simulations. This approach enables the use of a Deep Reinforcement Learning algorithm to train a control strategy. On this work, we considered a one-dimensional heat conduction problem, in which temperature distribution feeds a control system. Such control system has the objective of reacing and maintaining constant temperature value at a specific location of the 1D problem by activating a heat source; the desired location somehow cannot be directly measured so, the PINN approach allows to estimate its temperature with a minimum computational workload. With this approach, the control training becomes much faster without the need of performing numerical simulations or laboratory measurements.

Abstract Energy sources are critical in the industrial sector, particularly as population growth intensifies the pressure on industries to scale up production to meet increasing demands. Integrating renewable energy sources such as biomass, biogas, and photovoltaic systems in the palm oil production process can be considered a pivotal strategy for mitigating carbon emissions. However, load fluctuations due to mill size and insufficient supply of biogas and photovoltaic are significant challenges in energy management for the palm oil industry. Adopting renewable energy solutions is hindered by process disturbances and climatic variability caused by weather, ambient temperature, relative humidity, and solar radiation. These factors pose constraints on the implementation of renewable energy solutions. Therefore, this study aims to fill the gap in terms of renewable energy management within the palm oil industry by applying reinforcement learning to achieve effective energy management and reduce carbon emissions under five climatic scenarios. The result shows that reinforcement learning effectively combines renewable energy sources to generate electricity and steam while efficiently managing energy storage without exceeding predefined levels. Additionally, deploying a photovoltaic system contributes to energy savings from biomass and biogas of 2.9% and reduces average daily carbon dioxide emissions by 3137 kg. The proposed method has proved beneficial across all climate scenarios for energy savings and emission reduction.

Recent work on decentralized computational trust models for open multi-agent systems has resulted in the development of CA, a biologically inspired model which focuses on the trustee’s perspective. This new model addresses a serious unresolved problem in existing trust and reputation models, namely the inability to handle constantly changing behaviors and agents’ continuous entry and exit from the system. In previous work, we compared CA to FIRE, a well-known trust and reputation model, and found that CA is superior when the trustor population changes, whereas FIRE is more resilient to the trustee population changes. Thus, in this paper, we investigate how the trustors can detect the presence of several dynamic factors in their environment and then decide which trust model to employ in order to maximize utility. We frame this problem as a machine learning problem in a partially observable environment, where the presence of several dynamic factors is not known to the trustor, and we describe how an adaptable trustor can rely on a few measurable features so as to assess the current state of the environment and then use Deep Q-Learning (DQL), in a single-agent reinforcement learning setting, to learn how to adapt to a changing environment. We ran a series of simulation experiments to compare the performance of the adaptable trustor with the performance of trustors using only one model (FIRE or CA) and we show that an adaptable agent is indeed capable of learning when to use each model and, thus, perform consistently in dynamic environments.

Copyright of Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım ve Teknoloji is the property of Gazi University and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In the complex and dynamic landscape of cyber threats, organizations require sophisticated strategies for managing Cybersecurity Operations Centers and deploying Security Information and Event Management systems. Our study enhances these strategies by integrating the precision of well-known biomimetic optimization algorithms—namely Particle Swarm Optimization, the Bat Algorithm, the Gray Wolf Optimizer, and the Orca Predator Algorithm—with the adaptability of Deep Q-Learning, a reinforcement learning technique that leverages deep neural networks to teach algorithms optimal actions through trial and error in complex environments. This hybrid methodology targets the efficient allocation and deployment of network intrusion detection sensors while balancing cost-effectiveness with essential network security imperatives. Comprehensive computational tests show that versions enhanced with Deep Q-Learning significantly outperform their native counterparts, especially in complex infrastructures. These results highlight the efficacy of integrating metaheuristics with reinforcement learning to tackle complex optimization challenges, underscoring Deep Q-Learning's potential to boost cybersecurity measures in rapidly evolving threat environments. [ABSTRACT FROM AUTHOR]

With the development of artificial intelligence technology, deep reinforcement learning (DRL) has become a major approach to the design of intelligent vehicle-to-everything (V2X) routing protocols for vehicular ad hoc networks (VANETs). However, if the V2X routing protocol does not consider both real-time traffic conditions and historical vehicle trajectory information, the source vehicle may not transfer its packet to the correct relay vehicles and, finally, to the destination. Thus, this kind of routing protocol fails to guarantee successful packet delivery. Using the greater network flexibility and scalability of the software-defined network (SDN) architecture, this study designs a two-phase integrated DQN and RF Packet Routing Framework (IDRF) that combines the deep Q-learning network (DQN) and random forest (RF) approaches. First, the IDRF offline phase corrects the vehicle's historical trajectory information using the vehicle trajectory continuity algorithm and trains the DQN model. Then, the IDRF real-time phase judges whether vehicles can meet each other and makes a real-time routing decision to select the most appropriate relay vehicle after adding real-time vehicles to the VANET. In this way, the IDRF can obtain the packet transfer path with the shortest end-to-end delay. Compared to two DQN-based approaches, i.e., TDRL-RP and VRDRT, and traditional VANET routing algorithms, the IDRF exhibits significant performance improvements for both sparse and congested periods during intensive simulations of the historical GPS trajectories of 10,357 taxis within Beijing city. Performance improvements in the average packet delivery ratio, end-to-end delay, and overhead ratio of the IDRF over TDRL-RP and VRDRT under different numbers of pairs and transmission ranges are at least 3.56%, 12.73%, and 5.14% and 6.06%, 11.84%, and 7.08%, respectively. [ABSTRACT FROM AUTHOR]

Purpose: Traditional techniques for automating the planning of brain electrode placement based on multi-objective optimization involving many parameters are subject to limitations, especially in terms of sensitivity to local optima, and tend to be replaced by machine learning approaches. This paper explores the feasibility of using deep reinforcement learning (DRL) in this context, starting with the single-electrode use-case of deep brain stimulation (DBS). Methods: We propose a DRL approach based on deep Q-learning where the states represent the electrode trajectory and associated information, and actions are the possible motions. Deep neural networks allow to navigate the complex state space derived from MRI data. The chosen reward function emphasizes safety and accuracy in reaching the target structure. The results were compared with a reference (segmented electrode) and a conventional technique. Results: The DRL approach excelled in navigating the complex anatomy, consistently providing safer and more precise electrode placements than the reference. Compared to conventional techniques, it showed an improvement in accuracy of 2.3% in average proximity to obstacles and 19.4% in average orientation angle. Expectedly, computation times rose significantly, from 2 to 18 min. Conclusion: Our investigation into DRL for DBS electrode trajectory planning has showcased its promising potential. Despite only delivering modest accuracy gains compared to traditional methods in the single-electrode case, its relevance for problems with high-dimensional state and action spaces and its resilience against local optima highlight its promising role for complex scenarios. This preliminary study constitutes a first step toward the more challenging problem of multiple-electrodes planning. [ABSTRACT FROM AUTHOR]

The decision-making of generator start-up sequence plays a pivotal role in the power system restoration process following the blackout. In this paper, an optimized deep Q-learning network (DQN) algorithm is proposed to address this challenge. The generator start-up process is modeled as a Markov Decision Process (MDP) based on its characteristics. The DQN is tasked with deciding both the generator start-up sequence and the corresponding restoration path. To address the limitations of DQN, such as low exploration efficiency and slow convergence, the study incorporates the Artificial Potential Field (APF) algorithm to refine the reward function of it. This integration results in the development of the F-DQN (APF-DQN) algorithm, which enhances training efficiency. The effectiveness of this proposed method is demonstrated through the IEEE 39-bus test system. The results reveal that the DQN algorithm is capable of efficiently solving the model of the generator start-up sequence after the blackout. Moreover, the F-DQN algorithm exhibits superior learning efficiency, faster convergence, and higher-quality optimal solutions compared to the DQN. This paper also discusses the applicability of this method under partial blackouts. When compared to other decision-making algorithms, the proposed method offers a restoration scheme that is both time-efficient and results in increased electricity generation. [ABSTRACT FROM AUTHOR]

The diversity of services and infrastructure in metropolitan edge-to-cloud network(s) is rising to unprecedented levels. This is causing a rising threat of a wider range of cyber attacks coupled with a growing integration of a constrained range of infrastructure, particularly seen at the network edge. Deep reinforcement-based learning is an attractive approach to detecting attacks, as it allows less dependency on labeled data with better ability to classify different attacks. However, current approaches to learning are known to be computationally expensive (cost), and the learning experience can be negatively impacted by the presence of outliers and noise (quality). This work tackles both the cost and quality challenges with a novel service-based federated deep reinforcement learning solution, enabling anomaly detection and attack classification at a reduced data cost and with better quality. The federated settings in the proposed approach enable multiple edge units to create clusters that follow a bottom-up learning approach. The proposed solution adapts a deep Q-learning network (DQN) for service-tunable flow classification and introduces a novel federated DQN (FDQN) for federated learning. Through such targeted training and validation, variation in data patterns and noise is reduced. This leads to improved performance per service with lower training cost. Performance and cost of the solution, along with sensitivity to exploration parameters, are evaluated using examples of publicly available datasets (UNSW-NB15 and CIC-IDS2018). Evaluation results show the proposed solution to maintain detection accuracy in the range of ≈75–85% with lower data supply while improving the classification rate by a factor of ≈2. [ABSTRACT FROM AUTHOR]

The continuous evolution of artificial intelligence and cyber–physical systems has presented promising opportunities for optimizing traffic signal control in densely populated urban areas, with the aim of alleviating traffic congestion. One area that has garnered significant interest from both researchers and practitioners is the application of deep reinforcement learning (DRL) in traffic signal control. However, DRL-based algorithms often suffer from instability due to the dynamic nature of traffic flows. Discrepancies between the environments used for training and those encountered during deployment often lead to operational failures. Moreover, conventional DRL-based traffic signal control algorithms tend to reveal vulnerabilities when faced with unforeseen events, such as sensor failure. These challenges highlight the need for innovative solutions to enhance the robustness and adaptability of such systems. To address these pertinent issues, this paper introduces StageLight, a novel two-stage multiscale learning approach, which involves learning optimal timings on a coarse time scale in stage 1, while finetuning them on a finer time scale in stage 2. Our experimental results demonstrate StageLight's remarkable capability to generalize across diverse traffic conditions and its robustness to various sensor-failure scenarios. [ABSTRACT FROM AUTHOR]

This paper proposes a deep reinforcement learning-based framework for robot autonomous grasping and assembly skill learning. Meanwhile, a deep Q-learning-based robot grasping skill learning algorithm and a PPO-based robot assembly skill learning algorithm are presented, where a priori knowledge information is introduced to optimize the grasping action and reduce the training time and interaction data needed by the assembly strategy learning algorithm. Besides, a grasping constraint reward function and an assembly constraint reward function are designed to evaluate the robot grasping and assembly quality effectively. Finally, the effectiveness of the proposed framework and algorithms was verified in both simulated and real environments, and the average success rate of grasping in both environments was up to 90%. Under a peg-in-hole assembly tolerance of 3 mm, the assembly success rate was 86.7% and 73.3% in the simulated environment and the physical environment, respectively. [ABSTRACT FROM AUTHOR]

Economic and policy factors are driving the continuous increase in the adoption and usage of electrical vehicles (EVs). However, despite being a cleaner alternative to combustion engine vehicles, EVs have negative impacts on the lifespan of microgrid equipment and energy balance due to increased power demands and the timing of their usage. In our view, grid management should leverage on EV scheduling flexibility to support local network balancing through active participation in demand response programs. In this paper, we propose a model-free solution, leveraging deep Q-learning to schedule the charging and discharging activities of EVs within a microgrid to align with a target energy profile provided by the distribution system operator. We adapted the Bellman equation to assess the value of a state based on specific rewards for EV scheduling actions and used a neural network to estimate Q-values for available actions and the epsilon-greedy algorithm to balance exploitation and exploration to meet the target energy profile. The results are promising, showing the effectiveness of the proposed solution in scheduling the charging and discharging actions for a fleet of 30 EVs to align with the target energy profile in demand response programs, achieving a Pearson coefficient of 0.99. This solution also demonstrates a high degree of adaptability in effectively managing scheduling situations for EVs that involve dynamicity, influenced by various state-of-charge distributions and e-mobility features. Adaptability is achieved solely through learning from data without requiring prior knowledge, configurations, or fine-tuning. [ABSTRACT FROM AUTHOR]

Traffic congestion has become a major problem in this rapidly growing world. Everyone operating a vehicle, as well as the traffic police in charge of managing the traffic, finds it difficult to become stuck in heavy traffic. For this a set, predetermined timing for traffic flow for each direction at the junction is utilized by traditional traffic light controllers. However, the concept of a fixed time traffic signal controller does not work well in places with uneven traffic. A dynamic traffic control system is therefore required, which regulates the traffic signals in accordance with the volume of traffic. This paper proposes a model that uses reinforcement learning (RL) along with deep neural networks (DNN) to manage discretions (signal status) for an environment with the help of Simulation of Urban MObility (SUMO). A simulation of real-world environment consisting a network of Four-way crossroad junction that contains 4 arriving lanes and 4 exiting lanes is used to train the agent. The main objective of this research study is to construct a model that can independently determine the best course of action and aims to provide better traffic management that will decrease the average waiting time, cause lower congestion, and provide a smooth flow of traffic. [ABSTRACT FROM AUTHOR]

Small cell technology for future 6G networks allows network operators to increase network capacity by reducing the distance between Base Stations (BSs) and users, thereby increasing wireless channel gains. However, it also leads to significant computational complexity to optimally mitigate inter-cell and/or inter-beam interference by dynamically managing beamforming, transmit power and user scheduling. In this paper, we formulate an optimization problem aiming to maximize the sum utility of users where decision variables are beam pattern selection, user scheduling and transmit power allocation in small cell networks. Next, we capture room for performance enhancement and low computational complexity that existing studies have overlooked by proposing i) a novel decision making process of DQN (Deep Q-Network) to jointly learn all decision variables in a single DRL (Deep Reinforcement Learning) model without a curse of dimensionality by adopting a user-specific state to each agent with distributed interference approximation meaning that interferences to all users in all neighbor BSs can be abstracted by a single user, and ii) a novel reward design so that the reward is judged by the result of a practical optimization-based solution. Finally, we show the superiority of the proposed DQL (Deep Q-Learning) algorithm compared to the existing interference management algorithms via simulations and provide insights for network providers who will leverage DQL in future small cell networks through in-depth performance analysis compared with conventional DQL algorithm and practical optimization algorithms.

Recently, energy efficiency (EE) has been pointed out as one of the key requirements within mobile networks. The development of intelligent algorithms providing Radio Access Networks (RAN) with EE features is possible when having access to the Key Performance Indicators (KPIs), and proper control actions, e.g., cell on/off switching, or Radio Frequency (RF) channel reconfiguration. These features together with a Machine Learning (ML) framework are available in the O-RAN architecture. This paper provides an overview of the EE framework according to the use cases specified by the O-RAN ALLIANCE. It is followed by the implementation of the Energy Saving rApp (ES-rApp). The rApp utilizes Deep Q-Learning (DQL) to increase EE through intelligent RF channel reconfiguration. Simulation results show up to a 24.8% EE gain over the static RF Channel Configuration (RCC).

Traffic signal control is an important part of intelligent transportation. Efficient traffic signal control strategies not only alleviate traffic congestion, improve vehicle traffic efficiency, but also reduce exhaust pollution during the waiting period. The traditional multiple intersection signal light control method are generally faced with passive control and can not adapt to complex changes. In view of this, the study first combines reinforcement learning and Markov decision to construct a signal control problem model for multiple intersections. Secondly, deep Q-learning networks are introduced for decision solving, and multi-head attention mechanisms and graph convolutional networks are further introduced for optimization and improvement. Finally, a spatial lightweight model for multiple intersection adaptive graph convolution is proposed. The experimental results show that the maximum throughput of the multi-intersection adaptive control model can reach 195 vehicles/min, the shortest queue length is 4.31 vehicles, the shortest average delay is 1.23 minutes, the shortest average travel time is 2.07 minutes, and the attention value of adjacent intersections is close to 80%. The average green wave time under this model is 4 minutes, and the longest green wave section can reach 5 minutes. It can be seen that the new multi-intersection signal adaptive control model proposed in this study has advantages and feasibility compared with similar methods in various indexes, and can provide new ideas and methods for the further development of traffic signal automatic control technology.

In the era of fifth-generation (5G) cellular networks and beyond, network sharing has emerged as a promising approach to address the escalating demand for spectrum and infrastructure resources. Intelligent Neutral Host (INH) is an advanced network-sharing method facilitated by Open Radio Access Network (O-RAN) capabilities. This paper addresses the challenge of Radio Resource Management (RRM) in a multi-operator, multi-slice scenario. We propose an algorithm based on Q-learning and deep Q-learning, particularly concerning different Physical Resource Block (PRB) types to cater to diverse operator requirements. Implemented as an xApp on the Colosseum platform, our algorithm introduces a dynamic resource allocation strategy that adheres to Service Level Agreement (SLA) constraints and optimizes real-time Key Performance metrics (KPMs), including throughput, buffer occupancy, and PRB utilization. We assess the performance and efficacy of our algorithm in a complex traffic scenario to demonstrate how it effectively allocates resources among operators’ slices to satisfy their respective SLA while ensuring optimal resource utilization. The experimental results show that our proposed algorithm can efficiently allocate resources to individual slices while satisfying the SLA. Compared to traditional algorithms, our approach significantly minimizes SLA violations, reducing them to 2.5% for enhanced Mobile Broadband (eMBB) slices and eliminating them entirely for Ultra-Reliable Low-Latency Communications (URLLC) slices.

The continuous evolution of artificial intelligence and cyber–physical systems has presented promising opportunities for optimizing traffic signal control in densely populated urban areas, with the aim of alleviating traffic congestion. One area that has garnered significant interest from both researchers and practitioners is the application of deep reinforcement learning (DRL) in traffic signal control. However, DRL-based algorithms often suffer from instability due to the dynamic nature of traffic flows. Discrepancies between the environments used for training and those encountered during deployment often lead to operational failures. Moreover, conventional DRL-based traffic signal control algorithms tend to reveal vulnerabilities when faced with unforeseen events, such as sensor failure. These challenges highlight the need for innovative solutions to enhance the robustness and adaptability of such systems. To address these pertinent issues, this paper introduces StageLight, a novel two-stage multiscale learning approach, which involves learning optimal timings on a coarse time scale in stage 1, while finetuning them on a finer time scale in stage 2. Our experimental results demonstrate StageLight’s remarkable capability to generalize across diverse traffic conditions and its robustness to various sensor-failure scenarios.

Adaptive bitrate video streaming (ABRVS) and edge caching are two techniques that hold the potential to improve user-perceived video viewing experience. In this paper, we investigate the content caching, transcoding and transmission for ABRVS in cache-enabled cellular networks. Considering the dynamic characteristics of video popularity distribution and wireless network environment, to improve energy efficiency and minimize system energy consumption, we begin by formulating a long-term optimization problem that focuses on both video caching and user association (UA). The problem is then transformed into a Markov decision process (MDP), which is solved by designing a deep Q-learning network (DQN)-based algorithm. Using this algorithm, we can obtain the optimal video caching and UA solutions. Since the action space of the MDP is huge, to cope with the “curse of dimensionality”, linear approximation is integrated into the designed algorithm. Finally, the proposed algorithm’s convergence and effectiveness in reducing long-term system energy consumption are demonstrated through extensive simulations.

A DC microgrid’s tightly regulated DC/DC converter encounters significant challenges in voltage stability, primarily due to the negative incremental resistance of constant power loads (CPLs). Conventional controllers often struggle with load variations and changes in system parameters. Therefore, there has been growing interest in adaptive machine learning algorithms, such as Deep Reinforcement Learning (DRL), to improve voltage regulation. This paper presents an end-to-end DRL framework based on a modified Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The framework is designed to directly control power switches for regulating the voltage of a DC/DC buck converter that supplies power to CPLs. Real-time experiments were conducted using OPAL-RT to validate the approach under diverse load cycles and converter parameter changes.Comparative analysis against other DRL-based control strategies, including Deep Q-learning (DQN) and Deep Deterministic Policy Gradient (DDPG), demonstrated the superior static and dynamic voltage response of the proposed modified TD3 DRL controller, particularly in scenarios involving load and parametric variations.

Path planning is a fundamental task for autonomous mobile robots (AMRs). Classic approaches provide an analytical solution by searching for the trajectory with the shortest distance; however, reinforcement learning (RL) techniques have been proven to be effective in solving these problems with the experiences gained by agents in real time. This study proposes a reward function that motivates an agent to select the shortest path with fewer turns. The solution to the RL technique is obtained via dynamic programming and Deep Q-Learning methods. In addition, a path-tracking control design is proposed based on the Lyapunov candidate function. The results indicate that RL algorithms show superior performance compared to classic A* algorithms. The number of turns is reduced by 50%, resulting in a decrease in the total distance ranging from 3.2% to 36%.

This paper demonstrates the potential of reinforcement learning techniques for the secure joint relay and jammer selection in the dual-hop wireless cooperative network. The existing techniques solve the optimization problem to find near-optimal secrecy. However, due to the computational complexity, most of these algorithms are not scalable to large networks. In this work, two model-free reinforcement learning-based secure joint relay and jammer selection techniques are developed. The objective is to maximize the physical layer security (PLS) of the dual-hop wireless network. The secure joint relay and jammer selection is transformed into a prediction-based problem and solved using reinforcement learning. We introduced Q-learning-based joint relay and jammer selection (QL-JRJS) and deep Q-learning-based joint relay and jammer selection (DQL-JRJS) algorithms. The secrecy performance of the introduced algorithms is compared with the existing techniques. The simulation results demonstrated that the proposed DQL-JRJS algorithm on average achieves 38.7%, 32.1%, and 21.6% less secrecy outage probability (SOP) as compared to the greedy method (GM), vector alignment method (VAM), and QL-JRJS algorithm. However, QL-JRJS only mimics the performance of DQL-JRJS for small values of state action pairs. The proposed algorithms are best suited for situations where the exact global channel state information (CSI) for all the links is unavailable. [ABSTRACT FROM AUTHOR]

A reliable transit system is essential and offers a lot of advantages. However, traffic has always been an issue in major cities, and one of the main causes of congestion in these places is intersections. To reduce traffic, a reliable traffic control system must be put in place. This research sheds light on how to consider dynamic traffic at intersections and minimize traffic congestion using an end-to-end deep reinforcement learning approach. The goal of the model is to reduce waiting times at these crossings by controlling traffic in various scenarios after receiving the necessary training. [ABSTRACT FROM AUTHOR]

Exothermic reactions carried out in batch reactors need a lot of attention to operate because any insufficient condition can lead to thermal runaway causing an explosion in the worst case. Therefore, a well-designed intervention action is necessary to avoid non-desired events. For this problem, we propose to use resilience-based reinforcement learning, where the artificial agent can decide whether to intervene or not based on the current state of the system. One of our goals is to design resilient systems, which means designing systems that can recover after a disruption. Therefore, we developed the resilience calculation method for reactors, where we suggest the use of dynamic predictive time to failure and recover to better resilience evaluation. Moreover, if the process state is out of the design parameters, then we do not suggest calculating with the adaptation and recovery phase. We suggest using Deep Q-learning to learn when to intervene in the system to avoid catastrophic events, where we propose to use the resilience metric as a reward function for the learning process. The results show that the proposed methodology is applicable to develop resilient-based mitigation systems, and the agent can effectively distinguish between normal and hazardous states. [ABSTRACT FROM AUTHOR]

With the recent boom of the gig economy, urban delivery systems have experienced substantial demand growth. In such systems, orders are delivered to customers from local distribution points respecting a delivery time promise. An important example is a restaurant meal delivery system, where delivery times are expected to be minutes after an order is placed. The system serves orders by making use of couriers that continuously perform pickups and deliveries. Operating such a rapid delivery system is very challenging, primarily because of the high service expectations and the considerable uncertainty in both demand and delivery capacity. Delivery providers typically plan courier shifts for an operating period based on a demand forecast. However, because of the high demand volatility, it may at times during the operating period be necessary to adjust and dynamically add couriers. We study the problem of dynamically adding courier capacity in a rapid delivery system and propose a deep reinforcement-learning approach to obtain a policy that balances the cost of adding couriers and the cost-of-service quality degradation because of insufficient delivery capacity. Specifically, we seek to ensure that a high fraction of orders is delivered on time with a small number of courier hours. A computational study in the meal delivery space shows that a learned policy outperforms policies representing current practice and demonstrates the potential of deep learning for solving operational problems in highly stochastic logistic settings. History: This paper has been accepted for the Transportation Science Special Issue on Machine-Learning Methods and Applications in Large-Scale Route Planning Problems. Funding: This work was supported by Agencia Nacional de Investigación y Desarrollo [72180404]. Supplemental Material: The e-companion is available at https://doi.org/10.1287/trsc.2022.0042. [ABSTRACT FROM AUTHOR]

Vanadium redox-flow batteries (VRFBs) have played a significant role in hybrid energy storage systems (HESSs) over the last few decades owing to their unique characteristics and advantages. Hence, the accurate estimation of the VRFB model holds significant importance in large-scale storage applications, as they are indispensable for incorporating the distinctive features of energy storage systems and control algorithms within embedded energy architectures. In this work, we propose a novel approach that combines model-based and data-driven techniques to predict battery state variables, i.e., the state of charge (SoC), voltage, and current. Our proposal leverages enhanced deep reinforcement learning techniques, specifically deep q-learning (DQN), by combining q-learning with neural networks to optimize the VRFB-specific parameters, ensuring a robust fit between the real and simulated data. Our proposed method outperforms the existing approach in voltage prediction. Subsequently, we enhance the proposed approach by incorporating a second deep RL algorithm—dueling DQN—which is an improvement of DQN, resulting in a 10% improvement in the results, especially in terms of voltage prediction. The proposed approach results in an accurate VFRB model that can be generalized to several types of redox-flow batteries. [ABSTRACT FROM AUTHOR]

PhotoVoltaic (PV) panels have been increasingly favored by residential users in recent years, due to noticeable reductions in their costs. The PV systems become more effective when combined with battery packages, which store the energy produced by the PV systems for later use. This way, the PV systems are able to provide flexible and reliable services even when the peak demand for electricity misalign with the window of most efficient PV power generation. In this study, we develop an integrated charge/discharge scheme for lithium-ion batteries to maximize their total expected benefit. Specifically, we develop a Markov Decision Process (MDP) model to maximize the battery utilization, subject to uncertainty in weather conditions and electricity demands, while accounting for battery degradation due to calendar aging and charging/discharging cycles. Due to the extremely slow rate of degradation in batteries, the state space of the MDP is excessively large. To solve the problem efficiently, we establish structural properties of the MDP and exploit them to solve the problem. We improve the backward induction algorithm with established structural properties. We further improve the Deep Q-network (DQN) algorithm by proposing two novel algorithms, the augmented DQN (ADQN) algorithm and the stochastic augmented DQN (SADQN) algorithm. Computational results indicate that ADQN and SADQN solve the problem much faster than DQN, with better solution qualities. The ADQN and SADQN algorithms provide flexibility for practitioners in real-world implementations. [ABSTRACT FROM AUTHOR]

This study introduces a control strategy based on intersection capacity. The optimisation technique is formulated from available space at discharge routes. The downstream policy utilises density and speed (k-v) measurements to guide a deep Q-learning agent (DQLA) in managing a signalised junction using a constrained local communication protocol. Testing of the DQLA k-v strategy against other control methods is carried out in a simulated micro-model of a real urban traffic network. Though the adaptive signal system design is decentralised, statistical analyses explicitly prove the effectiveness of the discharge-based controller in mitigating operation at a global scale. The DQLA k-v controller has achieved significant cost savings in waiting time (10%−36%) and travel time (5%−25%) and asserted the highest mean travel speed (3.4 m/s). Consequently, vehicular traffic experienced the least time loss when traversing routes and witnessed fewer stops leading to close to optimum network operation at a 0.80 clearance ratio. [ABSTRACT FROM AUTHOR]

Multi-access edge computing (MEC) is a technology that enables devices with limited processing capabilities to handle computationally intensive tasks efficiently. The challenge with MEC is how to schedule multiple tasks rationally and efficiently, mainly when each device generates several tasks. A critical aspect of task scheduling is considering the spatial relationships between devices in the network to reduce long-term losses. To address these challenges, this article proposes a deep reinforcement learning (DRL)-based Mayfly Taylor optimization algorithm (MTOA) that uses graph attention neural networks (GATs). The algorithm creates a planning agent for each end device that collects timing-related characteristics of activities and makes decisions and predictions using a recurrent gated unit and a graph representation agent (GRU). The GRU contains possible spatial elements from the scenario and incorporates them into the decision-making process. A novel approach, the Mayfly Taylor optimization algorithm (MTOA), for addressing the challenges in Multi-access Edge Computing (MEC) task scheduling. The key innovation lies in the integration of deep reinforcement learning (DRL) with graph attention neural networks (GATs) to create an effective scheduling strategy. The proposed algorithm outperforms several baseline methods by exploiting the spatial positional relationships between devices. It significantly reduces average latency, bandwidth, and dropout rates and increases in link usage efficiency. Overall, the proposed approach provides a solution to the task-scheduling problem in the MEC scenario that considers separable and time-sensitive activities while benefiting from the spatial relationships between devices. [ABSTRACT FROM AUTHOR]

Autonomous robots can be employed in exploring unknown environments and performing many tasks, such as, e.g. detecting areas of interest, collecting target objects, etc. Deep reinforcement learning (RL) is often used to train this kind of robot. However, concerning the artificial environments aimed at testing the robot, there is a lack of available data sets and a long time is needed to create them from scratch. A good data set is in fact usually produced with high effort in terms of cost and human work to satisfy the constraints imposed by the expected results. In the first part of this paper, we focus on the specification of the properties of the solutions needed to build a data set, making the case of environment exploration. In the proposed approach, rather than using imperative programming, we explore the possibility of generating data sets using constraint programming in Prolog. In this phase, geometric predicates describe a virtual environment according to inter-space requirements. The second part of the paper is focused on testing the generated data set in an AI gym via space search techniques. We developed a Neuro-Symbolic agent built from the following: (i) A deep Q-learning component implemented in Python, able to address via RL a search problem in the virtual space; the agent has the goal to explore a generated virtual environment to seek for a target, improving its performance through a RL process. (ii) A symbolic component able to re-address the search when the Q-learning component gets stuck in a part of the virtual environment; these components stimulate the agent to move to and explore other parts of the environment. Wide experimentation has been performed, with promising results, and is reported, to demonstrate the effectiveness of the approach. [ABSTRACT FROM AUTHOR]

This paper discusses the challenges of mobile edge computing with low latency and energy consumption caused by the explosive growth of communication traffic and data generated by mobile devices. To address the issue of selecting edge servers, a multi-user-oriented edge server selection strategy is proposed. The strategy minimizes the weighted sum of network latency and total energy consumption and develops an improved SPEA2-based algorithm to choose the most suitable edge server. The resource allocation problem is also considered, and a resource allocation strategy is proposed that maximizes the energy efficiency ratio. A DQN-based resource allocation strategy is devised to find the best resource allocation strategy. Experimental results demonstrate that the proposed server selection strategy reduces system overhead in terms of energy consumption and latency while improving resource utilization. The resource allocation strategy improves the computational efficiency of edge servers while reducing latency and total energy consumption. [ABSTRACT FROM AUTHOR]

Ensuring the classification of water bodies suitable for fish habitat is essential for animal preservation and commercial fish farming. However, existing supervised machine learning models for predicting water quality lack specificity regarding fish survival. This study addresses this limitation and presents a novel model for forecasting fish viability in open aquaculture ecosystems. The proposed model combines reinforcement learning through Q-learning and deep feed-forward neural networks, enabling it to capture intricate patterns and relationships in complex aquatic environments. Moreover, the model’s reinforcement learning capability reduces the reliance on labeled data and offers potential for continuous improvement over time. By accurately classifying water bodies based on fish suitability, the proposed model provides valuable insights for sustainable aquaculture management and environmental conservation. Experimental results show a significantly improved accuracy of 96% for the proposed DQN-based model, outperforming existing Gaussian Naive Bayes (78%), Random Forest (86%), and K-Nearest Neighbors (92%) classifiers on the same dataset. These findings highlight the effectiveness of the proposed approach in forecasting fish viability and its potential to address the limitations of existing models. [ABSTRACT FROM AUTHOR]