Search

Q-learning has long been one of the most popular reinforcement learning algorithms, and theoretical analysis of Q-learning has been an active research topic for decades. Although researches on asymptotic convergence analysis of Q-learning have a long tradition, non-asymptotic convergence has only recently come under active study. The main goal of this paper is to investigate a new finite-time analysis of asynchronous Q-learning under Markovian observation models via a control system viewpoint. In particular, we introduce a discrete-time time-varying switching system model of Q-learning with diminishing step-sizes for our analysis, which significantly improves recent development of the switching system analysis with constant step-sizes, and leads to $\mathcal {O}(\sqrt {{\log k}/{k}})$ convergence rate that is comparable to or better than most of the state of the art results in the literature. In the meanwhile, we consider the continuous-time Lyapunov equation to avoid the difficulty in the analysis posed by using diminishing step-sizes in discrete-time. The proposed analysis brings in additional insights, covers different scenarios, and provides new simplified templates for analysis to deepen our understanding on Q-learning via its unique connection to discrete-time switching systems.

Current multi-objective reinforcement learning (MORL) research often struggles to balance multiple objectives and manage the stability and performance of learning algorithms, especially in complex environments. To address this, we propose a new multi-objective deep Q-network (MO-DQN) framework that integrates linear scalarization in MORL. This framework has the following features: First, we provide immediate feedback by incorporating linear scalarization into reward processing. Compared with some complex multi-objective optimization methods, this approach is relatively easy to understand and implement, offering greater convenience for practical applications. Additionally, linear scalarization accelerates the learning process and enhances the algorithm’s ability to dynamically adjust strategies. Second, we develop the Linear Scalarized Multi-objective Deep Q-Network (LSMO-DQN) under different reward mechanisms, improving MO-DQN’s ability to balance multiple objectives effectively. Immediate reward strategies accelerate learning and enable rapid adjustments, benefiting dynamic environments. In contrast, delayed reward strategies help understand long-term action impacts and promote strategic decision-making. Experiments are conducted in two different multi-objective environments, where our proposed method slightly outperforms other techniques, indicating its robustness and adaptability. Specifically, LSMO-DQN achieves higher cumulative rewards and demonstrates improved stability across various reward structures. The findings suggest that integrating linear scalarization in reward processing not only enhances learning performance but also provides a more straightforward approach to managing the trade-offs in multi-objective settings. These results highlight the potential of LSMO-DQN to improve learning performance, particularly in scenarios with data imbalances.

Maintaining player engagement in serious management games is a challenge due to the repetitive nature of traditional predetermined difficulty levels. A dynamic difficulty adjustment (DDA) system is introduced in this study to address this issue by integrating fuzzy logic and Q-learning. Player ennui is frequently the consequence of static difficulty adjustments. In order to dynamically adjust game complexity in accordance with player performance and preferences, our DDA system utilizes a diverse array of performance metrics, adaptive narrative elements, and real-time feedback, as well as fuzzy logic and Q-learning algorithms. According to empirical assessments, players were 28% more effective overall, and play sessions lasted an average of 35% longer. Player satisfaction and involvement were also much improved. Customers played the game longer and were less bored because of the higher degree of difficulty and customization. The integration of fuzzy logic and Q-learning in DDA systems greatly enhances the ability to maintain long-term player engagement in essential management games. This approach offers a long-lasting alternative for creating constantly captivating gaming experiences by effectively reducing the repetitiveness of traditional difficulty adjustments.

In distributed Software-defined networking (SDN), control plane functions are partitioned across multiple controller instances to enhance fault tolerance and scalability. However, the dynamic nature of network traffic and rapid network events, such as link failures and controller node failures, can lead to uneven workload distribution among controller nodes. This research aims to adjust switch-to-controller mapping to address load imbalance dynamically. We model flow arrivals at switches and subsequent actions within a Markov decision process (MDP) framework. In MDP, precise knowledge of the arrival rate is required, however, such an assumption is impractical in dynamic environments. Reinforcement learning (RL) learns policies from environment interactions, enabling autonomous decision-making in complex domains by adeptly navigating uncertainties. The proposed scheme uses RL to monitor SDN flow dynamics and maintain system load balance through switch migration. Herein, the proposed scheme generates migration triplets specifying the source controller, the destination controller for migration, and the switch to be migrated. The scheme considers the cost of migrating the flows in terms of the flow arrival rate and hop count between the switch and the controllers. Experimental results confirm that the framework effectively achieves load balancing across different network topologies and diverse traffic load distributions on switches.

This paper investigates the secrecy performance of a multi-relay non-orthogonal multiple access (NOMA) network. Considering the presence of eavesdroppers, an optimal relay selection scheme and a jamming signal transmission scheme for both amplify-and-forward (AF) and decode-and-forward (DF) strategies are proposed. However, the resource allocation problem aimed at maximizing the effective secure throughput (EST) of the system is non-convex. It is difficult to directly solve this optimization problem using conventional methods. As such, a Q-learning approach to solve the resource allocation problem in this system is applied, and an innovative reward function that can maximize the communication quality of edge users while ensuring secure communication of nearby users is designed. According to the analysis of the simulation results, the convergence of the proposed scheme is verified. Under the same conditions, DF relays achieve a higher signal-to-noise ratio (SINR) at the user terminal, and the EST is closely related to the transmission power of the source node and the relays. The numerical results also show that compared to conventional power allocation methods, the proposed method achieves a larger average EST and provides better confidentiality performance.

Trajectory planning for unmanned aerial vehicles (UAVs) presents a formidable challenge, particularly in environments where details about the terrain and objectives are not known beforehand. This uncertainty has propelled the adoption of UAVs across diverse sectors such as logistics, weather forecasting, security surveillance, and autonomous driving, where their ability to navigate autonomously is crucial. Recent advancements have leveraged deep learning for robust object detection and tracking, significantly enhancing data acquisition from aerial imagery and UAV-based applications. Despite numerous innovations, the critical issue of optimizing UAV trajectories in unknown environments remains largely unaddressed. This paper introduces a novel approach, the Instructed Reinforcement Q-Learning Algorithm (IR-QLA), which utilizes Received Signal Strength (RSS) as a dynamic reward metric to refine trajectory planning continuously. By integrating a unique instructed reinforcement principle, this method significantly accelerates the learning process, improving the UAVs’ navigational efficiencies. Our findings demonstrate that IR-QLA not only surpasses traditional algorithms like Ant Colony and Particle Swarm Optimization in terms of convergence speed but also minimizes the path length and iteration counts needed for trajectory optimization. Future studies will explore alternative reward sources and advanced algorithms to further enhance the precision and speed of UAV trajectory planning, aiming to establish new benchmarks in the field.

Multivariate time series (MTS) forecasting is a rapidly expanding field with diverse and futuristic applications. However, traditional statistical learning models need more prediction accuracy when faced with dynamic variability, non-linearity, and non-stationarity, as well as the challenge of selecting MTS data for classification. Moreover, the existing methods for early classification of multivariate time series data suffer from numerous severe challenges, including evaluating the length of testing the MTS data component, which must be equal to the training MTS data component, and the availability of faulty data components in MTS. To address this issue, we propose a novel framework for early MTS forecasting using double Q-learning-based ensemble techniques to improve prediction accuracy. Our framework uses Q-learning agents to select optimal actions, which results in maximum rewards and accurate prediction. We investigate the ensemble behavior of learned agents using double Q-learning and Gaussian Process Classifiers (GPC) for early forecasting of MTS data. We also determine the minimum required time-series length for classifying faulty data components using the probabilistic Auto-Regressive Integrated Moving Average (ARIMA) model, enhancing framework robustness and mitigating miss-classification accuracy. Our proposed framework achieves 99.89% accuracy for early forecasting, surpassing existing methods based on different benchmark settings and publicly available multivariate time-series datasets. The framework provides a promising solution to the challenges of accurate MTS forecasting and offers insights into the early prediction of recent stock market trading data.

The underwater wireless sensor network (UWSNs) is an important communication facility supporting underwater monitoring applications. However, the transmission channel has the characteristics of high bit error rate, strong multipath effect, and many interference factors, and the network node has the characteristics of high energy consumption, difficult energy supply, and the node position vulnerable to change, which makes it extremely difficult for UWSNs to realize the reliable and efficient packet forwarding. To address the problem, we propose the Stackelberg Q-learning based multi-hop cooperative routing algorithm (SQMCR). The SQMCR builds the transmission routes based on the Q-learning algorithm, considering factors such as the delay, the remaining energy, and the network topology, which improves the rationality and adaptability of selecting the next-hop node. By balancing the packet forwarding benefits and the energy consumption costs based on the Stackelberg Q-learning algorithm, the SQMCR establishes the cooperative communication policy to ensure both the reliability and efficiency of underwater communications. It also adopts initializing Q-values and dynamic exploration probabilities optimization methods to further improve the performance of routing algorithms. Experimental results show that the SQMCR can help UWSNs increase the packet forwarding reliability and prolong the network lifetime by 17%. It has a better environment and application adaptability and is more suitable for underwater high-reliability applications.

During the process of waste collection, various unpredicted disturbances might occur. Meanwhile, vehicle transport is an important source of carbon emissions. In this study, an energy-efficient multi-trip dynamic vehicle routing model is established for waste collection, which introduces two types of dynamic events: new collecting requirements of waste sites and vehicle breakdowns. A Q-learning-based hyperheuristic particle swarm optimization (QLHPSO) is proposed as a dynamic rescheduling method to solve the model. A set of low-level heuristics (LLHs) are designed by combining the learning operators in particle swarm optimization and local search operators. A Q-learning-based high-level strategy is developed to find a suitable LLH for each evolutionary state based on historical performance of LLHs. When rescheduling is triggered, a response mechanism is incorporated to construct initial population by utilizing the features of dynamic events and historical elites. Extensive experimental results on one real instance and nine synthetic instances show that QLHPSO can react to the environmental changes rapidly, and reschedule the vehicle routes with lower cost and carbon emissions compared to the state-of-the-art algorithms.

In the era of fifth-generation (5G) cellular networks and beyond, network sharing has emerged as a promising approach to address the escalating demand for spectrum and infrastructure resources. Intelligent Neutral Host (INH) is an advanced network-sharing method facilitated by Open Radio Access Network (O-RAN) capabilities. This paper addresses the challenge of Radio Resource Management (RRM) in a multi-operator, multi-slice scenario. We propose an algorithm based on Q-learning and deep Q-learning, particularly concerning different Physical Resource Block (PRB) types to cater to diverse operator requirements. Implemented as an xApp on the Colosseum platform, our algorithm introduces a dynamic resource allocation strategy that adheres to Service Level Agreement (SLA) constraints and optimizes real-time Key Performance metrics (KPMs), including throughput, buffer occupancy, and PRB utilization. We assess the performance and efficacy of our algorithm in a complex traffic scenario to demonstrate how it effectively allocates resources among operators’ slices to satisfy their respective SLA while ensuring optimal resource utilization. The experimental results show that our proposed algorithm can efficiently allocate resources to individual slices while satisfying the SLA. Compared to traditional algorithms, our approach significantly minimizes SLA violations, reducing them to 2.5% for enhanced Mobile Broadband (eMBB) slices and eliminating them entirely for Ultra-Reliable Low-Latency Communications (URLLC) slices.

Students are a gold asset for each country. Proper guidance/recommendation to the students regarding their education-related issues can ultimately result in uplifting the economy of a country. Different education models are followed in the world, out of which Outcome Based Education (OBE) is the one. OBE education model comprises three main components, which include Program Educational Objectives (PEOs), Program Learning Outcomes (PLOs), and Course Learning Outcomes (CLOs). CLOs are outcomes that a student achieves after studying a course. A single course may contain one or more CLOs. These CLOs are then mapped to PLOs and PLOs are then mapped to PEOs. Therefore, our objective in this work is to improve deficient/weak CLOs of students by suggesting online resources. Whereas, in the absence of this proposed system, students have to find out these resources by themselves or course teacher recommends relevant online resources. To achieve this objective, we created a dataset for the OBE education model, as to date no standard dataset exists on the OBE education model. From this dataset, we created a Student-to-CLO matrix and performed bi-clustering on this matrix to find groups of students having similar performance in different CLOs. So far, Bi-clustering has been used in the Bio-informatics field to determine similarity in gene expression data. Generated bi-clusters are sorted according to their homogeneity of contained values. These sorted bi-clusters are then mapped to a 2D grid to formulate a reinforcement-learning environment. The start state of the recommendation agent is determined using cosine similarity. If an agent visits a state, deficient CLOs of that state are recommended to the student. The agent can visit only those states that are nearby to its current state and accessible through its legal action space. An optimal sequence of actions to visit different states of a 2D grid, which can improve student’s performance, is determined using Q-learning. Online resources including research articles, YouTube videos, books, and online tutorials are suggested to the student to improve his deficient CLOs using a mobile app.

The burgeoning growth of vehicular traffic, fuelled by rapid urbanization and an ever-expanding population, has resulted in congested road networks. Traditional and sensor-based adaptive traffic light management systems have shown commendable progress in some scenarios, but they suffer from inherent disadvantages that hinder their effectiveness and scalability. To combat these challenges, a dynamic and adaptable traffic control system is imperative to optimize traffic flow. The present paper explores the limitations of sensor-based adaptive traffic light management and advocates for integrating a novel algorithm to overcome the existing drawbacks. It proposes an intelligent traffic management algorithm called IntelliSignal that leverages the map service for fetching real-time traffic information to calculate the optimal green time and a penalty-based road selection to optimize traffic flow. The proposed IntelliSignal is designed to provide equal chances for all roads while prioritizing higher-density roads with more green time, effectively mitigating traffic congestion and improving overall transportation efficiency. It incorporates Q-learning, a reinforcement learning technique that enables the system to adapt and learn from traffic patterns. The proposed IntelliSignal’s performance is assessed through rigorous evaluations conducted on the simulation platform SUMO. The acquired results demonstrate substantial enhancements across various crucial metrics, including average waiting time, vehicle density, travel time, CO2 emissions, and queue length. Furthermore, the simulation results demonstrate that the proposed IntelliSignal algorithm exhibits a remarkable 30.52% increment in system throughput compared to the traditional approach. This significant enhancement underscores the efficacy of the proposed IntelliSignal in optimizing system performance and merits consideration for practical implementation.

An increasing number of users are expected to be served by wireless network with heterogeneous requirements. Unmanned aerial vehicles (UAV) can be deployed to augment the heterogeneous network from aerial area by taking advantages of the characteristics of UAV such as mobility, manoeuvrability, low cost, and Line-of-Sight (LoS) communication. However, the deployment of UAV can also cause problems. For example, in UAV-assisted HetNet, the number of handover (HO) will increase because of the dense distribution of small base stations (SBS) and UAV base stations (UBS). Also, because of the small coverage of SBS and the LoS communication links from neighbour UBSs, the number of unnecessary HO will also rise. Frequent HO and unnecessary HO can result in interruption, increased overhead and energy consumption, which is not desirable for battery powered UAVs. In this paper, to solve the problem, a HO decision making algorithm adopting TOPSIS and Q-learning (QL) is proposed with the aim to reduce HO number and improve energy efficiency. Q-learning can be applied to address decision-making challenges in communication systems widely. However, a large volume of training data can pose challenges and complexities, therefore, the TOPSIS is utilised to reduce the size of the action space in Q-learning. The proposed hybrid TOPSIS-Q-learning method enhances both the handover performance and the scalability. In the method, signal to interference and noise ratio (SINR), time of stay (ToS) and average energy efficiency (EE) are taken into account. The simulation results show that the number of HO and unnecessary HO is remarkably reduced and the average EE is notably improved in comparison with other existing methods.

Exploration is an integral part of learning dynamics which allows algorithms to search a space of solutions. When many algorithms simultaneously explore, this can lead to counter-intuitive effects. This paper contributes an analysis of the influence that exploration has on a multi-agent system of $Q$ -learners in a famous congestion dilemma, the Braess paradox. I find ranges of the exploration rate for which $\epsilon $ -greedy $Q$ -learners show chaotic and oscillatory dynamics which do not converge, and yield better than Nash equilibrium results. I decouple the dynamics endogenous to $Q$ -learning from the exogenous exploration rate $\epsilon $ , and find that $Q$ -learners implicitly coordinate with low exploration rates $\epsilon \in (0, 0.1)$ , but are disrupted in their coordination for larger exploration rates $\epsilon > 0.1$ . The best implicit coordination leads to a 20% reduction in average travel times which approaches the social optimum. I discuss how our results may inform multi-agent algorithm design, fit within a cognitive science perspective of cognitive noise during learning, and provide a mechanistic hypothesis for the lack of empirical evidence of the Braess Paradox in traffic systems.

Various methods are employed in computer vision applications to identify individuals, including using face recognition as a human visual feature helpful in tracking or searching for a person. However, tracking systems that rely solely on facial information encounter limitations, particularly when faced with occlusions, blurred images, or faces oriented away from the camera. Under these conditions, the system struggles to achieve accurate tracking-based face recognition. Therefore, this research addresses this issue by fusing descriptions of the face visual with body visual features. When the system cannot find the target face, the CNN+LSTM hybrid method assists in multi-feature body visual recognition, narrowing the search space and speeding up the search process. The results indicate that the combination of the CNN+LSTM method yields higher accuracy, recall, precision, and F1 scores (reaching 89.20%, 87.36%, 91.02%, and 88.43%, respectively) compared to the single CNN method (reaching 88.84%, 74.00%, 67.00%, and 69.00% respectively). However, the combination of these two visual features requires high computation. Thus, it is necessary to add a tracking system to reduce the computational load and predict the location. Furthermore, this research utilizes the Q-Learning algorithm to make optimal decisions in automatically tracking objects in dynamic environments. The system considers factors such as face and body visual features, object location, and environmental conditions to make the best decisions, aiming to enhance tracking efficiency and accuracy. Based on the conducted experiments, it is concluded that the system can adjust its actions in response to environmental changes with better outcomes. It achieves an accuracy rate of 91.5% and an average of 50 fps in five different videos, as well as a video benchmark dataset with an accuracy of 84% and an average error of 11.15 pixels. Utilizing the proposed method speeds up the search process and optimizes tracking decisions, saving time and computational resources.

Traffic congestion has become an increasingly severe problem and needs to be solved urgently. Reinforcement learning (RL) is an advanced data-driven approach for large-scale adaptive traffic signal control (ATSC) in the complex urban traffic network. The decentralized multi-agent RL (MARL) framework is feasible for multi-intersection scenarios. However, the environment becomes partially observable and nonstationary without effective communication among agents from the perspective of each local agent. Previous studies solve this problem by improving the observation of each local agent and using a spatial discount factor. In this paper, we are motivated by achieving coordination among intersections in the RL settings and better performance in ATSC and solve the problem of traffic signal control at multiple intersections in large-scale networks based on RL techniques. Pressure is used as an indicator to represent the traffic conditions of an intersection considering its neighbor intersections. A new reward function with pressure and vehicle waiting time is designed to achieve coordination among intersections. The above three control policies are innovatively applied to a multi-agent Q-learning method for ATSC at multiple intersections. Experiments are carried out on a large synthetic traffic grid under simulated peak-hour traffic dynamics. Results demonstrate that our approach outperforms the state-of-the-art decentralized MARL algorithms in average queue length and intersection delay.

In reinforcement learning (RL) an agent interacts with the environment based on sequential decisions. This agent receives a reward from the environment according to decisions and tries to maximize the reward. RL is used in several domains, such as production, autonomous driving, business management, education, games, healthcare, natural language processing, robotics, and among others. RL methodologies require processing large volumes of data and computational power. To speed up these applications, field-programmable gate array (FPGA) are widely employed in the literature. This letter proposes an accelerator for the Markov decision process (MDP) implemented in the AI-Toolbox public library using high-level synthesis tools, using the tiger-antelope problem as use case. Our approach shows an acceleration greater than $7\times $ compared to the original version. [ABSTRACT FROM AUTHOR]

Solving constrained multi-objective optimization problems (CMOPs) is one of the most popular research topics in the multi-objective optimization community. Various approaches based on different algorithmic strategies have been proposed for benchmark CMOPs with different features and challenges. Although most of these algorithms employ one or more fixed strategies, determining the most suitable strategy is challenging and problem-dependent. This work presents a general multitasking framework comprising a main task for the original CMOP and an arbitrary number of auxiliary tasks based on different strategies for the auxiliary problems. Reinforcement learning techniques are utilized according to the population state to choose the most suitable auxiliary task during the evolutionary process. As instantiations, Q-Learning and Deep Q-Learning are successfully applied in the framework. Accordingly, two novel algorithms are developed to solve CMOPs. The results demonstrate the effectiveness of the proposed approaches on five benchmark CMOP test suites and real-world problems compared to 11 advanced methods. [ABSTRACT FROM AUTHOR]

With the growing emphasis on mitigating climate change, the power industry is moving toward renewable energy sources as an alternative to fossil fuel-based power plants. The transition to renewable energy has created numerous challenges, one of which is the low levels of inertia that impact the stability of power systems. Therefore, inertia monitoring has become an integral part of power system operation to dispatch renewable energy sources while maintaining frequency stability. This article presents an online method to continuously estimate the inertia of a power system. The inertia is computed from data provided by Phasor Measurement Units (PMUs) using small variations in frequency and power under ambient conditions. The method uses electrical and kinetic energy variations to compute inertia. In addition, a $Q$ -learning-based method is presented to identify mechanical power changes to discard invalid inertia estimates. The method is demonstrated using the IEEE-39 bus system to monitor the regional inertia of the test system.

Unmanned aerial vehicles (UAVs) are receiving increasing attention due to their wide range of applications. Flying ad-hoc networks (FANETs) enable inter-communication between UAVs, but their dynamics and fast changes in mobility and topology make it challenging to achieve stable routing. Clustering based FANETs provide a self-organizing approach for routing, but clustering is a decentralized process based on different utilities that can affect performance. In this work, we propose a centralized agent placed in a remote ground station to adjust the role of each utility in the cluster-head arbitration. Our approach uses six utilities, including node centrality, residual energy, link holding time, velocity similarity, buffer occupancy, and diversity. The developed reinforcement learning (RL)-clustering approach enables routing over multiple planes, with each plane representing a network cell with a different coverage radius of nodes, including Femto, Pico, and Micro planes. We propose a novel reward formulation for effective learning of agents that includes indicators of stability based on role change, energy consumption, and confirmation message through the selected node. For evaluation, the developed RL based agent was evaluated on three types of agents, namely, Q-learning, DQN, and DDPG and it was compared with random agent. Results have shown the superiority of our developed RL based clustering in terms of stability, energy consumption and most network metrics.

The current application of control theory is commonly carried out in systems with a model or known system dynamics. However, in practice this is a formidable task to achieve as not all state information can be known. The use of the Output Feedback (OPFB) scheme in the field of control systems also possesses a weakness because it requires the use of an observer. This appears rather contradictory as the use of an observer requires system dynamics information. This research proposes an optimal control scheme using Deep Recurrent Q-Networks (DRQN) to generate an optimal control signal trajectory based on a collection of input and output data from the system itself. The approach proposed in this study is based on the Q-Learning method from the Reinforcement Learning (RL) scheme. The Long-Short Term Memory (LSTM) is used to approximate the Q-function and determine the control signals for a system without a known model. The method that we proposed in this study has been tested on four case studies. The control signal trajectory generated from our proposed algorithm, is much smaller than the control signal that generated from classical Q-Learning scheme. The results of this research are certainly relevant to the aim of OPFB, namely that the controller is designed to be able to regulate (bring the state trajectory to zero) and minimize control signal energy. It is empirically discovered that the same result is proven by the norm values resulting from the Q-function trajectory. The norm of Q-function trajectory for our proposed algorithm on the 1st, 2nd, 3rd, and 4th case studies are 2.11E-08, 3.15E-06, 3.79E-09, and 1.59E-13, respectively.

Fluid resuscitation (therapy) is used to maintain tissue perfusion and restore cardiac functions in critical care. Automated fluid therapy can result in faster care, fewer dosing errors, and less cognitive burden on healthcare providers, ultimately improving patient outcomes. Despite a few attempts at automating this process, fluid management is an open research area for which optimal, personalized strategies are yet to be developed. This work presents a novel, model-free, subject-specific dose adjustment tool for fluid resuscitation. The proposed approach is based on reinforcement learning (RL) where a Q-learning algorithm automatically recommends subject-specific fluid infusion dosages in different hemorrhaging scenarios without having the knowledge of dose-response models. Comparison studies against two model-free fluid resuscitation controllers, i.e., fuzzy and proportional-integral-derivative (PID), within a verified simulated environment demonstrated the superior performance of the proposed approach in the closed-loop control of fluid resuscitation. Statistical analyses of performance measures indicated that the RL approach, with lower average resuscitation rates, can achieve more desired mean arterial pressure (MAP) responses than the fuzzy and PID controller for all virtual subjects. Additionally, simulation results demonstrated the higher robustness of our approach than the other two methods against external disturbances in resuscitation scenarios. These results confirm the potential of RL in the closed-loop control of hemodynamic responses in fluid therapy.

Multi-agent reinforcement learning (MARL) for cooperative tasks has been extensively researched over the past decade. The prevalent framework for MARL algorithms is centralized training and decentralized execution. Q-learning is often employed as a centralized learner. However, it requires finding the maximum value by comparing the Q-value of each joint action a’ in the next state s’ to update the Q-value of the last visited state-action pair (s,a). When the joint action space is extensive, the maximization operation involving comparisons becomes time-consuming and becomes the dominant computational burden of the algorithm. To tackle this issue, we propose an algorithm to reduce the number of comparisons by saving the joint actions with the top 2 Q-values (T2Q). Updating the top 2 Q-values involves seven cases, and the T2Q algorithm can avoid traversing the Q-table to update the Q-value in five of these seven cases, thus alleviating the computational burden. Theoretical analysis demonstrates that the upper bound of the expected ratio of comparisons between T2Q and Q-learning decreases as the number of agents increases. Simulation results from two-stage stochastic games are consistent with the theoretical analysis. Furthermore, the effectiveness of the T2Q algorithm is validated through the distributed sensor network task and the target transportation task. The T2Q algorithm successfully completes both tasks with a 100% success rate and minimal computational overhead.

Value-based Reinforcement Learning algorithms achieve superior performance by utilizing experiences gathered in the past to update their so-called value-function. In most cases, it is accomplished by applying a sampling strategy to an experience buffer, in which state transitions are stored during the training process. However, the design of such methods is not so intuitive. General theoretic approaches tend to determine the expected learning progress from each experience, based on which the update of neural networks can be carried out efficiently. Proper choice of these methods can not only accelerate, but also stabilize the training significantly by increasing sampling efficiency, which indirectly leads to a reduction in time and computing capacity requirements. As one of the most critical aspects of using Machine Learning (ML) based techniques originates from the lack of decent computing power, thus endeavour to find optimal solutions has long been a researched topic in the field of Reinforcement Learning. Therefore the main focus of this research has been to develop an experience prioritization method acquiring competitive performance, besides having the overall cost of training considerably lowered. In this paper, we propose a novel priority value assignment concept for experience prioritization in Reinforcement Learning, based on the Upper Confidence Bound algorithm. Furthermore, we present empirical findings of our solution, that it outperforms current state-of-the-art in terms of sampling efficiency, while enabling faster and more cost-efficient training processes.

The SAS-CBRS framework is being tested to share the federally held spectrum with licensed users and opportunistic users to maximize the underutilized spectrum’s utility and overcome spectrum scarcity. In the SAS-CBRS framework, radio resources are assigned to the incumbent access (IA), primary access licensees (PAL), and general authorized access (GAA) users according to the given priority. The SAS-CBRS three-tier framework is different from the conventional cognitive radio networks (CRN) as it involves a central entity that acts as a server called a spectrum access system (SAS). The methods to assign the resources using the SAS are still in the research phase. Yet, no standard method is defined by the FCC for resource allocation. The current CRN methods cannot be directly applied because of the addition of the third tier and a central server. Moreover, strict rules are defined for using the 3.5 GHz spectrum band for communication. In this paper, a novel DDRQ-SAS algorithm integrated with the double auction (DA) algorithm is proposed that uses deep recurrent double Q-learning. The DDRQ-SAS is used by the SAS to hold a spectrum auction and create a spectrum pool to get information on PAL channels. PAL operators use the DA algorithm to generate the asking prices intelligently for their available idle channels and the GAA users will use the DA algorithm to intelligently bid for their preferred channels. The DDRQ-SAS-DA algorithm allows the GAA users to get the guaranteed QoS offered by the PAL operators in an auction. GAA users maintain the preference list of the PAL reserved idle channels and bid intelligently based on the available QoS. SAS completes the transaction by allocating the channels to the winning GAAs. The defined problem is also modeled using the double auction multi-winner multi-channel technique and the TDSA-PS algorithm. Numerical results show that the proposed DDRQ-SAS-DA algorithm provides up to 20% better QoS at higher loads for GAA users, generates 24% more revenue for PAL operators, and is 1.6 times more efficient in assigning 500 GAA users.

As the adoption of cryptocurrencies, especially Bitcoin (BTC) continues to rise in today’s digital economy, understanding their unpredictable nature becomes increasingly critical. This research paper addresses this need by investigating the volatile nature of the cryptocurrency market, mainly focusing on Bitcoin trend prediction utilizing on-chain data and whale-alert tweets. By employing a Q-learning algorithm, a type of reinforcement learning, we analyze variables such as transaction volume, network activity, and significant Bitcoin transactions highlighted in whale-alert tweets. Our findings indicate that the algorithm effectively predicts Bitcoin trends when integrating on-chain and Twitter data. Consequently, this study offers valuable insights that could potentially guide investors in informed Bitcoin investment decisions, thereby playing a pivotal role in the realm of cryptocurrency risk management.

Dynamic pricing for preemptible cloud services (DPPCS) is highly demanded to effectively utilize the excess capacity in cloud computing. However, the dynamic nature of excess capacity exhibits high non-stationarity, which is characterized by multi-temporal stochastic patterns with time-varying statistical properties. The non-stationarity results in the DPPCS problem being a Non-Stationary Markov Decision Process (NSMDP) with unknown transition probabilities. Moreover, DPPCS is constrained by a certain maximum preemption rate, further complicating the DPPCS problem as a Constrained NSMDP (CNSMDP). We transform the CNSMDP into a piecewise Lagrangian dual model, which converts the CNSMDP into an unconstrained optimization problem. To solve the above problem, we propose a novel Q-Learning approach for DPPCS. We first present estimation methods for the unknown environment parameters, including a detection method for identifying temporal pattern changes, and a diffusion approximation method for estimating the actual preemption rate. Then, we introduce a Lagrange multiplier updating method, which can strike a balance between revenue and the preemption rate in the reward function. Building upon the above methods, we develop a Constrained Non-Stationary Q-Learning (CNSQL) algorithm for DPPCS, which dynamically adjusts its learning process to adapt to the multi-temporal patterns. Through simulated experiments, we demonstrate the effectiveness of our proposed approach compared to state-of-the-art algorithms. It performs well in improving revenue generated from excess capacity while maintaining the actual preemption rate within the specified constraint.

Learning activities are considerably supported and improved by the rapid advancement of e-learning systems. This gives students a tremendous chance to participate in learning activities worldwide. The Massive Open Online Courses (MOOCs) platform has emerged as one of the most significant platforms for e-learning as a result of the rapid growth of network information technologies. Due to the increased number of online courses available, it is harder for individual learners to choose the appropriate courses, activities, and learning paths for the actual necessities they want, which reduces their learning performances. Moreover, a sequential Recommender System (RS) can identify the learner’s future interest and suggest the subsequent item or learning content given a sequence of past interactions. This is in contrast to interactive recommendation methods that can create recommendations based on the learner’s feedback via constant interactions. To address these challenges, the goal of this paper is to propose a Reinforcement Learning (RL) based smart e-learning framework with Markov Decision Process (MDP) that has the potential to enhance the learning experience for each student by providing them with a personalized and effective learning path. Applying the MDP and RL-based techniques such as Q-learning for Sequential Path Recommendation (SPR) and learning development is more achievable. This is because the MDP allows for adjusting the recommendations method to find new activities and learning paths based on the learners’ feedback on recommendations results. Experimental findings reveal that the suggested model obtains significant improvements and provides viable performance under different parameters optimization. Furthermore, we also show that the proposed method outperforms a long session (long-term rewards such that they maximize learning progress while minimizing frustration and disengagement). This demonstrates the model’s improvements in simulating the learner’s sequential behavior, learning activities, various learning materials, and learning paths simultaneously. These promising initial results provide a possible solution to assess this challenge further in future work.

The ever-increasing and diverse user demands as well as the need for uninterrupted access to the medium with minimum latency in dense machine type communication networks, are the key driving forces to a holistic network redesign. In this context, fifth-generation and beyond (5G/B5G) networks, incorporate various advanced physical layer techniques, such as relaying-assisted transmission, aiming to improve network performance hl and extend the coverage area of multicellular orientations. However, the deployment of such techniques in a cellular environment characterized by high interference levels and multi-variate channel representations, leads to increased computational complexity for radio resource management (RRM) tasks. Machine learning (ML), and especially Deep Learning (DL), is proposed as an efficient way to support end-to-end user applications in highly complex environments, since ML/DL models can relax the R hl RM-associated computational burden. In this paper, we consider the joint problem of relay node (RN) placement and selection subject to subcarrier allocation and power management constraints in 5G/B5G hl networks. Various DL-based methods are examined and combined to solve both sub-problems. The performance of these schemes is evaluated for various relaying-assisted transmission approaches, hl either considering known channel state information (CSI) or not. According to the derived results, total system energy efficiency (EE) and spectral efficiency (SE) can be improved by up to 30%, when considering only the DL-based RN placement scheme compared to state-of-the-art non-ML schemes. The deployment of the reinforcement learning (RL) model for RN selection, can improve EE up to 80%, while SE can be improved up to 75%, compared to a system with only DL-enabled RN placement.

The combination of model predictive control (MPC) and learning methods has been gaining increasing attention as a tool to control systems that may be difficult to model. Using MPC as a function approximator in reinforcement learning (RL) is one approach to reduce the reliance on accurate models. RL is dependent on exploration to learn, and currently, simple heuristics based on random perturbations are most common. This paper considers variance-based exploration in RL geared towards using MPC as function approximator. We propose to use a non-probabilistic measure of uncertainty of the value function approximator in value-based RL methods. Uncertainty is measured by a variance estimate based on inverse distance weighting (IDW). The IDW framework is computationally cheap to evaluate and therefore well-suited in an online setting, using already sampled state transitions and rewards. The gradient of the variance estimate is then used to perturb the policy parameters in a direction where the variance of the value function estimate is increasing. The proposed method is verified on two simulation examples, considering both linear and nonlinear system dynamics, and compared to standard exploration methods using random perturbations.

Ultra-dense heterogeneous networks (UDHN) based on small cells are a requisite part of the future cellular networks as they are proposed as one of the enabling technologies to handle coverage and capacity problems. But co-tier and cross-tier interferences in UDHN severely degrade the quality of service due to K-tiered architecture. Machine learning based radio resource management either through independent learning or cooperative learning is a proven efficient scheme for interference mitigation and quality of service provision in UDHN in a both distributive and cooperative manner. However, an optimal learning paradigm selection, i.e., either independent or cooperative learning and optimal cooperative cluster size in cooperative learning for efficient radio resource management in UDHN is still an open research problem. In this article, a Q-learning based radio resource management scheme is proposed and evaluated for both distributive and cooperative schemes using independent and cooperative learning. The proposed Q-learning solution follows the $\epsilon -$ greedy policy for optimal convergence. The simulation results for the UDHN in an urban setup show that in comparison to the independent learning paradigm, cooperative learning has no significant impact on macro cell user capacity. However, there is a significant improvement in small cell user capacity and the sum capacity of the cooperating small cells in the cluster. A significant increase of 48.57% and 37.9% is observed in the small cell user capacity, and sum capacity of the cooperating small cells, respectively, using cooperative learning as compared to independent learning which sets cooperative learning as an optimal learning strategy in UDHN. The improvement in small cell user capacity is at cost of increased computational time which is directly proportional to the number of cooperating small cells. To solve the issue of computational time in cooperative learning, an optimal clustering algorithm is proposed. The proposed optimal clustering reduced the computational time by four times in cooperative Q-learning.

The 6TiSCH (IPv6 over IEEE802.15.4e time-slotted channel hopping mode) wireless sensor network architecture utilizes control packets to construct network formation. These control packets are essential for establishing communication links between nodes and configuring network settings. The trickle timer algorithm is utilized to broadcast the DIO control packet. DIO carries information about the available parent nodes, which is then used to form the routing tree. Sensors transmit control packets in one cell on each TSCH slotframe, called the minimal cell. This leads to the problem that RPL trickle timer algorithm encounters congestion in DIO control packet transmission with other control messages, particularly in dense networks. Moreover, high traffic transmission also leads to high queue usage, which then drops the DIO control packet. Failed DIO transmission can increase network formation time and energy consumption. To address this issue, we propose Q-Trickle, an adaptive trickle timer algorithm based on Q-learning that determines the optimal policy for transmitting or suppressing DIO based on minimal cell and transmission queue conditions. Q-Trickle adaptively selects a redundancy constant value and transmission interval that promotes fair transmission distribution and considers network condition. Additionally, a control scheme over minimal cell transmission is formulated to lower transmission congestion and faster synchronization. The proposed methods were assessed using simulation and actual testing on the FIT IoT-LAB testbed. The results indicated that Q-Trickle performed better than the benchmark methods. Q-Trickle decreases joining time, energy consumption, and number of failed DIO compared with the original algorithm by −13%, −11%, and −43%, respectively.

Global path planning techniques have been widely employed in solving path planning problems, however they have been found to be unsuitable for unknown environments. Contrarily, the traditional Q-learning method, which is a common reinforcement learning approach for local path planning, is unable to complete the task for multiple targets. To address these limitations, this paper proposes a modified Q-learning method, called Vector Field Histogram based Q-learning (VFH-QL) utilized the VFH information in state space representation and reward function, based on a 2D LiDAR sensor. We compared the performance of our proposed method with the classical Q-learning method (CQL) through training experiments that were conducted in a simulated environment with a size of 400 square pixels, representing a 20-meter square map. The environment contained static obstacles and a single mobile robot. Two experiments were conducted: experiment A involved path planning for a single target, while experiment B involved path planning for multiple targets. The results of experiment A showed that VFH-QL method had 87.06% less training time and 99.98% better obstacle avoidance compared to CQL. In experiment B, VFH-QL method was found to have an average training time that was 95.69% less than that of the CQL method and 83.99% better path quality. The VFH-QL method was then evaluated using a benchmark dataset. The results indicated that the VFH-QL exhibited superior path quality, with efficiency of 94.89% and improvements of 96.91% and 96.69% over CQL and SARSA in the task of path planning for multiple targets in unknown environments.

Planning precise manipulation in robotics to perform grasp and release-related operations, while interacting with humans is a challenging problem. Reinforcement learning (RL) has the potential to make robots attain this capability. In this paper, we propose an affordance-based human-robot interaction (HRI) framework, aiming to reduce the action space size that would considerably impede the exploration efficiency of the agent. The framework is based on a new algorithm called Contextual Q-learning (CQL). We first show that the proposed algorithm trains in a reduced amount of time (2.7 seconds) and reaches an 84% of success rate. This suits the robot’s learning efficiency to observe the current scenario configuration and learn to solve it. Then, we empirically validate the framework for implementation in HRI real-world scenarios. During the HRI, the robot uses semantic information from the state and the optimal policy of the last training step to search for relevant changes in the environment that may trigger the generation of a new policy.

In recent years, unmanned aerial vehicles (UAVs) have gained significant popularity and are used for many applications, from entertainment to surveillance and the modern battlefield. As regulation demands arose worldwide, controling and reacting to unauthorized flights of UAVs became a pressing issue. In this work, we present an algorithm to accelerate the training of a reinforcement learning drone agent for a counter unmanned aerial system (C-UAS). The main objective of this C-UAS is to guide an invading drone to a safe-killing zone (SZ) using a hunter quadrotor drone. The hunter quadrotor launches a spoofing, or meaconing, attack on the GNSS receiver of the invading drone. The proposed algorithms employ an abstraction of the C-UAS problem to accelerate the training step and enable training during the mission. Results for different SZ radii are discussed using a software-in-the-loop simulation for ground truth based on a detailed model of the UAV embedded system and flight dynamics, including error metrics and action time. We show that a 99% probability of successful target steering to the SZ can be achieved considering a SZ radius of 75 meters and a Q-table trained with the proposed accelerated training model.

Cyber-physical system security for electric distribution systems is critical. In direct switching attacks, often coordinated, attackers seek to toggle remote-controlled switches in the distribution network. Due to the typically radial operation, certain configurations may lead to outages and/or voltage violations. Existing optimization methods that model the interactions between the attacker and the power system operator (defender) assume knowledge of the attacker’s parameters. This reduces their usability. Furthermore, the trend with coordinated cyberattack detection has been the use of centralized mechanisms, correlating data from dispersed security systems. This can be prone to single point failures. In this paper, novel mathematical models are presented for the attacker and the defender. The models do not assume any knowledge of the attacker’s parameters by the defender. Instead, a machine learning (ML) technique implemented by a multi-agent system correlates detected attacks in a decentralized manner, predicting the targets of the attacker. Furthermore, agents learn optimal mitigation of the communication level through Q-learning. The learned attacker motive is also used by the defender to determine a new configuration of the distribution network. Simulations of the technique have been performed using the IEEE 123-Node Test Feeder. The simulation results validate the capability and performance of the algorithm.

This paper proposes a FAst paRAllel and pipeliNE Q-learning accelerator (FARANE-Q) for a configurable Reinforcement Learning (RL) algorithm implemented in a System on Chip (SoC). The proposed work offers flexibility, configurability, and scalability while maintaining computation speed and accuracy to overcome the challenges of a dynamic environment and increasing complexity. The proposed method includes a Hardware/Software (HW/SW) design methodology for the SoC architecture to achieve flexibility. We also propose joint optimizations on the algorithm, architecture, and implementation to obtain optimum (high efficiency) performance, specifically in energy and area efficiency. Furthermore, we implemented the proposed design in a real-time Zynq Ultra96-V2 FPGA platform to evaluate the functionality with an actual use case of smart navigation. Experimental results confirm that the proposed accelerator FARANE-Q outperforms state-of-the-art works by achieving a throughput of up to 148.55 MSps. It corresponds to the energy efficiency of 1747.64 MSps/W per agent for 32-bit and 2424.33 MSps/W per agent for 16-bit FARANE-Q. Moreover, the proposed 16-bit FARANE-Q outperforms other related works by an improvement of at least $1.23\times $ in energy efficiency. The designed system also maintains an error accuracy of less than 0.4% with optimized bit precision for more than eight fraction bits. The proposed FARANE-Q also offers a speed up of processing time up to $1795\times $ compared to embedded SW computation executed on ARM Zynq processor and $280\times $ of computation of full software executed on i7 processor. Hence, the proposed work has the potential to be used for smart navigation, robotic control, and predictive maintenance.

This letter applies a multiagent fuzzy Q-learning (FQL) algorithm, incorporated with a model-free nonlinear controller (MFNC), entitled FQL-MFNC for stabilized controlling of a recently introduced grid-connected nine-level Packed E-Cell (PEC9) inverter under dynamical operation. Unlike previous tuning schemes, which concentrate on extracting mathematical formulation of a controlled plant, this letter investigates a fuzzy Q-learning agent for optimal design of PEC9. In first step, the fuzzy reinforcement learning is adopted to tune the MFNC controller in the simulation environment. In fact, the FQL algorithm finds the optimal policy based on a reward function for adjustment of the MFNC control coefficients to guarantee the grid-connectivity requirements under PEC9 dynamical operation are met. The experimental tests are conducted to assure efficiency and practicability of the designed multi-agent FQL-MFNC scheme on the single-phase grid-connected PEC9 inverter. [ABSTRACT FROM AUTHOR]

Given the rapid growth of user-generated videos, internet traffic has been heavily dominated by online video streaming. Caching videos on edge servers in close proximity to users has been an effective approach to reduce the backbone traffic and the request response time, as well as to improve the video quality on the user side. Video popularity, however, can be highly dynamic over time. The cost of cache replacement at edge servers, particularly that related to service interruption during replacement, is not yet well understood. This paper presents a novel lightweight video caching algorithm for edge servers, seeking to optimize the hit rate with real-time decisions and minimized cost. Inspired by recent advances in deep Q-learning, our DQN-based online video caching (DQN-OVC) makes effective use of the rich and readily available information from users and networks. We decompose the Q-value function as a product of the video value function and the action function, which significantly reduces the state space. We instantiate the action function for cost-aware caching decisions with low complexity so that the cached videos can be updated continuously and instantly with dynamic video popularity. We used video traces from Tencent, one of the largest online video providers in China, to evaluate the performance of our DQN-OVC and to compare it with state-of-the-art solutions. The results demonstrate that DQN-OVC significantly outperforms the baseline algorithms in the edge caching context. [ABSTRACT FROM AUTHOR]

Data collection is a fundamental operation in energy harvesting industrial Internet of Things networks. To this end, we consider a hybrid access point (HAP) or controller that is responsible for charging and collecting $L$ bits from sensor devices. The problem at hand is to optimize the transmit power allocation of the HAP over multiple time frames. The main challenge is that the HAP has causal channel state information to devices. In this article, we outline a novel two-step reinforcement learning with Gibbs sampling (TSRL-Gibbs) strategy, where the first step uses Q-learning and an action space comprising transmit power allocation sampled from a multidimensional simplex. The second step applies Gibbs sampling to further refine the action space. Our results show that TSRL-Gibbs requires up to 28.5% fewer frames than competing approaches. [ABSTRACT FROM AUTHOR]

This article addresses the dynamic positioning control problem of a nonlinear unmanned marine vehicle (UMV) system subject to network communication constraints and deny-of-service (DoS) attack, where the dynamics of UMV are described by a Takagi–Sugeno (T-S) fuzzy system (TSFS). In order to save limited communication resource, a new intelligent event-triggering mechanism is proposed, in which the event triggering threshold is optimized by a $Q$ -learning algorithm. Then, a switched system approach is proposed to deal with the aperiodic DoS attack occurring in the communication channels. With a proper piecewise Lyapunov function, some sufficient conditions for global exponential stability (GES) of the closed-loop nonlinear UMV system are derived, and the corresponding observer and controller gains are designed via solving a set of matrix inequalities. A benchmark nonlinear UMV system is adopted as an example in simulation, and the simulation results validate the effectiveness of the proposed control method. [ABSTRACT FROM AUTHOR]

This article presents an adaptive control framework for dynamically reconfigurable battery (DRB) systems based on the deep reinforcement learning method. The proposed adaptive control framework relies on deep Q-network to learn the DRB system operations. By utilizing its model-free nature, the proposed framework can significantly reduce the complexity of building experiences or expert models for DRB systems as well as improve battery operating time by ensuring cell balancing. Extensive simulation and experimental study has been carried out with data gathered from a real-world DRB testbed, and the results show the effectiveness and efficiency of the proposed control framework. [ABSTRACT FROM AUTHOR]

In this brief, cluster synchronization of Boolean Networks (BNs) under state-flipped control is considered. We show how the cluster synchronization problem can be transformed into a set stabilization problem, based on which we give a theorem to judge whether the cluster synchronization of BNs can be achieved under a given flip set. Moreover, when the network structure is unknown, the $Q$ -learning $(QL)$ algorithm, a model-free reinforcement learning algorithm, is developed to search control sequences to achieve cluster synchronization. Some numerical examples are used to verify the validity of the theoretical results at the end. [ABSTRACT FROM AUTHOR]

The rising aging population, inequality of medical resources, and severe COVID-19 infection rate raise inevitable individual and social contradictions. One of the representative developing technologies, smart wearables, is dedicated to offering accurate personal healthcare. Nevertheless, energy constraints as well as unpredictable data transmission are critical in the development of wearable devices. In this regard, we investigate the key concerns of energy life and quality of service (QoS) for smart clothing. Unlike general wireless sensing networks (WSNs), the wireless body area network (WBAN) embedded in smart clothing is highly affected by human postural changes. In this article, we formulate the smart clothing with multiposture participated from two perspectives: 1) for energy life, we address the energy consumption, the energy harvested by the nodes, and the battery discharge and 2) the QoS involves the path loss and time delay. Moreover, five typical daily activity states have been discussed to model the impact of posture changes. Under the influence of the posture state, the tradeoff between the collected tribological electrical energy and the consumed energy is also presented in the article. We parameterize the path loss, transmission delay, energy consumption, and collection in each posture and integrally formulate the energy problem and QoS to a joint optimization problem. Particle swarm optimization (PSO), sine cosine algorithm (SCA), and Q-learning algorithm are adopted to optimize the overall cost, time delay, and energy consumption. In addition, a comparison of the battery power of the nodes is conducted. Simulation results show that each algorithm achieves certain optimization effects, for example, PSO, SCA, and Q-learning reduce total costs by 14%, 22%, and 30%, respectively. Q-learning is also effectively decreasing latency and energy consumption and improving battery life. [ABSTRACT FROM AUTHOR]

Link failure and network congestion have a serious impact on the performance degradation of network on chip (NoC). Adaptive routing algorithms, which have the capability of fault tolerance and congestion awareness, can make NoCs adapt to complex and changeable applications. The proposed algorithm in this letter is an adaptive routing algorithm based on $Q$ -learning. By introducing $H - Q$ table to learn the congestion and fault information of the paths between the source node and the destination node, it can avoid the unnecessary detour of data in the fault area and select the path with less congestion. Aiming at the problem caused by the untimely updating of some values in $Q$ -learning, the $Q$ -value decay and dynamic learning rate mechanism are proposed. Experimental results demonstrate that the proposed routing algorithms still can obtain a good performance under a high failure rate (Fault rate >25%) with modest extra hardware overhead. [ABSTRACT FROM AUTHOR]

This article proposes robust inverse $Q$ -learning algorithms for a learner to mimic an expert’s states and control inputs in the imitation learning problem. These two agents have different adversarial disturbances. To do the imitation, the learner must reconstruct the unknown expert cost function. The learner only observes the expert’s control inputs and uses inverse $Q$ -learning algorithms to reconstruct the unknown expert cost function. The inverse $Q$ -learning algorithms are robust in that they are independent of the system model and allow for the different cost function parameters and disturbances between two agents. We first propose an offline inverse $Q$ -learning algorithm which consists of two iterative learning loops: 1) an inner $Q$ -learning iteration loop and 2) an outer iteration loop based on inverse optimal control. Then, based on this offline algorithm, we further develop an online inverse $Q$ -learning algorithm such that the learner mimics the expert behaviors online with the real-time observation of the expert control inputs. This online computational method has four functional approximators: a critic approximator, two actor approximators, and a state-reward neural network (NN). It simultaneously approximates the parameters of $Q$ -function and the learner state reward online. Convergence and stability proofs are rigorously studied to guarantee the algorithm performance. [ABSTRACT FROM AUTHOR]

As the mainstream solid-state storage technology, NAND flash has the advantages of tiny size, cost-effective, and high performance, which make it a promising candidate to be embedded into the shingled magnetic recording (SMR) disk to build a faster, denser, and cheaper storage system. However, such an embedded flash with SMR (EF-SMR) disk system suffers from lengthy tail-latency due to “reclamation issues” in both the NAND flash and the SMR disk. Our preliminary observations reveal that tremendous idle time intervals exist in real-world scenarios, and few prior works have focused on addressing the tail-latency issue in the EF-SMR disk. In this article, we propose a novel method termed MAID-Q to fully exploit the idle time intervals to minimize the lengthy tail-latency of the EF-SMR disk based on a lightweight reinforcement learning model (i.e., the $Q$ -learning model). In addition, fine-grained block-level space management and a parallel reclamation strategy are proposed to improve the reclamation efficiency and hide the reclamation overheads. The effectiveness of our proposed design was evaluated with realistic I/O traces, and the results show that the proposed design can remedy the tail-latency by 88.31% and improve the overall performance by 79.03%. [ABSTRACT FROM AUTHOR]

The ongoing digitalization enables stable control processes and smooth operations of Industrial Control Systems (ICSs). A direct consequence of the highly interconnected architecture of ICSs is the introduced cyber vulnerability and increasing cyber security threats to ICSs. Numerous researches pay attention to the security problem of ICSs. However, most current studies face two challenges. First, the interaction problem between the cyber layer and the physical layer of ICSs may result in incorrect attack response strategies. Second, ICSs are real-time systems, but existing defense decision algorithms based on game theory or reinforcement learning techniques have high computational complexity, which prevents them from making decisions quickly. In this paper, we design a new multi-attribute based method for quantifying rewards and propose a multi-attribute based Q-learning algorithm to resolve the interaction problem. In addition, to overcome the limitation of slow convergence, we develop an effective parallel Q-learning (PQL) algorithm to quickly find the optimal strategy. The experimental results show the effectiveness of the PQL algorithm. Compared with the Q-learning algorithm (QL) and the deep Q-network (DQN) algorithm, our proposed solution can reduce the average completion time by 12.5 to 37 percent. [ABSTRACT FROM AUTHOR]

The uncertainties from the distributed energy resources (DERs) and the prosumer’s heterogeneous characteristics bring great challenges to the operation of the community energy market. In this paper, the real-time energy sharing and management in the community market are decomposed into two-layered sub-problems: the household appliance scheduling problem and the internal energy trading pricing problem. A hierarchical deep reinforcement learning (HDRL) based scheme is proposed for the community energy trading with multiple households, containing a two-stage learning process. In the inner stage, a multi-agent deep reinforcement learning (MADRL) based approach is developed to learn the real-time appliances scheduling policy based on the local observations and given internal electricity price in a decentralized way. In the outer stage, a deep reinforcement learning (DRL) based pricing approach is proposed to determine the real-time internal electricity prices based on the participants’ historical net power and external energy supplier’s electricity prices. The scheduling policy and households’ state are not required in the pricing process. Finally, the simulations with real-world datasets demonstrate that the internal energy trading can significantly reduce the prosumers’ daily cost and the control performance of the proposed scheme is superior to the existing studies. [ABSTRACT FROM AUTHOR]

To address the coordination issue of connected automated vehicles (CAVs) at urban scenarios, a game-theoretic decision-making framework is proposed that can advance social benefits, including the traffic system efficiency and safety, as well as the benefits of individual users. Under the proposed decision-making framework, in this work, a representative urban driving scenario, i.e. the unsignalized intersection, is investigated. Once the vehicle enters the focused zone, it will interact with other CAVs and make collaborative decisions. To evaluate the safety risk of surrounding vehicles and reduce the complexity of the decision-making algorithm, the driving risk assessment algorithm is designed with a Gaussian potential field approach. The decision-making cost function is constructed by considering the driving safety and passing efficiency of CAVs. Additionally, decision-making constraints are designed and include safety, comfort, efficiency, control and stability. Based on the cost function and constraints, the fuzzy coalitional game approach is applied to the decision-making issue of CAVs at unsignalized intersections. Two types of fuzzy coalitions are constructed that reflect both individual and social benefits. The benefit allocation in the two types of fuzzy coalitions is associated with the driving aggressiveness of CAVs. Finally, the effectiveness and feasibility of the proposed decision-making framework are verified with three test cases. [ABSTRACT FROM AUTHOR]