Search

The exposure of zero trust security in the Industrial Internet of Things (IIoT) increased in importance in the era where there is a huge risk of injection of malicious entities and owning the device by an unauthorized user. The gap in the existing approach of zero trust security is that continuous verification of devices is a time-consuming process and adversely affects the promising nature of the zero-trust model. Every time the node enters, even if the node is a member of the network, authorization of the node is necessary to ensure authenticity. This verification section of zero trust hinders the seamless working of the IIoT infrastructure. Therefore, the main objective of this paper is to propose the solution for the above-mentioned problem by enabling "device profiling" via deep reinforcement learning so that the same device can be identified and permitted access without hindering the working of Industrial Internet of Things infrastructure. The overall proposed approach works in different phases including the compression function for ensuring data confidentiality and integrity, then the device profiling is performed based on the features a device possesses, and lastly, deep reinforcement learning for anomaly detection. To test and validate the proposed approach, extensive experimentations were performed using measures such as false positive rate, data confidentiality rate, data integrity rate, and network access time, and results showed that the proposed technique titled "MMODPAD-DRL" outperforms the existing approaches in false positive rate by 27%, data confidentiality rate by 4% and data integrity rate by 3%, in addition, lessen the network access time by 20%. [ABSTRACT FROM AUTHOR]

In the harsh mobile edge computing (HMEC) environment, there are many dynamic changes such as interference from noise, the impact of extreme environmental conditions, and the mobility of devices. It is a great challenge to the online realtime task offloading scheduling for delay-sensitive applications. However, the dynamic changes in HMEC environment have been ignored in almost all previous studies. Therefore, we propose the federated deep reinforcement learning-based online task offloading and resource allocation (FD-OTR) algorithm to address the task offloading in HMEC. Additionally, the FD-OTR algorithm performs resource allocation for offloaded tasks. The task offloading part of FD-OTR algorithm can be divided into two layers: the deep reinforcement learning (DRL) layer and the federated learning (FL) layer. The online algorithm in the DRL layer can adapt to the dynamic HMEC environment and make real-time task offloading decisions. In the FL layer, federated learning with low communication overhead is used for model aggregation to form a better global model. Resource allocation is done by using a new meta-heuristic algorithm: the Sparrow Search Algorithm (SSA). Finally, the simulation results demonstrate that the FD-OTR algorithm performs well in HMEC. The convergence speed of FD-OTR is three times faster than the centralized method. Compared to the baseline algorithms, FD-OTR reduces costs by 14.3%, 11.2% and 9.28%, respectively. [ABSTRACT FROM AUTHOR]

Control–structure interaction (CSI) plays a significant role in active control systems. Popular methods incorporate actuator dynamics into an integrated control system to account for CSI, leading to a situation where existing structural control algorithms that ignore CSI cannot be applied directly. To address this issue, this study proposes a deep reinforcement learning (DRL) based active mass driver (AMD) decoupled control framework, in which a structural control algorithm is employed to generate the control force command without consideration of CSI, while a DRL agent is utilized to attenuate the CSI effects of AMD systems and achieve a desired control force. The DRL‐based AMD control framework is verified through a series of numerical experiments. Furthermore, the applicability of the control framework is confirmed in a wind‐excited 76‐story benchmark building. Comprehensive analysis indicates that the proposed control framework can effectively eliminate the CSI effects and apply accurate control force to the structure in various scenarios, which allows for an ideal structural response control. [ABSTRACT FROM AUTHOR]

• Performance of object detection tends to degrade on low-quality images. • Choosing suitable enhancement algorithms can improve the detection performance. • Reinforcement learning can autonomously select algorithms based on the scenario. • Reinforcement learning avoid the issue of data labeling and retraining. The task of object detection for low-quality images in multiple scenarios remains a challenging task. To alleviate this problem, a novel approach to improve the detection performance of the model on low-quality images using deep reinforcement learning is proposed. The approach differs from methods that involve data labeling and retraining, as the detection results of the model are utilized to formulate a reward function. This enables deep reinforcement learning to transform low-quality images into high-quality images through image enhancement, ultimately leading to an improvement in the model's recognition performance on low-quality images. An image enhancement tool chain (IETC) has been prepared, offering a flexible and lightweight algorithm selection for the Dueling Deep Q-Network-based Tool Selector (DDQN-TS). By adding a "Pass" option, Enable DDQN-TS to learn the relative relationship between performing and not performing image enhancement. To mitigate the potential inconsistency between the output of the detection model and the actual situation caused by negative samples, which can lead to incorrect decisions made by DDQN-TS, the introduction of a "thresh" parameter regulates the level of intervention in target detection by DIE. The experimental results demonstrate the effectiveness and portability of the DIE model in environments with noise, fog, and uneven brightness. [ABSTRACT FROM AUTHOR]

The frequently evolving manufacturing system necessitates real-time large-scale data analytics, surpassing the capabilities of classical systems or human skills, necessitating knowledge development and self-adaptive control. Unpredictable real-time events in smart factories cause changes in the effectiveness of planned schedules or tasks; even slight interruptions can accumulate to make the preplanned schedule unoptimized, if not impossible. To meet the required production efficiency, typical dynamic scenarios on the shop floor, such as failures, random job arrivals, and machine setup, must be addressed quickly. Hence, this study proposes a novel Triple Deep Q Network (TDQN) approach for learning high-quality dispatching rules for addressing the Flexible Job Shop Scheduling Problem (FJSSP) in uncertainty. For the dispatching rule-based FJSSP, the Markov Decision Process (MDP) to choose a suitable operation-machine (O-M) pair is formulated to allow operation selection and resource assignment resolutions to be made simultaneously. The performance of the TDQN method is investigated against the Double Deep Q Network (DDQN) and Deep Q Network (DQN) approach and found to be more stable. Moreover, among the 29 different shopfloor configuration settings, TDQN performed better in 62.07% of the overall instances, while DDQN and DQN performed better in only 31.03% and 6.9% of instances, respectively. [ABSTRACT FROM AUTHOR]

In recent years, space research has shifted heavily its focus towards enhanced autonomy on-board spacecrafts for on-orbit servicing activities (OOS). OOS and proximity operations include a variety of activities: the focal point of this work is the autonomous guidance of a chaser spacecraft for the shape reconstruction of an artificial uncooperative object. Adaptive guidance depends on the ability of the system to build a map of the uncertain environment, figuring out its location inside of it and accordingly determining the control law. Thus, autonomous navigation is framed as an active Simultaneous Localization and Mapping (SLAM) problem and modeled as a Partially Observable Markov Decision Process (POMDP). A state-of-the-art Deep Reinforcement Learning (DRL) method, Proximal Policy Optimization (PPO), is investigated to develop an agent capable of cleverly planning the shape reconstruction of the target. Starting from previous research on the topic, this work develops further proposing a continuous action space, such that the agent is no more forced to choose between a predefined set of possible discrete actions, fixed both in magnitude and direction. In this way any combination of the three-dimensional thrust vector components is available. The chaser spacecraft is a small satellite mounting an electric propulsion engine defining the action space range, in linearized eccentric relative motion with the selected uncooperative object. Through a rendered triangular mesh, the agent capabilities of geometry reconstruction and mapping are evaluated, considering the number of quality pictures made for each face. Extensive training tests are performed with random initial conditions to verify the generalizing capability of the DRL agent. The results are then validated in a comprehensive testing campaign, whose primary focus is the introduction of noisy measurements coming from navigation, affecting pose estimation. The sensitivity of the proposed method to this condition is analyzed and the efficiency of a retraining procedure is examined. The applicability of DRL methods and neural networks to support autonomous guidance in a close proximity scenario is corroborated and the technique employed is vastly tested and verified. [ABSTRACT FROM AUTHOR]

With the widespread adoption of electric vehicles (EVs), the demand for public charging services is steadily increasing. Consequently, the development of effective charging scheduling strategies, aimed at optimizing the utilization of limited charging infrastructure, has become a key problem. Considering the diversity of user demands, we propose a Cost-Aware Charging Scheduling Architecture (CASA). This architecture considers both urgent and nonurgent charging customers by designing two charging modes with different power levels and associated costs. However, optimizing multiple objectives simultaneously while ensuring the interests of all parties involved in the charging demand response presents a challenge. Moreover, the uncertainty in customer charging demands and Time-of-Use (TOU) tariff further complicates the establishment of the model. To address the aforementioned challenges, this study formulates EV charging scheduling as a Markov Decision Process (MDP) based on deep reinforcement learning (DRL), employing the Deep Q-Network (DQN) algorithm for solution derivation. The objective is to minimize the operational costs of charging stations while ensuring the quality of service (QoS) requirements for customers. The simulation results demonstrate that CASA exhibits superior performance in optimizing both the average response time and service success rate, compared to commonly used baselines for charging scheduling. Furthermore, the CASA approach achieves a significant reduction in operating costs of EV charging station. [ABSTRACT FROM AUTHOR]

As the bias between the estimated value and the true value, overestimation is a basic problem in reinforcement learning, which leads to a lower total reward because of the incorrect action decisions. In order to reduce the impact of overestimation on reinforcement learning, we propose a double Actor-Critic learning system embedding improved Monte Carlo Tree Search (DAC-IMCTS). The proposed learning system consists of a reference module, a simulation module and an outcome module. The reference module and the simulation module are designed to compute the upper bound and lower bound of the expected reward of the agent, respectively. And the outcome module is developed to learn the agent's control policies. The reference module, constructed based on the Actor-Critic framework, provides an upper confidence bound of the expected reward. Different from the classic Actor-Critic learning system, we introduce a simulation module into the new learning system to estimate the lower confidence bound of the expected reward. We propose an improved MCTS in this module to sample the policy distribution more efficiency. Based on the lower and upper confidence bounds, we propose a confidence interval weighted estimation algorithm (CIWE) in the outcome module for generating the target expected reward. We then prove that the target expected reward generated by our method has zero expectation bias, which reduces the overestimation that exists in the classic Actor-Critic learning system. We evaluate our learning system on OpenAI Gym experimental tasks. The experimental results show that our proposed model and algorithm outperform the state-of-the-art learning systems. [ABSTRACT FROM AUTHOR]

The IEEE 802.11ah standard is introduced to address the growing scale of internet of things (IoT) applications. To reduce contention and enhance energy efficiency in the system, the restricted access window (RAW) mechanism is introduced in the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, to achieve optimized network performance, it is necessary to appropriately determine the RAW parameters, including the number of RAW groups, the number of slots in each RAW, and the duration of each slot. In this paper, we optimize the configuration of RAW parameters in the uplink IEEE 802.11ah-based IoT network. To improve network throughput, we analyze and establish a RAW parameters optimization problem. To effectively cope with the complex and dynamic network conditions, we propose a deep reinforcement learning (DRL) approach to determine the preferable RAW parameters to optimize network throughput. To enhance learning efficiency and stability, we employ the proximal policy optimization (PPO) algorithm. We construct network environments with periodic and random traffic in an NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. The simulation results reveal that using the PPO-based DRL algorithm, optimized RAW parameters can be obtained under different network conditions, and network throughput can be improved significantly. [ABSTRACT FROM AUTHOR]

The unmanned combat system-of-systems (UCSoS) in modern warfare is comprised of various interconnected entities that work together to support mission accomplishment. The soaring number of entities makes the UCSoS fragile and susceptible to triggering cascading effects when exposed to uncertain disturbances such as attacks or failures. Reconfiguring the UCSoS to restore its effectiveness in a self-coordinated and adaptive manner based on the battlefield situation and operational requirements has attracted increasing attention. In this paper, we focus on the UCSoS reconstruction with heterogeneous costs, where the collaboration nodes may have different reconstruction costs. Specifically, we adopt the heterogeneous network to capture the interdependencies among combat entities and propose a more representative metric to evaluate the UCSoS reconstruction effectiveness. Next, we model the combat network reconstruction problem with heterogeneous costs as a nonlinear optimization problem and prove its NP-hardness. Then, we propose an approach called SoS-Restorer, which is based on deep reinforcement learning (DRL), to address the UCSoS reconstruction problem. The results show that SoS-Restorer can quickly generate reconstruction strategies and improve the operational capabilities of the UCSoS by about 20∼60% compared to the baseline algorithm. Furthermore, even when the size of the UCSoS exceeds that of the training data, SoS-Restorer exhibits robust generalization capability and can efficiently produce satisfactory results in real time. [ABSTRACT FROM AUTHOR]

Vehicular Edge Computing (VEC) is a crucial application of Mobile Edge Computing (MEC) in vehicular networks. In VEC networks, the computation tasks of vehicle terminals (VTs) can be offloaded to nearby MEC servers, overcoming the limitations of VTs' processing power and reducing latency caused by distant cloud communication. However, a mismatch between VTs' demanding tasks and MEC servers' limited resources can overload MEC servers, impacting Quality of Service (QoS) for computationally intensive tasks. Additionally, vehicle mobility can disrupt communication with static MEC servers, further affecting VTs' QoS. To address these challenges, this paper proposes a vehicle collaborative partial computation offloading model. This model allows VTs to offload tasks to two types of service nodes: collaborative vehicles and MEC servers. Factors like a vehicle's mobility, remaining battery power, and available computational power are also considered when evaluating its suitability for collaborative offloading. Furthermore, we design a deep reinforcement learning-based strategy for collaborative partial computation offloading that minimizes overall task delay while meeting individual latency constraints. Experimental results demonstrate that compared to traditional approaches without vehicle collaboration, this scheme significantly reduces latency and achieves a significant reduction (around 2%) in the failure rate under tighter latency constraints. [ABSTRACT FROM AUTHOR]

Recently, unmanned aerial vehicles (UAVs) have been widely used in mobile edge computing (MEC) scenarios due to their flexibility, rapid deployment, and ability to expand communication coverage. This paper explores how UAVs can compute offloading more effectively when serving as mobile edge servers in the scenario of UAVs-assisted MEC. We aim to maximize the utility of the UAV operator by joint flight trajectory of UAVs, users' offloading decisions, and collaborative strategy among UAVs while considering the latency and energy consumption. To this end, we first propose a dynamic latency-aware collaborative framework based on deep reinforcement learning. This framework utilizes a queuing model to implement the dynamic task backlog and models the reward of latency-sensitive tasks using a service level agreement. Then, considering the dynamics of the environment and the complexity of the problem, we describe the problem as a markov decision process and propose a proximal policy optimization-based computation offloading algorithm to solve the problem. The experimental results finally demonstrate that the proposed offloading algorithm converges rapidly and exhibits better performance compared to benchmark algorithms. [ABSTRACT FROM AUTHOR]

To address the current challenge of smart devices in healthcare Internet of things (IoT) struggling to efficiently process intensive applications in real-time, a collaborative cloud-edge offloading model tailored for ultra-dense edge computing (UDEC) networks is developed. While numerous studies have delved into the optimization of offloading in mobile edge computing (MEC), it is imperative to consider non-orthogonal multiple access (NOMA) as a physical technology when addressing the offloading optimization process in MEC. The multiuser sharing of spectrum resources in NOMA can enhance the network spectrum utilization and reduce the computational delay when users transmit computing tasks. Consequently, a model for NOMA-assisted UDEC systems is proposed. The model takes into account joint offloading decisions, computational resources, and sub-channel resources and is modeled as a complex nonlinear mixed-integer programming problem. The aim is to decrease the task execution delay and energy consumption of smart devices while ensuring that users' maximum acceptable delay for processing medical computational tasks is met efficiently and in a timely manner. Deep deterministic policy gradient (DDPG), a deep reinforcement learning method, is employed to solve the joint optimization problem. The final simulation results show that the algorithm converges well. The proposed offloading scheme can reduce the system cost by 54.5 and 69.9% in comparison with scenarios where users solely perform local computations and offload their tasks to the base station (BS). The application of NOMA communication in our offloading scheme boosts network spectrum utilization and trims down the system cost by 87.09% when contrasted with orthogonal multiple access (OMA). [ABSTRACT FROM AUTHOR]

Summary: Cognitive radio networks (CRNs) have emerged as a promising solution to address the spectrum scarcity problem by allowing unlicensed secondary users (SUs) to opportunistically access the underutilized licensed spectrum bands. However, efficient channel assignment in CRNs, especially with user mobility and power control considerations, remains a challenging problem. Traditional channel assignment algorithms often struggle to adapt to changing network conditions, resulting in suboptimal performance. To address these challenges, this research presents a novel Taylor Tuna Optimizer‐based hybrid deep Q‐network (TayTO‐based HDQNet) for channel assignment in CRNs with user mobility and power control considerations. The proposed TayTO‐based HDQNet combines the Elman neural network architecture with the reinforcement learning framework of deep Q‐networks, tuned using the Taylor Tuna Optimizer. This hybrid approach enables intelligent channel assignment decisions while considering power consumption constraints, leading to efficient communication in CRNs. To train the HDQNet agent, a comprehensive dataset is generated through simulations of various CRN scenarios, incorporating channel availability, interference levels, and power consumption for different channel assignment decisions. The HDQNet agent undergoes iterative training using this dataset to develop an optimal channel assignment policy. Experimental results demonstrate the effectiveness of the proposed TayTO‐based HDQNet approach done by MATLAB, achieving a high success rate of 98.47% and surpassing the reliability scores of previous studies. This highlights the improved performance and reliability of the proposed approach for channel assignment in CRNs. [ABSTRACT FROM AUTHOR]

The increasing prevalence of Internet of Things (IoT) systems has made them attractive targets for malicious actors. To address the evolving threats and the growing complexity of detection, there is a critical need to search for and develop new algorithms that are fast and robust in detecting and classifying dangerous network traffic. In this context, deep reinforcement learning (DRL) is gaining recognition as a prospective solution in numerous fields as it enables autonomous agents to cooperate with their environment for decision-making without relying on human experts. This article presents an innovative approach to intrusion detection in IoT systems using an adversarial reinforcement learning (RL) algorithm known for its exceptional predictive capabilities. The predictive process relies on a classifier, implemented as a streamlined and highly efficient neural network. Embedded within this classifier is a policy function meticulously trained using an innovative RL model. Importantly, this model ensures that the environment's behavior is dynamically fine-tuned simultaneously with the learning process, improving the overall effectiveness of the intrusion detection approach. The efficiency of our proposal was assessed using the Bot-IoT database, consisting of a mixture of legitimate IoT network traffic and simulated attack scenarios. Our scheme shows superior performance compared to existing ones. Therefore, our approach to IoT intrusion detection can be considered a valuable alternative to existing methods, capable of significantly improving the IoT systems' security. [ABSTRACT FROM AUTHOR]

Decentralized execution is a widely used framework in multi-agent reinforcement learning. However, it has a well-known but neglected shortcoming, redundant computation, that is, the same/similar computation is performed redundantly in different agents owing to their overlapping observations. This study proposes a novel method, the locally centralized team transformer (LCTT), to address this problem. This method first proposes a locally centralized execution framework that autonomously determines some agents as leaders that generate instructions and other agents as workers to act according to the received instructions without running their policy networks. For the LCTT, we subsequently propose the team-transformer (T-Trans) structure, which enables leaders to generate targeted instructions for each worker, and the leadership shift, which enables agents to determine those that should instruct or be instructed by others. The experimental results demonstrated that the proposed method significantly reduces redundant computations without decreasing rewards and achieves faster learning convergence. [ABSTRACT FROM AUTHOR]

The proliferation of shared electric scooters (E-scooters) has brought convenience to urban transportation but has also introduced challenges such as disorderly parking and an imbalance between supply and demand. Given the current inconsistent quantity and spatial distribution of shared E-scooters, coupled with inadequate research on deployment stations selection, we propose a novel maximal covering location problem (MCLP) based on distance tolerance. The model aims to maximize the coverage of user demand while minimizing the sum of distances from users to deployment stations. A deep reinforcement learning (DRL) was devised to address this optimization model. An experiment was conducted focusing on areas with high concentrations of shared E-scooter trips in Chicago. The solutions of location selection were obtained by DRL, the Gurobi solver, and the genetic algorithm (GA). The experimental results demonstrated the effectiveness of the proposed model in optimizing the layout of shared E-scooter deployment stations. This study provides valuable insights into facility location selection for urban shared transportation tools, and showcases the efficiency of DRL in addressing facility location problems (FLPs). [ABSTRACT FROM AUTHOR]

In unstructured environments, robots need to deal with a wide variety of objects with diverse shapes, and often, the instances of these objects are unknown. Traditional methods rely on training with large-scale labeled data, but in environments with continuous and high-dimensional state spaces, the data become sparse, leading to weak generalization ability of the trained models when transferred to real-world applications. To address this challenge, we present an innovative maximum entropy Deep Q-Network (ME-DQN), which leverages an attention mechanism. The framework solves complex and sparse reward tasks through probabilistic reasoning while eliminating the trouble of adjusting hyper-parameters. This approach aims to merge the robust feature extraction capabilities of Fully Convolutional Networks (FCNs) with the efficient feature selection of the attention mechanism across diverse task scenarios. By integrating an advantage function with the reasoning and decision-making of deep reinforcement learning, ME-DQN propels the frontier of robotic grasping and expands the boundaries of intelligent perception and grasping decision-making in unstructured environments. Our simulations demonstrate a remarkable grasping success rate of 91.6%, while maintaining excellent generalization performance in the real world. [ABSTRACT FROM AUTHOR]

Optimization of industrial processes such as manufacturing or processing of specific materials constitutes a point of interest for many researchers, and its application can lead not only to speeding up the processes in question, but also to reducing the energy cost incurred during them. This article presents a novel approach to optimizing the spindle motion of a computer numeric control (CNC) machine. The proposed solution is to use deep learning with reinforcement to map the performance of the reference points realization optimization (RPRO) algorithm used in the industry. A detailed study was conducted to see how well the proposed method performs the targeted task. In addition, the influence of a number of different factors and hyperparameters of the learning process on the performance of the trained agent was investigated. The proposed solution achieved very good results, not only satisfactorily replicating the performance of the benchmark algorithm, but also speeding up the machining process and providing significantly higher accuracy. [ABSTRACT FROM AUTHOR]

Addressing the exploration challenges faced by mobile robots using deep reinforcement learning for autonomous navigation in cluttered, obstacle-dense complex environments, this paper proposed the trajectory-guided navigation policy optimization (TGNPO) algorithm. Firstly, it employed an imitation learning approach to train an expert policy for a mobile robot, which could provide both expert demonstration behavior and navigation trajectory prediction and aimed to comprehensively guide the training of deep reinforcement learning. Secondly, it fused the predicted navigation trajectory from the expert policy with real-time images perceived by the mobile robot at the current moment. Combining the coordinate attention mechanism, it extracted feature regions which would guide the robot’s future navigation, thereby enhancing the learning performance of the navigation model. Finally, it utilized the navigation trajectory predicted by the expert policy to constrain the policy trajectory of the mobile robot, mitigating ineffective exploration and erroneous decision-making during navigation process By deploying the proposed algorithm on both simulation and physical platforms, experimental results demonstrated significant advantages in navigation learning efficiency and trajectory smoothness compared to existing state-of-the-art methods which fully proves the proposed algorithm’s capability to efficiently and safely execute robot navigation tasks. [ABSTRACT FROM AUTHOR]

Multi-agent systems have a wide range of applications in many fields, such as autonomous driving, intelligent logistics, and medical collaboration, etc. However, due to technological advances and increased system requirements, these systems face challenges such as large scale and high complexity, and often suffer from inefficient training and poor adaptability. To address these problems, this paper proposed a multi-agent first-order meta proximal policy optimization (MAMPPO) method by extending gradient-based meta-learning to multi-agent deep reinforcement learning. The method learned the initial model parameters in the multi-agent system to provide a new perspective for improving the performance of multi-agent deep reinforcement learning. It made full use of the previous experience in the process of multi-agent reinforcement learning to find the most sensitive parameters in the direction of gradient descent through repeated adaptation, and learned the initial parameters so that the model training starts from the optimal starting point. This method effectively improved the decision-making efficiency of the joint policy, and led to a significant increase in the speed of its policy change, which significantly accelerated the speed of adaptation in the face of a new situation. Experimental results on StarCraft Ⅱ show that the MAMPPO method can significantly improve the training speed and adaptability, which provides a new solution for the subsequent improvement of the training efficiency and adaptability of multi-agent reinforcement learning. [ABSTRACT FROM AUTHOR]

The efficient planning of electric power systems is essential to meet both the current and future energy demands. In this context, reinforcement learning (RL) has emerged as a promising tool for control problems modeled as Markov decision processes (MDPs). Recently, its application has been extended to the planning and operation of power systems. This study provides a systematic review of advances in the application of RL and deep reinforcement learning (DRL) in this field. The problems are classified into two main categories: Operation planning including optimal power flow (OPF), economic dispatch (ED), and unit commitment (UC) and expansion planning, focusing on transmission network expansion planning (TNEP) and distribution network expansion planning (DNEP). The theoretical foundations of RL and DRL are explored, followed by a detailed analysis of their implementation in each planning area. This includes the identification of learning algorithms, function approximators, action policies, agent types, performance metrics, reward functions, and pertinent case studies. Our review reveals that RL and DRL algorithms outperform conventional methods, especially in terms of efficiency in computational time. These results highlight the transformative potential of RL and DRL in addressing complex challenges within power systems. [ABSTRACT FROM AUTHOR]

Assessments of the hosting capacity of electricity distribution networks are of paramount importance, as they facilitate the seamless integration of rooftop photovoltaic systems into the grid, accelerating the transition towards a more carbon neutral and sustainable system. This paper employs a deep reinforcement learning-based approach to evaluate the real-time hosting capacity of low voltage distribution networks in a model-free manner. The proposed approach only requires real-time customer voltage data and solar irradiation data to provide a fast and accurate estimate of real-time hosting capacity at each customer connection point. This study addresses the imperative for accurate electrical models, which are frequently unavailable, in evaluating the hosting capacity of electricity distribution networks. To meet this challenge, the proposed approach utilizes a deep neural network-based, data-driven model of a low-voltage electricity distribution network. This proposed methodology incorporates model-free elements, enhancing its adaptability and robustness. In addition, a comparative analysis between model-based and model-free hosting capacity assessment methods is presented, highlighting their respective strengths and weaknesses. The utilization of the proposed hosting capacity estimation model enables distribution network service providers to make well-informed decisions regarding grid planning, leading to cost minimization. [ABSTRACT FROM AUTHOR]

In this paper, we consider a low-latency Mobile Edge Computing (MEC) network where multiple User Equipment (UE) wirelessly reports to a decision-making edge server. At the same time, the transmissions are operated with Finite Blocklength (FBL) codes to achieve low-latency transmission. We introduce the task of Age upon Decision (AuD) aimed at the timeliness of tasks used for decision-making, which highlights the timeliness of the information at decision-making moments. For the case in which dynamic task generation and random fading channels are considered, we provide a task AuD minimization design by jointly selecting UE and allocating blocklength. In particular, to solve the task AuD minimization problem, we transform the optimization problem to a Markov Decision Process problem and propose an Error Probability-Controlled Action-Masked Proximal Policy Optimization (EMPPO) algorithm. Via simulation, we show that the proposed design achieves a lower AuD than baseline methods across various network conditions, especially in scenarios with significant channel Signal-to-Noise Ratio (SNR) differences and low average SNR, which shows the robustness of EMPPO and its potential for real-time applications. [ABSTRACT FROM AUTHOR]

Our increasingly connected world continues to face an ever-growing number of network-based attacks. An Intrusion Detection System (IDS) is an essential security technology used for detecting these attacks. Although numerous Machine Learning-based IDSs have been proposed for the detection of malicious network traffic, the majority have difficulty properly detecting and classifying the more uncommon attack types. In this paper, we implement a novel hybrid technique using synthetic data produced by a Generative Adversarial Network (GAN) to use as input for training a Deep Reinforcement Learning (DRL) model. Our GAN model is trained on the NSL-KDD dataset, a publicly available collection of labeled network traffic data specifically designed to support the evaluation and benchmarking of IDSs. Ultimately, our findings demonstrate that training the DRL model on synthetic datasets generated by specific GAN models can result in better performance in correctly classifying minority classes over training on the true imbalanced dataset. [ABSTRACT FROM AUTHOR]

Label accuracy plays an important role in supervised hyperspectral image classification problems. However, labelling can be prone to various types of noise, mainly including human errors and incorrect pseudo-label predictions. Learning from datasets with noisy labels is a critical challenge in practice. To address this issue, we first propose a deep reinforcement learning method named HyperRefine to tackle the label noise problem for hyperspectral image classification. The method utilizes an agent to learn the accuracy of each sample label and determine its suitability to be used in training the classifier model. The agent is trained using a reinforcement signal based on the reward obtained from a small validation set that reflects performance on the target task, where the validation set is generated from the training set by a momentum-based recruiter. Compared to complex hand-crafted methods of label noise cleaning, HyperRefine is easy to implement, yet highly effective. The robustness of the model to different types of label noises is significantly improved by further combining a novel label refinement approach. Experiments on artificially added label noise and pseudo-label noise hyperspectral datasets show that the proposed method can achieve promising performance. [ABSTRACT FROM AUTHOR]

Postprandial glucose control can be challenging for individuals with type 1 diabetes, and this can be attributed to many factors, including suboptimal therapy parameters (carbohydrate ratios, correction factors, basal doses) because of physiological changes, meal macronutrients and engagement in postprandial physical activity. This narrative review aims to examine the current postprandial glucose‐management strategies tested in clinical trials, including adjusting therapy settings, bolusing for meal macronutrients, adjusting pre‐exercise and postexercise meal boluses for postprandial physical activity, and other therapeutic options, for individuals on open‐loop and closed‐loop therapies. Then we discuss their challenges and future avenues. Despite advancements in insulin delivery devices such as closed‐loop systems and decision‐support systems, many individuals with type 1 diabetes still struggle to manage their glucose levels. The main challenge is the lack of personalized recommendations, causing suboptimal postprandial glucose control. We suggest that postprandial glucose control can be improved by (i) providing personalized recommendations for meal macronutrients and postprandial activity; (ii) including behavioural recommendations; (iii) using other personalized therapeutic approaches (e.g. glucagon‐like peptide‐1 receptor agonists, sodium‐glucose co‐transporter inhibitors, amylin analogues, inhaled insulin) in addition to insulin therapy; and (iv) integrating an interpretability report to explain to individuals about changes in treatment therapy and behavioural recommendations. In addition, we suggest a future avenue to implement precision recommendations for individuals with type 1 diabetes utilizing the potential of deep reinforcement learning and foundation models (such as GPT and BERT), employing different modalities of data including diabetes‐related and external background factors (i.e. behavioural, environmental, biological and abnormal events). [ABSTRACT FROM AUTHOR]

The space time-sensitive multi-target collaborative observation scheduling problem extends the traditional Earth Observation Satellite (EOS) scheduling by incorporating the time dimension. Multiple effective and collaborative observations over time are required to collect relevant information on time-sensitive space targets in high-speed motion. Furthermore, changes in target spatial relationships cause conflicts in resource allocation. Due to the urgency of the time-sensitive targets, the observation system must respond promptly and efficiently upon receiving a task. This study proposes a mathematical model and a Time-Sensitive Observation Scheduling (TSOS) algorithm to address these problem characteristics. Firstly, a mathematical model for scheduling is established by introducing the completion quality index and resource switching frequency index, given the requirement for collaborative multiple satellites in this problem. The TSOS algorithm consists of two parts: task pre-processing and a circular scheduling strategy. In the task pre-processing module, the entire scheduling is divided into multiple observation decision intervals. This enables the candidate satellite set corresponding to each target to remain stable, which reduces the complexity of the problem. An action mapping method is used in the circular scheduling module, and a decision network based on Deep Reinforcement Learning (DRL) called the Observation Pair Decision Network is proposed. Experimental results demonstrate that the trained TSOS algorithm can effectively schedule resources in scenarios with different conflict degrees and exhibits strong robustness. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning has proven to be effective in various video games, such as Atari games, StarCraft II, Google research football (GRF), and Dota II. We participated in the 2022 IEEE Conference on Games Football AI Competition and ranked in the top eight. Despite recent efforts, building agents for GRF still suffers from multi-agent coordination, sparse rewards, and stochastic environments. To address these issues and achieve good outcomes in the competition, we devised a reinforcement learning algorithm that uses deep reinforcement learning from demonstrations and policy distillation. In this study, we innovatively propose a two-stage algorithm named mimic-to-counteract reinforcement learning (MCRL) based on the historical game logs of opponents, we encountered during the warm-up session and formulated partner agents function similarly to human sparring partners, whereby they simulate opponents with diverse styles of play, enabling primary players to practice against a range of policies, they may encounter in real competitions. Additionally, we trained numerous mentor agents capable of restraining the sparring partners. We distilled their policies and amalgamated them to train a potent primary agent. Empirical results show that the proposed MCRL algorithm can efficiently search for valuable strategies with stable updates and balance the relationship between policy iteration and policy style deviation. Also, the primary agent can learn diverse but coordinated counteracting strategies and ranks in the top eight in the competition. [ABSTRACT FROM AUTHOR]

The study proposes a method for the trajectory tracking control of a fixed-wing unmanned aerial vehicle (UAV) based on the deep deterministic policy gradient (DDPG). First, the problem of controlling the trajectory of a fixed-wing UAV is combined with the reinforcement learning framework and transformed into a Markov decision process, and a DDPG agent is established in the framework of TensorFlow. Second, we conducted simulations to train and optimize the model in a 3D environment of trajectory tracking control and obtained an integrated DDPG-based trajectory tracking controller that can regulate functions ranging from the state of flight of the UAV to rudder control. Third, we constructed a digital simulation system to verify the proposed method while considering the influence of parametric uncertainties, measurement-induced noise, and delays in the response of the control system. The effectiveness and robustness of the proposed DDPG controller were verified by comparing its performance with that of traditional proportional-integral-derivative (PID) control. Unmanned aerial vehicles have gained widespread use across numerous sectors of society. However, the control process for fixed-wing UAVs is more intricate and challenging than that for rotor type UAVs due to the coupled and nonlinear nature of their dynamic models. To address this issue, decoupling and splitting hierarchical control methods are commonly employed. However, these methods can introduce errors and affect flight performance. In contrast, artificial intelligence, particularly deep learning and reinforcement learning, has made significant advancements and found successful applications in areas such as robot control and gaming. To address the challenges posed by complex forms, excessive measurement information, and manual parameter tuning associated with traditional control approaches for fixed-wing UAVs, this paper introduces a deep reinforcement learning algorithm, i.e., the deep deterministic policy gradient, into the control process of fixed-wing UAVs. The paper presents a trajectory tracking control policy for UAVs, which efficiently achieves the task of tracking a desired trajectory without the need for selecting multiple working state nodes or designing linear control laws. Further, through simulation verification, the proposed policy demonstrates adaptability to interference caused by parameter bias, measurement noise, and control system response delays while achieving control accuracy comparable with classical control methods. [ABSTRACT FROM AUTHOR]

Although Deep Reinforcement Learning (DRL) is promising in solving Job Shop Scheduling Problems (JSPs), existing DRL-based methods still have large optimality gaps when learning job-to-machine solutions. This is mainly because: (i) existing state representations autonomously learned from graph-structured data cannot fully capture node information to support agents in making optimal decisions; and (ii) existing reward functions cannot accurately reflect some actions that will seriously worsen the current state. Aiming to address these issues, we propose a knowledge-based DRL method that selects nine well-known priority dispatching rules (PDRs) as state features, which can achieve effective model training. To avoid feature over-redundancy, we discard significantly correlated features based on the Pearson correlation relationship analysis, which can help to identify the key factors that affect the agents’ decision-making. Furthermore, since it is difficult to design a reward function that can accurately distinguish actions, we mask poor-performing actions based on problem-specific knowledge to prevent them from being selected at the current decision point. Comprehensive experimental results demonstrate the superiority of our approach over four PDRs and four state-of-the-art methods on various benchmarks. [ABSTRACT FROM AUTHOR]

Schedule maintenance activities for hydrogen fuel cell vehicles are aimed to keep them in better condition in long-term operation. In this paper, we develop a reinforcement learning based schedule maintenance strategy, which is used to comprehensively consider the balance between safety and maintenance cost, so as to find the optimal maintenance strategy. A multi-level framework is established with remaining useful life of key components, fault tree analysis, and logistics cost model to generate an exploration environment that can express the operational stability, the failure rate of components, and the cost of repair and storage. The trained agent combines a deep neural network to explore the optimal strategy under the dynamic reliability of key components. Finite steps are the constraints. Safety rates, maintenance cost, and episode of operation are incorporated into a multi-objective reward function. The hydrogen supply circuit of fuel cell vehicle is simulated via Python. The trained agent is compared with traditional time-based schedule maintenance strategy and corrective maintenance strategy. The results show that the reinforcement learning based schedule maintenance agent has optimized the total reward, cost control and accident rate by 77%, 59% and 5%, respectively, compared with the traditional time-based schedule maintenance strategy. In the safe operation management of fuel cell vehicles, efficient and stable decision-making ability has been achieved. • An adaptive maintenance strategy of deep reinforcement learning is proposed. • An exploration environment of FCV is established. • The deep Q network is proposed to solve the maintenance problem. • The efficient and stable decision-making ability is much better than time-based scheduled maintenance strategy. [ABSTRACT FROM AUTHOR]

Reinforcement learning is a compelling area of research within machine learning because it enables the improvement of control strategies for future manipulation of dynamic systems, leveraging previous data even without a precise model of the system. It usually makes complex, model-free predictions from data alone, which is actually consistent with the purpose of control in that they both aim to design systems using richly structured perceptions to execute planning and control strategies that adequately adapt to changing environments. The robust trajectory tracking control of intricate mechanical systems presents a challenging problem that necessitates effective control methods. In this paper, we propose a novel nonlinear control strategy based on deep reinforcement learning to solve the trajectory tracking problem of a 3-degree-of-freedom (3-DOF) control moment gyroscope (CMG). First, dynamic modeling of the 3-DOF CMG is used as a policy solver for the reinforcement learning training environment, and transfer learning is employed to bridge the reality gap. Then, the hyperparameters and reward functions of the neural network are optimized using the asynchronous successive halving algorithm. Ultimately, the twin delay depth determination policy gradient algorithm is trained in simulation to yield an agent capable of tracking user-defined trajectory routes as a nonlinear controller for the system. Both simulation and experimental results show that the proposed method works well for both high-frequency and low-frequency varying trajectory tracking control, and that the proposed method has better response speed and robustness than classic linear parameter-varying control methods and the state-of-the-art nonlinear parameter-varying method and the neural network-based feedback linearization adaptive control method. [ABSTRACT FROM AUTHOR]

Autonomous driving vehicle decision-making system has direct influence on driving performance. It is one of the key challenges to be addressed to realize fully autonomous driving. To solve this problem, a driving decision-making system based on deep reinforcement learning algorithm deep deterministic policy gradient(DDPG) was proposed. Firstly, a total of 64 dimensions of state spaces information such as ego vehicle information, road information and obstacle vehicle information on the basis of a driver model were selected as input variables of the constructed model. Then the decision-making was trained and outputs reasonable driving behaviors and control variable values. Finally, aiming at the problems of reward value and control variable values saltation, the DDPG decision model was improved to optimize decision control effect. To verify the performance of the proposed decision making model, simulation experiments were conducted on the open racing car simulator ( TORCS) platform. The results show that the proposed decision-making model can output reasonable driving behaviors and accurate control quantities based on real-time state information of vehicles and environment. Compared with the DDPG model, the improved decision-making model has better control accuracy, significantly reduces vehicle lateral speed, improves vehicle comfort and stability. [ABSTRACT FROM AUTHOR]

This paper studies the flexible double shop scheduling problem (FDSSP) that considers simultaneously job shop and assembly shop. It brings about the problem of scheduling association of the related tasks. To this end, a reinforcement learning algorithm with a deep temporal difference network is proposed to minimize the makespan. Firstly, the FDSSP is defined as the mathematical model of the flexible job-shop scheduling problem joined to the assembly constraint level. It is translated into a Markov decision process that directly selects behavioral strategies according to historical machining state data. Secondly, the proposed ten generic state features are input into the deep neural network model to fit the state value function. Similarly, eight simple constructive heuristics are used as candidate actions for scheduling decisions. From the greedy mechanism, optimally combined actions of all machines are obtained for each decision step. Finally, a deep temporal difference reinforcement learning framework is established, and a large number of comparative experiments are designed to analyze the basic performance of this algorithm. The results showed that the proposed algorithm was better than most other methods, which contributed to solving the practical production problem of the manufacturing industry. [ABSTRACT FROM AUTHOR]

Millimeter-wave (mmWave) massive multiple-input multiple-output non-orthogonal multiple access (MIMO-NOMA) is proven to be a primary technique for sixth-generation (6G) wireless communication networks. However, the great increase in users and antennas brings challenges for interference suppression and resource allocation for mmWave massive MIMO-NOMA systems. This study proposes a spectrum-efficient and fast convergence deep reinforcement learning (DRL)-based resource allocation framework to optimize user grouping and allocation of subchannel and power. First, an enhanced K-means grouping algorithm is proposed to reduce the multi-user interference and accelerate the convergence. Then, a dueling deep Q-network (DQN) structure is proposed to perform subchannel allocation, which further improves the convergence speed. Moreover, a deep deterministic policy gradient (DDPG)-based power resource allocation algorithm is designed to avoid the performance loss caused by power quantization and improve the system's achievable sum-rate. The simulation results demonstrate that our proposed scheme outperforms other neural network-based algorithms in terms of convergence performance, and can achieve higher system capacity compared with the greedy algorithm, the random algorithm, the RNN algorithm, and the DoubleDQN algorithm. [ABSTRACT FROM AUTHOR]

Adaptive cruise control (ACC) enables efficient, safe, and intelligent vehicle control by autonomously adjusting speed and ensuring a safe following distance from the vehicle in front. This paper proposes a novel adaptive cruise system, namely the Safety-First Reinforcement Learning Adaptive Cruise Control (SFRL-ACC). This system aims to leverage the model-free nature and high real-time inference efficiency of Deep Reinforcement Learning (DRL) to overcome the challenges of modeling difficulties and lower computational efficiency faced by current optimization control-based ACC methods while simultaneously maintaining safety advantages and optimizing ride comfort. Firstly, we transform the ACC problem into a safe DRL formulation Constrained Markov Decision Process (CMDP) by carefully designing state, action, reward, and cost functions. Subsequently, we propose the Projected Constrained Policy Optimization (PCPO)-based ACC Algorithm SFRL-ACC, which is specifically tailored to solve the CMDP problem. PCPO incorporates safety constraints that further restrict the trust region formed by the Kullback–Leibler (KL) divergence, facilitating DRL policy updates that maximize performance while keeping safety costs within their limit bounds. Finally, we train an SFRL-ACC policy and compare its computation time, traffic efficiency, ride comfort, and safety with state-of-the-art MPC-based ACC control methods. The experimental results prove the superiority of the proposed method in the aforementioned performance aspects. [ABSTRACT FROM AUTHOR]

The swift advancements in robotics have rendered navigation an essential task for mobile robots. While map-based navigation methods depend on global environmental maps for decision-making, their efficacy in unfamiliar or dynamic settings falls short. Current deep reinforcement learning navigation strategies can navigate successfully without pre-existing map data, yet they grapple with issues like inefficient training, slow convergence, and infrequent rewards. To tackle these challenges, this study introduces an improved two-delay depth deterministic policy gradient algorithm (LP-TD3) for local planning navigation. Initially, the integration of the long–short-term memory (LSTM) module with the Prioritized Experience Re-play (PER) mechanism into the existing TD3 framework was performed to optimize training and improve the efficiency of experience data utilization. Furthermore, the incorporation of an Intrinsic Curiosity Module (ICM) merges intrinsic with extrinsic rewards to tackle sparse reward problems and enhance exploratory behavior. Experimental evaluations using ROS and Gazebo simulators demonstrate that the proposed method outperforms the original on various performance metrics. [ABSTRACT FROM AUTHOR]

In this paper, we consider an integrated sensing, communication, and computation (ISCC) system to alleviate the spectrum congestion and computation burden problem. Specifically, while serving communication users, a base station (BS) actively engages in sensing targets and collaborates seamlessly with the edge server to concurrently process the acquired sensing data for efficient target recognition. A significant challenge in edge computing systems arises from the inherent uncertainty in computations, mainly stemming from the unpredictable complexity of tasks. With this consideration, we address the computation uncertainty by formulating a robust communication and computing resource allocation problem in ISCC systems. The primary goal of the system is to minimize total energy consumption while adhering to perception and delay constraints. This is achieved through the optimization of transmit beamforming, offloading ratio, and computing resource allocation, effectively managing the trade-offs between local execution and edge computing. To overcome this challenge, we employ a Markov decision process (MDP) in conjunction with the proximal policy optimization (PPO) algorithm, establishing an adaptive learning strategy. The proposed algorithm stands out for its rapid training speed, ensuring compliance with latency requirements for perception and computation in applications. Simulation results highlight its robustness and effectiveness within ISCC systems compared to baseline approaches. [ABSTRACT FROM AUTHOR]

With the emergence of wireless rechargeable sensor networks (WRSNs), the possibility of wirelessly recharging nodes using mobile charging vehicles (MCVs) has become a reality. However, existing approaches overlook the effective integration of node energy replenishment and mobile data collection processes. In this paper, we propose a joint energy replenishment and data collection scheme (D-JERDG) for WRSNs based on deep reinforcement learning. By capitalizing on the high mobility of unmanned aerial vehicles (UAVs), D-JERDG enables continuous visits to the cluster head nodes in each cluster, facilitating data collection and range-based charging. First, D-JERDG utilizes the K-means algorithm to partition the network into multiple clusters, and a cluster head selection algorithm is proposed based on an improved dynamic routing protocol, which elects cluster head nodes based on the remaining energy and geographical location of the cluster member nodes. Afterward, the simulated annealing (SA) algorithm determines the shortest flight path. Subsequently, the DRL model multiobjective deep deterministic policy gradient (MODDPG) is employed to control and optimize the UAV instantaneous heading and speed, effectively planning UAV hover points. By redesigning the reward function, joint optimization of multiple objectives such as node death rate, UAV throughput, and average flight energy consumption is achieved. Extensive simulation results show that the proposed D-JERDG achieves joint optimization of multiple objectives and exhibits significant advantages over the baseline in terms of throughput, time utilization, and charging cost, among other indicators. [ABSTRACT FROM AUTHOR]

Network disintegration is a fundamental issue in the field of complex networks, with its core in identifying critical nodes or sets and removing them to weaken network functionality. The research on this problem has significant strategic value and has increasingly attracted attention, including in controlling the spread of diseases and dismantling terrorist organizations. In this paper, we focus on the problem of network disintegration with discrete entity resources from the attack view, that is, optimizing resource allocation to maximize the effect of network disintegration. Specifically, we model the network disintegration problem with limited entity resources as a nonlinear optimization problem and prove its NP-hardness. Then, we design a method based on deep reinforcement learning (DRL), Net-Cracker, which transforms the two-stage entity resource and network node selection task into a single-stage object selection problem. Extensive experiments demonstrate that compared with the benchmark algorithm, Net-Cracker can improve the solution quality by about 8∼62%, while enabling a 30-to-160-fold speed up. Net-Cracker also exhibits strong generalization ability and can find better results in a near real-time manner even when the network scale is much larger than that in training data. [ABSTRACT FROM AUTHOR]

Stagnation at local optima represents a significant challenge in bio-inspired optimization algorithms, often leading to suboptimal solutions. This paper addresses this issue by proposing a hybrid model that combines the Orca predator algorithm with deep Q-learning. The Orca predator algorithm is an optimization technique that mimics the hunting behavior of orcas. It solves complex optimization problems by exploring and exploiting search spaces efficiently. Deep Q-learning is a reinforcement learning technique that combines Q-learning with deep neural networks. This integration aims to turn the stagnation problem into an opportunity for more focused and effective exploitation, enhancing the optimization technique's performance and accuracy. The proposed hybrid model leverages the biomimetic strengths of the Orca predator algorithm to identify promising regions nearby in the search space, complemented by the fine-tuning capabilities of deep Q-learning to navigate these areas precisely. The practical application of this approach is evaluated using the high-dimensional Heartbeat Categorization Dataset, focusing on the feature selection problem. This dataset, comprising complex electrocardiogram signals, provided a robust platform for testing the feature selection capabilities of our hybrid model. Our experimental results are encouraging, showcasing the hybrid strategy's capability to identify relevant features without significantly compromising the performance metrics of machine learning models. This analysis was performed by comparing the improved method of the Orca predator algorithm against its native version and a set of state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Network dismantling is one of the most challenging problems in complex systems. This problem encompasses a broad array of practical applications. Previous works mainly focus on the metrics such as the number of nodes in the Giant Connected Component (GCC), average pairwise connectivity, etc. This paper introduces a novel metric, the accumulated 2-core size, for assessing network dismantling. Due to the NP-hard computational complexity of this problem, we propose SmartCore, an end-to-end model for minimizing the accumulated 2-core size by leveraging reinforcement learning and graph neural networks. Extensive experiments across synthetic and real-world datasets demonstrate SmartCore's superiority over existing methods in terms of both accuracy and speed, suggesting that SmartCore should be a better choice for the network dismantling problem in practice. [ABSTRACT FROM AUTHOR]

This paper introduces a paradigm shift in atomic force microscope (AFM) scan control, leveraging an artificial intelligence (AI)-based controller. In contrast to conventional control methods, which either show a limited performance, such as proportional integral differential (PID) control, or which purely focus on mathematical optimality as classical optimal control approaches, our proposed AI approach redefines the objective of control for achieving practical optimality. This presented AI controller minimizes the root-mean-square control deviations in routine scans by a factor of about 4 compared to PID control in the presented setup and also showcases a distinctive asymmetric response in complex situations, prioritizing the safety of the AFM tip and sample instead of the lowest possible control deviations. The development and testing of the AI control concept are performed on simulated AFM scans, demonstrating its huge potential. Highlights: Artificial intelligence-based asymmetric controller for the highest tip and sample safety with AFM scans. Potential to significantly outperform proportional integral derivative control in classical scan situations. Dynamic change between classically optimal and asymmetric behavior depending on the scan situation. [ABSTRACT FROM AUTHOR]

In this work, we demonstrate how differentiable stochastic sampling techniques developed in the context of deep reinforcement learning can be used to perform efficient parameter inference over stochastic, simulation-based, forward models. As a particular example, we focus on the problem of estimating parameters of halo occupation distribution (HOD) models that are used to connect galaxies with their dark matter haloes. Using a combination of continuous relaxation and gradient re-parametrization techniques, we can obtain well-defined gradients with respect to HOD parameters through discrete galaxy catalogue realizations. Having access to these gradients allows us to leverage efficient sampling schemes, such as Hamiltonian Monte Carlo, and greatly speed up parameter inference. We demonstrate our technique on a mock galaxy catalogue generated from the Bolshoi simulation using a standard HOD model and find near-identical posteriors as standard Markov chain Monte Carlo techniques with an increase of ∼8× in convergence efficiency. Our differentiable HOD model also has broad applications in full forward model approaches to cosmic structure and cosmological analysis. [ABSTRACT FROM AUTHOR]

Throughout evolution, bacteria and other microorganisms have learned efficient foraging strategies that exploit characteristic properties of their unknown environment. While much research has been devoted to the exploration of statisticalmodels describing the dynamics of foraging bacteria and other (micro-) organisms, little is known, regarding the question of how good the learned strategies actually are. This knowledge gap is largely caused by the absence of methods allowing to systematically develop alternative foraging strategies to compare with. In the present work, we use deep reinforcement learning to show that a smart run-and-tumble agent, which strives to find nutrients for its survival, learns motion patterns that are remarkably similar to the trajectories of chemotactic bacteria. Strikingly, despite this similarity, we also find interesting differences between the learned tumble rate distribution and the one that is commonly assumed for the run and tumble model. We find that these differences equip the agent with significant advantages regarding its foraging and survival capabilities. Our results uncover a generic route to use deep reinforcement learning for discovering search and collection strategies that exploit characteristic but initially unknown features of the environment. These results can be used, e.g., to program future microswimmers, nanorobots, and smart active particles for tasks like searching for cancer cells, microwaste collection, or environmental remediation. [ABSTRACT FROM AUTHOR]

In the manufacturing industry, the design of machining processes plays a pivotal role in determining the quality, efficiency, and cost of product manufacturing. Machining process design is progressing towards intelligence to meet the high demands for efficiency and effectiveness in smart manufacturing. Building upon traditional computer-aided process planning, intelligent technologies, such as machine learning and knowledge graphs, have emerged as key drivers in advancing intelligent process design. To address knowledge accumulation, inflexible reuse, and fragmented reasoning in machining process design, this paper organically integrates the structured representation characteristics of the knowledge graph and the perceptual reasoning capabilities of deep reinforcement learning. It introduces an intelligent generation method for machining process design based on knowledge graph and deep reinforcement learning, aiming to achieve unified representation and reasoning reuse for historical process cases and general process rules. The effectiveness of the proposed method is validated through a case study involving the generation of machining process solutions for a specific model of diesel engine components. This research contributes valuable insights to overcoming the limitations of traditional methods and enhancing the efficiency of the machining process design. [ABSTRACT FROM AUTHOR]