Search

Path planning and obstacle avoidance are fundamental problems in unmanned ground vehicle path planning. Aiming at the limitations of Deep Reinforcement Learning (DRL) algorithms in unmanned ground vehicle path planning, such as low sampling rate, insufficient exploration, and unstable training, this paper proposes an improved algorithm called Dual Priority Experience and Ornstein–Uhlenbeck Soft Actor-Critic (DPEOU-SAC) based on Ornstein–Uhlenbeck (OU noise) and double-factor prioritized sampling experience replay (DPE) with the introduction of expert experience, which is used to help the agent achieve faster and better path planning and obstacle avoidance. Firstly, OU noise enhances the agent's action selection quality through temporal correlation, thereby improving the agent's detection performance in complex unknown environments. Meanwhile, the experience replay is based on double-factor preferential sampling, which has better sample continuity and sample utilization. Then, the introduced expert experience can help the agent to find the optimal path with faster training speed and avoid falling into a local optimum, thus achieving stable training. Finally, the proposed DPEOU-SAC algorithm is tested against other deep reinforcement learning algorithms in four different simulation environments. The experimental results show that the convergence speed of DPEOU-SAC is 88.99% higher than the traditional SAC algorithm, and the shortest path length of DPEOU-SAC is 27.24, which is shorter than that of SAC. [ABSTRACT FROM AUTHOR]

In order to enhance the trajectory tracking accuracy of distributed-driven intelligent vehicles, this paper formulates the tasks of torque output control for longitudinal dynamics and steering angle output control for lateral dynamics as Markov decision processes. To dissect the requirements of action output continuity for longitudinal and lateral control, this paper adopts the deep deterministic policy gradient algorithm (DDPG) for longitudinal velocity control and the deep Q-network algorithm (DQN) for lateral motion control. Multi-agent reinforcement learning methods are applied to the task of trajectory tracking in distributed-driven vehicle autonomous driving. By contrasting with two classical trajectory tracking control methods, the proposed approach in this paper is validated to exhibit superior trajectory tracking performance, ensuring that both longitudinal velocity deviation and lateral position deviation of the vehicle remain at lower levels. Compared with classical control methods, the maximum lateral position deviation is improved by up to 90.5% and the maximum longitudinal velocity deviation is improved by up to 97%. Furthermore, it demonstrates excellent generalization and high computational efficiency, and the running time can be reduced by up to 93.7%. [ABSTRACT FROM AUTHOR]

Reinforcement Learning (RL) methods are regarded as effective for designing autonomous driving policies. However, even when RL policies are trained to convergence, ensuring their robust safety remains a challenge, particularly in long-tail data. Therefore, decision-making based on RL must adequately consider potential variations in data distribution. This paper presents a framework for highway autonomous driving decisions that prioritizes both safety and robustness. Utilizing the proposed Replay Buffer Constrained Policy Optimization (RECPO) method, this framework updates RL strategies to maximize rewards while ensuring that the policies always remain within safety constraints. We incorporate importance sampling techniques to collect and store data in a Replay buffer during agent operation, allowing the reutilization of data from old policies for training new policy models, thus mitigating potential catastrophic forgetting. Additionally, we transform the highway autonomous driving decision problem into a Constrained Markov Decision Process (CMDP) and apply our proposed RECPO for training, optimizing highway driving policies. Finally, we deploy our method in the CARLA simulation environment and compare its performance in typical highway scenarios against traditional CPO, current advanced strategies based on Deep Deterministic Policy Gradient (DDPG), and IDM + MOBIL (Intelligent Driver Model and the model for minimizing overall braking induced by lane changes). The results show that our framework significantly enhances model convergence speed, safety, and decision-making stability, achieving a zero-collision rate in highway autonomous driving. [ABSTRACT FROM AUTHOR]

Adaptive cruise control (ACC) is the core building block of future full autonomous driving. Numerous recent research demonstrated that Autonomous Vehicles (AVs) adopting shorter headways generally increase road capacity and may relieve congestion at bottlenecks for moderate demand scenarios. However, with high demand scenarios, bottlenecks can still be activated causing capacity breakdown. Therefore, extra control measures as dynamic traffic control near bottlenecks is necessary. The challenge is harder on urban freeways with consecutive bottlenecks which affect each other. This paper aims to improve the performance of ACC systems in a high demand scenario. A multi‐bottleneck dynamic headway control strategy based on deep reinforcement learning (DRL) that adapts headways to optimize traffic flow and minimize delay is proposed. The controller dynamically assigns an optimal headway for each controlled section, based on state measurement representing the current traffic conditions. The case study is a freeway stretch with three consecutive bottlenecks which is then extended to include eight bottlenecks. Three different RL agent configurations are presented and compared. It is quantitatively demonstrated that the proposed control strategy improves traffic and enhances the system delay by up to 22.30%, and 18.87% compared to shortest headway setting for the three‐bottleneck and the eight‐bottleneck networks, respectively. [ABSTRACT FROM AUTHOR]

Autonomous vehicles navigating urban roads require technology that combines low latency with high computing power. The limited resources of the vehicle itself compel it to offload task requirements to edge server (ES) for processing assistance. However, as the number of vehicles continues to increase, how edge servers reasonably allocate limited resources to autonomous vehicles becomes critical to the success of urban intelligent transportation services. This paper establishes an urban road scenario with multiple autonomous vehicles and an edge computing server and considers two main driving behaviour transition resource requests, namely car‐following behaviour requests and lane‐changing behaviour requests. Simultaneously, acknowledging that vehicles may encounter unforeseen traffic hazards when switching driving behaviours, a safety redundancy setting strategy is employed to allocate additional resources to the vehicle to ensure safety and model the vehicle resource allocation problem in the autonomous driving system. Double‐deep Q‐network (DDQN) is then used to solve this model and maximize the total system utility by comprehensively considering resource costs, system revenue, and autonomous vehicle safety. Finally, results from the simulation experiment indicate that the proposed dynamic resource allocation scheme, based on deep reinforcement learning for autonomous vehicles under edge computing, not only greatly improves the system's benefits and reduces processing delays compared to traditional greedy algorithms and value iteration, but also effectively ensures security. [ABSTRACT FROM AUTHOR]

The autonomous vehicle is an innovative field for the application of machine learning algorithms. Controlling an agent designed to drive safely in traffic is very complex as human behavior is difficult to predict. An individual's actions depend on a large number of factors that cannot be acquired directly by visualization. The size of the vehicle, its vulnerability, its perception of the environment and weather conditions, among others, are all parameters that profoundly modify the actions that the optimized model should take. The agent must therefore have a great capacity for adaptation and anticipation in order to drive while ensuring the safety of users, especially pedestrians, who remain the most vulnerable users on the road. Deep reinforcement learning (DRL), a sub-field that is supported by the community for its real-time learning capability and the long-term temporal aspect of its objectives looks promising for AV control. In a previous article, we were able to show the strong capabilities of a DRL model with a continuous action space to manage the speed of a vehicle when approaching a pedestrian crossing. One of the points that remains to be addressed is the notion of discrete decision-making intrinsically linked to speed control. In this paper, we will present the problems of AV control during a pedestrian crossing, starting with a modelization and a DRL model with hybrid action space adapted to the scalability of a vehicle-to-pedestrian (V2P) encounter. We will also present the difficulties raised by the scalability and the curriculum-based method. [ABSTRACT FROM AUTHOR]

The application of autonomous driving system (ADS) technology can significantly reduce potential accidents involving vulnerable road users (VRUs) due to driver error. This paper proposes a novel hierarchical deep reinforcement learning (DRL) framework for high-performance collision avoidance, which enables the automated driving agent to perform collision avoidance maneuvers while maintaining appropriate speeds and acceptable social distancing. The novelty of the DRL method proposed here is its ability to accommodate dynamic obstacle avoidance, which is necessary as pedestrians are moving dynamically in their interactions with nearby ADSs. This is an improvement over existing DRL frameworks that have only been developed and demonstrated for stationary obstacle avoidance problems. The hybrid A* path searching algorithm is first applied to calculate a pre-defined path marked by waypoints, and a low-level path-following controller is used under cases where no VRUs are detected. Upon detection of any VRUs, however, a high-level DRL collision avoidance controller is activated to prompt the vehicle to either decelerate or change its trajectory to prevent potential collisions. The CARLA simulator is used to train the proposed DRL collision avoidance controller, and virtual raw sensor data are utilized to enhance the realism of the simulations. The model-in-the-loop (MIL) methodology is utilized to assess the efficacy of the proposed DRL ADS routine. In comparison to the traditional DRL end-to-end approach, which combines high-level decision making with low-level control, the proposed hierarchical DRL agents demonstrate superior performance. [ABSTRACT FROM AUTHOR]

Autonomous driving in urban areas is challenging because it requires understanding vehicle movements, traffic rules, map topologies and unknown environments in the highly complex driving environment, and thus typical urban traffic scenarios include various potentially hazardous situations. Therefore, training self-driving cars by using traditional deep learning models not only requires the labelling of numerous datasets but also takes a large amount of time. Because of this, it is important to find better alternatives for effectively training self-driving cars to handle vehicle behavior and complex road shapes in dynamic environments and to follow line guidance information. In this paper, we propose a method for training a self-driving car in simulated urban traffic scenarios to be able to judge the road conditions on its own for crossing an unsignalized intersection. In order to identify the behavior of traffic flow at the intersection, we use the CARLA (CAR Learning to Act) self-driving car simulator to build the intersection environment and simulate the process of traffic operation. Moreover, we attempt to use the DDPG (Deep Deterministic Policy Gradient) and RDPG (Recurrent Deterministic Policy Gradient) learning algorithms of the DRL (Deep Reinforcement Learning) technology to train models based on the CNN (Convolutional Neural Network) architecture. Specifically, the observation image of the semantic segmentation camera installed on the self-driving car and the vehicle speed are used as the model input. Moreover, we design an appropriate reward mechanism for performing training according to the current situation of the self-driving car judged from sensing data of the obstacle sensor, collision sensor and lane invasion detector. Doing so can improve the convergence speed of the model to achieve the purpose of the self-driving car autonomously judging the driving paths so as to accomplish accurate and stable autonomous driving control. [ABSTRACT FROM AUTHOR]

As autonomous driving technology continues to advance and gradually become a reality, ensuring the safety of autonomous driving in complex traffic scenarios has become a key focus and challenge in current research. Model-free deep reinforcement learning (Deep Reinforcement Learning) methods have been widely used for addressing motion planning problems in complex traffic scenarios, as they can implicitly learn interactions between vehicles. However, current planning methods based on deep reinforcement learning exhibit limited robustness and generalization performance. They struggle to adapt to traffic conditions beyond the training scenarios and face difficulties in handling uncertainties arising from unexpected situations. Therefore, this paper addresses the challenges presented by complex traffic scenarios, such as signal-free intersections. It does so by first utilizing the historical trajectories of adjacent vehicles observed in these scenarios. Through a Variational Auto-Encoder (VAE) based on the Gated Recurrent Unit (GRU) recurrent neural network, it extracts driver style features. These driver style features are then integrated with other state parameters and used to train a motion planning strategy within an extended reinforcement learning framework. This approach ultimately yields a more robust and interpretable mid-to-mid motion planning method. Experimental results confirm that the proposed method achieves low collision rates, high efficiency, and successful task completion in complex traffic scenarios. [ABSTRACT FROM AUTHOR]

In the research of robot systems, path planning and obstacle avoidance are important research directions, especially in unknown dynamic environments where flexibility and rapid decision makings are required. In this paper, a state attention network (SAN) was developed to extract features to represent the interaction between an intelligent robot and its obstacles. An auxiliary actor discriminator (AAD) was developed to calculate the probability of a collision. Goal-directed and gap-based navigation strategies were proposed to guide robotic exploration. The proposed policy was trained through simulated scenarios and updated by the Soft Actor-Critic (SAC) algorithm. The robot executed the action depending on the AAD output. Heuristic knowledge (HK) was developed to prevent blind exploration of the robot. Compared to other methods, adopting our approach in robot systems can help robots converge towards an optimal action strategy. Furthermore, it enables them to explore paths in unknown environments with fewer moving steps (showing a decrease of 33.9%) and achieve higher average rewards (showning an increase of 29.15%). [ABSTRACT FROM AUTHOR]

In this paper, the vehicle position points obtained by multi-sensor fusion are taken as the observed values, and Kalman filter is combined with the vehicle kinematics equation to further improve the vehicle trajectory. To realize this, mathematical principles of deep reinforcement learning are analyzed, and the theoretical basis of reinforcement learning is also analyzed. It is proved that the controller based on dynamic model is better than the controller based on kinematics in deviation control, and the performance of the controller based on deep reinforcement learning is also verified. The simulation data show that the proportion integration differentiation (PID) controller has a better tracking effect, but it does not have the constraint ability, which leads to radical acceleration change, resulting in unstable acceleration and deceleration control. Therefore, the deep reinforcement learning controller is selected as the longitudinal velocity tracking controller. The effectiveness of lateral and longitudinal motion decoupling strategy is verified by simulation experiments. [ABSTRACT FROM AUTHOR]

To overcome the challenges of obstacle avoidance for Unmanned Aerial Vehicles (UAVs) in autonomous flights, this paper proposes the Dual Experience Attention Convolution Soft Actor-Critic (DAC-SAC) algorithm. This algorithm integrates a dual experience buffer pool, a self-attention mechanism, and the Soft-Actor-Critic algorithm with a convolutional network. The dual experience buffer pools are used to solve the problem of ineffective UAV training due to the scarcity of successful training data. To overcome the drawbacks of the original Soft Actor-Critic (SAC) algorithm in handling image data, a Convolutional Neural Network (CNN) is applied to reconstruct the actor and critic network, allowing for better image feature extraction and classification. Furthermore, a self-attention mechanism is employed by adding a convolutional self-attention layer to the network. This modification enables dynamic adjustments for the attention weights based on varying input image features, effectively addressing focus-related challenges. Two simulation experiments are performed and the DAC-SAC algorithm achieves a 99.5% success rate in a known environment and an 84.8% success rate when dealing with an unknown environment. These results confirm that the proposed algorithm enables autonomous obstacle avoidance for UAVs even when considering depth images as input. [ABSTRACT FROM AUTHOR]

In order to improve the autonomous navigation capability of autonomous underwater vehicles (AUV), this paper applies Deep Reinforcement Learning (DRL) to realize autonomous path planning. In response to the limited detection range and difficult tracking DRL policy in three-dimensional unknown environments, this paper proposes an improved deep reinforcement learning method. Firstly, performing AUV modeling and processing obstacle detection information. Second, under a framework of reinforcement learning with the control system, an improved environmental interaction method was designed by analyzing the motion characteristics and controller performance of the AUV. Finally, based on this method, a policy network that can generate paths that satisfy obstacle avoidance and navigation is trained in combination with a twin delayed deterministic policy gradient (TD3). The simulation results show that the proposed method can achieve AUV path planning in a three-dimensional unknown environment. The proposed framework can ensure that the planned path meets the dynamics constraints and is trackable. The improved algorithm can enhance the convergence of training and the attitude stability of AUV. • This article applies forward-looking sonar to achieve three-dimensional path planning in unknown environments based on deep reinforcement learning, relying on the limited obstacle detection range of AUV itself. • The proposed path planning algorithm considers the characteristics of the model and controller in the process of deep reinforcement learning. • In other algorithm details, the reciprocal obstacle distance, and dense reward function are designed. These methods can significantly improve the efficiency of training. [ABSTRACT FROM AUTHOR]

System identification (SI) is a research focus in the field of autonomous underwater vehicles (AUV) which can be considered as the estimation to the hydrodynamic coefficients (HCs) in nonlinear dynamic equations of the AUV. This paper presents an SI method based on deep reinforcement learning (DRL) to estimate the HCs of AUVs only through the utilization of position and altitude information of the AUV. Compared with most existing methods, fewer design parameters are required in the proposed SI method, thereby making the SI method simpler to perform. First, a sensitivity analysis of the HCs is performed, and then the action and reward mechanism in the origin deep deterministic policy gradient (DDPG) algorithm is improved according to the results of sensitivity analysis. The simulation results show that the HCs can be estimated with a low error level and greater accuracy than the other methods. Therefore, it can be concluded that the DRL SI method possesses effective performance in HC estimation and the improved-DDPG algorithm is more suitable for SI than the original one. • The AUV velocity information is unnecessary so that the method is simpler to perform. • In Improved-DDPG algorithm, system identification accuracy is improved. • The training of neural networks is set to be segmented. • The training of NNs is terminated upon the first attainment of the reward standard. [ABSTRACT FROM AUTHOR]

Interaction between the background vehicles (BVs) and automated vehicles (AVs) in scenario-based testing plays a critical role in evaluating the intelligence of the AVs. Current testing scenarios typically employ predefined or scripted BVs, which inadequately reflect the complexity of human-like social behaviors in real-world driving scenarios, and also lack a systematic metric for evaluating the comprehensive intelligence of AVs. Therefore, this paper proposes an evolving scenario generation method, employing deep reinforcement learning (DRL) to construct human-like BVs that interact with AVs, and this evolving scenario is designed to test and evaluate the intelligence of AVs. Firstly, a class of BV driver models with human-like competitive, mutual, and cooperative driving motivations is designed. Then, utilizing the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and an improved level-k training procedure, the three distinct driver models acquire game-based interactive driving policies. And these driver models are combined to generate evolving scenarios in which they can interact continuously and evolve diverse contents. Next, a framework including safety, driving efficiency, and interaction utility are presented to evaluate and quantify the intelligence performance of 3 systems under test (SUTs), indicating the effectiveness of the evolving scenario for intelligence testing. Finally, the complexity and fidelity of the proposed evolving testing scenario are validated. The results demonstrate that the proposed evolving scenario exhibits the highest level of complexity compared to other baseline scenarios and has more than 85% similarity to naturalistic driving data. This highlights the potential of the proposed method to facilitate the development and evaluation of high-level AVs in a realistic and challenging environment. • A method for constructing a cluster of BV driver models based on DRL is proposed. These human-like social driver models are endowed with one of the following game-like interaction policies: cooperative, mutual, or competitive. • Utilizing the BV driver models, an evolving scenario of high complexity and fidelity is generated. • A quantitative evaluation framework for the intelligence of automated driving systems is proposed. [ABSTRACT FROM AUTHOR]

This study proposes an autonomous aircraft taxi agent that can be used to recommend the pilot the optimal speed profile to achieve optimal fuel burn and to arrive on time at the target position on the taxiway while considering potential interactions with surrounding traffic. The problem is modeled as a control decision problem that is solved by training the agent under a Deep Reinforcement Learning (DRL) mechanism using the Proximal Policy Optimization (PPO) algorithm. The reward function is designed to consider the fuel burn, taxi time, and delay time. Accordingly, the trained agent will learn to taxi the aircraft between any pair of locations on the airport surface in a timely manner while maintaining safety and efficiency. As a result, in more than 97.8% of the evaluated sessions, the controlled aircraft reached the target position with a time difference falling within the range of -20 to 5 s. Moreover, compared to actual fuel burn, the proposed autonomous taxi agent demonstrated a reduction of 29.5%, equivalent to reducing 13.9 kg of fuel consumption per aircraft. This benefit in fuel burn reduction can complement the emission reductions achieved by solving other sub-problems, such as pushback control and taxi-route assignments, to achieve much higher performance. [ABSTRACT FROM AUTHOR]

Limitations are identified in the expressive capabilities of the deep feature extraction network employed in deep reinforcement learning (DRL), particularly in complex scenarios. Additionally, learning performance is negatively impacted by compound errors, with consideration given to the potential critical role of sample quality in learning outcomes. Therefore, how to improve the collision avoidance decision-making adaptive and effectiveness for autonomous ships navigating in various scenarios is crucial. To address these issues and overcome data acquisition difficulties, a method called rule-guided vision supervised learning (RGVSL) is proposed in this paper. Through a static collision avoidance decision-making task, a comparison is drawn between a deep feature extraction network and the Nature CNN in DQN, revealing shortcomings in the feature extraction of DRL. Additionally, a collision avoidance decision-making environment with the capability of generalization for maritime encounters is proposed, and the advantages of the method in terms of adaptability and learning cost are validated. Finally, it is demonstrated that the RGVSL method is equivalent to a DQN with γ set to 0, indicating a significant performance improvement without compound errors. Achieving an adaptive decision accuracy of over 90% in various encounter scenarios without retraining, this research substantially reduces learning costs. It can provide innovative and practical solutions for the technological development in the field of autonomous ship collision avoidance decision-making. • Revolutionizes DRL data by using explicit rules for diverse scenarios, boosting model performance. • Exposes Nature CNN's DQN inadequacy, emphasizing the need for effective feature expression in DRL. • RGVSL achieves 90% accuracy in diverse maritime scenarios, cutting retraining costs, enhancing performance. [ABSTRACT FROM AUTHOR]

Taking the velocity regulation of autonomous driving as an example, this paper developed an enhancing active disturbance rejection design employing the deep reinforcement learning-deep deterministic policy gradient (DDPG). In this scheme, the active disturbance rejection control (ADRC) is adopted to online estimate and compensate disturbances and uncertainties, and feasible regions of control parameters are obtained through the Lyapunov method. Then DDPG is combined with ADRC to online adaptively tune control parameters in response to the changing environments, where safety, comfort, and energy-saving are considered in the reward design, and mapping relation between the defined action and state is constructed for the maximal reward. Besides, numerical simulations demonstrate the better performance and stronger robustness of the enhancing design when facing uncertainties, sensor noise, and mechanical faults, and comfort and energy consumption have also been improved to some extent compared with the general ADRC and model predictive control (MPC). • Combining LADRC and DDPG, an enhancing design is proposed for autonomous vehicle. • The adaptive regulation scheme shows strong capability to changing environments. • It can achieve better performance considering safety, comfort and energy-saving. [ABSTRACT FROM AUTHOR]

The application of Deep Reinforcement Learning (DRL) has significantly impacted the development of autonomous driving technology in the field of intelligent transportation. However, in mixed traffic scenarios involving both human-driven vehicles (HDVs) and connected and autonomous vehicles (CAVs), challenges arise, particularly concerning information sharing and collaborative control among multiple intelligent agents using DRL. To address this issue, we propose a novel framework, namely Spatial-Temporal Deep Reinforcement Learning (ST-DRL), that enables collaborative control among multiple CAVs in mixed traffic scenarios. Initially, the traffic states involving multiple agents are constructed as graph-formatted data, which is then sequential created to represent continuous time intervals. With the data representation, interactive behaviors and dynamic characteristics among multiple intelligent agents are implicitly captured. Subsequently, to better represent the spatial relationships between vehicles, a graph enabling network is utilize to encode the vehicle states, which can contribute to the improvement of information sharing efficiency among multiple intelligent agents. Additionally, a spatial-temporal feature fusion network module is designed, which integrates graph convolutional networks (GCN) and gated recurrent units (GRU). It can effectively fuse independent spatial-temporal features and further enhance collaborative control performance. Through extensive experiments conducted in the SUMO traffic simulator and comparison with baseline methods, it is demonstrated that the ST-DRL framework achieves higher success rates in mixed traffic scenarios and exhibits better trade-offs between safety and efficiency. The analysis of the results indicates that ST-DRL has increased the success rate of the task by 15.6 % compared to the baseline method, while reducing model training and task completion times by 26.6 % respectively. [ABSTRACT FROM AUTHOR]

In real-world scenarios, making navigation decisions for autonomous driving involves a sequential set of steps. These judgments are made based on partial observations of the environment, while the underlying model of the environment remains unknown. A prevalent method for resolving such issues is reinforcement learning, in which the agent acquires knowledge through a succession of rewards in addition to fragmentary and noisy observations. This study introduces an algorithm named deep reinforcement learning navigation via decision transformer (DRLNDT) to address the challenge of enhancing the decision-making capabilities of autonomous vehicles operating in partially observable urban environments. The DRLNDT framework is built around the Soft Actor-Critic (SAC) algorithm. DRLNDT utilizes Transformer neural networks to effectively model the temporal dependencies in observations and actions. This approach aids in mitigating judgment errors that may arise due to sensor noise or occlusion within a given state. The process of extracting latent vectors from high-quality images involves the utilization of a variational autoencoder (VAE). This technique effectively reduces the dimensionality of the state space, resulting in enhanced training efficiency. The multimodal state space consists of vector states, including velocity and position, which the vehicle's intrinsic sensors can readily obtain. Additionally, latent vectors derived from high-quality images are incorporated to facilitate the Agent's assessment of the present trajectory. Experiments demonstrate that DRLNDT may achieve a superior optimal policy without prior knowledge of the environment, detailed maps, or routing assistance, surpassing the baseline technique and other policy methods that lack historical data. [ABSTRACT FROM AUTHOR]

Most of the current studies on autonomous vehicle decision-making and control based on reinforcement learning are conducted in simulated environments. The training and testing of these studies are carried out under the condition of rule-based microscopic traffic flow, with little consideration regarding migrating them to real or near-real environments. This may lead to performance degradation when the trained model is tested in more realistic traffic scenes. In this study, we propose a method to randomize the driving behavior of surrounding vehicles by randomizing certain parameters of the car-following and lane-changing models of rule-based microscopic traffic flow. We trained policies with deep reinforcement learning algorithms under the domain-randomized rule-based microscopic traffic flow in freeway and merging scenes and then tested them separately in rule-based and high-fidelity microscopic traffic flows. The results indicate that the policies trained under domain-randomized traffic flow have significantly better success rates and episodic rewards compared to those trained under non-randomized traffic flow. [ABSTRACT FROM AUTHOR]

This study focuses on a crucial task in the field of autonomous driving, autonomous lane change. Autonomous lane change plays a pivotal role in improving traffic flow, alleviating driver burden, and reducing the risk of traffic accidents. However, due to the complexity and uncertainty of lane-change scenarios, the functionality of autonomous lane change still faces challenges. In this research, we conducted autonomous lane-change simulations using both deep reinforcement learning (DRL) and model predictive control (MPC). Specifically, we used the parameterized soft actor–critic (PASAC) algorithm to train a DRL-based lane-change strategy to output both discrete lane-change decisions and continuous longitudinal vehicle acceleration. We also used MPC for lane selection based on the smallest predictive car-following costs for the different lanes. For the first time, we compared the performance of DRL and MPC in the context of lane-change decisions. The simulation results indicated that, under the same reward/cost function and traffic flow, both MPC and PASAC achieved a collision rate of 0%. PASAC demonstrated a comparable performance to MPC in terms of average rewards/costs and vehicle speeds. [ABSTRACT FROM AUTHOR]

Intelligent decisions for autonomous lane-changing in vehicles have consistently been a focal point of research in the industry. Traditional lane-changing algorithms, which rely on predefined rules, are ill-suited for the complexities and variabilities of real-world road conditions. In this study, we propose an algorithm that leverages the deep deterministic policy gradient (DDPG) reinforcement learning, integrated with a long short-term memory (LSTM) trajectory prediction model, termed as LSTM-DDPG. In the proposed LSTM-DDPG model, the LSTM state module transforms the observed values from the observation module into a state representation, which then serves as a direct input to the DDPG actor network. Meanwhile, the LSTM prediction module translates the historical trajectory coordinates of nearby vehicles into a word-embedding vector via a fully connected layer, thus providing predicted trajectory information for surrounding vehicles. This integrated LSTM approach considers the potential influence of nearby vehicles on the lane-changing decisions of the subject vehicle. Furthermore, our study emphasizes the safety, efficiency, and comfort of the lane-changing process. Accordingly, we designed a reward and penalty function for the LSTM-DDPG algorithm and determined the optimal network structure parameters. The algorithm was then tested on a simulation platform built with MATLAB/Simulink. Our findings indicate that the LSTM-DDPG model offers a more realistic representation of traffic scenarios involving vehicle interactions. When compared to the traditional DDPG algorithm, the LSTM-DDPG achieved a 7.4% increase in average single-step rewards after normalization, underscoring its superior performance in enhancing lane-changing safety and efficiency. This research provides new ideas for advanced lane-changing decisions in autonomous vehicles. [ABSTRACT FROM AUTHOR]

With recent technological advancements, the commercialization of autonomous vehicles (AVs) is expected to be realized soon. However, it is anticipated that a mixed traffic of AVs and human-driven vehicles (HVs) will persist for a considerable period until the Market Penetration Rate reaches 100%. During this phase, AVs and HVs will interact and coexist on the roads. Such an environment can cause unpredictable and dynamic traffic conditions due to HVs, which results in traffic problems including traffic congestion. Therefore, the routes of AVs must be controlled in a mixed traffic environment. This study proposes a multi-objective vehicle routing control method using a deep Q-network to control the driving direction at intersections in a mixed traffic environment. The objective is to distribute the traffic flow and control the routes safely and efficiently to their destination. Simulation results showed that the proposed method outperformed existing methods in terms of the driving distance, time, and waiting time of AVs, particularly in more dynamic traffic environments. Consequently, the traffic became smooth as it moved along optimal routes. [ABSTRACT FROM AUTHOR]

With the advent of IoT, cities will soon be populated by autonomous vehicles and managed by intelligent systems capable of actively interacting with city infrastructures and vehicles. In this work, we propose a model based on reinforcement learning that teaches to autonomous connected vehicles how to save resources while navigating in such an environment. In particular, we focus on budget savings in the context of auction-based intersection management systems. We trained several models with Deep Q-learning by varying traffic conditions to find the most performance-effective variant in terms of the trade-off between saved currency and trip times. Afterward, we compared the performance of our model with previously proposed and random strategies, even under adverse traffic conditions. Our model appears to be robust and manages to save a considerable amount of currency without significantly increasing the waiting time in traffic. For example, the learner bidder saves at least 20% of its budget with heavy traffic conditions and up to 74% in lighter traffic with respect to a standard bidder, and around three times the saving of a random bidder. The results and discussion suggest practical adoption of the proposal in a foreseen future real-life scenario. [ABSTRACT FROM AUTHOR]

The development of autonomous driving models through reinforcement learning has gained significant traction. However, developing obstacle avoidance systems remains a challenge. Specifically, optimising path completion times while navigating obstacles is an underexplored research area. Amazon Web Services (AWS) DeepRacer emerges as a powerful infrastructure for engineering and analysing autonomous models, providing a robust foundation for addressing these complexities. This research investigates the feasibility of training end-to-end self-driving models focused on obstacle avoidance using reinforcement learning on the AWS DeepRacer autonomous race car platform. A comprehensive literature review of autonomous driving methodologies and machine learning model architectures is conducted, with a particular focus on object avoidance, followed by hands-on experimentation and the analysis of training data. Furthermore, the impact of sensor choice, reward function, action spaces, and training time on the autonomous obstacle avoidance task are compared. The results of the best configuration experiment demonstrate a significant improvement in obstacle avoidance performance compared to the baseline configuration, with a 95.8% decrease in collision rate, while taking about 79% less time to complete the trial circuit. [ABSTRACT FROM AUTHOR]

Autonomous ships is a key to avoid accidents caused by human errors and improve maritime safety. However, unlike the autonomous vehicles counterpart, collision avoidance for autonomous ships faces many challenges due to the hash driving environments, difficult ship control and large stopping distance. In this paper, we investigate a collision avoidance system for autonomous ships under complex encounter scenarios, such as busy ports. In the system various sensors are used to detect objects and perceive the maritime environments. To help the autonomous ships handle the complex and dynamic scenarios that may be encountered, a collision map used to describe the ships encounter scenarios is generated and utilized as the input of a deep reinforcement learning (DRL) model. The DRL model is applied to make collision avoidance and safe driving decisions. New reward functions are proposed to train the DRL model to generate safe ship maneuver actions to reduce collisions and ensure compliance with the Convention on the International Regulations for Preventing Collisions at Sea (COLREGs). Furthermore, a self-adaptive parameters sharing approach is designed for fast convergence and collision avoidance performance of the DRL model, where the parameters of the fully connected layers are shared and the correlation layers are self adapted for the DRL critic and actor networks. Simulation results show that the proposed system has high DRL convergence speed and excellent collision avoidance. • A novel sharing mechanism with hybrid structure for policy network and value network in DRL model is proposed to accelerate training speed. • According to the characteristics of ship navigation, the concept of collision map with different layers is proposed. • Both the dynamic futures and regional feature of ship navigation are considered in collision map to have comprehensive understanding of ship navigation for DRL model. • Several reward functions considering risk, COLREGs, navigation characteristics are designed to ensure safety of ship navigation. [ABSTRACT FROM AUTHOR]

In this work we present a multiagent deep reinforcement learning approach for autonomous driving vehicles that is able to operate in traffic networks with unsignalized intersections. The key aspects of the proposed study are the introduction of route-agents as the main building block of the system, as well as a collision term that allows the cooperation among vehicles and the construction of an efficient reward function. These have the advantage of establishing an enhanced collaborative multiagent deep reinforcement learning scheme that manages to control multiple vehicles and navigate them safely and efficiently-economically to their destination. In addition, it provides the beneficial flexibility to lay down a platform for transfer learning and reusing knowledge from the agents' policies in handling unknown traffic scenarios. We provide several experimental results in simulated road traffic networks of variable complexity and diverse characteristics using the SUMO environment that empirically illustrate the efficiency of the proposed multiagent framework. [ABSTRACT FROM AUTHOR]