Search

The connected and autonomous vehicles combined with deep reinforcement learning (DRL) are capable of handling complex driving scenarios. However, due to the random exploration feature of reinforcement learning (RL), unexpected actions and collisions that would be inevitable in the real world occur during training, resulting in property damage, injury, and loss of life. To address this issue, in this article, we propose a sophisticated safe DRL in autonomous lane changing that benefits from both exploration and optimization capabilities. The key idea is first to integrate the safety constraints into the RL algorithm to limit the actions that the agent can take during training, which is implemented by designing a vehicle convex occupancy approximation to estimate the candidate action set. Then, adaptive exploration strategies are used, in which an imaginary environment based on domain randomization is built to explore areas of the action–state space where it is uncertain about the outcomes. We present a Monte Carlo tree search to replace unsafe with safe action. The twin-delayed deep deterministic policy gradient is used as the RL algorithm to train action space agents. Experimental results show that our proposed framework significantly enhances safety during the lane-change process with faster and more stable learning than the other methods.

The intersection network constitutes the basic skeleton of the urban traffic environment, and informative representation of the intersection plays an important role in supporting the wide variety of applications in the intelligent transportation system. In this paper, we propose the Intersection to Vector model, named Intersec2vec, to achieve an accurate, efficient, and low-dimensional representation of each intersection in the large-scale intersection network. It introduces structural and temporal modules with attention mechanisms to specifically represent the evolution of intersection features in the spatiotemporal dimension, ensuring that intersections with stronger correlations have a higher probability of co-occurrence. Furthermore, our proposed Intersec2vec model is integrated into a traffic signal control method based on deep reinforcement learning by supporting more precise sub-area divisions, named Intersec2vec-TSC. For each sub-area, Intersec2vec-TSC adopts a hierarchical structure to design agents, where the upper agent determines the common cycle length based on the overall states of the sub-area, and lower agents jointly train a centralized evaluation network to achieve optimization of green time for each intersection. We conduct the experiment on 108 signalized intersections using real online car-hailing data, and the experimental results show that our proposed method significantly improves the stability of sub-area division and reduces the waiting time of cars during peak hours compared with other comparison methods.

Wi-Fi signals-based passive gesture tracking is making quick progress in promoting human–computer interaction. The state-of-the-arts have achieved promising cm-level tracking accuracies via the Fresnel ellipses, while they depend heavily on the transceiver positions and the gesture initial positions. This problem is known as the position-dependence problem, which hinders the actual deployment of sensing applications. In this article, we propose EasyTrack, which aims at position-independency passive gesture tracking based on Wi-Fi channel state information (CSI). EasyTrack develops a novel incremental motion tracking model to correlate the length variations and the angle information of the reflection paths with the gesture traces. This model eliminates the dependence on the prior knowledge of transceiver positions and the gesture initial positions, by shifting the sensing observation from the traditional transceiver view to the antenna array-oriented view. We propose a series of tracking refinement approaches, including trace segmentation, tracking smoothing, and motion detection to improve the tracking accuracy. We prototype EasyTrack using off-the-shelf Wi-Fi radios and extensive experiments attest EasyTrack can achieve tracking accuracy of 0.9 cm under various users and environment conditions.