Search

Handling long tail corner cases is a major challenge faced by autonomous vehicles (AVs). While large language models (LLMs) hold great potentials to handle the corner cases with excellent generalization and explanation capabilities and received increasing research interest on application to autonomous driving, there are still technical barriers to be tackled, such as strict model performance and huge computing resource requirements of LLMs. In this paper, we investigate a new approach of applying remote or edge LLMs to support autonomous driving. A key issue for such LLM assisted driving system is the assessment of LLMs on their understanding of driving theory and skills, ensuring they are qualified to undertake safety critical driving assistance tasks for CAVs. We design and run driving theory tests for several proprietary LLM models (OpenAI GPT models, Baidu Ernie and Ali QWen) and open-source LLM models (Tsinghua MiniCPM-2B and MiniCPM-Llama3-V2.5) with more than 500 multiple-choices theory test questions. Model accuracy, cost and processing latency are measured from the experiments. Experiment results show that while model GPT-4 passes the test with improved domain knowledge and Ernie has an accuracy of 85% (just below the 86% passing threshold), other LLM models including GPT-3.5 fail the test. For the test questions with images, the multimodal model GPT4-o has an excellent accuracy result of 96%, and the MiniCPM-Llama3-V2.5 achieves an accuracy of 76%. While GPT-4 holds stronger potential for CAV driving assistance applications, the cost of using model GPT4 is much higher, almost 50 times of that of using GPT3.5. The results can help make decision on the use of the existing LLMs for CAV applications and balancing on the model performance and cost.

Offloading computing to edge servers is a promising solution to support growing video understanding applications at resource-constrained IoT devices. Recent efforts have been made to enhance the scalability of such systems by reducing inference costs on edge servers. However, existing research is not directly applicable to pixel-level vision tasks such as video semantic segmentation (VSS), partly due to the fluctuating VSS accuracy and segment bitrate caused by the dynamic video content. In response, we present Penance, a new edge inference cost reduction framework. By exploiting softmax outputs of VSS models and the prediction mechanism of H.264/AVC codecs, Penance optimizes model selection and compression settings to minimize the inference cost while meeting the required accuracy within the available bandwidth constraints. We implement Penance in a commercial IoT device with only CPUs. Experimental results show that Penance consumes a negligible 6.8% more computation resources than the optimal strategy while satisfying accuracy and bandwidth constraints with a low failure rate., Comment: Accepted by ACM Multimedia 2023

Multi-vehicle pursuit (MVP) such as autonomous police vehicles pursuing suspects is important but very challenging due to its mission and safety critical nature. While multi-agent reinforcement learning (MARL) algorithms have been proposed for MVP problem in structured grid-pattern roads, the existing algorithms use randomly training samples in centralized learning, which leads to homogeneous agents showing low collaboration performance. For the more challenging problem of pursuing multiple evading vehicles, these algorithms typically select a fixed target evading vehicle for pursuing vehicles without considering dynamic traffic situation, which significantly reduces pursuing success rate. To address the above problems, this paper proposes a Progression Cognition Reinforcement Learning with Prioritized Experience for MVP (PEPCRL-MVP) in urban multi-intersection dynamic traffic scenes. PEPCRL-MVP uses a prioritization network to assess the transitions in the global experience replay buffer according to the parameters of each MARL agent. With the personalized and prioritized experience set selected via the prioritization network, diversity is introduced to the learning process of MARL, which can improve collaboration and task related performance. Furthermore, PEPCRL-MVP employs an attention module to extract critical features from complex urban traffic environments. These features are used to develop progression cognition method to adaptively group pursuing vehicles. Each group efficiently target one evading vehicle in dynamic driving environments. Extensive experiments conducted with a simulator over unstructured roads of an urban area show that PEPCRL-MVP is superior to other state-of-the-art methods. Specifically, PEPCRL-MVP improves pursuing efficiency by 3.95% over TD3-DMAP and its success rate is 34.78% higher than that of MADDPG. Codes are open sourced.

Multi-robot system for manufacturing is an Industry Internet of Things (IIoT) paradigm with significant operational cost savings and productivity improvement, where Unmanned Aerial Vehicles (UAVs) are employed to control and implement collaborative productions without human intervention. This mission-critical system relies on 3-Dimension (3-D) scene recognition to improve operation accuracy in the production line and autonomous piloting. However, implementing 3-D point cloud learning, such as Pointnet, is challenging due to limited sensing and computing resources equipped with UAVs. Therefore, we propose a Digital Twin (DT) empowered Knowledge Distillation (KD) method to generate several lightweight learning models and select the optimal model to deploy on UAVs. With a digital replica of the UAVs preserved at the edge server, the DT system controls the model sharing network topology and learning model structure to improve recognition accuracy further. Moreover, we employ network calculus to formulate and solve the model sharing configuration problem toward minimal resource consumption, as well as convergence. Simulation experiments are conducted over a popular point cloud dataset to evaluate the proposed scheme. Experiment results show that the proposed model sharing scheme outperforms the individual model in terms of computing resource consumption and recognition accuracy., Comment: 16 pages, 12 figures, journal

Despite the prevalence of GPS services, they still suffer from intermittent positioning with poor accuracy in partially shadowed regions like urban canyons, flyover shadows, and factories' indoor areas. Existing wisdom relies on hardware modifications of GPS receivers or power-hungry infrastructures requiring continuous plug-in power supply which is hard to provide in outdoor regions and some factories. This paper fills the gap with GPSMirror, the first GPS-strengthening system that works for unmodified smartphones with the assistance of newly-designed GPS backscatter tags. The key enabling techniques in GPSMirror include: (i) a careful hardware design with microwatt-level power consumption that pushes the limit of backscatter sensitivity to re-radiate extremely weak GPS signals with enough coverage approaching the regulation limit; and (ii) a novel GPS positioning algorithm achieving meter-level accuracy in shadowed regions as well as expanding locatable regions under inadequate satellites where conventional algorithms fail. We build a prototype of the GPSMirror tags and conduct comprehensive experiments to evaluate them. Our results show that a GPSMirror tag can provide coverage up to 27.7 m. GPSMirror achieves median positioning accuracy of 3.7 m indoors and 4.6 m in urban canyon environments, respectively., Comment: 13 pages, 26 figures, to appear in MobiCom 2023

Deep learning has shown impressive performance in semantic segmentation, but it is still unaffordable for resource-constrained mobile devices. While offloading computation tasks is promising, the high traffic demands overwhelm the limited bandwidth. Existing compression algorithms are not fit for semantic segmentation, as the lack of obvious and concentrated regions of interest (RoIs) forces the adoption of uniform compression strategies, leading to low compression ratios or accuracy. This paper introduces STAC, a DNN-driven compression scheme tailored for edge-assisted semantic video segmentation. STAC is the first to exploit DNN's gradients as spatial sensitivity metrics for spatial adaptive compression and achieves superior compression ratio and accuracy. Yet, it is challenging to adapt this content-customized compression to videos. Practical issues include varying spatial sensitivity and huge bandwidth consumption for compression strategy feedback and offloading. We tackle these issues through a spatiotemporal adaptive scheme, which (1) takes partial strategy generation operations offline to reduce communication load, and (2) propagates compression strategies and segmentation results across frames through dense optical flow, and adaptively offloads keyframes to accommodate video content. We implement STAC on a commodity mobile device. Experiments show that STAC can save up to 20.95% of bandwidth without losing accuracy, compared to the state-of-the-art algorithm., Comment: 10 pages, INFOCOM 2022

The emergence of Intelligent Connected Vehicles (ICVs) shows great potential for future intelligent traffic systems, enhancing both traffic safety and road efficiency. However, the ICVs relying on data driven perception and driving models face many challenges, including the lack of comprehensive knowledge to deal with complicated driving context. In this paper, we are motivated to investigate cooperative knowledge sharing for ICVs. We propose a secure and efficient directed acyclic graph (DAG) blockchain based knowledge sharing framework, aiming to cater for the micro-transaction based vehicular networks. The framework can realize both local and cross-regional knowledge sharing. Then, the framework is applied to autonomous driving applications, wherein machine learning based models for autonomous driving control can be shared. A lightweight tip selection algorithm (TSA) is proposed for the DAG based knowledge sharing framework to achieve consensus and identity verification for cross-regional vehicles. To enhance model accuracy as well as minimizing bandwidth consumption, an adaptive asynchronous distributed learning (ADL) based scheme is proposed for model uploading and downloading. Experiment results show that the blockchain based knowledge sharing is secure, and it can resist attacks from malicious users. In addition, the proposed adaptive ADL scheme can enhance driving safety related performance compared to several existing algorithms., Comment: 12 pages, 13 figures

Scanned receipts OCR and key information extraction (SROIE) represent the processeses of recognizing text from scanned receipts and extracting key texts from them and save the extracted tests to structured documents. SROIE plays critical roles for many document analysis applications and holds great commercial potentials, but very little research works and advances have been published in this area. In recognition of the technical challenges, importance and huge commercial potentials of SROIE, we organized the ICDAR 2019 competition on SROIE. In this competition, we set up three tasks, namely, Scanned Receipt Text Localisation (Task 1), Scanned Receipt OCR (Task 2) and Key Information Extraction from Scanned Receipts (Task 3). A new dataset with 1000 whole scanned receipt images and annotations is created for the competition. In this report we will presents the motivation, competition datasets, task definition, evaluation protocol, submission statistics, performance of submitted methods and results analysis.

Research on the sixth generation cellular networks (6G) is gaining huge momentum to achieve ubiquitous wireless connectivity. Connected autonomous driving (CAV) is a critical vertical envisioned for 6G, holding great potentials of improving road safety, road and energy efficiency. However the stringent service requirements of CAV applications on reliability, latency and high speed communications will present big challenges to 6G networks. New channel access algorithms and intelligent control schemes for connected vehicles are needed for 6G supported CAV. In this paper, we investigated 6G supported cooperative driving, which is an advanced driving mode through information sharing and driving coordination. Firstly we quantify the delay upper bounds of 6G vehicle to vehicle (V2V) communications with hybrid communication and channel access technologies. A deep learning neural network is developed and trained for fast computation of the delay bounds in real time operations. Then, an intelligent strategy is designed to control the inter-vehicle distance for cooperative autonomous driving. Furthermore, we propose a Markov Chain based algorithm to predict the parameters of the system states, and also a safe distance mapping method to enable smooth vehicular speed changes. The proposed algorithms are implemented in the AirSim autonomous driving platform. Simulation results show that the proposed algorithms are effective and robust with safe and stable cooperative autonomous driving, which greatly improve the road safety, capacity and efficiency.

We are on the cusp of a new era of connected autonomous vehicles with unprecedented user experiences, tremendously improved road safety and air quality, highly diverse transportation environments and use cases, as well as a plethora of advanced applications. Realizing this grand vision requires a significantly enhanced vehicle-to-everything (V2X) communication network which should be extremely intelligent and capable of concurrently supporting hyper-fast, ultra-reliable, and low-latency massive information exchange. It is anticipated that the sixth-generation (6G) communication systems will fulfill these requirements of the next-generation V2X. In this article, we outline a series of key enabling technologies from a range of domains, such as new materials, algorithms, and system architectures. Aiming for truly intelligent transportation systems, we envision that machine learning will play an instrumental role for advanced vehicular communication and networking. To this end, we provide an overview on the recent advances of machine learning in 6G vehicular networks. To stimulate future research in this area, we discuss the strength, open challenges, maturity, and enhancing areas of these technologies.

With 5G mobile communication systems been commercially rolled out, research discussions on next generation mobile systems, i.e., 6G, have started. On the other hand, vehicular technologies are also evolving rapidly, from connected vehicles as coined by V2X (vehicle to everything) to autonomous vehicles to the combination of the two, i.e., the networks of connected autonomous vehicles (CAV). How fast the evolution of these two areas will go head-in-head is of great importance, which is the focus of this paper. Based on a brief overview on the technological evolution of V2X to CAV and 6G key technologies, this paper explores two complementary research directions, namely, 6G for CAVs versus CAVs for 6G. The former investigates how various 6G key enablers, such as THz, cell free communication and artificial intelligence (AI), can be utilized to provide CAV mission-critical services. The latter discusses how CAVs can facilitate effective deployment and operation of 6G systems. This paper attempts to investigate the interactions between the two technologies to spark more research efforts in these areas.

Using the galaxy catalog built from ELUCID N-body simulation and the semi-analytical galaxy formation model, we have built a mock HI intensity mapping map. We have implemented the Finger-of-God (FoG) effect in the map by considering the galaxy HI gas velocity dispersion. By comparing the HI power spectrum in the redshift space with the measurement from IllustrisTNG simulation, we have found that such FoG effect can explain the discrepancy between current mock map built from N-body simulation and Illustris TNG simulation. Then we built a parameter-free FoG model and a shot-noise model to calculate the HI power spectrum. We found that our model can accurately fit both the monopole and quadrupole moments of the HI matter power spectrum. Our method of building the mock HI intensity map and the parameter-free FoG model will be widely useful for the up-coming 21cm intensity mapping experiments, such as CHIME, Tianlai, BINGO, FAST and SKA. It is also crucial for us to study the non-linear effects in 21cm intensity mapping., Comment: 10 pages, 5 figures, 1 table, published on ApJ, updated to match the published version