Search

Mobile Augmented Reality (MAR) integrates computer-generated virtual objects with physical environments for mobile devices. MAR systems enable users to interact with MAR devices, such as smartphones and head-worn wearables, and perform seamless transitions from the physical world to a mixed world with digital entities. These MAR systems support user experiences using MAR devices to provide universal access to digital content. Over the past 20 years, several MAR systems have been developed, however, the studies and design of MAR frameworks have not yet been systematically reviewed from the perspective of user-centric design. This article presents the first effort of surveying existing MAR frameworks (count: 37) and further discusses the latest studies on MAR through a top-down approach: (1) MAR applications; (2) MAR visualisation techniques adaptive to user mobility and contexts; (3) systematic evaluation of MAR frameworks, including supported platforms and corresponding features such as tracking, feature extraction, and sensing capabilities; (4) and underlying machine learning approaches supporting intelligent operations within MAR systems. Finally, we summarise the development of emerging research fields and the current state-of-the-art and discuss the important open challenges and possible theoretical and technical directions. This survey aims to benefit both researchers and MAR system developers alike.

Screen lock is a critical security feature for smartphones to prevent unauthorized access. Although various screen unlocking technologies, including fingerprint and facial recognition, have been widely adopted, they still have some limitations. For example, fingerprints can be stolen by special material stickers and facial recognition systems can be cheated by 3D-printed head models. In this paper, we propose EmgAuth, a novel electromyography(EMG)-based smartphone unlocking system based on the Siamese network. EmgAuth enables users to unlock their smartphones by leveraging the EMG data of the smartphone users collected from Myo armbands. When training the Siamese network, we design a special data augmentation technique to make the system resilient to the rotation of the armband, which makes EmgAuth free of calibration. We conduct extensive experiments including 80 participants and the evaluation results verify that EmgAuth can effectively authenticate users with an average true acceptance rate of 91.81% while keeping the average false acceptance rate of 7.43%. In addition, we also demonstrate that EmgAuth can work well for smartphones with different screen sizes and for different scenarios when users are placing smartphones at different locations and with different orientations. EmgAuth shows great promise to serve as a good supplement for existing screen unlocking systems to improve the safety of smartphones.

With the emerging demands of new communication services, the contradiction between capacity demand and spectrum shortage of railway communication systems becomes more severe. How to provide broadband communication services has become the key goal of future smart high-speed railway (HSR) systems. Millimeter wave (mm-wave) frequency band has abundant spectrum resources and can provide communication services with large bandwidth. However, due to the high-speed of the train as well as the complexity and dynamics of environments, the communication link may be blocked randomly for a short time and will also lead to frequent handovers. In this paper, we adopt the promising intelligent reflecting surface (IRS) technology for a mm-wave HSR communication system. In order to improve system capacity, IRS is deployed to improve reflection transmission links, and optimization algorithms are designed for transceiver beamforming and IRS phase shift. In addition, given the specificity of the HSR scenario, we also formulate the average system ergodic capacity maximization problem and obtain upper bound on the average system ergodic capacity with statistical channel state information (CSI). Through extensive simulations, we verify that the proposed scheme performs significantly better than the other two baseline schemes in terms of average system throughput and average system ergodic capacity.

With their high mobility, unmanned aerial vehicles (UAVs) become appealing data collectors in hard-to-reach wide-area distributed sensor networks. Different from existing works focusing on the perspective of UAVs for service order optimization and UAV utility maximization, we consider the utilities of both sensors and UAVs, and innovatively model the competition among sensors (buyers) for the service of UAVs (sellers) as an auction game. A “unity makes strength” strategy is exploited. That is, to strengthen the bidding competitiveness, a group-buying coalition auction method that encourages sensors to form coalitions to bid for UAV service is proposed. Besides, we propose a parallel variable neighborhood ascent search algorithm, we can quickly determine the approximately optimal group-buying coalition structure. Numerical results show that the proposed method outperforms the joint trajectory design-task scheduling (TDTS) UAV-to-community method and the single coalition formation game (CFG) method.

Wearable devices have been widely utilized by the general public for tracking physical activities. Many complex machine learning models leverage wearable devices to address application problems, such as predicting pedestrian behaviors and health management. These models often incur heavy computing load and energy cost, which is challenging for wearable devices. However, aggregating the data from different wearable devices to a central server introduces privacy concerns. To address these challenges, we propose an architecture, CloudyFL, by deploying cloudlets close to wearable devices. In CloudyFL, each cloudlet forms a trusted zone covering a subset of nearby wearable devices. Models are trained in this trusted zone, and then, only the model parameters are transmitted to a centralized aggregator using a federated learning framework. We additionally propose an LSTM-based model for user behavior sensing, with a neural network design to adjust to the non-IID data distribution on multiple cloudlets. Experimental results show that our training model within the CloudyFL architecture can achieve a performance better than existing methodologies.

Real-time holographic video communications enable immersive experiences for next-generation video services in the future metaverse era. However, high-fidelity holographic videos require high bandwidth and significant computation resources, which exceed the transferring and computing capacity of 5G networks. This article reviews state-of-the-art holographic point cloud video transmission techniques and highlights the critical challenges of delivering such immersive services. We further implement a preliminary prototype of an AI-driven holographic video communication system and present critical experimental results to evaluate its performance. Finally, we identify future research directions and discuss potential solutions for providing real-time and high-quality holographic experiences.

Mobile Augmented Reality (MAR) applications enable users to interact with physical environments through overlaying digital information on top of camera views. Detecting and classifying complex objects in the real world presents a critical challenge to enable immersive user experiences in MAR applications. Aiming to provide continuous MAR experiences, we address a key challenge of continuous object recognition, which requires accommodating an increasing number of recognition requests on different types of images in MAR systems and possible new types of images in emerging applications. Inspired by the latest advance in continual learning approaches in computer vision, this paper presents a novel MAR system to enhance its scalability with continual learning in realistic scenarios. Our experiments demonstrate that 1) the system enables efficiently recognising objects without requiring retraining from scratch; and 2) edge computing further reduces latency for continual object recognition.

Augmented Reality (AR) provides immersive user experiences by overlaying digital information on physical environments. Context-awareness is crucial for delivering relevant augmentations that best suit users’ requirements and their en-vironments. In this article, we combine context-aware reasoning with emerging AR applications to provide the most relevant information according to user and environment contexts. To support the best possible quality of experience, 5G edge computing enables the distribution of computation-intensive AR tasks to edge servers through 5G networks. We develop ConAR, a context-aware head-mounted display AR system that is deployed on the edge and cloud leveraging both environmental sensors and user profile context for navigation. ConAR is composed of a HoloLens application and a paired mobile client, which contains a context model for air quality forecasting, and rendering recommendations on holograms through a HoloLens 2 device. We evaluate our system performance by deploying our proposed air quality prediction algorithm on the edge and cloud while communicating to them using 5G and LTE connections. We measure network quality metrics and find the deployment on the edge with 5G connections significantly outperforms alternative solutions. Our results demonstrate that the 5G edge computing is suitable for supporting latency-sensitive analysis tasks for context-aware AR.

In a postpandemic world, remaining vigilant and maintaining social distancing are still crucial so societies can contain the virus and the public can avoid disproportionate health impacts. Augmented reality (AR) can visually assist users in understanding the distances in social distancing. However, integrating external sensing and analysis is required for social distancing beyond the users’ local environment. We present DistAR, an android-based application for social distancing leveraging AR and smart sensing using on-device analysis of optical images and environment crowdedness from smart campus data. Our prototype is one of the first efforts to combine AR and smart sensing technologies to create a real-time social distancing application.

The Third Generation Partnership Project (3GPP) is accelerating 5G new radio (NR) global standards aimed at significant enhancement of wireless system performance for higher date rate, better energy efficiency, and higher reliability than the current 4G cellular systems. The operators, manufacturers, and test equipment vendors have worked together to develop standardized over-the-air (OTA) test methodologies for the overall performance evaluation of 5G NR devices. 3GPP is taking the lead in standardizing the OTA testing of 5G NR under fading channel conditions. In 3GPP specifications, test methods have been studied to verify the multiple-input multiple-output (MIMO) performance of 5G NR user equipments (UEs) in OTA mode. This article follows the 3GPP standardization work and discusses the MIMO OTA test methodologies for 5G NR UEs working at frequency range 1 (FR1) and FR2, with a focus on its new challenges and solutions compared to 4G MIMO OTA testing methods. Then the OTA throughput testing results of real 5G NRUEs are demonstrated under the standard channel models. Finally, the challenges and limitations of standard 5G MIMO OTA test solutions are also highlighted.

This article contributes a research vision for using edge computing to deliver the computing infrastructure for emerging smart megacities, with use cases, key requirements, and reflections on the state of the art. We also address edge server placements, a key challenge for edge computing adoption.

Mobile Augmented Reality (MAR) mixes physical environments with user-interactive virtual annotations. Immersive MAR experiences are supported by computation-intensive tasks, which are typically offloaded to cloud or edge servers. Such offloading introduces additional network traffic and influences the motion-to-photon latency (a determinant of user-perceived quality of experience). Therefore, proper multimedia protocols are crucial to minimise transmission latency and ensure sufficient throughput to support MAR performance. Relatedly, 5G is a potential MAR supporting technology and is widely believed to be faster and more efficient than its predecessors. However, the suitability and performance of existing multimedia protocols for MAR in the 5G edge context have not been explored. In this work, we present a detailed evaluation of several popular multimedia protocols (HLS, MPEG-DASH, RTP, RTMP, RTMFP, and RTSP) and transport protocols (QUIC, UDP, and TCP) with a MAR system on a real-world 5G edge testbed. The evaluation results indicate that RTMP has the lowest median client-to-server packet latency on 5G and LTE for all image resolutions. In terms of individual image resolutions, from 144p to 480p over 5G and LTE, RTMP has the lowest median packet latency of 14.03±1.05ms. Whereas for jitter, HLS has the smallest median jitter across all image resolutions over LTE and 5G with medians of 2.62 ms and 1.41 ms, respectively. Our experimental results indicate that RTMP and HLS are the most suitable protocols for MAR.

Littering is a significant challenge for environmental sustainability and a major burden for cities and densely populated areas. Current solutions for litter monitoring, such as litter watch campaigns and city-operated litter collection, are costly and challenging to conduct at a large scale. This article presents a vision for using autonomous ground vehicles (AGVs) for litter monitoring and removal and introduces a mechanism for AGVs that uses thermal dissipation resulting from sunlight to identify and remove litter objects. We identify and highlight key challenges for deploying the envisioned solution on a city scale, and demonstrate the feasibility of the solution through extensive experiments.

With the explosive growth of vehicle applications, vehicular networks based on millimeter wave (mmWave) bands have attracted interests from both academia and industry. mmWave communications are able to utilize the huge available bandwidth to provide multiple Gbps transmission rates among vehicles. In this paper, we address the content distribution scheduling problem in mmWave vehicular networks. It has been challenging for all vehicles in the same network to complete content downloading due to the limited communication resources of roadside units (RSUs) and the high mobility of vehicles. We propose a joint vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) scheduling scheme to minimize the total number of content distribution time slots from a global optimization perspective. In the V2I phase, the RSU serially transmits integrity content to vehicles, which are selected according to the vehicular network topology and transmission scheduling scheme. In the V2V phase, full-duplex communications and concurrent transmissions are exploited to achieve content sharing between vehicles and improve transmission efficiency. Performance evaluations demonstrate that our proposed scheme reduces the number of time slots and significantly improves system throughput when compared with other schemes, especially under large-size file transfers and a large number of vehicles.

Accurate and timely traffic flow prediction is crucial for intelligent transportation systems (ITS). Recent advances in graph-based neural networks have achieved promising prediction results. However, some challenges remain, especially regarding graph construction and the time complexity of models. In this paper, we propose a multi-stream feature fusion approach to extract and integrate rich features from traffic data and leverage a data-driven adjacent matrix instead of the distance-based matrix to construct graphs. We calculate the Spearman rank correlation coefficient between monitor stations to obtain the initial adjacent matrix and fine-tune it while training. As to the model, we construct a multi-stream feature fusion block (MFFB) module, which includes a three-channel network and the soft-attention mechanism. The three-channel networks are graph convolutional neural network (GCN), gated recurrent unit (GRU) and fully connected neural network (FNN), which are used to extract spatial, temporal and other features, respectively. The soft-attention mechanism is utilized to integrate the obtained features. The MFFB modules are stacked, and a fully connected layer and a convolutional layer are used to make predictions. We conduct experiments on two real-world traffic prediction tasks and verify that our proposed approach outperforms the state-of-the-art methods within an acceptable time complexity.

Unmanned aerial vehicles (UAVs) enable promising solutions in assisting data collection in wide-area distributed sensor networks, leveraging their advanced properties of high mobility and line-of-sight communication links. However, existing UAV-assisted data collection methods mainly focus on unilaterally maximizing the utility of UAVs or sensors. Unfortunately, the problem driven by the market economy is ignored, namely the game between buyer and seller, in the process of sensors competing for UAV services. To address this problem, we propose a group-buying coalition auction method that encourages sensors to form coalitions to bid for UAV data collection services. Then, a parallel variable neighborhood ascent search algorithm is designed to quickly search the approximately optimal group-buying coalition structure. We further propose a novel group-buying coalition auction method, named TRUST, which can ensure the economical properties, i.e., truthfulness, individual rationality, and maximization of social welfare. Numerical results show that the sensors’ average age of information (AoI) under the proposed method is reduced by 16.7% and 44.5% compared with the coalition formation game (CFG) and joint trajectory design-task scheduling (TDTS) UAV-to-community methods. To our best knowledge, this is the first effort on truthful coalition formation-based group-buying auction.