Search

In the realm of AR applications and particularly camera-to-screen interactions, camera tracking stands as a crucial technology. However, the ever-increasing demand for tracking accuracy makes it essential to explore a six-degrees of freedom (6-DoF) tracking technology with ultra-high precision to facilitate micro-motion sensing. In this paper, we propose a novel sensing method MoiréTracker to achieve camera’s 6-DoF pose tracking with ultra-high precision. MoiréTracker outputs camera’s continuous 3-DoF trajectory and 3-DoF posture changes according to the captured moiré patterns, which can be produced by the superposition of camera’s Color Filter Array (CFA) and the projection of screen raster on the CFA plane. Thanks to moiré pattern’s high sensitivity to 6-DoF motions, we characterize the relationship between moiré features and camera’s micro pose changes, so as to realize the continuous 6-DoF pose tracking for camera with ultra-high precision. Moreover, our proposal involves a thumbnail-based method aimed at expanding the working range of MoiréTracker, enabling the pervasive camera-to-screen interactions. We implement a prototype system and evaluate its performance in real-world environments. Extensive experiment results show that MoiréTracker achieves the average trajectory error of 1.20 cm and the posture error of 1.07°.

The detection of objects via neural networks plays a key role in various video analytics, but consumes huge resources. Due to the limited computing capability at edges, such real-time detections should be precisely used for the objects that need the most attention. Unfortunately, as the target objects keep moving, existing systems fail to consider both the distribution of objects and object movements over regions, and existing tracking mechanisms are easily affected by background content. Therefore, we propose spatial and temporal detection with attention for analytics, to increase the quality of detections for those targets. However, the attention shift over regions, the uncertainty of detections, and the constrained edge resources essentially hamper us from efficient analytics. We propose an adaptive partition planner to divide the frame into regions to achieve spatial attention. Afterwards, we design a detection planner to orchestrate the detection model temporally for each region by an online mechanism, via a queue-based adaptation. The spatial and temporal attention are integrated to maximize the accumulative detection accuracy. Via rigorous proof, both dynamic regret regarding detection accuracy and the real-time requirement for the video analytics are ensured. The testbed experiments confirm the superiority of our approach over multiple state-of-the-art algorithms.

Deep convolutional neural network (NN)-based object detectors are not appropriate for straightforward inference on high-resolution videos at edge devices, as maintaining high accuracy often brings about prohibitively long latency. Although existing solutions have attempted to reduce on-device inference latency by selecting a cheaper configuration (e.g., choosing a more lightweight NN or scaling a frame to a smaller size before inference) or eliminating a background containing no object, they often ignore various high-resolution features and fail to optimize for those videos. We thus present AdaPyramid, a framework to reduce as much on-device inference latency as possible, especially for high-resolution videos, while achieving the accuracy demand approximately. We observe that the cheapest configuration to achieve the accuracy demand varies significantly across both different frames and different regions in a frame. The underlying reason is that object features (e.g., the location, size and category of objects) are more uneven in high-resolution videos, both temporally and spatially. Moreover, we observe that the object size presents a prominent hierarchical distribution in high-resolution frames. AdaPyramid thus partitions each frame hierarchically just like a pyramid and chooses a content-aware configuration for each region, which is adapted online based on the feedback. We evaluate the performance of AdaPyramid on a public dataset and our collected real-world videos. The obtained results show that under comparable accuracy to the state-of-the-art solutions, AdaPyramid can decrease inference latency by 40% on average, with up to 2.5× speed-up.

Nowadays, to realize the intelligent manufacturing in Industrial Internet of Things (IIoT) scenarios, novel approaches in computer vision are in great demand to tackle the new challenges in IIoT environment. These approaches, which we call Industrial Vision, are expected to offer customized solutions for intelligent manufacturing in an accurate, time efficient and robust manner. In this paper, we propose a novel approach to industrial vision, called Edge-Eye, to rectify the edge deviation automatically for Irradiated Cross-linked Polyethylene Foam (IXPE) production with millimeter-level accuracy. We deploy a commercial camera with mobile edge node in front of the IXPE sheet to continuously detect and rectify the edge deviation. Particularly, to handle the complex production environment when extracting the edge of IXPE sheet, we deploy a pair of reference bars with high-contrast colors to efficiently differentiate the sheet edge from the background. Then, we propose a Bi-direction Edge Tracking method to perform the edge detection from both vertical and horizontal aspects. To realize the rectification using mobile edge nodes with limited computing resources, we reduce the cost of computation by extracting the Minimized Region of Interest, i.e., the edge area overlapped with the higher contrast reference bar on both sides. We further design a negative feedback control system with multi-stage feedback regulation mechanism, keeping the edge deviation within millimeter-level. The experimental results show that Edge-Eye achieves the average accuracy of 5 mm for the edge deviation rectification, with the average latency of 200 ms for edge deviation detection.

With the rising of demands for novel Human-Computer Interaction (HCI) approaches in the 3D space, a number of intelligent approaches have been proposed to achieve the HCI by tracking the translation and rotation of the target devices. In this article, we propose to realize a light-weight, battery-free, 3D motion tracking solution by leveraging a spinning linearly polarized antenna to track a passive RFID tag array. Instead of using the fixed antennas, which can only receive stable signal in some specific environments due to the unpredictable multipath effect, we propose to mitigate the multipath effect and the ambient interference by continuously spinning a linearly polarized antenna, and then extract the most distinctive features based on the optimal reading conditions of the spinning antenna. In particular, because the phase variation around the matching direction is more stable while the RSSI variation around the mismatching direction is more distinctive, we leverage such matching/mismatching property of the linearly polarized antenna to extract the most distinctive features for motion tracking. To depict the property, we build a theoretical model to explain the RSSI and the phase variation of the RFID tag along with the spinning of the antenna, and further extend the model from a single RFID tag to an RFID tag array. Based on the model, we can extract the distinctive RSSI features for the rotation tracking and the stable phase features for the translation tracking. Moreover, to tackle the low rate of feature extraction due to the spinning of antenna, we further propose to enhance the unstable phase features based on the overall trend of other tags with interpolation, such that the sampling rate can be efficiently improved. Finally, we propose a LSTM (Long Short Term Memory)-based network to track the 3D motion based on the signal features extracted based on the polarization model. The experimental results show that our system can achieve an average error of 10.45 cm in the translation tracking, and an average error of $6.02^\circ$6.02∘ in the rotation tracking in the 3D space.

The usage of live streaming services has led to a substantial increase in live video traffic. However, the perceived quality of experience of users is frequently limited by variations in the upstream bandwidth of streamers. To address this issue, several adaptive bitrate (ABR) algorithms have been developed to mitigate bandwidth variations. Nevertheless, the ability of users to enjoy high-quality live streams remains limited. While neural-enhanced approaches, such as super-resolution, offer significant quality improvements, frame-oriented super-resolution leads to excessive inference delay that violates the real-time feature of live streaming. In response, we propose ViChaser, which examines block-oriented super-resolution for live streaming. ViChaser performs neural super-resolution on potential blocks of interest in the media server, corresponding to the user’s viewpoint, and uses online learning to adapt to the dynamic content of the video. Additionally, ViChaser utilizes the Lyapunov framework to efficiently allocate uplink bandwidth for original low-quality live video and high-quality labels. The experimental results demonstrate that ViChaser achieves 1.2–1.5 dB higher video quality in Peak-Signal-to-Noise-Ratio than WebRTC and increases processing speed by 11–16 fps relative to LiveNAS.

Over the last decade, the mobile crowdsourcing has become a paradigm to conduct the manual annotation and further analytics by recruited workers, with their rewards depending on the result quality. Existing dispatchers cannot precisely capture the resource-quality trade-off for video analytics, because the configurations supported by recruited workers are limited, and workers’ availability changes over time. To determine the most suitable configurations as well as workers for video analytics, we formulate a non-linear mixed program in long term, maximizing the crowdsourcing profit. Based on previous results under various configurations and workers, we design an algorithm via a series of subproblems to decide the configurations adaptively upon the prediction of workers’ feedbacks. Such prediction is based on volatile multi-armed bandit to capture workers’ availability and stochastic changes on resource uses. Furthermore, we extend the proposed algorithms to the multi-worker selection scenario where the platform needs to determine a candidate worker set instead of a single worker for video analytics. Via rigorous proof, the regret is ensured upon the Lyapunov optimization and the bandit, measuring the gap between the online decisions and the offline optimum. Extensive trace-driven experiments show that our proposed algorithm improves the profit by 37% compared with other algorithms.

Video streaming is one of the most popular Internet applications that makes up a large amount of Internet traffic. A fundamental mechanism in video streaming is adaptive bitrate (ABR) selection which decides the proper compression level for each chunk of a video to optimize the users’ quality of experience (QoE). The existing ABR algorithms require significant tuning and do not generalize to diverse network conditions and personalized QoE objectives. In this article, we propose a novel framework for meta-learning based ABR design and discuss challenges of deploying learning based ABR mechanism in real-world video streaming systems. We utilize the proposed framework to design MetaABR, a novel adaptive bitrate selection algorithm based on meta-reinforcement learning to maximize users’ QoE. By jointly training multiple learning tasks with a shared meta-critic, it can provide transferrable meta-knowledge to supervise bitrate selection across tasks, and can be applied to efficiently learn a new task in unseen environment with only a few trials. We implement MetaABR on an emulation platform which connects to the Linux network protocol stack through virtual network interfaces. Extensive experiments based on real-world traces and wireless testbed show that MetaABR achieves the best comprehensive QoE compared with the state-of-the-art ABR algorithms in a variety of network environments.