Search

Watching sports events via 3D-instead of two-dimensional video streaming allows for increased immersion, e.g. via mixed reality headsets in comparison to traditional screens. So far, capturing 3D video of sports events required expensive outside-in tracking with numerous cameras. This study demonstrates the feasibility of streaming sports content to mixed reality headsets as holographs in real-time using inside-out tracking and low-cost equipment only. We demonstrate our system by streaming a race car on an indoor track as 3D models, which are then rendered in an Magic Leap One headset. An onboard camera, mounted on the race car provides the video stream used to localize the car via computer vision. The localization is estimated by an end-to-end convolutional neural network (CNN). The study compares three state-of-the-art CNN models in their respective accuracy and execution time, with PoseNet + LSTM achieving position and orientation accuracy of 0.35m and 3.95°. The total streaming latency in this study was 1041ms, suggesting technical feasibility of streaming 3D sports content, e.g. on large playgrounds, in near real-time onto mixed-reality headsets.

© 2020 IEEE. Generating 3D point clouds is challenging yet highly desired. This work presents a novel autoregressive model, PointGrow, which can generate diverse and realistic point cloud samples from scratch or conditioned on semantic contexts. This model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points, allowing inter-point correlations to be well-exploited and 3D shape generative processes to be better interpreted. Since point cloud object shapes are typically encoded by long-range dependencies, we augment our model with dedicated self-attention modules to capture such relations. Extensive evaluations show that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to realism and diversity. Several important applications, such as unsupervised feature learning and shape arithmetic operations, are also demonstrated.

Hand gesture recognition enables non-tactile interfaces for human-machine interactions. Cameras are currently powerful tools to recognize these gestures, however, use of cameras is constrained by privacy concerns and need for welllit, line of sight implementation. In this study, we propose an alternate method to recognize gestures using a passive data-glove augmented with passive RFID tags. We envision passive tags-based gesture recognition will have applications in improving operator safety around machines, activity monitoring in factories and sign to speech recognition, etc. Low-level reader information (RSSI, Phase and Doppler frequency) can be used to capture changes to the tags in the environment, therefore generating enough information to infer gestures. In this paper, we present a technique to enable fast feature recognition using low-level reader data by correcting for inconsistencies in phase data due to frequency hopping. We experimented with four different classifiers on the low-level reader data and our Fully-Connected Neural Network (FCCN) classifier is able to learn gestures from tag-data with 98% accuracy.

We present an Augmented Reality (AR) visualization and interaction tool for users to control Internet of Things (IoT) devices with hand gestures. Today, smart IoT devices are becoming increasingly ubiquitous with diverse forms and functions, yet most user controls over them are still limited to mobile devices and web interfaces. Recently, AR has been developed rapidly, and provided immersive solutions to enhance user experience of applications in many fields. Its capability to create immersive interactions allows AR to improve the way smart devices are controlled via more direct visual feedback. In this paper, we create a functional prototype of one such system, enabling seamless interactions with sound and lighting systems through the use of augmented hand-controlled interaction panels. To interpret users' intentions, we implement a standard 2D convolution neural network (CNN) for real-time hand gesture recognition and deploy it within our system. Our prototype is also equipped with a simple but effective object detector which can identify target devices within a proper range by analyzing geometric features. We evaluate the performance of our system qualitatively and quantitatively and demonstrate it on two smart devices.