Search

Three-dimensional integration offers architectural and performance benefits for scaling augmented/virtual reality (AR/VR) models on highly resource-constrained edge devices. Two-dimensional off-chip memory interfaces are too prohibitively energy intensive and bandwidth (BW) limited for AR/VR devices. To solve this, we propose using advanced 3-D stacking technology for high-density vertical integration to local memory and compute, increasing memory capacity within the same footprint at iso-BW with improvements in energy and latency. We evaluate 3-D architectures for a prototype AR/VR accelerator to demonstrate up to 3.9× latency reduction and 1.6× lower energy compared to a 2-D configuration within a smaller/similar footprint. Additionally, we show the feasibility of deploying higher resolution AR/VR models by stacking multiple tiers of memory, providing a pathway to break the footprint constraints of 2-D architectures. The use of high-density 3-D interconnects allows us to demonstrate localized benefits at the accelerator-level compared with standard system-on-chip memory disaggregation techniques/architectures. [ABSTRACT FROM AUTHOR]

We propose a new methodology to estimate the 3D displacement field of deformable objects from video sequences using standard monocular cameras. We solve in real time the complete (possibly visco-)hyperelasticity problem to properly describe the strain and stress fields that are consistent with the displacements captured by the images, constrained by real physics. We do not impose any ad-hoc prior or energy minimization in the external surface, since the real and complete mechanics problem is solved. This means that we can also estimate the internal state of the objects, even in occluded areas, just by observing the external surface and the knowledge of material properties and geometry. Solving this problem in real time using a realistic constitutive law, usually non-linear, is out of reach for current systems. To overcome this difficulty, we solve off-line a parametrized problem that considers each source of variability in the problem as a new parameter and, consequently, as a new dimension in the formulation. Model Order Reduction methods allow us to reduce the dimensionality of the problem, and therefore, its computational cost, while preserving the visualization of the solution in the high-dimensionality space. This allows an accurate estimation of the object deformations, improving also the robustness in the 3D points estimation. [ABSTRACT FROM AUTHOR]

While $360^{\circ }$ 360 ∘ cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make visual recognition non-trivial. Ideally, $360^{\circ }$ 360 ∘ imagery could inherit the deep convolutional neural networks (CNNs) already trained with great success on perspective projection images. However, spherical images cannot be projected to a single plane without significant distortion, and existing methods to transfer CNNs from perspective to spherical images introduce significant computational costs and/or degradations in accuracy. We propose to learn a Spherical Convolution Network (SphConv) that translates a planar CNN to the equirectangular projection of $360^{\circ }$ 360 ∘ images. Given a source CNN for perspective images as input, SphConv learns to reproduce the flat filter outputs on $360^{\circ }$ 360 ∘ data, sensitive to the varying distortion effects across the viewing sphere. The key benefits are 1) efficient and accurate recognition for $360^{\circ }$ 360 ∘ images, and 2) the ability to leverage powerful pre-trained networks for perspective images. We further proposes two instantiation of SphConv—Spherical Kernel learns location dependent kernels on the sphere for SphConv, and Kernel Transformer Network learns a functional transformation that generates SphConv kernels from the source CNN. Among the two variants, Kernel Transformer Network has a much lower memory footprint at the cost of higher computational overhead. Validating our approach with multiple source CNNs and datasets, we show that SphConv using KTN successfully preserves the source CNN’s accuracy, while offering efficiency, transferability, and scalability to typical image resolutions. We further introduce a spherical Faster R-CNN model based on SphConv and show that we can learn a spherical object detector without any object annotation in $360^{\circ }$ 360 ∘ images. [ABSTRACT FROM AUTHOR]

With the full development of intelligent mobile communications, wireless mixed reality (MR) provides a more visually immersive experience and stronger interaction with environments than virtual reality (VR) and augmented reality (AR). However, the asymmetric characteristic of wireless MR traffic creates a huge challenge to current mobile networks. Dynamic time division duplex (D-TDD) is considered as a promising technology to improve wireless MR users’ quality of experience (QoE) due to its potentials and advantages in delivering asymmetric traffic. Therefore, in this paper, we propose a QoE-driven distributed multidimensional resource allocation (MRA) supplemented by inter-cell interference (ICI) mitigation scheme for wireless MR in multi-cell D-TDD systems. First, to improve QoE of MR users, we formulate the joint optimization of subframe configuration, channel assignment and computation offloading as a mixed-integer nonlinear programming problem. A novel fully-decentralized multi-agent deep Q-network (DQN) algorithm is developed to solve the problem. Then, to mitigate ICI, a water filling based power control algorithm is investigated to minimize the total power of each small base station and its associated MR users. Simulation results demonstrate that our proposed scheme improves QoE of MR users in a realizable way as compared to existing schemes. [ABSTRACT FROM AUTHOR]

Multiple deep neural networks (DNNs) are increasingly used in real-world intelligent applications, such as intelligent robotics and autonomous vehicles to collectively complete complicated tasks running on edge devices. Because each layer of the subtasks prefers a distinct dataflow due to the heterogeneity in shape and scale of the network layers, a variable dataflow approach on the DNN accelerators is urgently required. On DNN accelerators that enable multiple dataflows, however, we detect a dimension mismatch between parallel processing under the dataflow approach and linear data memory arrangement. When multiple DNN tasks share partial features or weights, the issue is further exacerbated. During processing, this mismatch causes a sluggish data supply from both off-chip and on-chip memory. Consequently, the overall throughput, performance, and energy efficiency suffer since DNN models are sensitive to data density. In this work, we reveal the mechanism behind this data dimension mismatch and present a series of metrics that quantify the influence on system performance. On this foundation, we offer a framework that tracks the data tensor dimension conversion and employs a flexible data arrangement over multi-DNN computation to adapt to dataflow variability. An accelerator architecture named data arrangement multi-DNN accelerator (DARMA) that features a data arrangement and distribution circuit and hierarchical memory for data dimension conversion is also presented. Since the mismatch is mitigated, the suggested accelerator outperforms current accelerators in terms of bandwidth and processing unit utilization. Through tests on VR/AR, MLperf, and other multitask applications, the evaluation results show that the proposed architecture provides both energy-efficiency and throughput improvements. [ABSTRACT FROM AUTHOR]

In this article, we design and develop a 3-D virtual museum with holistic metadata documentation and a variety of reptile behaviors and movements. First, we reconstruct the reptile's mesh in high resolution, and then create its rigged/skinned digital counterpart. We acquire the movement of two subjects using an optical motion capture system, accelerometers, and RGB-vision cameras; these movements are then segmented and annotated to various behaviors. The 3-D environment, virtual reality (VR), and augmented reality (AR) functionalities of our online repository serve as tools for interactively educating the public on animals, which are difficult to observe and study in their natural environment. It also reveals important information regarding animals' intangible characteristics (e.g., behavior), that is critical for the preservation of wildlife. Our museum is publicly accessible, enabling motion data reusability, and facilitating learning applications through gamification. We conducted a user study that confirms the naturalness and realism of our reptiles, along with the ease of use and usefulness of our museum. [ABSTRACT FROM AUTHOR]

In this article, we propose an interactive visualization method for motor design. We have proposed a visualization system for education using augmented reality (AR) and virtual reality (VR). As a further development of this, we are working on motor design as an industrial application. There are two issues in achieving high-speed interactivity. One is a fast mesh modification method in finite element analysis, and the other is a fast solving method. We propose a high-speed mesh modification method that utilizes regularly arranged grid points and coordinate transformation. In the proposed design system, the user can immediately observe the analysis result of the design change. Furthermore, if the proposed system is executed in cooperation with the projector, designers can change the design of the motor with their fingers and discuss it on the board. Furthermore, since this system can be executed on a Web browser, it can also be executed on a smartphone or tablet PC. [ABSTRACT FROM AUTHOR]

Video processing is foundational to a spectrum of important workloads: video sharing, video conferencing, photos/video archival, virtual/augmented reality, cloud gaming, and live streaming. The exponential growth in these workloads, coincident with a slowing Moore's law, poses significant challenges to deliver more computing at higher efficiencies. In this article, we present the design of a new accelerator—the video coding unit—targeted at warehouse-scale (cloud) video transcoding. Our deployment at scale in Google, the first of its kind, has demonstrated significant benefits on several products (YouTube, Meet, Stadia, Photos, etc.) serving billions of users. [ABSTRACT FROM AUTHOR]

Visual–inertial odometry (VIO) is widely studied and used in autonomous robots. This article proposes a novel tightly coupled monocular VIO system based on point-line constraints (PLC-VIO). In the front end, PLC-VIO presents a line segment extraction and merging algorithm based on the EDLines method and achieves real-time feature tracking based on the geometric constraints between feature points and lines. In the back end, PLC-VIO reconstructs new 3-D landmarks of feature lines through points on the line and optimizes the states by minimizing a cost function that combines the preintegrated inertial measurement unit (IMU) error term together with the point and line reprojection error terms in a sliding window optimization framework. A loop closure module is also integrated, which enables relocalization and drift elimination. The corresponding experimental evaluations are conducted using public datasets to validate the effectiveness and robustness of the proposed system, and the results show that PLC-VIO can achieve good performance when compared with other state-of-the-art systems and, at the same time, with no compromise to real-time performance. Note to Practitioners—Visual–inertial odometry (VIO) can estimate the states of the rigid body (including position, attitude, and velocity) that is widely used in robotic navigation, autonomous driving, virtual reality (VR), and augmented reality (AR). Aiming at the problem of estimating the states of autonomous robots in the GPS-denied environment, this article proposes a novel VIO system based on the point-line constraints (PLC-VIO). PLC-VIO can not only achieve accurate pose estimation for robots due to the introduction of the line features but also make no concession to real-time performance. Furthermore, PLC-VIO can also enrich the texture features of the environment during the 3-D mapping construction. The corresponding experiments are implemented in public datasets to evaluate the effectiveness, efficiency, and robustness of the proposed system. We believe that PLC-VIO can be widely used in robotic navigation and AR/VR fields to provide accurate position and environment information in real time. [ABSTRACT FROM AUTHOR]

Emerging applications such as augmented reality and tactile Internet are compute-intensive and latency sensitive, which hampers their running in constrained end devices alone or in the distant cloud. The stringent requirements of such applications have given rise to the realization of edge computing, wherby computation is offloaded near to users. Compute-less networking is an extension of edge computing that aims to reduce computation and abridge communication by adopting in-network computing and computation reuse. Computation reuse aims to cache the result of computations and use them to perform similar tasks in the future and, therefore, avoid redundant calculations and optimize the use of resources. In this article, we focus on the correctness of the final output produced by computation reuse. Since the input might not be identical but similar, the reuse of previous computation raises questions about the accuracy of the final results. To this end, we implement a proof of concept to study and gauge the effectiveness and efficiency of computation reuse. We are able to reduce task completion time by up to 80 percent while ensuring high correctness. We further discuss open challenges and highlight future research directions. [ABSTRACT FROM AUTHOR]

With the development of multimedia technology, Augmented Reality (AR) has become a promising next-generation mobile platform. The primary value of AR is to promote the fusion of digital contents and real-world environments, however, studies on how this fusion will influence the Quality of Experience (QoE) of these two components are lacking. To achieve better QoE of AR, whose two layers are influenced by each other, it is important to evaluate its perceptual quality first. In this paper, we consider AR technology as the superimposition of virtual scenes and real scenes, and introduce visual confusion as its basic theory. A more general problem is first proposed, which is evaluating the perceptual quality of superimposed images, i.e., confusing image quality assessment. A ConFusing Image Quality Assessment (CFIQA) database is established, which includes 600 reference images and 300 distorted images generated by mixing reference images in pairs. Then a subjective quality perception experiment is conducted towards attaining a better understanding of how humans perceive the confusing images. Based on the CFIQA database, several benchmark models and a specifically designed CFIQA model are proposed for solving this problem. Experimental results show that the proposed CFIQA model achieves state-of-the-art performance compared to other benchmark models. Moreover, an extended ARIQA study is further conducted based on the CFIQA study. We establish an ARIQA database to better simulate the real AR application scenarios, which contains 20 AR reference images, 20 background (BG) reference images, and 560 distorted images generated from AR and BG references, as well as the correspondingly collected subjective quality ratings. Three types of full-reference (FR) IQA benchmark variants are designed to study whether we should consider the visual confusion when designing corresponding IQA algorithms. An ARIQA metric is finally proposed for better evaluating the perceptual quality of AR images. Experimental results demonstrate the good generalization ability of the CFIQA model and the state-of-the-art performance of the ARIQA model. The databases, benchmark models, and proposed metrics are available at: https://github.com/DuanHuiyu/ARIQA. [ABSTRACT FROM AUTHOR]

We propose a vision-based framework for dynamic sky replacement and harmonization in videos. Different from previous sky editing methods that either focus on static photos or require real-time pose signal from the camera’s inertial measurement units, our method is purely vision-based, without any requirements on the capturing devices, and can be well applied to either online or offline processing scenarios. Our method runs in real-time and is free of manual interactions. We decompose the video sky replacement into several proxy tasks, including motion estimation, sky matting, and image blending. We derive the motion equation of an object at infinity on the image plane under the camera’s motion, and propose “flow propagation”, a novel method for robust motion estimation. We also propose a coarse-to-fine sky matting network to predict accurate sky matte and design image blending to improve the harmonization. Experiments are conducted on videos diversely captured in the wild and show high fidelity and good generalization capability of our framework in both visual quality and lighting/motion dynamics. We also introduce a new method for content-aware image augmentation and proved that this method is beneficial to visual perception in autonomous driving scenarios. Our code and animated results are available at https://github.com/jiupinjia/SkyAR [ABSTRACT FROM AUTHOR]

A holistic understanding of dynamic scenes is of fundamental importance in real-world computer vision problems such as autonomous driving, augmented reality and spatio-temporal reasoning. In this paper, we propose a new computer vision benchmark: Video Panoptic Segmentation (VPS). To study this important problem, we present two datasets, Cityscapes-VPS and VIPER together with a new evaluation metric, video panoptic quality (VPQ). We also propose VPSNet++, an advanced video panoptic segmentation network, which simultaneously performs classification, detection, segmentation, and tracking of all identities in videos. Specifically, VPSNet++ builds upon a top-down panoptic segmentation network by adding pixel-level feature fusion head and object-level association head. The former temporally augments the pixel features while the latter performs object tracking. Furthermore, we propose panoptic boundary learning as an auxiliary task, and instance discrimination learning which learns spatio-temporally clustered pixel embedding for individual thing or stuff regions, i.e., exactly the objective of the video panoptic segmentation problem. Our VPSNet++ significantly outperforms the default VPSNet, i.e., FuseTrack baseline, and achieves state-of-the-art results on both Cityscapes-VPS and VIPER datasets. The datasets, metric, and models are publicly available at https://github.com/mcahny/vps. [ABSTRACT FROM AUTHOR]

Due to the increased popularity of augmented and virtual reality experiences, the interest in capturing high-resolution real-world point clouds has never been higher. Loss of details and irregularities in point cloud geometry can occur during the capturing, processing, and compression pipeline. It is essential to address these challenges by being able to upsample a low Level-of-Detail (LoD) point cloud into a high LoD point cloud. Current upsampling methods suffer from several weaknesses in handling point cloud upsampling, especially in dense real-world photo-realistic point clouds. In this paper, we present a novel geometry upsampling technique, PU-Dense, which can process a diverse set of point clouds including synthetic mesh-based point clouds, real-world high-resolution point clouds, real-world indoor LiDAR scanned objects, as well as outdoor dynamically acquired LiDAR-based point clouds. PU-Dense employs a 3D multiscale architecture using sparse convolutional networks that hierarchically reconstruct an upsampled point cloud geometry via progressive rescaling and multiscale feature extraction. The framework employs a UNet type architecture that downscales the point cloud to a bottleneck and then upscales it to a higher level-of-detail (LoD) point cloud. PU-Dense introduces a novel Feature Extraction Unit that incorporates multiscale spatial learning by employing filters at multiple sampling rates and receptive fields. The architecture is memory efficient and is driven by a binary voxel occupancy classification loss that allows it to process high-resolution dense point clouds with millions of points during inference time. Qualitative and quantitative experimental results show that our method significantly outperforms the state-of-the-art approaches by a large margin while having much lower inference time complexity. We further test our dataset on high-resolution photo-realistic datasets. In addition, our method can handle noisy data well. We further show that our approach is memory efficient compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Objective: We designed, developed, and evaluated a 3D augmented reality (AR) weather visualization to investigate whether it could enhance communication about weather in general aviation (GA) education. Background: Evaluations of GA weather training identified gaps in training where students lack the ability to correlate weather knowledge to inflight decision making. Literature review: 3D AR learning objects have been used in the sciences to make representations of multidimensional natural phenomena more accessible in classroom settings, and they offer the promise of enhancing communication about weather. Research question: Can smartphone- and tablet-based 3D AR weather visualizations be effective tools to enhance current GA weather education? Methods: A 3D AR thunderstorm cell lifecycle visualization was designed and developed. A preliminary evaluation of the application for GA weather training was conducted with one certified flight instructor, one university aviation meteorology instructor, one university thunderstorm expert, and three students to assess whether the AR thunderstorm visualization can communicate weather theory and whether the interfaces are usable for learning and task completion. Results: Students’ knowledge of thunderstorms increased after using the visualization to explore the dynamics of the thunderstorm lifecycle and various aspects of thunderstorms. Experts felt that the learning experience met their expectations of what they wanted to communicate about thunderstorm theory. The AR interfaces were rated as usable for learning interactions and produced low levels of workload. Conclusion: The communication of thunderstorm theory was supported by the animation and interactivity of the visualization, and has the potential to enhance current general aviation weather education. [ABSTRACT FROM AUTHOR]

Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted a growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this paper, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research. [ABSTRACT FROM AUTHOR]

Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds. Practical visual localization approaches need to be robust to a wide variety of viewing conditions, including day-night changes, as well as weather and seasonal variations, while providing highly accurate six degree-of-freedom (6DOF) camera pose estimates. In this paper, we extend three publicly available datasets containing images captured under a wide variety of viewing conditions, but lacking camera pose information, with ground truth pose information, making evaluation of the impact of various factors on 6DOF camera pose estimation accuracy possible. We also discuss the performance of state-of-the-art localization approaches on these datasets. Additionally, we release around half of the poses for all conditions, and keep the remaining half private as a test set, in the hopes that this will stimulate research on long-term visual localization, learned local image features, and related research areas. Our datasets are available at visuallocalization.net , where we are also hosting a benchmarking server for automatic evaluation of results on the test set. The presented state-of-the-art results are to a large degree based on submissions to our server. [ABSTRACT FROM AUTHOR]

A relevant solution for the high demand for new multimedia applications and services is millimeter wave (mmWave) frequency band in 5G. However, in order to face the technological challenges of the present and those that will appear in the short-term future, it is necessary to improve the spectral efficiency of 5G systems. In particular, the Radio Resource Management (RRM) module is considered an essential component. Nevertheless, resource allocation techniques that combine orthogonal multiplexing (OMA) schemes, such as Time Division Multiple Access (TDMA), with Non-Orthogonal Multiple Access (NOMA) techniques have not been studied in-depth. Therefore, this article designs and evaluates different RRM models that combine TDMA with NOMA for innovative applications in the 5G mmWave frequency bands. To this end, the advantages and challenges associated with mmWave frequency bands and potential future applications are introduced. An innovative use case has been designed to test the RRM models. The use case is based on the on-demand distribution of multimedia content in high-density environments, such as museum halls. The on-demand content aims at offering Augmented Reality (AR) services to unicast users in order to provide personalized services. The models have been evaluated in terms of throughput, capacity, and availability. According to the results, the combined RRM techniques offer up to 50% more capacity than the single multiplexing technology models and can provide video-on-demand service to practically the entire cell. [ABSTRACT FROM AUTHOR]

Modern machines continuously log status reports over long periods of time, which are valuable data to optimize working routines. Data visualization is a commonly used tool to gain insights into these data, mostly in retrospective (e.g., to determine causal dependencies between the faults of different machines). We present an approach to bring such visual analyses to the shop floor to support reacting to faults in real time. This approach combines spatio-temporal analyses of time series using a handheld touch device with augmented reality for live monitoring. Important information augments machines directly in their real-world context, and detailed logs of current and historical events are displayed on the handheld device. In collaboration with an industry partner, we designed and tested our approach on a live production line to obtain feedback from operators. We compare our approach for monitoring and analysis with existing solutions that are currently deployed. [ABSTRACT FROM AUTHOR]

Our built world is one of the most important factors for a livable future, accounting for massive impact on resource and energy use, as well as climate change, but also the social and economic aspects that come with population growth. The architecture, engineering, and construction industry is facing the challenge that it needs to substantially increase its productivity, let alone the quality of buildings of the future. In this article, we discuss these challenges in more detail, focusing on how digitization can facilitate this transformation of the industry, and link them to opportunities for visualization and augmented reality research. We illustrate solution strategies for advanced building systems based on wood and fiber. [ABSTRACT FROM AUTHOR]

We present a generative model-based point cloud generation method using deep adversarial local features. The proposed generative adversarial network (GAN) can reduce computational load and increase the accuracy in three-dimensional (3D) acquisition, reconstruction, and rendering processes. To train the proposed GAN, we first optimize the latent space using an autoencoder to extract local features. The training process provides an accurate estimation of local context from the latent variables and robust point cloud generation. The main contribution of this work is a novel deep learning-based 3D point cloud generation, which significantly reduces computational load to render augmented reality (AR) and mixed reality (MR) contents. Additional contribution in the deep learning field is twofold: i) The autoencoder in the proposed network avoids the vanishing gradient problem using hierarchically linked features in different layers, and ii) the complexity of the network is significantly reduced by removing the transformation network that estimates the affine transformation matrix of the point cloud. [ABSTRACT FROM AUTHOR]

With the development of virtual reality and augmented reality, the pixel sizes of displays are reaching the sub-micrometer regime. One of the main challenges is the short-channel effect of the thin-film transistors (TFTs) in the driving circuits. In this work, we report the short-channel effects of indium-gallium-zinc-oxide source–gated transistor (IGZO SGT). The simulation results show that when reducing the channel length down to 500 nm, the output impedance of the IGZO SGT is still high up to 4.13 $\times \,\,10^{{8}}\,\, \Omega \!\cdot \!\mu \text{m}$ at gate–source voltage (${V}_{\text {GS}}$) of 5 V and drain–source voltage (${V}_{\text {DS}}$) of 7 V, 437 times higher than that of a conventional IGZO TFT of the same dimension. It is found that the Schottky source contact in SGTs can effectively suppress the back-channel current as compared with TFTs. The threshold voltage of short-channel SGTs remains almost the same when applying different drain voltages in contrast to TFTs. When ${V}_{\text {DS}}$ changes from 5 to 7 V at ${V}_{\text {GS}} =5$ V, the drain current of the IGZO TFT increases 23.7%, while the drain current of the SGT only increases 0.8%. The experimental results of IGZO TFTs and SGTs with a channel length of 500 nm showed similar dependence on the drain voltage to the simulation results. Such insensitivity to short-channel effects makes SGTs a promising candidate as current driving transistors in high-pixel-density display circuits. [ABSTRACT FROM AUTHOR]

Augmented reality (AR) has been presented lately as one of the key technology fields in which 5G networks can become a disruptive tool, raising interest from both industry and academia. The main goal of this article is to extend the current state of the art of distributed AR studies and implementations by extracting the main AR algorithms' offloading requirements individually. This extension is further achieved by depicting the data flow between these algorithms and their hardware requirements. From the obtained results, we estimate a preliminary set of network key performance indicators (KPIs) for a subset of three examples of distributed AR implementations highlighting the necessity of 5G technologies and their ecosystem to unveil the full potential of AR. Finally, based on these KPls, we propose a set of 5G configuration parameters for a successful distributed AR implementation. As most of the described algorithms are also used in virtual reality (VR) applications, our contributions can facilitate future distributed implementations of both AR and VR applications. [ABSTRACT FROM AUTHOR]

Manual thickness measurement of thin-walled parts is largely demanded in industrial applications. Repetitive manual interventions and poor consistency of measurement data are two of the critical existing problems for manual thickness measurement, which reduce the measuring accuracy and efficiency. To address these problems, this article combines digital ultrasonic thickness with an augmented reality (AR)-based system to assist the manual thickness measurement of the thin-walled parts. A simple adaptive pattern projection method is proposed to replace the manual drawing of grids and thickness states on the part. A multifunctional tool is developed to automatically visualize and record the measured thickness data in real-time, which guarantees the consistency of measurement data. The communication within the proposed system is established to enhance the human–machine interaction experience, allowing the operator to focus on the thickness measurement itself. The accuracy of the projection position is analyzed and the maximum deviation is 1.5 mm. A case study is conducted to validate the proposed system. The proposed system largely improves digitization and decreases the workload of manual thickness measurement. The measurement time is shortened compared to the conventional thickness measurement method. [ABSTRACT FROM AUTHOR]

Beyond-5G wireless systems require significant improvement to enable the Internet of Everything, offering ultra-reliability, ultra-low-latency and high data-rates for holographic telepresence, immersive augmented and virtual reality, and cyber-physical systems in Industry 4.0. The mmWave frequency band (30–300 GHz) provides the required bandwidths, but very challenging propagation conditions exist. Conventional co-located multi-antenna systems counter higher path loss, but are insufficient in challenging real-life scenarios with frequent non-line-of-sight conditions. For distributed massive MIMO systems or large intelligent surfaces, we advocate optically-enabled distributed antenna systems (DAS) to alleviate these issues. To ensure tight synchronization and scalability, we propose a mmWave-over-fiber based architecture with low-complexity high-performance remote antenna units (RAUs). Strategically distributing and integrating RAUs in the user equipments' environment yield high throughput and reliable coverage. We demonstrate a mmWave-over-fiber DAS yielding multi-Gb/s mmWave communication in a harsh indoor environment with non-line-of-sight conditions, measuring wireless data rates up to 24 Gb/s, by selecting the RAU yielding the best link quality, and up to 48 Gb/s, by leveraging distributed MIMO techniques. [ABSTRACT FROM AUTHOR]

$360^{o}$ video is becoming an increasingly popular technology on commercial social platforms and vital part of emerging Virtual Reality/Augmented Reality (VR/AR) applications. However, the delivery of $360^{o}$ video content in mobile networks is challenging because of its size. The encoding of $360^{o}$ video into multiple quality layers and tiles and edge cache-assisted video delivery have been proposed as a remedy to the excess bandwidth requirements of $360^{o}$ video delivery systems. Existing works using the above tools have shown promising performance for Video-on-Demand (VoD) $360^{o}$ delivery, but they cannot be straightforwardly extended in a live-streaming setup. Motivated by the above, we study edge cache-assisted $360^{o}$ live video streaming to increase the overall quality of the delivered $360^{o}$ videos to users and reduce the service cost. We employ Long Short-Term Memory (LSTM) networks to forecast the evolution of the content requests and prefetch content to caches. To further enhance the delivered video quality, users located in the overlap of the coverage areas of multiple Small Base Stations (SBSs) are allowed to receive data from any of these SBSs. We evaluate and compare the performance of our algorithm with Least Frequently Used (LFU), Least Recently Used (LRU), and First In First Out (FIFO) algorithms. The results show the superiority of the proposed approach in terms of delivered video quality, cache-hit-ratio and backhaul link usage. [ABSTRACT FROM AUTHOR]

5G and beyond mobile systems target a plethora of emerging industrial and entertainment verticals that incur extra overhead to the network. These verticals are characterized by vigorous, continuous, and conflicting requirements that make the desired system's mission strenuous and more challenging. These verticals, such as autonomous driving, will accommodate immersive services, including virtual reality/augmented reality (VR/AR) and Holography services. Immersive services, in particular, have strict requirements for latency, throughput, and positioning. This article discusses VR-based remote services' potential, which occupies an important place among immersive services such as remote surgery, remote space control, and remote driving of Unmanned Aerial Vehicles (UAVs) or cars. Such services require an ultra-low Glass-to-Glass latency to avoid any failure and accidents when remotely controlling devices. We evaluate an immersive remote control service from the end-to-end communication perspectives using different camera devices that stream real-time 360° videos to a VR Head Mounted Device (HMD). The obtained results demonstrate the challenges of such service and the need for more advanced and optimized techniques, devices, and protocols to achieve less than 20 ms of Glass-to-Glass latency. [ABSTRACT FROM AUTHOR]

Recent development toward innovative applications and technologies like self-driving, augmented reality, smart cities, and various other applications leads to excessive growth in the number of devices. These devices have finite computation resources and cannot handle the applications that require extensive computation with minimal delay. To overcome this, the mobile edge cloud (MEC) emerges as a practical solution that allows devices to offload their extensive computation to MEC located in their vicinity; this will lead to succeeding the arduous delay of the millisecond scale: requirement of $5^{th}$ generation communication system. This work examines the convex optimization problem. The objective is to minimize the task duration by optimal allocation of the resources like local and edge computational capabilities, transmission power, and optimal task segmentation. For optimal allocation of resources, an algorithm name Estimation of Optimal Resource Allocator (EORA) is designed to optimize the function by keeping track of statistics of each candidate of the population. Using EORA, a comparative analysis of the hybrid approach (partial offloading) and edge computation only is performed. Results reveal the fundamental trade-off between both of these models. Simultaneously, the impact of devices’ computational capability, data volume, and computational cycles requirement on task segmentation is analyzed. Simulation results demonstrate that the hybrid approach: partial offloading scheme reduces the task's computation time and outperforms edge computing only. [ABSTRACT FROM AUTHOR]

Point set registration (PSR) is an essential problem in the field of surgical navigation and augmented reality (AR). In surgical navigation, the aim of registration is mapping the pre-operative space to the intra-operative space. This article introduces a reliable hybrid mixture model, in which the reliability of the normal vectors in the generalized point set (GPS) is examined and exploited. The motivation of considering the reliability of orientation information is that normal vectors cannot be estimated or measured accurately in the clinic. The point set (PS) is divided into two subsets according to the reliability of normal vectors. PSR is cast into the maximum likelihood estimation (MLE) problem. The expectation maximization (EM) framework is used to solve the MLE problem. In the E-step, the posterior probabilities between points in two PSs are computed. In the M-step, the transformation matrix and model components are updated by optimizing the objective function. We have demonstrated through extensive experiments on the human femur bone PS that the proposed algorithm outperforms the state-of-the-art ones in terms of accuracy, robustness, and convergence speed. [ABSTRACT FROM AUTHOR]

Previous work on interactive 3D labeling focused on improving user experience based on virtual/augmented reality and, thereby, speeding-up the labeling of scenes. In this article, we present a novel interactive, collaborative VR-based 3D labeling system for live-captured scenes by multiple remotely connected users based on sparse multi-user input with automatic label propagation and completion. Hence, our system is particularly beneficial in the case of multiple users that are able to label different scene parts from the respectively adequate views in parallel. Our proposed system relies on 1) the RGB-D capture of an environment by a user, 2) a reconstruction client that integrates this stream into a 3D model, 3) a server that gets scene updates and manages the global 3D scene model as well as client requests and the integration/propagation of labels, 4) labeling clients that allow an independent VR-based scene exploration and labeling for each user, and 5) remotely connected users that provide a sparse 3D labeling used to control the label propagation over objects and the label prediction to other scene parts. Our evaluation demonstrates the intuitive collaborative 3D labeling experience as well as its capability to meet the efficiency constraints regarding reconstruction speed, data streaming, visualization, and labeling. [ABSTRACT FROM AUTHOR]

Hand pose understanding is essential to applications such as human computer interaction and augmented reality. Recently, deep learning based methods achieve great progress in this problem. However, the lack of high-quality and large-scale dataset prevents the further improvement of hand pose related tasks such as 2D/3D hand pose from color and depth from color. In this paper, we develop a large-scale and high-quality synthetic dataset, PBRHand. The dataset contains millions of photo-realistic rendered hand images and various ground truths including pose, semantic segmentation, and depth. Based on the dataset, we firstly investigate the effect of rendering methods and used databases on the performance of three hand pose related tasks: 2D/3D hand pose from color, depth from color and 3D hand pose from depth. This study provides insights that photo-realistic rendering dataset is worthy of synthesizing and shows that our new dataset can improve the performance of the state-of-the-art on these tasks. This synthetic data also enables us to explore multi-task learning, while it is expensive to have all the ground truth available on real data. Evaluations show that our approach can achieve state-of-the-art or competitive performance on several public datasets. [ABSTRACT FROM AUTHOR]

This article discusses a systematic review of the state-of-the-art on augmented reality (AR) and virtual reality (VR) in spinal navigation, where early clinical validations have shown promising outlook on accuracy and scalability parameters. The objective of this research is to evaluate clinical relevance for AR–VR enabled spinal surgical technologies and develop an economic feasibility model for stakeholders, such as patients, hospitals, and research organizations with technology adoption. From the influencing parameters, we identified the research gaps that can be explored going forward and a list of high priority research challenges that could provide an attractive research and development investment case for industry players. [ABSTRACT FROM AUTHOR]

Dynamic Adaptive Streaming over HTTP (DASH) has been widely used by several popular streaming services, such as YouTube, Netflix, and Facebook. Adopting DASH requires to pre-determine a set of encoding configurations, called encoding ladder, to generate a set of representations stored on the streaming server. These representations are adaptively requested by clients according to their network conditions during streaming sessions. In this article, we aim to solve the optimal laddering problem that determines the optimal encoding ladder to maximize the client viewing quality. In particular, we consider video models, viewing probability, and client distribution to formulate the mathematical problem. We use a divide-and-conquer approach to decompose the problem into two subproblems: (i) per-class optimization for clients with different bandwidths and (ii) global optimization to maximize the overall viewing quality under the storage limit of the streaming server. We propose two algorithms for each of the per-class optimization and global optimization problems. Analytical analysis and real experiments are conducted to evaluate the performance of our proposed algorithms, compared to other state-of-the-art algorithms. Based on the results, we recommend a combination of the proposed algorithms to solve the optimal laddering problem. The evaluation results show the merits of our recommended algorithms, which: (i) outperform the state-of-the-art algorithms by up to 52.17 and 26.35 in Viewport Video Multi-Method Assessment Fusion (V-VMAF) in per-class optimization, (ii) outperform the state-of-the-art algorithms by up to 43.14 in V-VMAF for optimal laddering in global optimization, (iii) achieve good scalability under different storage limits and number of bandwidth classes, and (iv) run faster than the state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Instead of relying on remote clouds, today’s Augmented Reality (AR) applications usually send videos to nearby edge servers for analysis (such as objection detection) so as to optimize the user’s quality of experience (QoE), which is often determined by not only detection latency but also detection accuracy, playback fluency, etc. Therefore, many studies have been conducted to help adaptively choose best video configuration, e.g., resolution and frame per second (fps), based on network bandwidth to further improve QoE. However, we notice that the video content itself has significant impacts on the configuration selection, e.g., the videos with high-speed objects must be encoded with a high fps to meet the user’s fluency requirement. In this article, we aim to adaptively select configurations that match the time-varying network condition as well as the video content. We design Cuttlefish, a system that generates video configuration decisions using reinforcement learning (RL). Cuttlefish trains a neural network model that picks a configuration for the next encoding slot based on observations collected by AR devices. Cuttlefish does not rely on any pre-programmed models or specific assumptions on the environments. Instead, it learns to make configuration decisions solely through observations of the resulting performance of historical decisions. Cuttlefish automatically learns the adaptive configuration policy for diverse AR video streams and obtains a gratifying QoE. We compared Cuttlefish to several state-of-the-art bandwidth-based and velocity-based methods using trace-driven and real world experiments. The results show that Cuttlefish achieves a 18.4-25.8 percent higher QoE than the others. [ABSTRACT FROM AUTHOR]

Contribution: Significant gender differences are observed on primary school students’ perception of self-efficacy and test anxiety in mathematics. Girls perceive themselves to be significantly worse than boys in mathematics and report higher test anxiety toward mathematics exams. Gender differences in self-efficacy become more pronounced as students grow up, and test anxiety increases for all students. However, the present study shows that teachers’ do not perceive differences in self-efficacy in mathematics between boys and girls. Background: The low presence of women in science, technology, engineering, and mathematics (STEM) might be explained by the attitude of young students toward mathematics. Different studies show that girls are less interested in STEM areas than boys during secondary school. A study on the reasons for this fact pointed out that the early years of education can provide a relevant insight to reverse the situation. Research Questions: Is there any age-dependent gender difference in primary school students in aspects related to mathematics? Are teachers aware of students’ perceptions? Methodology: This work presents a study of over 2000 primary school students (6–12 years old) and 200 teachers in Aragón (Spain). The study consists of a survey on aspects that influence the experience of female and male students with mathematics and Spanish language for comparison purposes and teacher’s awareness of students’ perception. Findings: The present study shows that during primary school, girls are more likely to experiment a negative attitude toward mathematics than boys as they grow up, and teachers may not perceive girls’ situation. [ABSTRACT FROM AUTHOR]

Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. As a consequence, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. The main contribution of this paper is to reveal how exemplary workflows are redefined by taking full advantage of head-mounted displays when entirely co-registered with the imaging system at all times. The awareness of the system from the geometric and physical characteristics of X-ray imaging allows the exploration of different human-machine interfaces. Our system achieved an error of 4.76 ± 2.91mm for placing K-wire in a fracture management procedure, and yielded errors of 1.57 ± 1.16° and 1.46 ± 1.00° in the abduction and anteversion angles, respectively, for total hip arthroplasty (THA). We compared the results with the outcomes from baseline standard operative and non-immersive AR procedures, which had yielded errors of [4.61mm, 4.76°, 4.77°] and [5.13mm, 1.78°, 1.43°], respectively, for wire placement, and abduction and anteversion during THA. We hope that our holistic approach towards improving the interface of surgery not only augments the surgeon’s capabilities but also augments the surgical team’s experience in carrying out an effective intervention with reduced complications and provide novel approaches of documenting procedures for training purposes. [ABSTRACT FROM AUTHOR]

A major research area in Computer Assisted Intervention (CAI) is to aid laparoscopic surgery teams with Augmented Reality (AR) guidance. This involves registering data from other modalities such as MR and fusing it with the laparoscopic video in real-time, to reveal the location of hidden critical structures. We present the first system for AR guided laparoscopic surgery of the uterus. This works with pre-operative MR or CT data and monocular laparoscopes, without requiring any additional interventional hardware such as optical trackers. We present novel and robust solutions to two main sub-problems: the initial registration, which is solved using a short exploratory video, and update registration, which is solved with real-time tracking-by-detection. These problems are challenging for the uterus because it is a weakly-textured, highly mobile organ that moves independently of surrounding structures. In the broader context, our system is the first that has successfully performed markerless real-time registration and AR of a mobile human organ with monocular laparoscopes in the OR. [ABSTRACT FROM AUTHOR]

Future machine learning (ML) powered applications, such as autonomous driving and augmented reality, involve training and inference tasks with timeliness requirements and are communication- and computation-intensive, which demands the edge learning framework. The real-time requirements drive us to go beyond accuracy for ML. In this article, we introduce the concept of timely edge learning, aiming to achieve accurate training and inference while minimizing the communication and computation delay. We discuss key challenges and propose corresponding solutions from data, model, and resource management perspectives to meet the timeliness requirements. In particular, for edge training, we argue that the total training delay rather than rounds should be considered, and propose data or model compression, and joint device scheduling and resource management schemes for both centralized training and federated learning systems. For edge inference, we explore the dependency between accuracy and delay for communication and computation, and propose dynamic data compression and flexible pruning schemes. Two case studies show that the timeliness performance, including the training accuracy under a given delay budget and the completion ratio of inference tasks within deadline, are highly improved with the proposed solutions. [ABSTRACT FROM AUTHOR]

Objective: In minimally invasive surgery (MIS), in situ augmented reality (AR) navigation systems are usually implemented using a glasses-free 3D display to represent the preoperative tissue structure, and can provide intuitive see-through guidance information. However, due to changes in intraoperative tissue, the preoperative tissue structure is not able to exactly correspond to reality, which influences the precision of in situ AR navigation. To solve this problem, we propose a method to update the tissue structure for in situ AR navigation in such way to reflect changes in intraoperative tissue. Methods: The proposed method to update the tissue structure is based on the calibrated ultrasound and two-level surface warping technologies. Firstly, the particle filter-based calibration is implemented to perform ultrasound calibration and obtain intraoperative position of anatomical points. Secondly, intraoperative positions of anatomical points are inputted in the two-level surface warping method to update the preoperative tissue structure. Finally, the glasses-free real 3-D display of the updated tissue structure is finished, and is superimposed onto a patient by a translucent mirror for in situ AR navigation. Results: we validated the proposed method by simulating liver tissue intervention, and achieved the tissue updating accuracy of 92.86%. Furthermore, the targeting error of AR navigation based on the proposed method was also evaluated through minimally invasive liver surgery, and the acquired mean targeting error was 1.92 mm. Conclusion: The results demonstrate that the proposed AR navigation method is effective. Significance: The proposed method can facilitate MIS, as it provides accurate 3D navigation. [ABSTRACT FROM AUTHOR]

Autonomous driving has the potential to make transportation systems safer, greener and more efficient. To realize autonomous driving, a great deal of in-car cutting-edge applications such as augmented reality, dynamic path planning and cognitive driving systems are required, which need significant computational resources and near realtime response. To cope with this new paradigm, vehicular fog computing (VFC) has emerged recently, which migrates the computing from congested base stations (or cloud servers) to nearby vehicles with under-utilized computational resources. in VFC, the designs of server recruitment and task offloading strategies under information asymmetry and uncertainty pose new technical challenges. in this article, we propose a two-stage VFC framework to address these challenges. The framework consists of a contract theory based vehicular computational resource management mechanism, and a matching-learning based task offloading mechanism. Simulation results demonstrate that the proposed framework can boost the performance of VFC in terms of resource utilization efficiency and task offloading delay. [ABSTRACT FROM AUTHOR]

5G and Blockchain are potentially revolutionizing future technologies. 5G promises high rates and QoS to the users and blockchain guarantees a high level of trust and security among the peers. Applications that would be using 5G have varying needs in terms of speed, bandwidth, latency and various other factors. Augmented reality, self-driving vehicles and other ioT applications tend to use 5G for reliable and fast communication. To work seamlessly and securely in such scenarios a more specialized and efficient approach would be required. in this article, we have identified the specific areas where blockchain could be utilized to enhance the security and privacy of the 5G services offered to the users. The current challenges faced in deployment and upliftment of 5G and their related solutions based on blockchain are discussed. A model for Multi-Operator Network Slicing in 5G using blockchain is also presented along with 5G blockchain implementation. [ABSTRACT FROM AUTHOR]

In this study, we focus on improving the energy efficiency of multiple mobile augmented reality (MAR) devices (MDs) with different MAR applications, which are connected to a single multi-access edge computing (MEC) server, through the centralized orchestration of the MEC server. To achieve this, a trade-off between the accuracy, latency, and energy consumption of each MD is considered as intertwined costs. Accordingly, we minimize a sum of intertwined costs including the energy consumption, latency and accuracy loss of multiple MDs, while satisfying the maximum latency constraint and the minimum accuracy constraint of each MAR application. With rigorous analysis, we design a theoretical framework for the optimization of MD performance by jointly managing the frame resolution of the image to offload to the MEC server from MDs, and the computation capacity distribution over MDs in the case of the limited computation capacity of the MEC server. Our numerical analysis validates that the proposed scheme significantly reduces the number of service-level agreement violations, such as exceeding the maximum latency constraint, compared with the existed algorithm. Specifically, the proposed work significantly improve costs of service latency, accuracy loss, and energy consumption of MDs for MEC-assisted MAR services compared with baseline schemes. [ABSTRACT FROM AUTHOR]

We report the design and fabrication of a high-speed and ultranarrow pitch gate driver with amorphous indium-gallium-zinc-oxide thin-film transistors (a-IGZO TFTs). At a supply voltage of 10 V, the gate driver operates with a pulsewidth of 1}~μs and corresponds to an operating frequency of 500 kHz, which is compatible with an 8k4k resolution display operated at 240 Hz. It is also ultrasmall in physical size with merely 18.6~μm in width (pitch) and 530~μm in length that makes it suitable for small size, ultrahigh definition (UHD), and narrow bezel displays with pixel density up to 2700 pixel per inch (ppi) for augmented reality and virtual reality (AR/VR) applications. The fabricated gate driver exhibits the rising time of ~360 ns and falling time of ~430 ns at 10 V with no ripples and degradation after the 480th stage. This effort demonstrates the potential of dual-gate a-IGZO TFTs for high-speed high-density display circuit integration. [ABSTRACT FROM AUTHOR]

An instrument for remote control of the robot by wearable brain–computer interface (BCI) is proposed for rehabilitating children with attention-deficit/hyperactivity disorder (ADHD). Augmented reality (AR) glasses generate flickering stimuli, and a single-channel electroencephalographic BCI detects the elicited steady-state visual evoked potentials (SSVEPs). This allows benefiting from the SSVEP robustness by leaving available the view of robot movements. Together with the lack of training, a single channel maximizes the device’s wearability, fundamental for the acceptance by ADHD children. Effectively controlling the movements of a robot through a new channel enhances rehabilitation engagement and effectiveness. A case study at an accredited rehabilitation center on ten healthy adult subjects highlighted an average accuracy higher than 83%, with information transfer rate (ITR) up to 39 b/min. Preliminary further tests on four ADHD patients between six- and eight-years old provided highly positive feedback on device acceptance and attentional performance. [ABSTRACT FROM AUTHOR]

Objective: Augmented reality (AR) navigation using a position sensor in endoscopic surgeries relies on the quality of patient–image registration and hand–eye calibration. Conventional methods collect the necessary data to compute two output transformation matrices separately. However, the AR display setting during surgery generally differs from that during preoperative processes. Although conventional methods can identify optimal solutions under initial conditions, AR display errors are unavoidable during surgery owing to the inherent computational complexity of AR processes, such as error accumulation over successive matrix multiplications, and tracking errors of position sensor. Methods: We propose the simultaneous optimization of patient–image registration and hand–eye calibration in an AR environment before surgery. The relationship between the endoscope and a virtual object to overlay is first calculated using an endoscopic image, which also functions as a reference during optimization. After including the tracking information from the position sensor, patient–image registration and hand–eye calibration are optimized in terms of least-squares. Results: Experiments with synthetic data verify that the proposed method is less sensitive to computation and tracking errors. A phantom experiment with a position sensor is also conducted. The accuracy of the proposed method is significantly higher than that of the conventional method. Conclusion: The AR accuracy of the proposed method is compared with those of the conventional ones, and the superiority of the proposed method is verified. Significance: This study demonstrates that the proposed method exhibits substantial potential for improving AR navigation accuracy. [ABSTRACT FROM AUTHOR]

The next generation of networks will need to support massive requests for services that have very stringent performance requirements, and their delivery involves significant computing and storage resources. Prominent examples are the mobile augmented and virtual reality services, processing of large-scale IoT data, and mobile data analytics. In order to address these arising challenges, mobile network operators need innovative solutions, and one such is the close collaboration with the cloud service providers. In this article, we propose a novel system architecture that integrates the infrastructure of Mobile Network Operator (MNO) and Cloud Service Provider (CSP), leveraging recent developments in network softwarization that allows the unified control of the network, computing, and storage resources. We envision a multi-tier architecture where the CSP and MNO exchange real-time information about their resource availability and the users' needs, and jointly devise the servicing policy. We present a blueprint of the architecture, several application examples, and a numerical case study that focuses on distributed computing. [ABSTRACT FROM AUTHOR]

Search tasks in production environments have been shown to be significantly improved by attention guiding techniques in augmented reality (AR). In mobile use cases, head-mounted displays support context-sensitive and flexible search tasks. To improve search time and navigation efficiency, first concepts like attention funnel and spherical wave based guidance techniques have been tested successfully in comparison to simple arrow navigation. Existing research indicates that for two-dimensional interfaces, a differentiation into coarse and fine navigation will result in further performance improvements. In this article, a laboratory study is presented with a stereoscopic head-mounted display that examines whether an additional arrow guiding technique for coarse navigation could improve both search time and efficiency of existing techniques. Results show objectively superior performance for one of the combined techniques in comparison to the unimodal attention guiding techniques and subjective preference by the participants. Further research should investigate the performance in a real environment. [ABSTRACT FROM AUTHOR]

Omnidirectional (or 360°) images and videos are emergent signals being used in many areas, such as robotics and virtual/augmented reality. In particular, for virtual reality applications, they allow an immersive experience in which the user can interactively navigate through a scene with three degrees of freedom, wearing a head-mounted display. Current approaches for capturing, processing, delivering, and displaying 360° content, however, present many open technical challenges and introduce several types of distortions in the visual signal. Some of the distortions are specific to the nature of 360° images and often differ from those encountered in classical visual communication frameworks. This paper provides a first comprehensive review of the most common visual distortions that alter 360° signals going through the different processing elements of the visual communication pipeline. While their impact on viewers’ visual perception and the immersive experience at large is still unknown—thus, it is an open research topic—this review serves the purpose of proposing a taxonomy of the visual distortions that can be encountered in 360° signals. Their underlying causes in the end-to-end 360° content distribution pipeline are identified. This taxonomy is essential as a basis for comparing different processing techniques, such as visual enhancement, encoding, and streaming strategies, and allowing the effective design of new algorithms and applications. It is also a useful resource for the design of psycho-visual studies aiming to characterize human perception of 360° content in interactive and immersive applications. [ABSTRACT FROM AUTHOR]

The task of determining the physical coordinates of a target in indoor environments is still a key factor for many applications, including people and robot navigation, user tracking, location-based advertising, augmented reality, gaming, emergency response, or ambient-assisted living environments. Among the different possibilities for indoor positioning, acoustic local positioning systems (ALPSs) have the potential for centimeter-level positioning accuracy with coverage distances up to tens of meters. In addition, acoustic transducers are small, low cost, and reliable thanks to the room constrained propagation of these mechanical waves. Waveform design (coding and modulation) is usually incorporated into these systems to facilitate the detection of the transmitted signals at the receiver. The aperiodic correlation properties of the emitted signals have a large impact on how the ALPSs cope with common impairment factors, such as multipath propagation, multiple access interference, Doppler shifting, near-far effect, or ambient noise. This article analyzes three of the most promising families of codes found in the literature for ALPS: Kasami codes, Zadoff–Chu, and orthogonal chirp signals. The performance of these codes is evaluated in terms of time of arrival accuracy and characterized by means of model simulation under realistic conditions and by means of experimental tests in controlled environments. The results derived from this study can be of interest for other applications based on spreading sequences, such as underwater acoustic systems, ultrasonic imaging, or even code division multiple access (CDMA) communications systems. [ABSTRACT FROM AUTHOR]