Search

Estimating the depth of virtual content has proven to be a challenging task in Augmented Reality (AR) applications. Existing studies have shown that the visual system makes use of multiple depth cues to infer the distance of objects, occlusion being one of the most important ones. The ability to generate appropriate occlusions becomes particularly important for AR applications that require the visualization of augmented objects placed below a real surface. Examples of these applications are medical scenarios in which the visualization of anatomical information needs to be observed within the patient's body. In this regard, existing works have proposed several focus and context (F+C) approaches to aid users in visualizing this content using Video See-Through (VST) Head-Mounted Displays (HMDs). However, the implementation of these approaches in Optical See-Through (OST) HMDs remains an open question due to the additive characteristics of the display technology. In this article, we, for the first time, design and conduct a user study that compares depth estimation between VST and OST HMDs using existing in-situ visualization methods. Our results show that these visualizations cannot be directly transferred to OST displays without increasing error in depth perception tasks. To tackle this gap, we perform a structured decomposition of the visual properties of AR F+C methods to find best-performing combinations. We propose the use of chromatic shadows and hatching approaches transferred from computer graphics. In a second study, we perform a factorized analysis of these combinations, showing that varying the shading type and using colored shadows can lead to better depth estimation when using OST HMDs.

We introduce NeuroConstruct, a novel end-to-end application for the segmentation, registration, and visualization of brain volumes imaged using wide-field microscopy. NeuroConstruct offers a Segmentation Toolbox with various annotation helper functions that aid experts to effectively and precisely annotate micrometer resolution neurites. It also offers an automatic neurites segmentation using convolutional neuronal networks (CNN) trained by the Toolbox annotations and somas segmentation using thresholding. To visualize neurites in a given volume, NeuroConstruct offers a hybrid rendering by combining iso-surface rendering of high-confidence classified neurites, along with real-time rendering of raw volume using a 2D transfer function for voxel classification score versus voxel intensity value. For a complete reconstruction of the 3D neurites, we introduce a Registration Toolbox that provides automatic coarse-to-fine alignment of serially sectioned samples. The quantitative and qualitative analysis show that NeuroConstruct outperforms the state-of-the-art in all design aspects. NeuroConstruct was developed as a collaboration between computer scientists and neuroscientists, with an application to the study of cholinergic neurons, which are severely affected in Alzheimer's disease.

Genomic research emerges from collaborative work within and across different scientific disciplines. A diverse range of visualisation techniques has been employed to aid this research, yet relatively little is known as to how these techniques facilitate collaboration. We conducted a case study of collaborative research within a biomedical institute to learn more about the role visualisation plays in genomic mapping. Interviews were conducted with molecular biologists (N = 5) and bioinformaticians (N = 6). We found that genomic research comprises a variety of distinct disciplines engaged in complex analytic tasks that each resist simplification, and their complexity influences how visualisations were used. Visualisation use was impacted by group-specific interactions and temporal work patterns. Visualisations were also crucial to the scientific workflow, used for both question formation and confirmation of hypotheses, and acted as an anchor for the communication of ideas and discussion. In the latter case, two approaches were taken: providing collaborators with either interactive or static imagery representing a viewpoint. The use of generic software for simplified visualisations, and quick production and curation was also noted. We discuss these findings with reference to group-specific interactions and present recommendations for improving collaborative practices through visual analytics.

Past studies have shown that when a visualization uses pictographs to encode data, they have a positive effect on memory, engagement, and assessment of risk. However, little is known about how pictographs affect one's ability to understand a visualization, beyond memory for values and trends. We conducted two crowdsourced experiments to compare the effectiveness of using pictographs when showing part-to-whole relationships. In Experiment 1, we compared pictograph arrays to more traditional bar and pie charts. We tested participants' ability to generate high-level insights following Bloom's taxonomy of educational objectives via 6 free-response questions. We found that accuracy for extracting information and generating insights did not differ overall between the two versions. To explore the motivating differences between the designs, we conducted a second experiment where participants compared charts containing pictograph arrays to more traditional charts on 5 metrics and explained their reasoning. We found that some participants preferred the way that pictographs allowed them to envision the topic more easily, while others preferred traditional bar and pie charts because they seem less cluttered and faster to read. These results suggest that, at least in simple visualizations depicting part-to-whole relationships, the choice of using pictographs has little influence on sensemaking and insight extraction. When deciding whether to use pictograph arrays, designers should consider visual appeal, perceived comprehension time, ease of envisioning the topic, and clutteredness.

The state-of-the-art optical see-through head-mounted displays (OST-HMD) for augmented reality applications lack the ability to render correct light interaction behavior between digital and physical objects, known as mutual occlusion capability. This paper presents a novel optical architecture for enabling a compact, high performance, occlusion-capable optical see-through head-mounted display (OCOST-HMD) with correct, pupil-matched viewing perspective. The proposed design utilizes a single-layer, double-pass architecture, offering a compact OCOST-HMD solution that is capable of rendering per-pixel mutual occlusion, a correctly pupil-matched viewing between virtual and real views, and a very wide see-through field of view (FOV). Based on this architecture, we demonstrate a design embodiment and a compact prototype implementation. The prototype offers a virtual display with an FOV of 34 by 22, an angular resolution of 1.06 arc minutes per pixel, and an average image contrast greater than 40% at the Nyquist frequency of 53 cycles/mm. Further, the prototype system affords a wide see-though FOV of 90 by 50, within which about 40 diagonally is occlusion-enabled, along with an angular resolution of 1.0 arc minutes, comparable to a 20/20 vision and a dynamic range greater than 100:1. Lastly, we composed a quantitative color study that compares the effects of occlusion between a conventional HMD system and our OCOST-HMD system and the resulting response exhibited in different studies.

Most commercially available optical see-through head-mounted displays (OST-HMDs) utilize optical combiners to simultaneously visualize the physical background and virtual objects. The displayed images perceived by users are a blend of rendered pixels and background colors. Enabling high fidelity color perception in mixed reality (MR) scenarios using OST-HMDs is an important but challenging task. We propose a real-time rendering scheme to enhance the color contrast between virtual objects and the surrounding background for OST-HMDs. Inspired by the discovery of color perception in psychophysics, we first formulate the color contrast enhancement as a constrained optimization problem. We then design an end-to-end algorithm to search the optimal complementary shift in both chromaticity and luminance of the displayed color. This aims at enhancing the contrast between virtual objects and the real background as well as keeping the consistency with the original displayed color. We assess the performance of our approach using a simulated OST-HMD environment and an off-the-shelf OST-HMD. Experimental results from objective evaluations and subjective user studies demonstrate that the proposed approach makes rendered virtual objects more distinguishable from the surrounding background, thereby bringing a better visual experience.

In spatial augmented reality, a widely dynamic projection mapping system has been developed as a novel approach to graphics presentation for widely moving objects in dynamic situations. However, this method necessitates a novel tracking marker design that is resistant to random and complex occlusion and out-of-focus blurring, which conventional markers have not achieved. This article presents a uniform circumferential marker that becomes an ellipse in perspective projection and expresses geometric information. It can track the relative posture of a dynamically moving sphere with high speed, high accuracy, and robustness owing to continuous contour lines, thereby supporting both wide-range movement in the depth direction and human interaction. Moreover, a biased circumferential marker is proposed to embed unique coding, where the absolute posture is decoded with a novel recognition algorithm. Moreover, rough initialization using the geometry of multiple ellipses is proposed for both markers to start the automatic and precise tracking. Real-time rotation visualization onto the surface of a moving sphere is made possible with the high-speed, widely dynamic projection mapping system. The tracking performance is demonstrated to exhibit sufficient basic tracking performance as well as robustness against blurring and occlusion compared to conventional dot-based markers.

High dynamic range (HDR) imaging is a technique that allows an extensive dynamic range of exposures, which is important in image processing, computer graphics, and computer vision. In recent years, there has been a significant advancement in HDR imaging using deep learning (DL). This study conducts a comprehensive and insightful survey and analysis of recent developments in deep HDR imaging methodologies. We hierarchically and structurally group existing deep HDR imaging methods into five categories based on (1) number/domain of input exposures, (2) number of learning tasks, (3) novel sensor data, (4) novel learning strategies, and (5) applications. Importantly, we provide a constructive discussion on each category regarding its potential and challenges. Moreover, we review some crucial aspects of deep HDR imaging, such as datasets and evaluation metrics. Finally, we highlight some open problems and point out future research directions., Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), main and suppl. material

Extracting and analyzing crime patterns in big cities is a challenging spatiotemporal problem. The hardness of the problem is linked to two main factors, the sparse nature of the crime activity and its spread in large spatial areas. Sparseness hampers most time series (crime time series) comparison methods from working properly, while the handling of large urban areas tends to render the computational costs of such methods impractical. Visualizing different patterns hidden in crime time series data is another issue in this context, mainly due to the number of patterns that can show up in the time series analysis. In this article, we present a new methodology to deal with the issues above, enabling the analysis of spatiotemporal crime patterns in a street-level of detail. Our approach is made up of two main components designed to handle the spatial sparsity and spreading of crimes in large areas of the city. The first component relies on a stochastic mechanism from which one can visually analyze probable×intensive crime hotspots. Such analysis reveals important patterns that can not be observed in the typical intensity-based hotspot visualization. The second component builds upon a deep learning mechanism to embed crime time series in Cartesian space. From the embedding, one can identify spatial locations where the crime time series have similar behavior. The two components have been integrated into a web-based analytical tool called CriPAV (Crime Pattern Analysis and Visualization), which enables global as well as a street-level view of crime patterns. Developed in close collaboration with domain experts, CriPAV has been validated through a set of case studies with real crime data in São Paulo - Brazil. The provided experiments and case studies reveal the effectiveness of CriPAV in identifying patterns such as locations where crimes are not intense but highly probable to occur as well as locations that are far apart from each other but bear similar crime patterns.

Realistic speech-driven 3D facial animation is a challenging problem due to the complex relationship between speech and face. In this paper, we propose a deep architecture, called Geometry-guided Dense Perspective Network (GDPnet), to achieve speaker-independent realistic 3D facial animation. The encoder is designed with dense connections to strengthen feature propagation and encourage the re-use of audio features, and the decoder is integrated with an attention mechanism to adaptively recalibrate point-wise feature responses by explicitly modeling interdependencies between different neuron units. We also introduce a non-linear face reconstruction representation as a guidance of latent space to obtain more accurate deformation, which helps solve the geometry-related deformation and is good for generalization across subjects. Huber and HSIC (Hilbert-Schmidt Independence Criterion) constraints are adopted to promote the robustness of our model and to better exploit the non-linear and high-order correlations. Experimental results on the public dataset and real scanned dataset validate the superiority of our proposed GDPnet compared with state-of-the-art model. The code is available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/GDPnet.

Using optical sensors to track hand gestures in virtual reality (VR) simulations requires issues such as occlusion, field-of-view, accuracy and stability of sensors to be addressed or mitigated. We introduce an optical hand-based interaction system that comprises two Leap Motion sensors mounted onto a VR headset at different orientations. Our system collects sensor data from the leap motions, combines and processes it to produce optimal hand tracking data, that minimizes the effect of sensor occlusion and noise. This contrasts with previous systems that do not use multiple head-mounted sensors or incorporate hand-data aggregation. We also present a study that compares the proposed system with glove-based and traditional motion controller-based interaction. We investigate hand interactions effect on the feeling of naturalness and immersion. The results show that the use of two head-mounted sensors and the data aggregation system increased the number of valid hands presented to the user and can be successfully applied to VR. The user study shows that there is a strong preference for the proposed system in terms of natural feeling and freeing interaction. The absence of an indirect interface such as gloves or controllers was found to aid in creating a more natural and immersive experience.

Due to the scale of data and the complexity of analysis tasks, insight discovery often requires coordinating multiple visualizations (views), with each view displaying different parts of data or the same data from different perspectives. For example, to analyze car sales records, a marketing analyst uses a line chart to visualize the trend of car sales, a scatterplot to inspect the price and horsepower of different cars, and a matrix to compare the transaction amounts in types of deals. To explore related information across multiple views, current visual analysis tools heavily rely on brushing and linking techniques, which may require a significant amount of user effort (e.g., many trial-and-error attempts). There may be other efficient and effective ways of displaying cross-view data relationships to support data analysis with multiple views, but currently there are no guidelines to address this design challenge. In this article, we present systematic design considerations for visualizing cross-view data relationships, which leverages descriptive aspects of relationships and usable visual context of multi-view visualizations. We discuss pros and cons of different designs for showing cross-view data relationships, and provide a set of recommendations for helping practitioners make design decisions.

Deep reinforcement learning (DRL) targets to train an autonomous agent to interact with a pre-defined environment and strives to achieve specific goals through deep neural networks (DNN). Recurrent neural network (RNN) based DRL has demonstrated superior performance, as RNNs can effectively capture the temporal evolution of the environment and respond with proper agent actions. However, apart from the outstanding performance, little is known about how RNNs understand the environment internally and what has been memorized over time. Revealing these details is extremely important for deep learning experts to understand and improve DRLs, which in contrast, is also challenging due to the complicated data transformations inside these models. In this article, we propose Deep Reinforcement Learning Interactive Visual Explorer (DRLIVE), a visual analytics system to effectively explore, interpret, and diagnose RNN-based DRLs. Having focused on DRL agents trained for different Atari games, DRLIVE accomplishes three tasks: game episode exploration, RNN hidden/cell state examination, and interactive model perturbation. Using the system, one can flexibly explore a DRL agent through interactive visualizations, discover interpretable RNN cells by prioritizing RNN hidden/cell states with a set of metrics, and further diagnose the DRL model by interactively perturbing its inputs. Through concrete studies with multiple deep learning experts, we validated the efficacy of DRLIVE.

Intrinsic image decomposition is the task of mapping image to albedo and shading. Classical approaches derive methods from spatial models. The modern literature stresses evaluation, by comparing predictions to human judgements ("lighter", "same as", "darker"). The best modern intrinsic image methods train a map from image to albedo using images rendered from computer graphics models and example human judgements. This approach yields practical methods, but obtaining rendered images can be inconvenient. Furthermore, the approach cannot explain how a one could learn to recover intrinsic images without geometric, surface and illumination models, as people and animals appear to do. This paper describes a method that learns intrinsic image decomposition without seeing human annotations, rendered data, or ground truth data. Instead, the method relies on paradigms - spatial models of albedo and of shading. Rather than finding the "best" albedo and shading for an image via optimization, our approach trains a neural network on synthetic images. The synthetic images are constructed by multiplying albedos and shading fields sampled from our models. The network is subject to a novel smoothing procedure that ensures good behavior at short scales on real images. An averaging procedure ensures that reported albedo and shading are largely equivariant - different crops and scalings of an image will report the same albedo and shading at shared points. This averaging procedure controls long scale error. The standard evaluation for an intrinsic image method is a WHDR score. Our method achieves WHDR scores competitive with those of strong recent methods allowed to see training WHDR annotations, rendered data, and ground truth data. Our method produces albedo and shading maps with attractive qualitative properties - for example, albedo fields do not suppress wood grain and represent narrow grooves in surfaces well. Because our method is unsupervised, we can compute estimates of the test/train variance of WHDR scores; these are quite large, and suggest is unsafe to rely small differences in reported WHDR.

In the field of information visualization, the concept of "tasks" is an essential component of theories and methodologies for how a visualization researcher or a practitioner understands what tasks a user needs to perform and how to approach the creation of a new design. In this article, we focus on the collection of tasks for tree visualizations, a common visual encoding in many domains ranging from biology to computer science to geography. In spite of their commonality, no prior efforts exist to collect and abstractly define tree visualization tasks. We present a literature review of tree visualization articles and generate a curated dataset of over 200 tasks. To enable effective task abstraction for trees, we also contribute a novel extension of the Multi-Level Task Typology to include more specificity to support tree-specific tasks as well as a systematic procedure to conduct task abstractions for tree visualizations. All tasks in the dataset were abstracted with the novel typology extension and analyzed to gain a better understanding of the state of tree visualizations. These abstracted tasks can benefit visualization researchers and practitioners as they design evaluation studies or compare their analytical tasks with ones previously studied in the literature to make informed decisions about their design. We also reflect on our novel methodology and advocate more broadly for the creation of task-based knowledge repositories for different types of visualizations. The Supplemental Material, which can be found on the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/TVCG.2021.3064037, will be maintained on OSF: https://osf.io/u5ehs/.

The increased availability of quantitative historical datasets has provided new research opportunities for multiple disciplines in social science. In this article, we work closely with the constructors of a new dataset, CGED-Q (China Government Employee Database-Qing), that records the career trajectories of over 340,000 government officials in the Qing bureaucracy in China from 1760 to 1912. We use these data to study career mobility from a historical perspective and understand social mobility and inequality. However, existing statistical approaches are inadequate for analyzing career mobility in this historical dataset with its fine-grained attributes and long time span, since they are mostly hypothesis-driven and require substantial effort. We propose CareerLens, an interactive visual analytics system for assisting experts in exploring, understanding, and reasoning from historical career data. With CareerLens, experts examine mobility patterns in three levels-of-detail, namely, the macro-level providing a summary of overall mobility, the meso-level extracting latent group mobility patterns, and the micro-level revealing social relationships of individuals. We demonstrate the effectiveness and usability of CareerLens through two case studies and receive encouraging feedback from follow-up interviews with domain experts.

Event sequences are central to the analysis of data in domains that range from biology and health, to logfile analysis and people's everyday behavior. Many visualization tools have been created for such data, but people are error-prone when asked to judge the similarity of event sequences with basic presentation methods. This article describes an experiment that investigates whether local and global alignment techniques improve people's performance when judging sequence similarity. Participants were divided into three groups (basic versus local versus global alignment), and each participant judged the similarity of 180 sets of pseudo-randomly generated sequences. Each set comprised a target, a correct choice and a wrong choice. After training, the global alignment group was more accurate than the local alignment group (98 versus 93 percent correct), with the basic group getting 95 percent correct. Participants' response times were primarily affected by the number of event types, the similarity of sequences (measured by the Levenshtein distance) and the edit types (nine combinations of deletion, insertion and substitution). In summary, global alignment is superior and people's performance could be further improved by choosing alignment parameters that explicitly penalize sequence mismatches.

Recent work in data visualization has demonstrated that small, perceptually-distinct color palettes-such as those used in categorical mapping-can connote significant affective qualities. Data that are mapped or otherwise visualized are also often emotive in nature, either inherently (e.g., climate change, disease mortality rates), or by design, such as can be found in visual storytelling. However, little is known about how the affective qualities of color interact with those of data context in visualization design. This article describes the results of a crowdsourced study on the influence of affectively congruent versus incongruent color schemes on categorical map-reading response. We report both objective (pattern detection; area comparison) and subjective (affective quality; appropriateness; preference) measures of map-reader response. Our results suggest that affectively congruent colors amplify perceptions of the affective qualities of maps with emotive topics, affective incongruence may cause confusion, and that affective congruence is particularly influential in maps of positive-leaning data topics. Finally, we offer preliminary design recommendations for balancing color congruence with other design factors, and for synthesizing color and affective context in thematic map design.

Omnidirectional images (also referred to as static 360{\deg} panoramas) impose viewing conditions much different from those of regular 2D images. How do humans perceive image distortions in immersive virtual reality (VR) environments is an important problem which receives less attention. We argue that, apart from the distorted panorama itself, two types of VR viewing conditions are crucial in determining the viewing behaviors of users and the perceived quality of the panorama: the starting point and the exploration time. We first carry out a psychophysical experiment to investigate the interplay among the VR viewing conditions, the user viewing behaviors, and the perceived quality of 360{\deg} images. Then, we provide a thorough analysis of the collected human data, leading to several interesting findings. Moreover, we propose a computational framework for objective quality assessment of 360{\deg} images, embodying viewing conditions and behaviors in a delightful way. Specifically, we first transform an omnidirectional image to several video representations using different user viewing behaviors under different viewing conditions. We then leverage advanced 2D full-reference video quality models to compute the perceived quality. We construct a set of specific quality measures within the proposed framework, and demonstrate their promises on three VR quality databases., Comment: 11 pages, 11 figure, 9 tables. This paper has been accepted by IEEE Transactions on Visualization and Computer Graphics

Data visualizations are often represented in public discourse as objective proof of facts. However, a visualization is only a single translation of reality, just like any other media, representation devices, or modes of representation. If we wish to encourage thoughtful, informed, and literate consumption of data visualizations, it is crucial that we consider why they are often presented and interpreted as objective. We reflect theoretically on data visualization as a system of representation historically anchored in science, rationalism, and notions of objectivity. It establishes itself within a lineage of conventions for visual representations which extends from the Renaissance to the present and includes perspective drawing, photography, cinema and television, as well as computer graphics. By examining our tendency to see credibility in data visualizations and grounding that predisposition in a historical context, we hope to encourage more critical and nuanced production and interpretation of data visualizations in the public discourse.

We present a new framework for online dense 3D reconstruction of indoor scenes by using only depth sequences. This research is particularly useful in cases with a poor light condition or in a nearly featureless indoor environment. The lack of RGB information makes long-range camera pose estimation difficult in a large indoor environment. The key idea of our research is to take advantage of the geometric prior of Manhattan scenes in each stage of the reconstruction pipeline with the specific aim to reduce the cumulative registration error and overall odometry drift in a long sequence. This idea is further boosted by local Manhattan frame growing and the local-to-global strategy that leads to implicit loop closure handling for a large indoor scene. Our proposed pipeline, namely ManhattanFusion, starts with planar alignment and local pose optimization where the Manhattan constraints are imposed to create detailed local segments. These segments preserve intrinsic scene geometry by minimizing the odometry drift even under complex and long trajectories. The final model is generated by integrating all local segments into a global volumetric representation under the constraint of Manhattan frame-based registration across segments. Our algorithm outperforms others that use depth data only in terms of both the mean distance error and the absolute trajectory error, and it is also very competitive compared with RGB-D based reconstruction algorithms. Moreover, our algorithm outperforms the state-of-the-art in terms of the surface area coverage by 10-40 percent, largely due to the usefulness and effectiveness of the Manhattan assumption through the reconstruction pipeline.

Head-mounted loupes can increase the user's visual acuity to observe the details of an object. On the other hand, optical see-through head-mounted displays (OST-HMD) are able to provide virtual augmentations registered with real objects. In this article, we propose AR-Loupe, combining the advantages of loupes and OST-HMDs, to offer augmented reality in the user's magnified field-of-vision. Specifically, AR-Loupe integrates a commercial OST-HMD, Magic Leap One, and binocular Galilean magnifying loupes, with customized 3D-printed attachments. We model the combination of user's eye, screen of OST-HMD, and the optical loupe as a pinhole camera. The calibration of AR-Loupe involves interactive view segmentation and an adapted version of stereo single point active alignment method (Stereo-SPAAM). We conducted a two-phase multi-user study to evaluate AR-Loupe. The users were able to achieve sub-millimeter accuracy ( 0.82 mm) on average, which is significantly ( ) smaller compared to normal AR guidance ( 1.49 mm). The mean calibration time was 268.46 s. With the increased size of real objects through optical magnification and the registered augmentation, AR-Loupe can aid users in high-precision tasks with better visual acuity and higher accuracy.

Simulating shadow interactions between real and virtual objects is important for augmented reality (AR), in which accurately and efficiently detecting real shadows from live videos is a crucial step. Most of the existing methods are capable of processing only scenes captured under a fixed viewpoint. In contrast, this paper proposes a new framework for shadow detection in live outdoor videos captured under moving viewpoints. The framework splits each frame into a tracked region, which is the region tracked from the previous video frame through optical flow analysis, and an emerging region, which is newly introduced into the scene due to the moving viewpoint. The framework subsequently extracts features based on the intensity profiles surrounding the boundaries of candidate shadow regions. These features are then utilized to both correct erroneous shadow boundaries for the tracked region and to detect shadow boundaries for the emerging region by a Bayesian learning module. To remove spurious shadows, spatial layout constraints are further considered for emerging regions. The experimental results demonstrate that the proposed framework outperforms the state-of-the-art shadow tracking and detection algorithms on a variety of challenging cases in real time, including shadows on backgrounds with complex textures, nonplanar shadows, fast-moving shadows with changing typologies, and shadows cast by nonrigid objects. The quantitative experiments show that our method outperforms the best existing method, achieving a 33.3% increase in the average F_{measure}. Coupled with an image-based shadow-casting method, the proposed framework generates realistic shadow interaction results. This capability will be particularly beneficial for supporting AR.

Mirror therapy is increasingly used in stroke rehabilitation to improve functional movements of the affected limb. However, the extent of mirroring in conventional mirror therapy is typically fixed (1:1) and cannot be tailored based on the patient’s impairment level. Further, the movements of the affected limb are not actively incorporated in the therapeutic process. To address these issues, we developed an immersive VR system using HTC Vive and Leap Motion, which communicates with our free and open-source software environment programmed using SteamVR and the Unity 3D gaming engine. The mirror therapy VR environment was incorporated with two novel features: (1) scalable mirroring and (2) shared control. In the scalable mirroring, mirror movements were programmed to be scalable between 0 and 1, where 0 represents no movements, 0.5 represents 50% mirroring, and 1 represents 100% mirroring. In shared control, the contribution of the mirroring limb to the movements was programmed to be scalable between 0 to 1, where 0 represents 100% contribution from the mirroring limb (i.e., no mirroring), 0.5 represents 50% of movements from the mirrored limb and 50% of movements from the mirroring limb, and 1 represents full mirroring (i.e., no shared movements). Validation experiments showed that these features worked appropriately. The proposed VR-based mirror therapy is the first fully developed system that is freely available to the rehabilitation science community. The scalable and shared control features can diversify mirror therapy and potentially augment the outcomes of rehabilitation, although this needs to be verified through future experiments.

The modeling of atmospheric clouds is one of the crucial elements in the natural phenomena visualization system. Over the years, a wide range of approaches has been proposed on this topic to deal with the challenging issues associated with visual realism and performance. However, the lack of recent review papers on the atmospheric cloud modeling methods available in computer graphics makes it difficult for researchers and practitioners to understand and choose the well-suited solutions for developing the atmospheric cloud visualization system. Hence, we conducted a comprehensive review to identify, analyze, classify, and summarize the existing atmospheric cloud modeling solutions. We selected 113 research studies from recognizable data sources and analyzed the research trends on this topic. We defined a taxonomy by categorizing the atmospheric cloud modeling methods based on the methods' similar characteristics and summarized each of the particular methods. Finally, we underlined several research issues and directions for potential future work. The review results provide an overview and general picture of the atmospheric cloud modeling methods that would be beneficial for researchers and practitioners.

Traditionally, reduced hair simulation methods are either restricted to heuristic approximations or bound to specific hairstyles. We introduce the first CNN-integrated framework for simulating various hairstyles. The approach produces visually realistic hairs with an interactive speed. To address the technical challenges, our hair simulation pipeline is designed as a two-stage process. First, we present a fully-convolutional neural interpolator as the backbone generator to compute dynamic weights for guide hair interpolation. Then, we adopt a second generator to produce fine-scale displacements to enhance the hair details. We train the neural interpolator with a dedicated loss function and the displacement generator with an adversarial discriminator. Experimental results demonstrate that our method is effective, efficient, and superior to the state-of-the-art on a wide variety of hairstyles. We further propose a performance-driven digital avatar system and an interactive hairstyle editing tool to illustrate the practical applications.

Textures contain a wealth of image information and are widely used in various fields such as computer graphics and computer vision. With the development of machine learning, the texture synthesis and generation have been greatly improved. As a very common element in everyday life, wallpapers contain a wealth of texture information, making it difficult to annotate with a simple single label. Moreover, wallpaper designers spend significant time to create different styles of wallpaper. For this purpose, this paper proposes to describe wallpaper texture images by using multi-label semantics. Based on these labels and generative adversarial networks, we present a framework for perception driven wallpaper texture generation and style transfer. In this framework, a perceptual model is trained to recognize whether the wallpapers produced by the generator network are sufficiently realistic and have the attribute designated by given perceptual description; these multi-label semantic attributes are treated as condition variables to generate wallpaper images. The generated wallpaper images can be converted to those with well-known artist styles using CycleGAN. Finally, using the aesthetic evaluation method, the generated wallpaper images are quantitatively measured. The experimental results demonstrate that the proposed method can generate wallpaper textures conforming to human aesthetics and have artistic characteristics., IEEE Transactions on Circuits and Systems for Video Technology

Skeleton‐based action recognition algorithms have been widely applied to human action recognition. Graph convolutional networks (GCNs) generalize convolutional neural networks (CNNs) to non‐Euclidean graphs and achieve significant performance in skeleton‐based action recognition. However, existing GCN‐based models have several issues, such as the topology of the graph is defined based on the natural skeleton of the human body, which is fixed during training, and it may not be applied to different layers of the GCN model and diverse datasets. Besides, the higher‐order information of the joint data, for example, skeleton and dynamic information is not fully utilised. This work proposes a novel multi‐stream adaptive spatial‐temporal attention GCN model that overcomes the aforementioned issues. The method designs a learnable topology graph to adaptively adjust the connection relationship and strength, which is updated with training along with other network parameters. Simultaneously, the adaptive connection parameters are utilised to optimise the connection of the natural skeleton graph and the adaptive topology graph. The spatial‐temporal attention module is embedded in each graph convolution layer to ensure that the network focuses on the more critical joints and frames. A multi‐stream framework is built to integrate multiple inputs, which further improves the performance of the network. The final network achieves state‐of‐the‐art performance on both the NTU‐RGBD and Kinetics‐Skeleton action recognition datasets. The simulation results prove that the proposed method reveals better results than existing methods in all perspectives and that shows the superiority of the proposed method.

Egocentric augmented reality (AR) interfaces are quickly becoming a key asset for assisting high precision activities in the peripersonal space in several application fields. In these applications, accurate and robust registration of computer-generated information to the real scene is hard to achieve with traditional Optical See-Through (OST) displays given that it relies on the accurate calibration of the combined eye-display projection model. The calibration is required to efficiently estimate the projection parameters of the pinhole model that encapsulate the optical features of the display and whose values vary according to the position of the user's eye. In this article, we describe an approach that prevents any parallax-related AR misregistration at a pre-defined working distance in OST displays with infinity focus; our strategy relies on the use of a magnifier placed in front of the OST display, and features a proper parameterization of the virtual rendering camera achieved through a dedicated calibration procedure that accounts for the contribution of the magnifier. We model the registration error due to the viewpoint parallax outside the ideal working distance. Finally, we validate our strategy on a OST display, and we show that sub-millimetric registration accuracy can be achieved for working distances of ±100 mm around the focal length of the magnifier.

The appearance of a real-world feather is the result of light interactions with complex, patterned structures of varying scale; however, these have not yet been modeled for accurate rendering of feathers in computer graphics. Previously published works related to production and research have presented simplified curve models to represent the appearance of feathers. In this work we present why these approaches are not sufficient using imaging from real feathers along with motivation for a specific appearance model for feathers. We also propose and compare a new technique that takes into account the detailed substructures of feathers and their role in rendering an accurate far-field appearance. Our proposed method lends a high degree of photorealism for rendering feathers in visual-effects and content-creation applications.

A commercial head-mounted display (HMD) for virtual reality (VR) presents three-dimensional imagery with a fixed focal distance. The VR HMD with a fixed focus can cause visual discomfort to an observer. In this article, we propose a novel design of a compact VR HMD supporting near-correct focus cues over a wide depth of field (from 18 cm to optical infinity). The proposed HMD consists of a low-resolution binary backlight, a liquid crystal display panel, and focus-tunable lenses. In the proposed system, the backlight locally illuminates the display panel that is floated by the focus-tunable lens at a specific distance. The illumination moment and the focus-tunable lens' focal power are synchronized to generate focal blocks at the desired distances. The distance of each focal block is determined by depth information of three-dimensional imagery to provide near-correct focus cues. We evaluate the focus cue fidelity of the proposed system considering the fill factor and resolution of the backlight. Finally, we verify the display performance with experimental results.

The representation of consistent mixed reality (XR) environments requires adequate real and virtual illumination composition in real-time. Estimating the lighting of a real scenario is still a challenge. Due to the ill-posed nature of the problem, classical inverse-rendering techniques tackle the problem for simple lighting setups. However, those assumptions do not satisfy the current state-of-art in computer graphics and XR applications. While many recent works solve the problem using machine learning techniques to estimate the environment light and scene’s materials, most of them are limited to geometry or previous knowledge. This paper presents a CNN-based model to estimate complex lighting for mixed reality environments with no previous information about the scene. We model the environment illumination using a set of spherical harmonics (SH) environment lighting, capable of efficiently represent area lighting. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. Unlike previous CNN-based lighting estimation methods, we propose using a highly optimized deep neural network architecture, with a reduced number of parameters, that can learn high complex lighting scenarios from real-world high-dynamic-range (HDR) environment images. We show in the experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of 7.85 × 10−4 when comparing SH lighting coefficients. We validate our model in a variety of mixed reality scenarios. Furthermore, we present qualitative results comparing relights of real-world scenes.

A growing number of longitudinal cohort studies are generating data with extensive patient observations across multiple timepoints. Such data offers promising opportunities to better understand the progression of diseases. However, these observations are usually treated as general events in existing visual analysis tools. As a result, their capabilities in modeling disease progression are not fully utilized. To fill this gap, we designed and implemented ThreadStates, an interactive visual analytics tool for the exploration of longitudinal patient cohort data. The focus of ThreadStates is to identify the states of disease progression by learning from observation data in a human-in-the-loop manner. We propose a novel Glyph Matrix design and combine it with a scatter plot to enable seamless identification, observation, and refinement of states. The disease progression patterns are then revealed in terms of state transitions using Sankey-based visualizations. We employ sequence clustering techniques to find patient groups with distinctive progression patterns, and to reveal the association between disease progression and patient-level features. The design and development were driven by a requirement analysis and iteratively refined based on feedback from domain experts over the course of a 10-month design study. Case studies and expert interviews demonstrate that ThreadStates can successively summarize disease states, reveal disease progression, and compare patient groups.

Semantic segmentation is a critical component in autonomous driving and has to be thoroughly evaluated due to safety concerns. Deep neural network (DNN) based semantic segmentation models are widely used in autonomous driving. However, it is challenging to evaluate DNN-based models due to their black-box-like nature, and it is even more difficult to assess model performance for crucial objects, such as lost cargos and pedestrians, in autonomous driving applications. In this work, we propose VASS, a Visual Analytics approach to diagnosing and improving the accuracy and robustness of Semantic Segmentation models, especially for critical objects moving in various driving scenes. The key component of our approach is a context-aware spatial representation learning that extracts important spatial information of objects, such as position, size, and aspect ratio, with respect to given scene contexts. Based on this spatial representation, we first use it to create visual summarization to analyze models' performance. We then use it to guide the generation of adversarial examples to evaluate models' spatial robustness and obtain actionable insights. We demonstrate the effectiveness of VASS via two case studies of lost cargo detection and pedestrian detection in autonomous driving. For both cases, we show quantitative evaluation on the improvement of models' performance with actionable insights obtained from VASS.

The interpretation of deep neural networks (DNNs) has become a key topic as more and more people apply them to solve various problems and making critical decisions. Concept-based explanations have recently become a popular approach for post-hoc interpretation of DNNs. However, identifying human-understandable visual concepts that affect model decisions is a challenging task that is not easily addressed with automatic approaches. We present a novel human-in-the-loop approach to generate user-defined concepts for model interpretation and diagnostics. Central to our proposal is the use of active learning, where human knowledge and feedback are combined to train a concept extractor with very little human labeling effort. We integrate this process into an interactive system, ConceptExtract. Through two case studies, we show how our approach helps analyze model behavior and extract human-friendly concepts for different machine learning tasks and datasets and how to use these concepts to understand the predictions, compare model performance and make suggestions for model refinement. Quantitative experiments show that our active learning approach can accurately extract meaningful visual concepts. More importantly, by identifying visual concepts that negatively affect model performance, we develop the corresponding data augmentation strategy that consistently improves model performance., Comment: 11 pages, 10 figures

The combination of diverse data types and analysis tasks in genomics has resulted in the development of a wide range of visualization techniques and tools. However, most existing tools are tailored to a specific problem or data type and offer limited customization, making it challenging to optimize visualizations for new analysis tasks or datasets. To address this challenge, we designed Gosling—a grammar for interactive and scalable genomics data visualization. Gosling balances expressiveness for comprehensive multi-scale genomics data visualizations with accessibility for domain scientists. Our accompanying JavaScript toolkit called Gosling.js provides scalable and interactive rendering. Gosling.js is built on top of an existing platform for web-based genomics data visualization to further simplify the visualization of common genomics data formats. We demonstrate the expressiveness of the grammar through a variety of real-world examples. Furthermore, we show how Gosling supports the design of novel genomics visualizations. An online editor and examples of Gosling.js, its source code, and documentation are available at https://gosling.js.org.

International audience; Situated visualization is an emerging concept within visualization, in which data is visualized in situ, where it is relevant to people. The concept has gained interest from multiple research communities, including visualization, human-computer interaction (HCI) and augmented reality. This has led to a range of explorations and applications of the concept, however, this early work has focused on the operational aspect of situatedness leading to inconsistent adoption of the concept and terminology. First, we contribute a literature survey in which we analyze 44 papers that explicitly use the term "situated visualization" to provide an overview of the research area, how it defines situated visualization, common application areas and technology used, as well as type of data and type of visualizations. Our survey shows that research on situated visualization has focused on technology-centric approaches that foreground a spatial understanding of situatedness. Secondly, we contribute five perspectives on situatedness (space, time, place, activity, and community) that together expand on the prevalent notion of situatedness in the corpus. We draw from six case studies and prior theoretical developments in HCI. Each perspective develops a generative way of looking at and working with situatedness in design and research. We outline future directions, including considering technology, material and aesthetics, leveraging the perspectives for design, and methods for stronger engagement with target audiences. We conclude with opportunities to consolidate situated visualization research.

The main purpose of this paper is to design an innovative ship handling simulation program by combining the development of Quest3D software and the Google earth harbor terrain software. The paper describes the usage of the Google earth program to retrieve the latitude and longitude elevation, through GIS conversion and 3D graphical model to generate the Quest 3D harbor construction blueprint. Projectors display the image with build-in formula calculated angle settings to achieve a better perspective for the projectors installation. A 3D virtual reality scene was built by using the Maya Model, and then we overlap with the harbor module to complete the famous world ports. The results will be tested in the ship handling simulator machine at the 3D virtual reality software development laboratory.

Significant work has been done towards deep learning (DL) models for automatic lung and lesion segmentation and classification of COVID-19 on chest CT data. However, comprehensive visualization systems focused on supporting the dual visual+DL diagnosis of COVID-19 are non-existent. We present COVID-view, a visualization application specially tailored for radiologists to diagnose COVID-19 from chest CT data. The system incorporates a complete pipeline of automatic lungs segmentation, localization/ isolation of lung abnormalities, followed by visualization, visual and DL analysis, and measurement/quantification tools. Our system combines the traditional 2D workflow of radiologists with newer 2D and 3D visualization techniques with DL support for a more comprehensive diagnosis. COVID-view incorporates a novel DL model for classifying the patients into positive/negative COVID-19 cases, which acts as a reading aid for the radiologist using COVID-view and provides the attention heatmap as an explainable DL for the model output. We designed and evaluated COVID-view through suggestions, close feedback and conducting case studies of real-world patient data by expert radiologists who have substantial experience diagnosing chest CT scans for COVID-19, pulmonary embolism, and other forms of lung infections. We present requirements and task analysis for the diagnosis of COVID-19 that motivate our design choices and results in a practical system which is capable of handling real-world patient cases., Comment: 11 pages, 10 figures, accepted to IEEE VIS 2021 conference and IEEE Transactions on Visualization and Computer Graphics

Knowing the robot's pose is a crucial prerequisite for mobile robot tasks such as collision avoidance or autonomous navigation. Using powerful predictive models to estimate transformations for visual odometry via downward facing cameras is an understudied area of research. This work proposes a novel approach based on deep learning for estimating ego motion with a downward looking camera. The network can be trained completely unsupervised and is not restricted to a specific motion model. We propose two neural network architectures based on the Early Fusion and Slow Fusion design principle: “EarlyBird” and “SlowBird”. Both networks share a Spatial Transformer layer for image warping and are trained with a modified structural similarity index (SSIM) loss function. Experiments carried out in simulation and for a real world differential drive robot show similar and partially better results of our proposed deep learning based approaches compared to a state-of-the-art method based on fast Fourier transformation.

A novel optical-tracking technique is presented for aerodynamic measurements on free-flying models in hypersonic wind tunnels. This technique is a generalization of the edge-fitting concept develop...

In recent years, virtual reality (VR) has been reaching its maturity level for practical applications amongst many fields of studies, especially in the exploration system of applied VR in cultural heritage that enables virtual walkthroughs. However, this study remains scattered and limited in the area of applied VR in cultural heritage. Therefore, a comprehensive systematic review is conducted to create a coherent taxonomy within this research scope through extensive article searches on VR, virtual heritage, and usability. Articles were searched from four databases, consisting of an expansive collection of literature covering studies from 2007 to January 2020. A total of 290 articles were obtained from the final data search and categorized into 4 taxonomy schemes classifications. The classes consist of VR-related review or survey articles, case studies, systems design and development, and evaluation studies. This paper also presents the robust and generic motivations based on the derived taxonomy of the integration of VR in cultural heritage in the context of software and hardware applications, designs, developments and assessments. The article searches also ascertained that applied VR in cultural heritage could broadly be utilized to provide customizable multimedia contents inside the VR environments, resulting in a higher degree of immersion, and this sense of presence, especially in the reconstruction of cultural heritage via applied VR. Henceforth, VR in cultural heritage has the power to extend its capabilities of influencing users by capturing their attention, which are essential functionalities in multimedia VR systems designed for transferring historical information in a more enriched, immersive, and effective manner.

The strong feature representation ability of deep learning enables content-based image retrieval (CBIR) to achieve higher retrieval accuracy, while there are still some challenges for CBIR such as high requirements of training labels and retrieve efficiency. In this paper, we propose an unsupervised adversarial image retrieval (UAIR) framework by breaking the limitation of training labels. The framework is composed of two opposite parts and is linked by an adversarial loss function. For each input image, a generative model is used to select “well-matched” images from the database; a discriminative model is used to distinguish whether the selected images are similar enough to the input image. During training, the generative model tries to convince the discriminative model that the selected images are similar and the discriminative model always challenges the results of the generative model. The performances of the UAIR have been compared with other state-of-the-art image retrieval methods, including recently reported GAN-based methods. Extensive experiments show that the UAIR achieves significant improvement in CBIR with unsupervised adversarial training.

Haze and fog are big reasons for road accidents. The haze occurrence in the air lowers the images quality captured by visible camera sensors. Haze brings inconvenience to numerous computer vision applications as it diminishes the scene visibility. Haze removal techniques recuperate the color and scene contrast. These haze removal techniques are extensively utilized in numerous applications like outdoor surveillance, object detection, consumer electronics, etc. Haze removal is commonly performed under the physical degradation model, which requires a solution of an ill-posed inverse issue. Different dehazing algorithms was recently proposed to relieve this difficulty and has acknowledged a great deal of consideration. Dehazing is basically accomplished through four major steps: hazy images acquisition process, estimation process (atmospheric light, transmission map, scattering phenomenon, and visibility or haze level), enhancement process (improved visibility level, reduce haze or noise level), restoration process (restore enhanced image, image reconstruction). This four-step dehazing process makes it possible to provide a step-by-step approach to the complex solution of the ill-posed inverse problem. Our detailed survey and experimental analysis on different dehazing methods that will help readers understand the effectiveness of the individual step of the dehazing process and will facilitate development of advanced dehazing algorithms. The overall objective of this review paper is to explore the various methods for efficiently removing the haze and short comings of the earlier presented techniques used in the revolutionary era of image processing applications.

Condition assessment is an essential process to comprehend the condition of the water pipelines and facilitate the maintenance as well as the renewal plans. Nowadays, varied in-pipe inspection platforms equipped with closed-circuit cameras are employed to capture the internal condition of the water pipelines. However, the automated platform often faces the challenge to negotiate with the installed valves during the inspection. To ensure continuous inspection, the platform needs identify the valve automatically and activate the control mechanism to pass through it. Thus, the valves need to be detected to facilitate the negotiation and ensure that the control mechanism can take an action in time. This paper focuses on real-time valve detection using Jetson TX2™ and a lightweight algorithm, namely YOLOv3-tiny. The performance of the implementation is compared with state-of-the-art real-time detection models. The experimental results demonstrate that YOLOv3-tiny has a high detection speed in frame per second for valve detection and outperforms the state-of-the-art real-time algorithms. Hence, the deployment YOLOv3-tiny into the embedded system will aid the automated platform to accomplish the uninterrupted inspection and enhance the capability for the condition assessment of the water pipelines.

Moving object detection in low-luminance Images is one of the most fundamental and difficult issues in machine vision. Therefore, in this paper, deep self-adaptive network (DSA-Net) is proposed to effectively detect moving objects in low-luminance scenes. Particularly, (1) three mechanisms are developed in this joint learning framework: graph-based unsupervised feature selection, feature representations ranking, and multiple-way feature interaction. (2) Both anti-occlusion and multi-object handling module are explored simultaneously in the unified DSA-Net model. (3) A weakly fine-tuning strategy is presented, including the easiness and group curriculum term. It leverages helpful prior-knowledge to guide the learner to select confident training samples. The experimental results show that DSA-Net outperforms the state-of-the-art methods.

Although human pose estimation has achieved great success, the ambiguity of joint prediction has not been well resolved, especially in complex situations (crowded scenes, occlusions, and unnormal poses). We think that is caused by the noisy information introduced by combining multi-level features by simply adding features at each position. To alleviate this problem, we propose a new structure of gated multi-scale feature fusion (GMSFF). This module aims to selectively import high-level features to make up for the missing semantic information of low-resolution feature maps. Inspired by the prior knowledge that the position information of joints can refer to each other, we propose a new fine-tuning strategy for pose estimation—spatial mutual information complementary module (SMICM). It can assist the model in better adjusting the current joint’s position by capturing the information contained in other joints and only adds a little computational cost. We evaluated our proposed method on four datasets: MPII Human Pose Dataset (MPII), COCO keypoint detection Dataset (COCO), Occluded Human Dataset (OCHuman), and CrowdPose Dataset. The experimental results show that with the deepening of the occlusion and crowding level of the datasets, the improvement becomes more and more obvious. In particular, a performance improvement of 2.2 AP was obtained on the OCHuman dataset. In addition, our modules are plug-and-play.

With the application of magnetic resonance imaging (MRI) in the diagnosis of brain diseases, MRI has become a powerful tool for clinical neuroimaging analysis, and whole brain segmentation is a very important step of neuroimaging analysis. In order to promote segmentation accuracy, this paper makes full use of the spatial constraint information between adjacent slice of MRI brain image sequence, and combines three consecutive images into one image named 2.5D slice image. Through 2.5D slice image, a triple U-Net-based whole brain segmentation method is proposed, which is composed of one main U-Net and two auxiliary U-Nets. Moreover, this network introduces three auxiliary outputs to provide inter-layer constraints for the final prediction, and uses a multi-auxiliary hybrid loss function based on binary cross-entropy loss and dice loss to optimize the training model. In this paper, a comprehensive comparative experiment is carried out on LPBA40 dataset and IBSR18 dataset. The experimental results show that the dice coefficient, specificity and sensitivity of this method are 98.23%, 99.62% and 98.54%, respectively, on LPBA40 dataset, and 97.01%, 99.45% and 98%, respectively, on IBSR18 dataset. The experiments show that the accuracy of whole brain segmentation is greatly improved by the proposed triple U-Net.

Multimedia data play an important role in many areas such as business, medicine, and entertainment. Multimedia contents are often vulnerable to malicious interventions due to the advancement in multimedia innovations. It is, therefore, highly necessary to verify the integrity and authenticity of multimedia content through appropriate security measures. This increases the need to establish a range of watermarking schemes for several security purposes, including copyrights and authentication. Digital watermarking has proven an effective and influential approach for 3D models among various other authentication strategies. In this study, an in-depth review of different types of 3D model representations, watermarking techniques in spectral and spatial domains along with most common attacks that can be applied to them are presented. The paper concludes with a set of observations along with some of the research issues and challenges which are needed to be studied further.

A traditional image steganography embeds secret information into a cover image to generate a secret-embedded image. The modification traces imposed on the cover image can be easily detected by steganalysis tools. Coverless steganography has been introduced to solve this problem. In this study, coverless steganography is combined with image style transfer, an arbitrary image style transfer network CSST-Net is put forward, and a secret information is encoded into the parameters (an adaptive steganography matrix) of CSST-Net, which is used to restrict the style transfer. Arbitrary image style transfer is performed instructed by the adaptive steganography matrix, and the image style transfer result driven by secret information is directly synthesized. Our experiments show that CSST-Net can not only synthesize any image style transfer result with good visual effect, but also achieve good performance in capacity, anti-steganalysis and security.