Search

Background: Redirected jumping (RDJ) allows users to explore virtual environments (VEs) naturally by scaling a small real-world jump to a larger virtual jump with virtual camera motion manipulation, thereby addressing the problem of limited physical space in VR applications. Previous RDJ studies have mainly focused on detection threshold estimation. However, the effect VE or selfrepresentation (SR) has on the perception or performance of RDJs remains unclear. Methods: In this paper, we report experiments to measure the perception (detection thresholds for gains, presence, embodiment, intrinsic motivation, and cybersickness) and physical performance (heart rate intensity, preparation time, and actual jumping distance) of redirected forward jumping under six different combinations of VE (low and high visual richness) and SRs (invisible, shoes, and human-like). Results: Our results indicated that the detection threshold ranges for horizontal translation gains were significantly smaller in the VE with high rather than low visual richness. When different SRs were applied, our results did not suggest significant differences in detection thresholds, but it did report longer actual jumping distances in the invisible body case compared with the other two SRs. In the high visual richness VE, the preparation time for jumping with a human-like avatar was significantly longer than that with other SRs. Finally, some correlations were found between perception and physical performance measures. Conclusions: All these findings suggest that both VE and SRs influence users' perception and performance in RDJ and must be considered when designing locomotion techniques.

In recent years, medical simulators have been widely applied to a broad range of surgery training tasks. However, most of the existing surgery simulators can only provide limited immersive environments with a few pre-processed organ models, while ignoring the instant modeling of various personalized clinical cases, which brings substantive differences between training experiences and real surgery situations. To this end, we present a virtual reality (VR) based surgery simulation system for personalized percutaneous coronary intervention (PCI). The simulation system can directly take patient-specific clinical data as input and generate virtual 3D intervention scenarios. Specially, we introduce a fiber-based patient-specific cardiac dynamic model to simulate the nonlinear deformation among the multiple layers of the cardiac structure, which can well respect and correlate the atriums, ventricles and vessels, and thus gives rise to more effective visualization and interaction. Meanwhile, we design a tracking and haptic feedback hardware, which can enable users to manipulate physical intervention instruments and interact with virtual scenarios. We conduct quantitative analysis on deformation precision and modeling efficiency, and evaluate the simulation system based on the user studies from 16 cardiologists and 20 intervention trainees, comparing it to traditional desktop intervention simulators. The results confirm that our simulation system can provide a better user experience, and is a suitable platform for PCI surgery training and rehearsal.

Background This study proposes a series of geometry and physics modeling methods for personalized cardiovascular intervention procedures, which can be applied to a virtual endovascular simulator. Methods Based on personalized clinical computed tomography angiography (CTA) data, mesh models of the cardiovascular system were constructed semi-automatically. By coupling 4D magnetic resonance imaging (MRI) sequences corresponding to a complete cardiac cycle with related physics models, a hybrid kinetic model of the cardiovascular system was built to drive kinematics and dynamics simulation. On that basis, the surgical procedures related to intervention instruments were simulated using specially-designed physics models. These models can be solved in real-time; therefore, the complex interactions between blood vessels and instruments can be well simulated. Additionally, X-ray imaging simulation algorithms and realistic rendering algorithms for virtual intervention scenes are also proposed. In particular, instrument tracking hardware with haptic feedback was developed to serve as the interaction interface of real instruments and the virtual intervention system. Finally, a personalized cardiovascular intervention simulation system was developed by integrating the techniques mentioned above. Results This system supported instant modeling and simulation of personalized clinical data and significantly improved the visual and haptic immersions of vascular intervention simulation. Conclusions It can be used in teaching basic cardiology and effectively satisfying the demands of intervention training, personalized intervention planning, and rehearsing.

Unsupervised co-segmentation for a set of 3D shapes is a challenging problem as no prior information is provided. The accuracy of the current approaches is necessarily restricted by the accuracy of the unsupervised face classification, which is used to provide an initialization for the following optimization to improve the consistency between adjacent faces. However, it is exceedingly difficult to obtain a satisfactory initialization pre-segmentation owing to variation in topology and geometry of 3D shapes. In this study, we consider the unsupervised 3D shape co-segmentation as an exemplar-based clustering problem, aimed at simultaneously discovering optimal exemplars and obtaining co-segmentation results. Therefore, we introduce a novel exemplar-based clustering method based on affinity propagation for 3D shape co-segmentation, which can automatically identify representative exemplars and patterns in 3D shapes considering the high-order statistics, yielding consistent and accurate co-segmentation results. Experiments using various datasets, especially large sets with 200 or more shapes that would be challenging to manually segment, demonstrate that our method exhibits a better performance compared to state-of-the-art methods.

Online 3D reconstruction of real-world scenes has been attracting increasing interests from both the academia and industry, especially with the consumer-level depth cameras becoming widely available. Recent most online reconstruction systems take live depth data from a moving Kinect camera and incrementally fuse them to a single high-quality 3D model in real time. Although most real-world scenes have static environment, the daily objects in a scene often move dynamically, which are non-trivial to reconstruct especially when the camera is also not still. To solve this problem, we propose a single depth camera-based real-time approach for simultaneous reconstruction of dynamic object and static environment, and provide solutions for its key issues. In particular, we first introduce a robust optimization scheme which takes advantage of raycasted maps to segment moving object and background from the live depth map. The corresponding depth data are then fused to the volumes, respectively. These volumes are raycasted to extract views of the implicit surface which can be used as a consistent reference frame for the next iteration of segmentation and tracking. Particularly, in order to handle fast motion of dynamic object and handheld camera in the fusion stage, we propose a sequential 6D pose prediction method which largely increases the registration robustness and avoids registration failures occurred in conventional methods. Experimental results show that our approach can reconstruct moving object as well as static environment with rich details, and outperform conventional methods in multiple aspects.

This paper presents a new method for 3D model labeling guided by weakly tagged 2D color images. Many previous methods on 3D model labeling achieve impressive performances using large training data sets. However, it is difficult and time-consuming to build such a carefully annotated data set. In order to solve this problem, we take advantage of the large number of weakly tagged color images to label the 3D models. In our approach, we first collect and tag the web color images with semantic annotations. Then we project the input 3D model into multiview projections. Through the multiview alignment, we transfer the semantic labels onto the model projections via a color-weighting process. Combining pre-segment information, we back-project the labels and get final labeling results. Experimental results between two benchmarks show that our approach could get comparable labeling accuracy compared to other two state-of-art methods without expensive training cost.

Despite growing progresses in recent years, cross-scenario person re-identification remains challenging, mainly due to the pedestrians commonly surrounded by highly-complex environment contexts. In reality, the human perception mechanism could adaptively find proper contextualized spatial-temporal clues towards pedestrian recognition. However, conventional methods fall short in adaptively leveraging the long-term spatial-temporal information due to ever-increasing computational cost. Moreover, CNN-based deep learning methods are hard to conduct optimization due to the non-differentiable property of the built-in context search operation. To ameliorate, this paper proposes a novel Context-Interactive CNN (CI-CNN) to dynamically find both spatial and temporal contexts by embedding multi-task Reinforcement Learning (MTRL). The CI-CNN streamlines the multi-task reinforcement learning by using an actor-critic agent to capture the temporal-spatial context simultaneously, which comprises a context-policy network and a context-critic network. The former network learns policies to determine the optimal spatial context region and temporal sequence range. Based on the inferred temporal-spatial cues, the latter one focuses on the identification task and provides feedback for the policy network. Thus, CI-CNN can simultaneously zoom in/out the perception field in spatial and temporal domain for the context interaction with the environment. By fostering the collaborative interaction between the person and context, our method could achieve outstanding performance on various public benchmarks, which confirms the rationality of our hypothesis, and verifies the effectiveness of our CI-CNN framework.

With the increasing availability of high-resolution images, videos, and 3D models, the demand for scalable large data processing techniques increases. We introduce a method of sparse dictionary learning for edit propagation of large input data. Previous approaches for edit propagation typically employ a global optimization over the whole set of pixels (or vertexes), incurring a prohibitively high memory and time-consumption for large input data. Rather than propagating an edit pixel by pixel, we follow the principle of sparse representation to obtain a representative and compact dictionary and perform edit propagation on the dictionary instead. The sparse dictionary provides an intrinsic basis for input data, and the coding coefficients capture the linear relationship between all pixels and the dictionary atoms. The learned dictionary is then optimized by a novel scheme, which maximizes the Kullback-Leibler divergence between each atom pair to remove redundant atoms. To enable local edit propagation for images or videos with similar appearance, a dictionary learning strategy is proposed by considering range constraint to better account for the global distribution of pixels in their feature space. We show several applications of the sparsity-based edit propagation, including video recoloring, theme editing, and seamless cloning, operating on both color and texture features. Our approach can also be applied to computer graphics tasks, such as 3D surface deformation. We demonstrate that with an atom-to-pixel ratio in the order of 0.01% signifying a significant reduction on memory consumption, our method still maintains a high degree of visual fidelity.

A majority of stock 3D models in modern shape repositories are assembled with many fine-grained components. The main cause of such data form is the component-wise modeling process widely practiced by human modelers. These modeling components thus inherently reflect some function-based shape decomposition the artist had in mind during modeling. On the other hand, modeling components represent an over-segmentation since a functional part is usually modeled as a multi-component assembly. Based on these observations, we advocate that labeled segmentation of stock 3D models should not overlook the modeling components and propose a learning solution to grouping and labeling of the fine-grained components. However, directly characterizing the shape of individual components for the purpose of labeling is unreliable, since they can be arbitrarily tiny and semantically meaningless. We propose to generate part hypotheses from the components based on a hierarchical grouping strategy, and perform labeling on those part groups instead of directly on the components. Part hypotheses are mid-level elements which are more probable to carry semantic information. A multiscale 3D convolutional neural network is trained to extract context-aware features for the hypotheses. To accomplish a labeled segmentation of the whole shape, we formulate higher-order conditional random fields (CRFs) to infer an optimal label assignment for all components. Extensive experiments demonstrate that our method achieves significantly robust labeling results on raw 3D models from public shape repositories. Our work also contributes the first benchmark for component-wise labeling., Accepted to SIGGRAPH Asia 2018. Corresponding Author: Kai Xu (kevin.kai.xu@gmail.com)

Lighting is a key factor in creating impressive artistic portraits. In this paper, we propose to analyze portrait lighting by learning templates of lighting styles. Inspired by the experience of artists, we first define several novel features that describe the local contrasts in various face regions. The most informative features are then selected with a stepwise feature pursuit algorithm to derive the templates of various lighting styles. After that, the matching scores that measure the similarity between a testing portrait and those templates are calculated for lighting style classification. Furthermore, we train a regression model by the subjective scores and the feature responses of a template to predict the score of a portrait lighting quality. Based on the templates, a novel face illumination descriptor is defined to measure the difference between two portrait lightings. Experimental results show that the learned templates can well describe the lighting styles, whereas the proposed approach can assess the lighting quality of artistic portraits as human being does.

Clouds are important components of the fascinating natural images. However, extracting cloud shapes from images remains a challenging task. This paper presents a calculation method for estimating the shape of a cumulus cloud from a single image suitable for flight simulations and games. The shape of the cloud is assumed to be symmetric. Based on this assumption, the intensities of pixels are correlated with the geometry of a cloud's surface via a simplified single scattering model. A propagation scheme is designed to derive the surface progressively, and mesh editing techniques are used to improve the surface. Finally, the cloud is represented by a particle system. The results show that the proposed method can generate realistic cumulus clouds that are similar to those found in the images in terms of the shape distribution.

We resolve the canopy-frame interactions for umbrella simulation in this paper. An umbrella is a heterogeneous articulated system that consists of a frame composed of elastic ribs and other rigid components, and a fabric-made canopy bound to the frame. The canopy interacts with the frame through the binding constraint and collisions. Since the flexible canopy and elastic ribs have different stiffness, we employ a bi-time-step scheme to obtain tractable performance. It evolves the frame with a smaller time step but the canopy with a larger time step. We resolve the canopy-frame interactions by changing the canopy positions and velocities at an interval of the larger time step, which is limited to be an integer multiple of the smaller time step. These displacements and velocity changes are transferred into opposite impulses and forces, and applied to the frame in the following smaller time steps. With these transformations, the solution of the canopy-frame interactions is independent of the difference of the two time steps. Moreover, we adopt a projection-based algorithm to accurately detect the canopy-frame collisions. We demonstrate the effectiveness of our method with several umbrella animations.

This paper proposes a novel image-based framework to manipulate the illumination of human face through adaptive layer decomposition. According to our framework, only a single reference image, without any knowledge of the 3D geometry or material information of the input face, is needed. To transfer the illumination effects of a reference face image to a normal lighting face, we first decompose the lightness layers of the reference and the input images into large-scale and detail layers through weighted least squares (WLS) filter with adaptive smoothing parameters according to the gradient values of the face images. The large-scale layer of the reference image is filtered with the guidance of the input image by guided filter with adaptive smoothing parameters according to the face structures. The relit result is obtained by replacing the largescale layer of the input image with that of the reference image. To normalize the illumination effects of a non-normal lighting face (i.e., face delighting), we introduce similar reflectance prior to the layer decomposition stage by WLS filter, which make the normalized result less affected by the high contrast light and shadow effects of the input face. Through these two procedures, we can change the illumination effects of a non-normal lighting face by first normalizing the illumination and then transferring the illumination of another reference face to it. We acquire convincing relit results of both face relighting and delighting on numerous input and reference face images with various illumination effects and genders. Comparisons with previous papers show that our framework is less affected by geometry differences and can preserve better the identification structure and skin color of the input face.

In this paper, we propose a novel method for inextensible cloth simulation. Our method introduces a neighborhood projection optimized with a relaxation factor. The neighborhood projection enforces inextensibility by modifying particle positions with a Jacobi-style iteration, leading to conservation of linear and angular quasi momenta. The relaxation factor is estimated according to the corrections and constraints, and is used to scale the corrections while keeping convergence to a smaller number of iterations. Experimental results demonstrate that our method increases the simulation efficiency, and stably handles inextensible cloth even in overconstrained situations. In addition to the simulation of hanging cloth and draping cloth, the simulated umbrella demonstrates the characters of our method for this type of objects.

This paper systematically advocates an interactive volumetric image manipulation framework, which can enable the rapid deployment and instant utility of patient-specific medical images in virtual surgery simulation while requiring little user involvement. We seamlessly integrate multiple technical elements to synchronously accommodate physics-plausible simulation and high-fidelity anatomical structures visualization. Given a volumetric image, in a user-transparent way, we build a proxy to represent the geometrical structure and encode its physical state without the need of explicit 3-D reconstruction. On the basis of the dynamic update of the proxy, we simulate large-scale deformation, arbitrary cutting, and accompanying collision response driven by a non-linear finite element method. By resorting to the upsampling of the sparse displacement field resulted from non-linear finite element simulation, the cut/deformed volumetric image can evolve naturally and serves as a time-varying 3-D texture to expedite direct volume rendering. Moreover, our entire framework is built upon CUDA Beihang University, Beijing, China and thus can achieve interactive performance even on a commodity laptop. The implementation details, timing statistics, and physical behavior measurements have shown its practicality, efficiency, and robustness. Copyright © 2013 John Wiley & Sons, Ltd.

This paper proposes a new generative model for flexible editing of human motion. Different from previous work, three intuitive factors of motion, namely, content, identity and style, can be manipulated directly with the new model. With the new generative model, motion editing can be achieved in various aspects, including transferring an unknown style from an actor to another, synthesizing other styles for an unknown actor and generating a new motion with other content. Copyright © 2013 John Wiley & Sons, Ltd.

Estimation of human shape from images has numerous applications ranging from graphics to surveillance. A single image provides insufficient constraints (e.g. clothing), making human shape estimation more challenging. We propose a method to simultaneously estimate a person's clothed and naked shapes from a single image of that person wearing clothing. The key component of our method is a deformable model of clothed human shape. We learn our deformable model, which spans variations in pose, body, and clothes, from a training dataset. These variations are derived by the non-rigid surface deformation, and encoded in various low-dimension parameters. Our deformable model can be used to produce clothed 3D meshes for different people in different poses, which neither appears in the training dataset. Afterward, given an input image, our deformable model is initialized with a few user-specified 2D joints and contours of the person. We optimize the parameters of the deformable model by pose fitting and body fitting in an iterative way. Then the clothed and naked 3D shapes of the person can be obtained simultaneously. We illustrate our method for texture mapping and animation. The experimental results on real images demonstrate the effectiveness of our method.

We propose a novel edit propagation algorithm for interactive image and video manipulations. Our approach uses the locally linear embedding (LLE) to represent each pixel as a linear combination of its neighbors in a feature space. While previous methods require similar pixels to have similar results, we seek to maintain the manifold structure formed by all pixels in the feature space. Specifically, we require each pixel to be the same linear combination of its neighbors in the result. Compared with previous methods, our proposed algorithm is more robust to color blending in the input data. Furthermore, since every pixel is only related to a few nearest neighbors, our algorithm easily achieves good runtime efficiency. We demonstrate our manifold preserving edit propagation on various applications.

Relighting a portrait in a single image is still a challenging problem, particularly when only a single artistic reference photograph or painting is provided. In this paper, we propose an artistic illumination transfer system for portraits based on a database of portrait images (photographs and paintings) associated with hand-drawn illumination templates (276) by artists. Users can select a reference portrait image in the database, and the corresponding illumination template is transferred to an input portrait using image warping. Users can also provide reference portrait images those are not in the database. Based on the Face Illumination Descriptor (FID), the system selects from the database the reference image with the closest illumination to that of the user-provided reference image and adjusts the corresponding illumination template to match the contrast of the user-provided reference image. Experiments on not only paintings but also photographs, paper-cuts and sketches demonstrate that convincing illumination transferred results can be rendered by our system. © 2012 Wiley Periodicals, Inc.

This paper presents an approach for video event inference from dozens of actions performed by multiple players. First, we constructed an And-Or graph to describe the different configurations of the event category such as shooting in soccer matches. We considered both temporal relations and role relations for the graph and encode them as vector parameters for each pair of graph nodes. Then, we developed an inference algorithm by using bottom-up and top-down processes. We found the proposals for each node during the bottom-up step by considering three terms of energies and refined the proposals during the top-down step by measuring the action-labeling similarity and the temporal misplacement penalty. The optimal proposal of the inferring event and its score are obtained as the result. In the experiments, we tested the inference performance of the approach for the shooting events on real soccer match videos. By our approach, we can infer different kinds of shooting events in one scenario and interpret them play-by-play in a flexible way. Copyright © 2012 John Wiley & Sons, Ltd.

Virtual calligraphic carving aims to generate 3D virtual carving works from 2D calligraphic images. This paper proposes a method of virtual calligraphic carving taking calligraphic carving work as a depth map composed of smoothness scalar field and brush-pressure distribution, which ensures the smoothness of carving surface and interprets the strength of calligraphy, respectively. On one hand, the smoothness scalar field is modeled as a bounded planar scalar field with sources and constructed by applying a series of algorithms. On the other hand, the brush-pressure distribution is represented by a map created by estimating the touching pressures employed by brush on paper. Then the depth map is composed of the smoothness scalar field and brush-pressure distribution map and rendered into a virtual calligraphic carving work. The experiments show that this method can generate virtual calligraphic carving works that are vivid and very similar to that of artists. Copyright © 2012 John Wiley & Sons, Ltd.

Rendering global illumination for objects with mesostructure surfaces is a time-consuming task, and cannot presently be applied to interactive graphics. This paper presents a real-time rendering method based on a mesostructure height gradient map (MHGM) to exhibit lighting effects on meso-scale details in dynamic environments. We approximate global illumination using a lighting model including three components: incident ambient light, direct light and single bounce indirect light. MHGM is introduced to create local apex sets, which would help us to compute the three components adaptively. Our approach runs entirely on the graphics hardware, and uses deferred shading and the graphics pipeline to accelerate computation. We achieve high quality results which can render meso-scale details with approximate global illumination even for low-resolution geometric models. Moreover, our approach fully supports dynamic scenes and deformable objects.

In this paper, we present a novel automatic image completion solution in a greedy manner inspired by a primal sketch representation model. Firstly, an image is divided into structure (sketchable) components and texture (non-sketchable) components, and the missing structures, such as curves and corners, are predicted by tensor voting. Secondly, the textures along structural sketches are synthesized with the sampled patches of some known structure components. Then, using the texture completion priorities decided by the confidence term, data term and distance term, the similar image patches of some known texture components are found by selecting a point with the maximum priority on the boundary of hole region. Finally, these image patches inpaint the missing textures of hole region seamlessly through graph cuts. The characteristics of this solution include: (1) introducing the primal sketch representation model to guide completion for visual consistency; (2) achieving fully automatic completion. The experiments on natural images illustrate satisfying image completion results.

This paper presents a semantic labeling framework with geodesic propagation (GP). Under the same framework, three algorithms are proposed, including GP, supervised GP (SGP) for image, and hybrid GP (HGP) for video. In these algorithms, we resort to the recognition proposal map and select confident pixels with maximum probability as the initial propagation seeds. From these seeds, the GP algorithm iteratively updates the weights of geodesic distances until the semantic labels are propagated to all pixels. On the contrary, the SGP algorithm further exploits the contextual information to guide the direction of propagation, leading to better performance but higher computational complexity than the GP. For video labeling, we further propose the HGP algorithm, in which the geodesic metric is used in both spatial and temporal spaces. Experiments on four public data sets show that our algorithms outperform several state-of-the-art methods. With the GP framework, convincing results for both image and video semantic labeling can be obtained.