Search

High dynamic range (HDR) panoramic environment maps are widely used to illuminate virtual objects to blend with real-world scenes. However, in common applications for augmented and mixed-reality (AR/MR), capturing 360�� surroundings to obtain an HDR environment map is often not possible using consumer-level devices. We present a novel light estimation method to predict 360�� HDR environment maps from a single photograph with a limited field-of-view (FOV). We introduce the Dynamic Lighting network (DLNet), a convolutional neural network that dynamically generates the convolution filters based on the input photograph sample to adaptively learn the lighting cues within each photograph. We propose novel Spherical Multi-Scale Dynamic (SMD) convolutional modules to dynamically generate sample-specific kernels for decoding features in the spherical domain to predict 360�� environment maps. Using DLNet and data augmentations with respect to FOV, an exposure multiplier, and color temperature, our model shows the capability of estimating lighting under diverse input variations. Compared with prior work that fixes the network filters once trained, our method maintains lighting consistency across different exposure multipliers and color temperature, and maintains robust light estimation accuracy as FOV increases. The surrounding lighting information estimated by our method ensures coherent illumination of 3D objects blended with the input photograph, enabling high fidelity augmented and mixed reality supporting a wide range of environmental lighting conditions and device sensors.

Corresponding lighting and reflectance between real and virtual objects is important for spatial presence in augmented and mixed reality (AR and MR) applications. We present a method to reconstruct real-world environmental lighting, encoded as a reflection map (RM), from a conventional photograph. To achieve this, we propose a stacked convolutional neural network (SCNN) that predicts high dynamic range (HDR) 360 ${}^\circ$ ∘ RMs with varying roughness from a limited field of view, low dynamic range photograph. The SCNN is progressively trained from high to low roughness to predict RMs at varying roughness levels, where each roughness level corresponds to a virtual object’s roughness (from diffuse to glossy) for rendering. The predicted RM provides high-fidelity rendering of virtual objects to match with the background photograph. We illustrate the use of our method with indoor and outdoor scenes trained on separate indoor/outdoor SCNNs showing plausible rendering and composition of virtual objects in AR/MR. We show that our method has improved quality over previous methods with a comparative user study and error metrics.

We present a novel mixed reality (MR) rendering and composition solution that illuminates and blends virtual objects into underwater 360�� videos (360-video) in real-time. Real-time underwater lighting (caustics, god rays, fog, and particulates) were developed to improve the overall lighting and blending quality. We also provide a MR toolkit, an interface to tune the parameters of the underwater lighting so the user can match the lighting observed in the 360-video. Our image based lighting provides automatic ambient and high frequency underwater lighting. This ensures that the virtual objects are lit and blend similarly to each frame of the video semi-automatically and in real-time. We conducted a user study by having participants rate our method based on the visual quality and presence using a five point Likert Scale. The results show that our underwater lighting is preferred over no underwater effects or using naive ambient lighting. We also have a few takeaways on what elements of our underwater lighting and interaction have a significant impact on visual quality and presence in underwater MR.

Convolutional Neural Networks using RGB-D images as input have shown superior performance in recent research in the field of semantic segmentation. In RGB-D data, the depth channel encodes information from the 3D spatial domain, which has an inherent difference with the color channels. It thus needs to be treated in a special way, rather than just processed as another channel of the input signal. Under this purpose, we propose a simple but not trivial edge-aware convolutional kernel to utilize the geometric information contained in the depth channel to extract feature maps in a more effective manner. The edge-aware convolutional kernel is built upon regular convolutional kernel, thus, it can be used to restructure existing CNN models to achieve stable and effective feature extraction for RGB-D data. We compare our result with a previous method that is closely related to our to show our method can provide more effective and stable feature extraction.

High dynamic range (HDR) environment maps (EMs) are spherical textures containing HDR pixels used for illuminating virtual scenes with high realism. Detecting as few necessary pixels as possible within the EM is important for a variety of tasks, such as real-time rendering and EM database management. To address this, we propose a shadow-based algorithm for detecting the most dominant light sources within an EM. This algorithm takes into account the relative impact of all other light sources within the upper-hemisphere of the texture. This is achieved by decomposing an EM into superpixels, sorting the superpixels from brightest to least, and using l0-norm minimisation to keep only the necessary superpixels that maintains the shadow quality of the EM with respect to the just noticeable difference (JND) principle. We show that our method improves upon prior methods in detecting as few lights as possible while still preserving the shadow-casting properties of EMs.

We present a novel approach for adapting existing visual object trackers (VOT) to work for equirectangular video, utilizing image reprojection. Our system can easily be integrated with existing VOT algorithms, significantly increasing the accuracy and robustness of tracking in spherical 360° environments without requiring retraining. Our adapted approach involves the orthographic projection of a subsection of the image centered around the tracked object each frame. Our projection reduces the distortion around the tracked object each frame, allowing the VOT algorithm to more easily track the object as it moves.

Path tracing provides photo-realistic rendering in many applications but intermediate previsualization often suffers from distracting noise. Since the fundamental underlying problem is insufficient samples, we exploit the coherence of the visual signal to reconstruct missing samples, using a low-rank matrix completion framework. We present novel methods to construct low rank matrices for incomplete images including missing pixel, missing sub-pixel, and multi-frame scenarios. A convolutional neural network provides fast pre-completion for initialising missing values, and subsequent weighted nuclear norm minimisation (WNNM) with a parameter adjustment strategy (PAWNNM) efficiently recovers missing values even in high frequency details. The result shows better visual quality than recent methods including compressed sensing based reconstruction.

This paper presents a novel immersive system called MR360 that provides interactive mixed reality (MR) experiences using a conventional low dynamic range (LDR) 360° panoramic video (360-video) shown in head mounted displays (HMDs). MR360 seamlessly composites 3D virtual objects into a live 360-video using the input panoramic video as the lighting source to illuminate the virtual objects. Image based lighting (IBL) is perceptually optimized to provide fast and believable results using the LDR 360-video as the lighting source. Regions of most salient lights in the input panoramic video are detected to optimize the number of lights used to cast perceptible shadows. Then, the areas of the detected lights adjust the penumbra of the shadow to provide realistic soft shadows. Finally, our real-time differential rendering synthesizes illumination of the virtual 3D objects into the 360-video. MR360 provides the illusion of interacting with objects in a video, which are actually 3D virtual objects seamlessly composited into the background of the 360-video. MR360 was implemented in a commercial game engine and tested using various 360-videos. Since our MR360 pipeline does not require any pre-computation, it can synthesize an interactive MR scene using a live 360-video stream while providing realistic high performance rendering suitable for HMDs.

Creating realistic 3D face models is a challenging problem in computer graphics because humans are so sensitive to facial abnormalities. The authors propose a method to synthesize a 3D face model using weighted blending of multiscale details from different face models. Using multiscale continuous displacement maps (CDMs), they achieve full correspondences across multiple scales in the parameter space. Their results demonstrate detail transfer across faces with highly different proportions, such as between humans and nonhuman creatures. An artist evaluation also indicated the proposed approach is intuitive and easy to use.

Creating an accurate virtual human body model is challenging but required in many fields. This study presents a method to create 3D human body models from medical image scans. Visible light scanning of articulated 3D objects such as the human hand has limitation due to the self-occlusion of surfaces in certain poses. We present a complete system to create a deformable articulated human body volume from multiple 3D MRI scans of a living person, which can produce accurate volume deformation containing inner anatomical layers. The method combines technologies involving medical imaging, and computer vision, as well as computer graphics, to address the practical issues involved in producing detailed volume models from living human scans. The results provide an occlusion free person-specific 3D human body model, asymptotically accurate inner tissue deformations, and realistic volume animation of articulated movements driven by standard joint control estimated from the actual skeleton.

This paper presents a multi-view acquisition system using multi-modal sensors, composed of time-of-flight (ToF) range sensors and color cameras. Our system captures the multiple pairs of color images and depth maps at multiple viewing directions. In order to ensure the acceptable accuracy of measurements, we compensate errors in sensor measurement and calibrate multi-modal devices. Upon manifold experiments and extensive analysis, we identify the major sources of systematic error in sensor measurement and construct an error model for compensation. As a result, we provide a practical solution for the real-time error compensation of depth measurement. Moreover, we implement the calibration scheme for multi-modal devices, unifying the spatial coordinate for multi-modal sensors.

This paper describes a complete system to create anatomically accurate example-based volume deformation and animation of articulated body regions, starting from multiple in vivo volume scans of a specific individual. In order to solve the correspondence problem across volume scans, a template volume is registered to each sample. The wide range of pose variations is first approximated by volume blend deformation (VBD), providing proper initialization of the articulated subject in different poses. A novel registration method is presented to efficiently reduce the computation cost while avoiding strong local minima inherent in complex articulated body volume registration. The algorithm highly constrains the degrees of freedom and search space involved in the nonlinear optimization, using hierarchical volume structures and locally constrained deformation based on the biharmonic clamped spline. Our registration step establishes a correspondence across scans, allowing a data-driven deformation approach in the volume domain. The results provide an occlusion-free person-specific 3D human body model, asymptotically accurate inner tissue deformations, and realistic volume animation of articulated movements driven by standard joint control estimated from the actual skeleton. Our approach also addresses the practical issues arising in using scans from living subjects. The robustness of our algorithms is tested by their applications on the hand, probably the most complex articulated region in the body, and the knee, a frequent subject area for medical imaging due to injuries.

This paper presents an effective approach to rendering virtual 3D objects using real-time image based lighting IBL with conventional 360$$^{\circ }$$ panoramic video. Raw 360$$^{\circ }$$ panoramic video captured in a low dynamic range setup is the only light source used for the real-time IBL rendering. Input video data is boosted to high dynamic range using inverse tone mapping. This converted video is then reconstructed into low-resolution diffuse radiance maps to speed up diffuse rendering. A mipmap-based specular sampling scheme provides fast GPU rendering even for glossy specular objects. Since our pipeline does not require any precomputation, it can support a live 360$$^{\circ }$$ panoramic video stream as the radiance map, and the process fits easily into a standard rasterization pipeline. The results provide sufficient performance for IBL via stereo head mounted display HMD, an ideal device for immersive augmented reality films and games using 360$$^{\circ }$$ panoramic videos as both the lighting and backdrop for illumination composition.

This paper presents a novel method to synthesise a high dynamic range (HDR) spherical radiance map (RM) from a single low dynamic range (LDR) legacy photograph of an outdoor scene. The synthesised HDR RM is then used for real-time image based lighting (IBL) to render 3D virtual objects with plausible illumination matching with the background outdoor scene. Seam Carving (SC) is utilised to extend the legacy photograph to a panoramic image having a wider field of view for both backdrop and RM. Since SC is a content-aware image resizing technique, it effectively preserves important details such as shape and location of the sun in the outdoor scene and minimises distortion while expanding. The boundary of the 360-degree panoramic image used as RM is refined to minimise the seam. The extended LDR RM is then converted into a HDR RM using our novel inverse tone mapping (ITM) that provides fine tuning to simulate sunlight fall-off for sunny outdoor scenes. The synthesised HDR RM provides believable real-world lighting for IBL. With the combination of extended backdrop and real-time IBL, our pipeline can provide a potential solution for augmented reality (AR) rendering via head mounted displays (HMDs), only from a single legacy photograph.

Although head-mounted displays (HMDs) are ideal devices for personal viewing of immersive stereoscopic content, exposure to VR applications on them results in significant discomfort for the majority of people, with symptoms including eye fatigue, headaches, nausea, and sweating. A conflict between accommodation and vergence depth cues on stereoscopic displays is a significant cause of visual discomfort. This article describes the results of an evaluation used to judge the effectiveness of dynamic depth-of-field (DoF) blur in an effort to reduce discomfort caused by exposure to stereoscopic content on HMDs. Using a commercial game engine implementation, study participants report a reduction of visual discomfort on a simulator sickness questionnaire when DoF blurring is enabled. The study participants reported a decrease in symptom severity caused by HMD exposure, indicating that dynamic DoF can effectively reduce visual discomfort.

A new method deforms a 3D liver mesh in an arbitrary phase of respiration. During preprocessing, the method step defines a patient-specific deformation space using two polar shapes of the liver during respiration. 3D magnetic resonance imaging captures patient livers during exhaling and inhaling. Next, using a fully automated nonrigid mesh registration, this method creates the two phases' corresponding surface meshes. Then, it defines the respiration's deformation space by extracting deformation gradients between the exhalation and inhalation meshes. At runtime, the method uses sparse local features suitably obtained from 2D ultrasound imaging to solve the constraint optimization problem that minimizes dissimilarity of deformation gradients between the target deformation and the patient-specific deformation space. Researchers used real patient data to evaluate this method, which could be applicable to image-guided tumor ablations.

The multivariate normal is widely used as the expected distribution of face shape. It has been used for face detection and tracking in computer vision, as a prior for facial animation editing in computer graphics, and as a model in psychological theory. In this contribution we consider the character of the multivariate normal in high dimensions, and show that these applications are not justified. While we provide limited evidence that facial proportions are not Gaussian, this is tangential to our conclusion: even if faces are truly “Gaussian”, maximum a posteriori and other applications and conclusions that assume that typical faces lie near the mean are not valid.

The seamless composition of synthetic objects into real-world scenes requires a high level of detail to convince viewers that the rendered object belongs in the scene. These details include illumination properties that perceptually match the real-world environment. To achieve such effects, image based lighting is often used to emulate real-world lighting. This paper explores the role of lighting, and measures effects of local illumination changes. A psychophysical study is presented that captures the thresholds of the human visual systems ability to perceive inconsistencies in illumination in image composites. Further, a set of optimised parameters for image based lighting is proposed. The results show a significant reduction in memory while maintaining the perceived visual quality of a composite render.

Blending face geometry in different expressions is a popular approach for facial animation in films and games. The quality of the animation relies on the set of blend shape expressions, and creating sufficient blend shapes takes a large amount of time and effort. This paper presents a complete pipeline to create a set of blend shapes in different expressions for a face mesh having only a neutral expression. A template blend shapes model having sufficient expressions is provided and the neutral expression of the template mesh model is registered into the target face mesh using a non-rigid ICP (iterative closest point) algorithm. Deformation gradients between the template and target neutral mesh are then transferred to each expression to form a new set of blend shapes for the target face. We solve optimization problem to consistently map the deformation of the source blend shapes to the target face model. The result is a new set of blend shapes for a target mesh having triangle-wise correspondences between the source face and target faces. After creating blend shapes, the blend shape animation of the source face is retargeted to the target mesh automatically.

We present a novel method for pose estimation of a depth camera through plane features. Since conventional features for color images, mainly points and lines, are not applicable to a depth image, we propose a new type of feature utilizing planar structures in a scene. To measure the accuracy of our method, we generated a synthetic scene and calculated a position error between an estimated location and its ground truth. We also applied our method to the real world scene captured by a depth camera verifying practical usage.

This paper presents a plausible real-time smooth mesh generation method from real scene images, captured from a conventional color and depth camera. We perform statistical outlier removal method to detect holes for the captured raw depth image and fill the detected holes with the weighted mean values of its neighbors. The 3D surface mesh of the real scene is then reconstructed. The initially generated noisy mesh is filtered with our novel multi-weight depth refinement algorithm. The entire pipeline is fully automated from capturing to mesh generation and working at interactive frame rates. Robust results are shown even in the dynamic scene captured from the moving camera.

We propose a novel method for detecting multiple planar structures in a scene from a depth image, and estimating their parametric models in real time. To realize this goal we split an entire depth image into small patches. Initially, we assume that all patches are on the planar structure and estimate their parametric models using least square fitting. We select patches where our assumption is valid and the selected planar patches are iteratively merged and refined in case that they are in the same planar structure. The qualitative and quantitative experiments shows that the proposed method has clear benefits in terms of accuracy and processing time.

We propose a novel markerless 3D facial motion capture system using only one common camera. This system is simple and easy to transfer facial expressions of a user's into virtual world. It has robustly tracking facial feature points associated with head movements. In addition, it estimates high accurate 3D points' locations. We designed novel approaches to the followings; Firstly, for precisely 3D head motion tracking, we applied 3D constraints using a 3D face model on conventional 2D feature points tracking approach, called Active Appearance Model (AAM). Secondly, for dealing with various expressions of a user's, we designed 2D face generic models from around 5000 images data and 3D shape data including symmetric and asymmetric facial expressions. Lastly, for accurately facial expression cloning, we invented a manifold space to successfully transfer 2D low dimensional feature points to 3D high dimensional points. The manifold space is defined by eleven facial expression bases.

This paper describes a complete pipe-line of a practical system for producing real-time facial expressions of a 3D virtual avatar controlled by an actor's live performances. The system handles practical challenges arising from markerless expression captures from a single conventional video camera. For robust tracking, a localized algorithm constrained by belief propagation is applied to the upper face, and an appearance matching technique using a parameterized generic face model is exploited for lower face and head pose tracking. The captured expression features then transferred to high dimensional 3D animation controls using our facial expression space which is a structure-preserving map between two algebraic structures. The transferred animation controls drive facial animation of a 3D avatar while optimizing the smoothness of the face mesh. An example-based face deformation technique produces non-linear local detail deformations on the avatar that are not captured in the movement of the animation controls.

This paper introduces an approach to performance animation that employs a small number of motion sensors to create an easy-to-use system for an interactive control of a full-body human character. Our key idea is to construct a series of online local dynamic models from a prerecorded motion database and utilize them to construct full-body human motion in a maximum a posteriori framework (MAP). We have demonstrated the effectiveness of our system by controlling a variety of human actions, such as boxing, golf swinging, and table tennis, in real time. Given an appropriate motion capture database, the results are comparable in quality to those obtained from a commercial motion capture system with a full set of motion sensors (e.g., XSens [2009]); however, our performance animation system is far less intrusive and expensive because it requires a small of motion sensors for full body control. We have also evaluated the performance of our system by leave-one-out-experiments and by comparing with two baseline algorithms.

This paper presents a novel approach to face modeling for realistic synthesis, powered by a probabilistic face diffuse model and a generic face specular map. We first construct a probabilistic face diffuse model for estimating the albedo and the normals of a face from an unknown input image. Then, we introduce a generic face specular map for estimating the specularity of the face. Using the estimated albedo, normal and specular information, we can synthesize the face under arbitrary lighting and viewing directions realistically. Unlike many existing face modeling techniques, our approach can retain both the diffuse and specular properties of the face without involving an elaborating 3D matching procedure. Thanks to the compact representation and the effective inference scheme, our technique can be applied to many practical applications, such as face normalization, avatar creation and de-identification.

Volume registration of articulated subjects such as the human knee region requires novel methods to accommodate the wide range of movement resulting from skeletal joint rotations. This paper addresses articulated volume deformation with a pair of techniques. Volume skeletal deformation (VSD) is applied first, producing a smooth but approximate deformation of the soft tissue as the underlying skeleton moves, while preserving rigid motion in the segmented bone regions. The resulting deformed volume is used as initialization for a non- rigid deformation step that locally and adaptively adds radial basis function control points to minimize the remaining registration error in any poorly aligned region. The resulting process registers a neutral pose volume to any given bent-knee pose MRI scan.

Articulated body animation with smooth skin deformation is an important topic in computer graphics. This paper presents a pipeline that extends articulated body deformation to the volume graphics domain. The pipeline consists of in-vivo volume scans, kinematic joint estimation, volumetric joint weight computation, soft-tissue volume deformation, and direct volume rendering. The result is a fully articulated body volume driven by intuitive joint control that respects rigid deformation of the bone structures and produces smooth deformations of both the skin surface and the interior soft tissue regions.

The human hand is an important interface with complex shape and movement. In virtual reality and gaming applications the use of an individualized rather than generic hand representation can increase the sense of immersion and in some cases may lead to more effortless and accurate interaction with the virtual world. We present a method for constructing a person-specific model from a single canonically posed palm image of the hand without human guidance. Tensor voting is employed to extract the principal creases on the palmar surface. Joint locations are estimated using extracted features and analysis of surface anatomy. The skin geometry of a generic 3D hand model is deformed using radial basis functions guided by correspondences to the extracted surface anatomy and hand contours. The result is a 3D model of an individual's hand, with similar joint locations, contours, and skin texture.