Search

Augmented reality (AR) marker hiding is a technique to visually remove AR markers in a real-time video stream. A conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on an AR marker region in a real-time frame, assuming that the AR marker is on a planar surface. However, this approach may cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries even under nonplanar background geometry without measuring it. For doing this, our method estimates the dense motion in the marker's background by analyzing the motion of sparse feature points around it, together with a smooth motion assumption, and deforms the background image according to it. Our experiments demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures.

Authoring video blogs requires a video editing process, which is cumbersome for ordinary users. Video summarization can automate this process by extracting important segments from original videos. Because bloggers typically have certain stories for their blog posts, video summaries of a blog post should take the author’s intentions into account. However, most prior works address video summarization by mining patterns from the original videos without considering the blog author’s intentions. To generate a video summary that reflects the blog author’s intention, we focus on supporting texts in video blog posts and present a text-based method, in which the supporting text serves as a prior to the video summary. Given video and text that describe scenes of interest, our method segments videos and assigns to each video segment its priority in the summary based on its relevance to the input text. Our method then selects a subset of segments with content that is similar to the input text. Accordingly, our method produces different video summaries from the same set of videos, depending on the input text. We evaluated summaries generated from both blog viewers’ and authors’ perspectives in a user study. Experimental results demonstrate the advantages to the proposed text-based method for video blog authoring.

Indirect augmented reality (IAR) presents pre-generated augmented images for achieving high-quality geometric and photometric registration between pre-captured images and virtual objects. Meanwhile, IAR causes spatial inconsistency between the real world and presented images when users move from the location where the real scene was captured. This paper describes a novel way to address the spatial inconsistency; namely, enabling viewpoint change in IAR. The key idea of this study is to employ an image-based rendering technique using pre-captured multi-view omnidirectional images to provide free-viewpoint navigation. For a pilot study, we have developed an IAR system representing a motion parallax effect using an optical-flow-based camera motion estimation.

The proliferation of off-the-shelf head-mounted displays (HMDs) let end-users enjoy virtual reality applications, some of which render a real-world scene using a novel view synthesis (NVS) technique. View-dependent texture mapping (VDTM) has been studied for NVS due to its photo-realistic quality. The VDTM technique renders a novel view by adaptively selecting textures from the most appropriate images. However, this process is computationally expensive because VDTM scans every captured image. For stereoscopic HMDs, the situation is much worse because we need to render novel views once for each eye, almost doubling the cost. This paper proposes light-weight VDTM tailored for an HMD. In order to reduce the computational cost in VDTM, our method leverages the overlapping fields of view between a stereoscopic pair of HMD images and pruning the images to be scanned. We show that the proposed method drastically accelerates the VDTM process without spoiling the image quality through a user study.

3D models of outdoor environments have been used for several applications such as a virtual earth system and a vision-based vehicle safety system. 3D data for constructing such 3D models are often measured by an on-vehicle system equipped with laser rangefinders, cameras, and GPS/IMU. However, 3D data of moving objects on streets lead to inaccurate 3D models when modeling outdoor environments. To solve this problem, this paper proposes a moving object detection method for point clouds by minimizing an energy function based on photometric and depth consistencies assuming that input data consist of synchronized point clouds, images, and camera poses from a single sequence captured with a moving on-vehicle system.

Augmented reality (AR) marker hiding is a technique to visually remove AR markers in a real-time video stream. A conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on an AR marker region in a real-time frame, assuming that the AR marker is on a planar surface. However, this approach may cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries even under nonplanar background geometry without measuring it. For doing this, our method estimates the dense motion in the marker's background by analyzing the motion of sparse feature points around it, together with a smooth motion assumption, and deforms the background image according to it. Our experiments demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures., journal article

Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible background textures in realtime. Most conventional methods based on image inpainting achieve diminished reality by assuming that the background around a target object is almost planar. This paper proposes a new diminished reality method that considers background geometries with less constraints than the conventional ones. In this study, we approximate the background geometry by combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture and limiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using the geometries and camera pose estimated by visual-simultaneous localization and mapping (SLAM). The mask region that includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments.

Non-photorealistic rendering (NPR) creates images with artistic styles of paintings. In this field, a number of methods of converting photographed images into non-photorealistic ones have been developed, and can be categorized into filter-based and exemplar-based approaches. In this paper, we focus on the exemplar-based approach and propose a novel method which transfers a style of a reference pictorial image to a photographed image. Specifically, we first input a pair of target and reference images. The target image is converted by minimizing an energy function which is defined based on the difference in intensities between an output image and a target image, and the pattern dissimilarity between an output image and a reference image. The proposed method transfers structures and colors of textures in the reference image and generates continuous textures by minimizing the energy function. In experiments, we demonstrate the effectiveness of the proposed method using a variety of images and examine the influence of parameter changes and intensity adjustment for pre-processing on resultant images.

Automatically reconstructing a 3D shape model of a nonrigid object using a sequence from a single commodity RGB-D sensor is a challenging problem. Some techniques use a 3D shape template of a target object; however, in order to generate the template automatically, the target object required to be stationary. Otherwise, a non-rigid ICP algorithm, which registers a pair of point clouds, can be used for reconstructing 3D geometry of a non-rigid object directly, but it often fails due to the ambiguity in point correspondences. This paper presents a method for generating a 3D shape template from a single RGB-D sequence. In order to reduce the ambiguity in point correspondences, our method leverages point trajectories obtained in the RGB images, which can be used for associating points in different point clouds. We demonstrate the capability of our method using deforming human bodies.

Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible background textures in realtime. Most conventional methods based on image inpainting achieve diminished reality by assuming that the background around a target object is almost planar. This paper proposes a new diminished reality method that considers background geometries with less constraints than the conventional ones. In this study, we approximate the background geometry by combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture and limiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using the geometries and camera pose estimated by visual-simultaneous localization and mapping (SLAM). The mask region that includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments., journal article

Augmented reality (AR) applications often require virtualized real objects, i.e., virtual objects that are built based on real objects and rendered from an arbitrary viewpoint. In this paper, we propose a method for real object virtualization and AR image generation based on view-dependent geometry modification and texture mapping. The proposed method is a hybrid of model- and image-based rendering techniques that uses multiple input images of the real object as well as the object's three-dimensional (3D) model obtained by an automatic 3D reconstruction technique. Even with state-of-the-art technology, the reconstructed 3D model's accuracy can be insufficient, resulting in such visual artifacts as false object boundaries. The proposed method generates a depth map from a 3D model of a virtualized real object and expands its region in the depth map to remove the false object boundaries. Since such expansion reveals the background pixels in the input images, which is particularly undesirable for AR applications, we preliminarily extract object regions and use them for texture mapping. With our GPU implementation for real-time AR image generation, we experimentally demonstrated that using expanded geometry reduces the number of required input images and maintains visual quality.

application/pdf, Recent popularization of camera devices, including action cams and smartphones, enables us to record videos in everyday life and share them through the Internet. Video blog is a recent approach for sharing videos, in which users enjoy expressing themselves in blog posts with attractive videos. Generating such videos, however, requires users to review vast amount of raw videos and edit them appropriately, which keeps users away from doing so. In this paper, we propose a novel video summarization method for helping users to create a video blog post. Unlike typical video summarization methods, the proposed method utilizes the text, which is written for a video blog post, and makes the video summary consistent with the content of the text. For this, we perform video summarization by solving an optimization problem, in which an objective function involves the content similarity between the summarized video and the text. Our user study with 20 participants has demonstrated that our proposed method is suitable to create video blog posts compared with conventional methods for video summarization.

application/pdf, Augmented Reality (AR) marker hiding is a technique to visually remove AR markers from a real-time video stream. The conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on the AR marker region in the real-time frame, assuming that the AR marker is on a planar surface. However, this approach can cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries under unknown and nonplanar background geometry. The proposed method estimates dense motion in the marker's background based on feature point correspondences around the marker, together with a smooth motion assumption, and deforms the background image according to the dense motion. Experimental results demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures.

application/pdf, This paper proposes an example-based method to increase the resolution of a low-resolution image. In the proposed method, we generate example images by a novel view synthesis technique using 3D geometry reconstruction and camera pose estimation from a video or images capturing the same scene. We then increase the resolution by minimizing an energy function by searching for the optimal example from the generated example images. The proposed method has less limitations on camera positions and geometry of the target scene than those in conventional methods. Experiments demonstrate the effectiveness of the proposed method by qualitatively comparing the results of the proposed and conventional methods.

Augmented Reality (AR) marker hiding is a technique to visually remove AR markers from a real-time video stream. The conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on the AR marker region in the real-time frame, assuming that the AR marker is on a planar surface. However, this approach can cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries under unknown and nonplanar background geometry. The proposed method estimates dense motion in the marker's background based on feature point correspondences around the marker, together with a smooth motion assumption, and deforms the background image according to the dense motion. Experimental results demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures., conference paper