Search

Augmented reality (AR) applications often require virtualized real objects, i.e., virtual objects that are built based on real objects and rendered from an arbitrary viewpoint. In this paper, we propose a method for real object virtualization and AR image generation based on view-dependent geometry modification and texture mapping. The proposed method is a hybrid of model- and image-based rendering techniques that uses multiple input images of the real object as well as the object's three-dimensional (3D) model obtained by an automatic 3D reconstruction technique. Even with state-of-the-art technology, the reconstructed 3D model's accuracy can be insufficient, resulting in such visual artifacts as false object boundaries. The proposed method generates a depth map from a 3D model of a virtualized real object and expands its region in the depth map to remove the false object boundaries. Since such expansion reveals the background pixels in the input images, which is particularly undesirable for AR applications, we preliminarily extract object regions and use them for texture mapping. With our GPU implementation for real-time AR image generation, we experimentally demonstrated that using expanded geometry reduces the number of required input images and maintains visual quality.

Augmented reality (AR) marker hiding is a technique to visually remove AR markers in a real-time video stream. A conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on an AR marker region in a real-time frame, assuming that the AR marker is on a planar surface. However, this approach may cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries even under nonplanar background geometry without measuring it. For doing this, our method estimates the dense motion in the marker's background by analyzing the motion of sparse feature points around it, together with a smooth motion assumption, and deforms the background image according to it. Our experiments demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures.

Authoring video blogs requires a video editing process, which is cumbersome for ordinary users. Video summarization can automate this process by extracting important segments from original videos. Because bloggers typically have certain stories for their blog posts, video summaries of a blog post should take the author’s intentions into account. However, most prior works address video summarization by mining patterns from the original videos without considering the blog author’s intentions. To generate a video summary that reflects the blog author’s intention, we focus on supporting texts in video blog posts and present a text-based method, in which the supporting text serves as a prior to the video summary. Given video and text that describe scenes of interest, our method segments videos and assigns to each video segment its priority in the summary based on its relevance to the input text. Our method then selects a subset of segments with content that is similar to the input text. Accordingly, our method produces different video summaries from the same set of videos, depending on the input text. We evaluated summaries generated from both blog viewers’ and authors’ perspectives in a user study. Experimental results demonstrate the advantages to the proposed text-based method for video blog authoring.

Detecting important regions in videos has been extensively studied for past decades for their wide variety of applications including video summarization and retargeting. Visual attention models draw much attention for this purpose, which find visually salient regions. However, visual attention models ignore intentionally captured regions (ICRs) derived from videographers' intentions, i.e., what the videographers wanted to capture in their videos. This paper proposes a Markov random field-based ICR model for finding them. Observing that a videographer's intention is embedded into camera motion together with objects' motion, our ICR model uses point trajectory-based features to distinguish ICRs from non-ICRs. It also leverages spatial and temporal consistency of ICRs to improve the performance. We have experimentally demonstrated our ICR model's performance and the difference between ICRs and visually salient regions., ICIP 2013 : 20th IEEE International Conference on Image Processing , Sep 15-18, 2013 , Melbourne, Australia

When observing a person (an actor) performing or demonstrating some activity for the purpose of learning the action, it is best for the viewers to be present at the same time and place as the actor. Otherwise, a video must be recorded. However, conventional video only provides two-dimensional (2D) motion, which lacks the original third dimension of motion. In the presence of some ambiguity, it may be hard for the viewer to comprehend the action with only two dimensions, making it harder to learn the action. This paper proposes an augmented reality system to reenact such actions at any time the viewer wants, in order to aid comprehension of 3D motion. In the proposed system, a user first captures the actor's motion and appearance, using a single RGB-D camera. Upon a viewer's request, our system displays the motion from an arbitrary viewpoint using a rough 3D model of the subject, made up of cylinders, and selecting the most appropriate textures based on the viewpoint and the subject's pose. We evaluate the usefulness of the system and the quality of the displayed images by user study., ICME 2014 : IEEE International Conference on Multimedia and Expo , Jul 14-18, 2014 , Chengdu, China

Augmented reality (AR) images with virtualized real objects can be used for various applications. However, such AR image generation requires hand-crafted 3D models of that objects, which are usually not available. This paper proposes a view-dependent texture (VDT)- and view-dependent geometry (VDG)-based method for generating high quality AR images, which uses 3D models automatically reconstructed from multiple images. Since the quality of reconstructed 3D models is usually insufficient, the proposed method inflates the objects in the depth map as VDG to repair chipped object boundaries and assigns a color to each pixel based on VDT to reproduce the detail of the objects. Background pixel exposure due to inflation is suppressed by the use of the foreground region extracted from the input images. Our experimental results have demonstrated that the proposed method can successfully reduce above visual artifacts., ISMAR 2013 : IEEE and ACM International Symposium on Mixed and Augmented Reality , Oct 1-4, 2013 , Adelaide, Australia

Automatically reconstructing a 3D shape model of a nonrigid object using a sequence from a single commodity RGB-D sensor is a challenging problem. Some techniques use a 3D shape template of a target object; however, in order to generate the template automatically, the target object required to be stationary. Otherwise, a non-rigid ICP algorithm, which registers a pair of point clouds, can be used for reconstructing 3D geometry of a non-rigid object directly, but it often fails due to the ambiguity in point correspondences. This paper presents a method for generating a 3D shape template from a single RGB-D sequence. In order to reduce the ambiguity in point correspondences, our method leverages point trajectories obtained in the RGB images, which can be used for associating points in different point clouds. We demonstrate the capability of our method using deforming human bodies., EI 2016 : IS&T International Symposium on Electronic Imaging , Feb 14-18, 2016 , San Francisco , CA , USA

Recent popularization of camera devices, including action cams and smartphones, enables us to record videos in everyday life and share them through the Internet. Video blog is a recent approach for sharing videos, in which users enjoy expressing themselves in blog posts with attractive videos. Generating such videos, however, requires users to review vast amount of raw videos and edit them appropriately, which keeps users away from doing so. In this paper, we propose a novel video summarization method for helping users to create a video blog post. Unlike typical video summarization methods, the proposed method utilizes the text, which is written for a video blog post, and makes the video summary consistent with the content of the text. For this, we perform video summarization by solving an optimization problem, in which an objective function involves the content similarity between the summarized video and the text. Our user study with 20 participants has demonstrated that our proposed method is suitable to create video blog posts compared with conventional methods for video summarization., ICME 2015 : IEEE International Conference on Multimedia and Expo , Jun 29-Jul 3, 2015 , Torino, Italy

The proliferation of off-the-shelf head-mounted displays (HMDs) let end-users enjoy virtual reality applications, some of which render a real-world scene using a novel view synthesis (NVS) technique. View-dependent texture mapping (VDTM) has been studied for NVS due to its photo-realistic quality. The VDTM technique renders a novel view by adaptively selecting textures from the most appropriate images. However, this process is computationally expensive because VDTM scans every captured image. For stereoscopic HMDs, the situation is much worse because we need to render novel views once for each eye, almost doubling the cost. This paper proposes light-weight VDTM tailored for an HMD. In order to reduce the computational cost in VDTM, our method leverages the overlapping fields of view between a stereoscopic pair of HMD images and pruning the images to be scanned. We show that the proposed method drastically accelerates the VDTM process without spoiling the image quality through a user study., ICME 2017: 18th IEEE International Conference on Multimedia and Expo, July 10-14, 2017, Hong Kong, China

An enormous number of images are currently shared through social networking services such as Facebook. These images usually contain appearance of people and may violate the people's privacy if they are published without permission from each person. To remedy this privacy concern, visual privacy protection, such as blurring, is applied to facial regions of people without permission. However, in addition to image quality degradation, this may spoil the context of the image: If some people are filtered while the others are not, missing facial expression makes comprehension of the image difficult. This paper proposes an image melding-based method that modifies facial regions in a visually unintrusive way with preserving facial expression. Our experimental results demonstrated that the proposed method can retain facial expression while protecting privacy., ICME 2015 : IEEE International Conference on Multimedia and Expo , Jun 29-Jul 3, 2015 , Torino, Italy

Augmented Reality (AR) marker hiding is a technique to visually remove AR markers from a real-time video stream. The conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on the AR marker region in the real-time frame, assuming that the AR marker is on a planar surface. However, this approach can cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries under unknown and nonplanar background geometry. The proposed method estimates dense motion in the marker's background based on feature point correspondences around the marker, together with a smooth motion assumption, and deforms the background image according to the dense motion. Experimental results demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures., ISMARW 2015 : IEEE International Symposium on Mixed and Augmented Reality Workshops , Sep 29-Oct 3, 2015 , Fukuoka, Japan