Search

Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible background textures in realtime. Most conventional methods based on image inpainting achieve diminished reality by assuming that the background around a target object is almost planar. This paper proposes a new diminished reality method that considers background geometries with less constraints than the conventional ones. In this study, we approximate the background geometry by combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture and limiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using the geometries and camera pose estimated by visual-simultaneous localization and mapping (SLAM). The mask region that includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments.

In the field of augmented reality (AR), many kinds of vision-based extrinsic camera parameter estimation methods have been proposed to achieve geometric registration between real and virtual worlds. Previously, a feature landmark-based camera parameter estimation method was proposed. This is an effective method for implementing outdoor AR applications because a feature landmark database can be automatically constructed using the structure-from-motion (SfM) technique. However, the previous method cannot work in real time because it entails a high computational cost or matching landmarks in a database with image features in an input image. In addition, the accuracy of estimated camera parameters is insufficient for applications that need to overlay CG objects at a position close to the user's viewpoint. This is because it is difficult to compensate for visual pattern change of close landmarks when only the sparse depth information obtained by the SfM is available. In this paper, we achieve fast and accurate feature landmark-based camera parameter estimation by adopting the following approaches. First, the number of matching candidates is reduced to achieve fast camera parameter estimation by tentative camera parameter estimation and by assigning priorities to landmarks. Second, image templates of landmarks are adequately compensated for by considering the local 3-D structure of a landmark using the dense depth information obtained by a laser range sensor. To demonstrate the effectiveness of the proposed method, we developed some AR applications using the proposed method.

This paper proposes a method to render free viewpoint images from omnidirectional videos using a deformable 3-D mesh model. In the proposed method, a 3-D mesh is placed in front of a virtual viewpoint and deformed by using the pre-estimated omnidirectional depth maps that are selected on the basis of position and posture of the virtual viewpoint. Although our approach is fundamentally based on the model-based rendering approach that renders a geometrically correct virtualized world, in order to avoid the hole problem, we newly employ a viewpoint-dependent deformable 3-D model instead of the use of a unified 3-D model that is generally used in the model based rendering approach. In experiments, free-viewpoint images are generated from the omnidirectional video captured by an omnidirectional multi camera system to show the feasibility of the proposed method for walk-through applications in the virtualized environment.

In this article, we propose an efficient method for estimating a depth map from long-baseline image sequences captured by a calibrated moving multi-camera system. Our concept for estimating a depth map is very simple; we integrate the counting of the total number of interest points (TNIP) in images with the original framework of multiple baseline stereo. Even by using a simple algorithm, the depth can be determined without computing similarity measures such as SSD (sum of squared differences) and NCC (normalized cross correlation) that have been used for conventional stereo matching. The proposed stereo algorithm is computationally efficient and robust for distortions and occlusions and has high affinity with omni-directional and multi-camera imaging. Although expected trade-off between accuracy and efficiency is confirmed for a naive TNIP-based method, a hybrid approach that uses both TNIP and SSD improve this with realizing high accurate and efficient depth estimation. We have experimentally verified the validity and feasibility of the TNIP-based stereo algorithm for both synthetic and real outdoor scenes.

Augmented reality (AR) applications often require virtualized real objects, i.e., virtual objects that are built based on real objects and rendered from an arbitrary viewpoint. In this paper, we propose a method for real object virtualization and AR image generation based on view-dependent geometry modification and texture mapping. The proposed method is a hybrid of model- and image-based rendering techniques that uses multiple input images of the real object as well as the object's three-dimensional (3D) model obtained by an automatic 3D reconstruction technique. Even with state-of-the-art technology, the reconstructed 3D model's accuracy can be insufficient, resulting in such visual artifacts as false object boundaries. The proposed method generates a depth map from a 3D model of a virtualized real object and expands its region in the depth map to remove the false object boundaries. Since such expansion reveals the background pixels in the input images, which is particularly undesirable for AR applications, we preliminarily extract object regions and use them for texture mapping. With our GPU implementation for real-time AR image generation, we experimentally demonstrated that using expanded geometry reduces the number of required input images and maintains visual quality.

Non-photorealistic rendering (NPR) creates images with artistic styles of paintings. In this field, a number of methods of converting photographed images into non-photorealistic ones have been developed, and can be categorized into filter-based and exemplar-based approaches. In this paper, we focus on the exemplar-based approach and propose a novel method which transfers a style of a reference pictorial image to a photographed image. Specifically, we first input a pair of target and reference images. The target image is converted by minimizing an energy function which is defined based on the difference in intensities between an output image and a target image, and the pattern dissimilarity between an output image and a reference image. The proposed method transfers structures and colors of textures in the reference image and generates continuous textures by minimizing the energy function. In experiments, we demonstrate the effectiveness of the proposed method using a variety of images and examine the influence of parameter changes and intensity adjustment for pre-processing on resultant images.

Common light sources such as an ordinary flashlight with lenses and/or reflectors make complex 4-D light field that cannot be represented by conventional isotropic distribution model nor point light source model. This paper describes a new approach to estimate 4-D light field using an illuminated diffuser. Unlike conventional works that capture a 4-D light field directly, our method decomposes observed intensities on the diffuser into intensities of 4-D light rays based on inverse rendering technique with prior knowledge. We formulate 4-D light field reconstruction problem as a non-smooth convex optimization problem for mathematically finding the global minimum.

This paper proposes a method for estimating extrinsic camera parameters using video images and position data acquired by GPS. In conventional methods, the accuracy of the estimated camera position largely depends on the accuracy of GPS positioning data because they assume that GPS position error is very small or normally distributed. However, the actual error of GPS positioning easily grows to the 10m level and the distribution of these errors is changed depending on satellite positions and conditions of the environment. In order to achieve more accurate camera positioning in outdoor environments, in this study, we have employed a simple assumption that true GPS position exists within a certain range from the observed GPS position and the size of the range depends on the GPS positioning accuracy. Concretely, the proposed method estimates camera parameters by minimizing an energy function that is defined by using the reprojection error and the penalty term for GPS positioning., ICPR 2010 : 20th International Conference on Pattern Recognition , Aug 23-26, 2010 , Istanbul, Turkey

This paper describes an application of augmented reality (AR) techniques to virtual cultural heritage reconstruction on the real sites of defunct constructs. To realize AR-based cultural heritage reconstruction, extrinsic camera parameter estimation is required for geometric registration of real and virtual worlds. To estimate extrinsic camera parameters, we use a pre-constructed feature landmark database of the target environment. Conventionally, a feature landmark database has been constructed in a large-scale environment using a structure -from-motion technique for omnidirectional image sequences. However, the accuracy of estimated camera parameters is insufficient for specific applications like AR-based cultural heritage reconstruction, which needs to overlay CG objects at the position close to the user’s viewpoint. This is due to the difficulty in compensation of the appearance change of close landmarks only from the sparse 3-D information obtained by structure-from-motion. In this paper, visual patterns of landmarks are compensated for by considering local shapes obtained by omnidirectional range finder to find corresponding landmarks existing close to the user. By using these landmarks with local shapes, accurate geometric registration is achieved for AR sightseeing in historic sites., ACCV 2010 : 10th Asian Conference on Computer Vision , Nov 8-12, 2010 , Queenstown, New Zealand

Augmented reality (AR) marker hiding is a technique to visually remove AR markers in a real-time video stream. A conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on an AR marker region in a real-time frame, assuming that the AR marker is on a planar surface. However, this approach may cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries even under nonplanar background geometry without measuring it. For doing this, our method estimates the dense motion in the marker's background by analyzing the motion of sparse feature points around it, together with a smooth motion assumption, and deforms the background image according to it. Our experiments demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures.

Authoring video blogs requires a video editing process, which is cumbersome for ordinary users. Video summarization can automate this process by extracting important segments from original videos. Because bloggers typically have certain stories for their blog posts, video summaries of a blog post should take the author’s intentions into account. However, most prior works address video summarization by mining patterns from the original videos without considering the blog author’s intentions. To generate a video summary that reflects the blog author’s intention, we focus on supporting texts in video blog posts and present a text-based method, in which the supporting text serves as a prior to the video summary. Given video and text that describe scenes of interest, our method segments videos and assigns to each video segment its priority in the summary based on its relevance to the input text. Our method then selects a subset of segments with content that is similar to the input text. Accordingly, our method produces different video summaries from the same set of videos, depending on the input text. We evaluated summaries generated from both blog viewers’ and authors’ perspectives in a user study. Experimental results demonstrate the advantages to the proposed text-based method for video blog authoring.

Surface completion is a technique for filling missing regions in 3D models measured by range scanners and videos. Conventionally, although missing regions were filled with the similar shape in a model, the completion process was fairly inefficient because the whole region in the model was searched for the similar shape. In this paper, the completion is efficiently performed using principal curvatures of local shape. In experiments, the effectiveness of the proposed method is successfully verified with subjective evaluation. In addition, the quantitative evaluation which has not been in the literature is newly performed., ICIP 2009 : 16th IEEE International Conference on Image Processing , Nov 7-10, 2009 , Cairo, Egypt

In this paper, we propose a novel surface completion method to generate plausible shapes and textures for missing regions of 3D models. The missing regions are filled in by minimizing two energy functions for shape and texture, which are both based on similarities between the missing region and the rest of the object; in doing so, we take into account the positive correlation between shape and texture. We demonstrate the effectiveness of the proposed method experimentally by applying it to two models., ICIP2011 : 18th IEEE International Conference on Image Processing , Sep 11-14, 2011 , Brussels, Belgium

This paper proposes an example-based method to increase the resolution of a low-resolution image. In the proposed method, we generate example images by a novel view synthesis technique using 3D geometry reconstruction and camera pose estimation from a video or images capturing the same scene. We then increase the resolution by minimizing an energy function by searching for the optimal example from the generated example images. The proposed method has less limitations on camera positions and geometry of the target scene than those in conventional methods. Experiments demonstrate the effectiveness of the proposed method by qualitatively comparing the results of the proposed and conventional methods., ICIP 2015 : 22th IEEE International Conference on Image Processing , Sep 27-30, 2015 , Quebec, Canada

Image inpainting techniques have been widely investigated to remove undesired objects in an image. Conventionally, missing parts in an image are completed by optimizing the objective function using pattern similarity. However, unnatural textures are easily generated due to two factors: (1) available samples in the image are quite limited, and (2) pattern similarity is one of the required conditions but is not sufficient for reproducing natural textures. In this paper, we propose a new energy function based on the pattern similarity considering brightness changes of sample textures (for (1)) and introducing spatial locality as an additional constraint (for (2)). The effectiveness of the proposed method is successfully demonstrated by qualitative and quantitative evaluation. Furthermore, the evaluation methods used in much inpainting research are discussed., PSIVT 2009 : 3rd Pacific-Rim Symposium on Image and Video Technology , Jan 13-16, 2009 , Tokyo, Japan

Three-dimensional (3-D) models of outdoor scenes are widely used for object recognition, navigation, mixed reality, and so on. Because such models are often made manually with high costs, automatic 3-D modeling has been investigated. A 3-D model is usually generated by using a stereo method. However such approaches cannot use several hundreds images together for dense depth estimation because it is difficult to accurately calibrate a large number of cameras. In this paper we propose a 3-D modeling method that first estimates extrinsic camera. parameters of a monocular image sequence captured by a moving video camera, and then reconstructs a 3-D model of a scene. We can acquire a 3-D model of an outdoor scene accurately by using several hundreds input images., ICPR 2002 : 16th International Conference on Pattern Recognition , Aug 11-15, 2002 , Quebec, Canada

Development of algorithms for vehicle safety systems, which support safety driving, takes a long period of time and a huge cost because it requires an evaluation stage where huge combinations of possible driving situations should be evaluated by using videos which are captured beforehand in real environments. In this paper, we address this problem by using free viewpoint images instead of the real images. More concretely, we generate free-viewpoint images from a combination of a 3D point cloud measured by laser scanners and an omni-directional image sequence acquired in a real environment. We basically rely on the 3D point cloud for geometrically correct virtual viewpoint images. In order to remove the holes caused by the unmeasured region of the 3D point cloud and to remove false silhouettes in surface reconstruction, we have developed a technique of free-viewpoint image generation that uses both a 3D point cloud and depth information extracted from images. In the experiments, we have evaluated our framework with a white line detection algorithm and experimental results have shown the applicability of free-viewpoint images for evaluation of algorithms., IV'14 : 2014 IEEE Intelligent Vehicles Symposium Proceedings , Jun 8-11, 2014 , Dearborn, MI, USA

This paper proposes a new method of diminished reality which removes AR markers from a user's view image in real time. To achieve natural marker hiding, assuming that an area around a marker is locally planar, the marker area in the first frame is inpainted using the rectified image to achieve high-quality inpainting. The unique inpainted texture is overlaid on the marker region in each frame according to camera motion for geometric consistency. Both global and local luminance changes around the marker are separately detected and reflected to the inpainted texture for photometric consistency., ICPR 2008 : 19th International Conference on Pattern Recognition , Dec 8-11, 2008 , Tampa, FL, USA

Several human navigation services are currently available on the cellular phones that uses embedded GPS and 2-D map. However, 2-D map based human navigation is not always easy to understand for users because that is not intuitive. In order to realize more intuitive human navigation, AR (Augmented Reality) based navigation where guiding information is overlaid in the real image is expected to be the next generation navigation system. For AR navigation, the key problem is how to acquire the accurate position and posture of the embedded camera on the cellular phone. Many researchers have intensively tackled to the camera parameter estimation problem for AR in recent years. However, most of these methods cannot be applied to the current mobile devices because they are designed to treat video sequence where temporal information like camera parameter of the previous frame is known. In this research, we propose a novel method that estimates camera parameters of single input image using SIFT features and voting scheme., SICE 2008 : International Conference on Instrumentation, Control and Information Technology , Aug 20-22, 2008 , Chofu, Tokyo, Japan

Technology that enables users to experience a remote site virtually is called telepresence. A telepresence system using real environment images is expected to be used in the field of entertainment, medicine, education and so on. This paper describes a novel telepresence system which enables users to walk through a photorealistic virtualized environment by actual walking. To realize such a system, a wide-angle high-resolution movie is projected on an immersive multi-screen display to present users the virtualized environments and a treadmill is controlled according to detected user's locomotion. In this study, we use an omnidirectional multi-camera system to acquire images real outdoor scene. The proposed system provides users with rich sense of walking in a remote site., Stereoscopic Displays and Virtual Reality Systems XI , Jan 19-22, 2004 , San Jose, CA, USA

Indirect augmented reality (IAR) utilizes pre-captured omnidirectional images and offline superimposition of virtual objects for achieving high-quality geometric and photometric registration. Meanwhile, IAR causes inconsistency between the real world and the pre-captured image. This paper describes the first-ever study focusing on the temporal inconsistency issue in IAR. We propose a novel IAR system which reflects real-world illumination change by selecting an appropriate image from a set of images pre-captured under various illumination. Results of a public experiment show that the proposed system can improve the realism in IAR., ISMAR 2014 : IEEE and ACM International Symposium on Mixed and Augmented Reality , Sep 10-12, 2014 , Munich, Germany

Telepresence systems using an omnidirectional image sensor enable us to experience remote sites with rich sensation. An omnidirectional multi-camera system is more useful to acquire high-resolution omnidirectional images of outdoor scenes than a monocular camera system. However, exact calibration of the camera system is necessary to generate a panoramic movie. In this paper, we describe panoramic movie generation method based on geometric and photometric calibration of the omnidirectional multi-camera system. Some prototypes of telepresence system are also shown as applications of generated panoramic movies. These high-resolution telepresence systems have been proven to enable us to experience remote sites with high presence., SCIA 2003 : 13th Scandinavian Conference on Image Analysis , Jun 29-Jul 2, 2003 , Halmstad, Sweden

When observing a person (an actor) performing or demonstrating some activity for the purpose of learning the action, it is best for the viewers to be present at the same time and place as the actor. Otherwise, a video must be recorded. However, conventional video only provides two-dimensional (2D) motion, which lacks the original third dimension of motion. In the presence of some ambiguity, it may be hard for the viewer to comprehend the action with only two dimensions, making it harder to learn the action. This paper proposes an augmented reality system to reenact such actions at any time the viewer wants, in order to aid comprehension of 3D motion. In the proposed system, a user first captures the actor's motion and appearance, using a single RGB-D camera. Upon a viewer's request, our system displays the motion from an arbitrary viewpoint using a rough 3D model of the subject, made up of cylinders, and selecting the most appropriate textures based on the viewpoint and the subject's pose. We evaluate the usefulness of the system and the quality of the displayed images by user study., ICME 2014 : IEEE International Conference on Multimedia and Expo , Jul 14-18, 2014 , Chengdu, China

Telepresence systems using an omnidirectional image sensor enable us to experience remote site. A omnidirectional multi-camera system is more useful to acquire outdoor scenes than a monocular camera system, because the multi-camera system can easily capture high-resolution omnidirectional images. However, exact calibration of the camera system is necessary to virtualize the real world accurately. In this paper, we describe a geometric and photometric camera calibration and a panorama movie generation method for the omnidirectional multi-camera system. In the geometric calibration, intrinsic and extrinsic parameters of each camera are estimated using a calibration board and a laser measurement system called total station. In the photometric calibration, the limb darkening and color balances among the cameras are corrected. The result of the calibration is used in the panorama movie generation. In experiments, we have actually calibrated the multi-camera system and have generated spherical panorama movies by using the estimated camera parameters. A telepresence system was prototyped in order to confirm that the panorama movie can be used for telepresence well. In addition, we have evaluated the discontinuity in generated panoramic images., Stereoscopic Displays and Virtual Reality Systems XII , Jan 21-23, 2003 , Santa Clara, CA, USA

This paper proposes a teleoperation interface by which an operator can control a robot from freely configured viewpoints using realistic images of the physical world. The viewpoints generated by the proposed interface provide human operators with intuitive control using a head-mounted display and head tracker, and assist them to grasp the environment surrounding the robot. A state-of-the-art free-viewpoint image generation technique is employed to generate the scene presented to the operator. In addition, an augmented reality technique is used to superimpose a 3D model of the robot onto the generated scenes. Through evaluations under virtual and physical environments, we confirmed that the proposed interface improves the accuracy of teleoperation., IROS 2013 : IEEE/RSJ International Conference on Intelligent Robots and Systems , Nov 3-7, 2013 , Tokyo, Japan

We present a novel method for estimating 3-D positions of near light sources by using highlights on the outside and inside of a single clear hollow sphere. Conventionally, the positions of near light sources have been estimated by using observed highlights on multiple reference objects, e.g. mirror balls., ICPR 2012 : 21th International Conference on Pattern Recognition , Nov 11-15, 2012 , Tsukuba, Japan

Augmented reality (AR) images with virtualized real objects can be used for various applications. However, such AR image generation requires hand-crafted 3D models of that objects, which are usually not available. This paper proposes a view-dependent texture (VDT)- and view-dependent geometry (VDG)-based method for generating high quality AR images, which uses 3D models automatically reconstructed from multiple images. Since the quality of reconstructed 3D models is usually insufficient, the proposed method inflates the objects in the depth map as VDG to repair chipped object boundaries and assigns a color to each pixel based on VDT to reproduce the detail of the objects. Background pixel exposure due to inflation is suppressed by the use of the foreground region extracted from the input images. Our experimental results have demonstrated that the proposed method can successfully reduce above visual artifacts., ISMAR 2013 : IEEE and ACM International Symposium on Mixed and Augmented Reality , Oct 1-4, 2013 , Adelaide, Australia

Recently, the telepresence which allows us to experience a remote site through a virtualized real world has been investigated. We have already proposed telepresence systems using omni-directional images which enable the user to look around a virtualized dynamic real scene. The system gives the feeling of high presence to the user. However, conventional systems are not networked but are implemented as stand-alone systems that use immersive displays. In this paper we propose a networked telepresence system which enables the user to see a virtualized real world easily in network environments. This paper describes the system implementation using real-time and stored omni-directional video streams as well as its experiments., PCM 2004 : Pacific Rim Conference on Multimedia , Nov 30-Dec 3, 2004 , Tokyo, Japan

3D mesh models generated with range scanner or video images often have holes due to many occlusions by other objects and the object itself. This paper proposes a novel method to fill the missing parts in the incomplete models. The missing parts are filled by minimizing the energy function, which is defined based on similarity of local shape between the missing region and the rest of the object. The proposed method can generate complex and consistent shapes in the missing region. In the experiment, the effectiveness of the method is successfully demonstrated by applying it to complex shape objects with missing parts., ICIP 2008 : 15th IEEE International Conference on Image Processing , Oct 12-15, 2008 , San Diego, CA, USA

Estimating camera position and posture can be applied to the fields of augmented reality and robot navigation. In these fields, to obtain absolute position and posture of the camera, sensor-based methods using GPS and magnetic sensors and vision-based methods using input images from the camera have been investigated. However, sensor-based methods are difficult to synchronize the camera and sensors accurately, and usable environments are limited according to selection of sensors. On the other hand, vision-based methods need to allocate many artificial markers otherwise an estimation error will accumulate. Thus, it is difficult to use such methods in large and natural environments. This paper proposes a vision-based camera position and posture estimation method for large environments, which does not require sensors and artificial markers by detecting natural feature points from image sequences taken beforehand and using them as landmarks., SCIA 2005 : 14th Scandinavian Conference on Image Analysis , Jun 19-22, 2005 , Joensuu, Finland

Recently, a number of methods for 3-D modeling from images have been developed. However, the accuracy of a reconstructed model depends on camera positions and postures with which the images are obtained. In most of conventional methods, some skills for adequately controlling the camera movement are needed for users to obtain a good 3-D model. In this study, we propose an interactive 3-D modeling interface in which special skills are not required. This interface consists of “indication of camera movement” and “preview of reconstruction result.” In experiments for subjective evaluation, we verify the usefulness of the proposed 3D modeling interfaces., SCIA 2005 : 14th Scandinavian Conference on Image Analysis , Jun 19-22, 2005 , Joensuu, Finland

Image inpainting, which removes undesired objects in a static image and fills in the missing regions with plausible textures, has been developed in the research fields of image processing. On the other hand, Diminished Reality (DR), which visually removes real objects from video images by filling in the missing regions with background textures in real time, is one of the growing topics in Virtual/Mixed Reality, and considered as the opposite of Augmented Reality. In this paper, we introduce the state-of-the-art of image inpainting methods and how to apply the image inpainting to diminished reality., VAMR 2014 : 6th International Conference Virtual, Augmented and Mixed Reality , Jun 22-27, 2014 , Heraklion, Crete, Greece

In the field of augmented reality (AR), geometric and photometric registration is routinely achieved in real time. However, real-time geometric registration often leads to misalignment (e.g., jitter and drift) due to the error from camera pose estimation. Due to limited resources on mobile devices, it is also difficult to implement state-of-the-art techniques for photometric registration on mobile AR systems. In order to solve these problems, we developed a mobile AR system in a significantly different way from conventional systems. In this system, captured omnidirectional images and virtual objects are registered geometrically and photometrically in an offline rendering process. The appropriate part of the prerendered omnidirectional AR image is shown to a user through a mobile device with online registration between the real world and the pre-captured image. In order to investigate the validity of our new framework for mobile AR, we conducted experiments using the prototype system on a real site in Todai-ji Temple, a famous world cultural heritage site in Japan., SA 2013 : SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications , Nov 19-22, 2013 , Hong Kong

Automatically reconstructing a 3D shape model of a nonrigid object using a sequence from a single commodity RGB-D sensor is a challenging problem. Some techniques use a 3D shape template of a target object; however, in order to generate the template automatically, the target object required to be stationary. Otherwise, a non-rigid ICP algorithm, which registers a pair of point clouds, can be used for reconstructing 3D geometry of a non-rigid object directly, but it often fails due to the ambiguity in point correspondences. This paper presents a method for generating a 3D shape template from a single RGB-D sequence. In order to reduce the ambiguity in point correspondences, our method leverages point trajectories obtained in the RGB images, which can be used for associating points in different point clouds. We demonstrate the capability of our method using deforming human bodies., EI 2016 : IS&T International Symposium on Electronic Imaging , Feb 14-18, 2016 , San Francisco , CA , USA

This paper describes a novel method for estimating positions and postures of an omnidirectional multi-camera system from its multiple image sequences and sparse position data acquired by GPS. The proposed method is based on a structure-from-motion technique which is enhanced by using multiple image sequences as well as GPS position data. Moreover, the position data are also used to remove mis-tracked features. The proposed method allows us to estimate position and posture without accumulative errors and annoying effects due to moving objects in outdoor environments. The validity of the method is demonstrated through experiments using both synthetic and real outdoor scenes., First Korea-Japan Joint Workshop on Pattern Recognition , Nov 23-24, 2006 , Jeju, Korea

Omnidirectional cameras usually cannot capture the entire direction of view due to a blind side. Thus, such an invisible part decreases realistic sensation in a telepresence system. In this study, an omnidirectional video without invisible areas is generated by filling in the missing region using an image inpainting technique for highly realistic sensation in telepresence. This paper proposes a new method that successfully inpaints a missing region by compensating for the change in appearance of textures caused by the camera motion and determining a searching area for similar textures considering the camera motion and the shape of the scene around the missing region. In experiments, the effectiveness of the proposed method is demonstrated by inpainting missing regions in a real image sequence captured with an omnidirectional camera and generating an omnidirectional video without invisible areas., ACCV 2009 : 9th Asian Conference on Computer Vision , Sep 23-27, 2009 , Xi'an, China

Telepresence systems using an omnidirectional image sensor enable us to experience remote sites with rich sensation. An omnidirectional multi-camera system is more useful for acquiring high-resolution omnidirectional images of outdoor scenes than a monocular camera system. However, exact calibration of the camera system is necessary to generate a panoramic movie. In this paper, we describe a panoramic movie generation method based on geometric and photometric calibration of the omnidirectional multi-camera system. A prototype of a telepresence system is also shown as applications of generated panoramic movies. This high-resolution telepresence system has been proven to enable us to experience remote sites with rich presence., MFI 2003 : International Conference on Multisensor Fusion and Integration for Intelligent Systems , Aug 1, 2003 , Tokyo, Japan

Recently, document and photograph digitization from a paper is very important for digital archiving and personal data transmission through the internet. Though many people wish to digitize documents on a paper easily, now heavy and large image scanners are required to obtain high quality digitization. To realize easy and high quality digitization of documents and photographs, we propose a novel digitization method that uses a movie captured by a hand-held camera. In our method, first, 6-DOF(Degree Of Freedom) position and posture parameters of the mobile camera are estimated in each frame by tracking image features automatically. Next, re-appearing feature points in the image sequence are detected and stitched for minimizing accumulated estimation errors. Finally, all the images are merged as a high-resolution mosaic image using the optimized parameters. Experiments have successfully demonstrated the feasibility of the proposed method. Our prototype system can acquire initial estimates of extrinsic camera parameters in real-time with capturing images., Computational Imaging II , Jan 18, 2004 , San Jose, CA, USA

This paper describes a novel telepresence system which enables users to walk through a photorealistic virtualized environment by actual walking. In this system, a wide-angle high-resolution video is projected on an immersive multi- screen display and a treadmill is controlled according to user’s locomotion. Calibration and motion estimation of a camera system are performed to enhance quality of pre- sented images. The proposed system provides users with rich sense of walking in a remote site., MVA 2005 : IAPR Conference on Machine Vision Applications , May 16-18, 2005 , Tsukuba, Japan

This paper describes a novel telepresence system which enables users to walk through a photorealistic virtualized environment by actual walking. To realize such a system, a wide-angle high-resolution movie is projected on an immersive multi-screen display to present users the virtualized environments and a treadmill is controlled according to detected user's locomotion. In this study, we use an omnidirectional multi-camera system to acquire images of a real outdoor scene. The proposed system provides users with rich sense of walking in a remote site., ICPR 2004 : 17th International Conference on Pattern Recognition , Aug 23-26, 2004 , Cambridge, UK

This paper proposes a new diminished reality method for 3D scenes considering background structures. Most conventional methods using image inpainting assumes that the background around a target object is almost planar. In this study, approximating the background structure by the combination of local planes, perspective distortion of texture is corrected and searching area is limited for improving the quality of image inpainting. The temporal coherence of texture is preserved using the estimated structures and camera pose estimated by Visual-SLAM., ISMAR 2013 : IEEE and ACM International Symposium on Mixed and Augmented Reality , Oct 1-4, 2013 , Adelaide, Australia

This paper proposes a new method of diminished reality which removes AR markers from a user's view image in real time. To achieve natural marker hiding, assuming that an area around a marker is locally planar, the marker area in the first frame is inpainted using the rectified image to achieve high-quality inpainting. The unique inpainted texture is overlaid on the marker region in each frame according to camera motion for geometric consistency. Both global and local luminance changes around the marker are separately detected and reflected to the inpainted texture for photometric consistency., ISMAR 2012: IEEE International Symposium on Mixed and Augmented Reality , Nov 5-8, 2012 , Atlanta, GA, USA

This paper describes a real-time pre-visualization method using augmented reality techniques for videographers. It enables them to test camera work without real actors in a real environment. As a substitute for real actors, virtual ones are superimposed on a live video in real-time according to a real camera motion and an illumination condition. The key technique of this method is real-time motion estimation of a camera, which can be applied to unknown complex environments including natural objects. In our method, geometric and photometric registration problems for such unknown environments are solved to realize the above visualization. A prototype system demonstrates availability of the previsualization method., ICME 2008 : IEEE International Conference on Multimedia & Expo , Jun 23-26, 2008 , Hannover, Germany

This paper proposes a novel method for estimating depth without similarity measures such as SSD and NCC. Our idea for estimating a depth map is very simple; only counting interest points in images is integrated with the framework of multi-baseline stereo. Even by a simple algorithm, depth can be determined without computing similarity measures such as SSD and NCC that have been used for conventional stereo matching. The proposed method realizes robust depth estimation against occlusions with lower computational cost. Though a naive TNIP based method can realize fast and robust depth estimation, the accuracy of estimated depth is lower than one by SSSD based method because TNIP uses sparse data. In this paper, we also show that accuracy of depth estimation can be increased by combining TNIP based method and SSSD based method. In experiments, the validity and feasibility of our algorithm are demonstrated for both synthetic and real outdoor scenes., SICE 2007 : International Conference on Instrumentation, Control and Information Technology , Sep 17-20, 2007 , Takamatsu, Japan

This paper proposes a linear estimation method for 4-D illumination light field using an inverse lighting framework. Although lighting environments have conventionally been modeled by the combination of point light sources or a spherical light field map, complex light sources such as an ordinary flashlight with lenses and reflectors, can not be correctly modeled by these approaches. In order to correctly model these complex light sources, we employ 4-D illumination light field. Unlike conventional work, we decompose observed intensities on a diffuse-reflection board into intensities of 4-D light rays by solving a linear system. We validate the effectiveness of the proposed method through the experiments on real and virtual environments., ACPR 2013 : 2nd IAPR Asian Conference on Pattern Recognition , Nov 5-8, 2013 , Okinawa, Japan

A 3D modeling technique for an urban environment can be applied to several applications such as landscape simulations, navigational systems, and mixed reality systems. In this field, the target environment is first measured using several types of sensors (laser rangefinders, cameras, GPS sensors, and gyroscopes). A 3D model of the environment is then constructed based on the results of the 3D measurements. In this 3D modeling process, 3D points that exist on moving objects become obstacles or outliers to enable the construction of an accurate 3D model. To solve this problem, we propose a method for detecting 3D points on moving objects from 3D point cloud data based on photometric consistency and knowledge of the road environment. In our method, 3D points on moving objects are detected based on luminance variations obtained by projecting 3D points onto omnidirectional images. After detecting 3D the points based on evaluation value, the points are detected using prior information of the road environment., ICIP 2013 : 20th IEEE International Conference on Image Processing , Sep 15-18, 2013 , Melbourne, Australia

Recent popularization of camera devices, including action cams and smartphones, enables us to record videos in everyday life and share them through the Internet. Video blog is a recent approach for sharing videos, in which users enjoy expressing themselves in blog posts with attractive videos. Generating such videos, however, requires users to review vast amount of raw videos and edit them appropriately, which keeps users away from doing so. In this paper, we propose a novel video summarization method for helping users to create a video blog post. Unlike typical video summarization methods, the proposed method utilizes the text, which is written for a video blog post, and makes the video summary consistent with the content of the text. For this, we perform video summarization by solving an optimization problem, in which an objective function involves the content similarity between the summarized video and the text. Our user study with 20 participants has demonstrated that our proposed method is suitable to create video blog posts compared with conventional methods for video summarization., ICME 2015 : IEEE International Conference on Multimedia and Expo , Jun 29-Jul 3, 2015 , Torino, Italy

This paper describes a novel method for estimating extrinsic camera parameters using both feature points on an image sequence and sparse position data acquired by GPS. Our method is based on a structure-from-motion technique but is enhanced by using GPS data so as to minimize accumulative estimation errors. Moreover, the position data are also used to remove mis-tracked features. The proposed method allows us to estimate extrinsic parameters without accumulative errors even from an extremely long image sequence. The validity of the method is demonstrated through experiments of estimating extrinsic parameters for both synthetic and real outdoor scenes., ACCV 2006 : 7th Asian Conference on Computer Vision , Jan 13-16, 2006 , Hyderabad, India

Image inpainting techniques have been widely used to remove undesired visual objects in images such as damaged portions of photographs and people who have accidentally entered into pictures. Conventionally, the missing parts of an image are completed by optimizing the objective function which is defined based on the sum of SSD (sum of squared differences). However, the naive SSD-based objective function is not robust against intensity change in an image. Thus, unnatural intensity change often appears in the missing parts. In addition, when an image has continuously changing texture patterns, the completed texture in a resultant image sometimes blurs due to inappropriate pattern matching. In this paper, in order to improve the image quality of the completed texture, the conventional objective function is newly extended by considering intensity changes and spatial locality to prevent unnatural intensity changes and blurs in a resultant image. By minimizing the extended energy function, the missing regions can be completed without unnatural intensity changes and blurs. In experiments, the effectiveness of the proposed method is successfully demonstrated by applying our method to various images and comparing the results with those obtained by the conventional method., Computational Imaging V , Jan 28, 2007 , San Jose, CA, USA

In the field of augmented reality, it is important to solve a geometric registration problem between real and virtual worlds. To solve this problem, many kinds of image based online camera parameter estimation methods have been proposed. As one of these methods, we have been proposed a feature landmark based camera parameter estimation method. In this method, extrinsic camera parameters are estimated from corresponding landmarks and image features. Although the method can work in large and complex environments, our previous method cannot work in real-time due to high computational cost in matching process. Additionally, initial camera parameters for the first frame must be given manually. In this study, we realize real-time and manual-initialization free camera parameter estimation based on feature landmark database. To reduce the computational cost of the matching process, the number of matching candidates is reduced by using priorities of landmarks that are determined from previously captured video sequences. Initial camera parameter for the first frame is determined by a voting scheme for the target space using matching candidates. To demonstrate the effectiveness of the proposed method, applications of landmark based real-time camera parameter estimation are demonstrated in outdoor environments., The Engineering Reality of Virtual Reality 2009 , Jan 18-22, 2009 , San Jose, CA, USA