Search

application/pdf, Surface completion is a technique for filling missing regions in 3D models measured by range scanners and videos. Conventionally, although missing regions were filled with the similar shape in a model, the completion process was fairly inefficient because the whole region in the model was searched for the similar shape. In this paper, the completion is efficiently performed using principal curvatures of local shape. In experiments, the effectiveness of the proposed method is successfully verified with subjective evaluation. In addition, the quantitative evaluation which has not been in the literature is newly performed.

application/pdf, In the field of augmented reality, it is important to solve a geometric registration problem between real and virtual worlds. To solve this problem, many kinds of image based online camera parameter estimation methods have been proposed. As one of these methods, we have been proposed a feature landmark based camera parameter estimation method. In this method, extrinsic camera parameters are estimated from corresponding landmarks and image features. Although the method can work in large and complex environments, our previous method cannot work in real-time due to high computational cost in matching process. Additionally, initial camera parameters for the first frame must be given manually. In this study, we realize real-time and manual-initialization free camera parameter estimation based on feature landmark database. To reduce the computational cost of the matching process, the number of matching candidates is reduced by using priorities of landmarks that are determined from previously captured video sequences. Initial camera parameter for the first frame is determined by a voting scheme for the target space using matching candidates. To demonstrate the effectiveness of the proposed method, applications of landmark based real-time camera parameter estimation are demonstrated in outdoor environments.

application/pdf, Image inpainting techniques have been widely investigated to remove undesired objects in an image. Conventionally, missing parts in an image are completed by optimizing the objective function using pattern similarity. However, unnatural textures are easily generated due to two factors: (1) available samples in the image are quite limited, and (2) pattern similarity is one of the required conditions but is not sufficient for reproducing natural textures. In this paper, we propose a new energy function based on the pattern similarity considering brightness changes of sample textures (for (1)) and introducing spatial locality as an additional constraint (for (2)). The effectiveness of the proposed method is successfully demonstrated by qualitative and quantitative evaluation. Furthermore, the evaluation methods used in much inpainting research are discussed.

Omnidirectional cameras usually cannot capture the entire direction of view due to a blind side. Thus, such an invisible part decreases realistic sensation in a telepresence system. In this study, an omnidirectional video without invisible areas is generated by filling in the missing region using an image inpainting technique for highly realistic sensation in telepresence. This paper proposes a new method that successfully inpaints a missing region by compensating for the change in appearance of textures caused by the camera motion and determining a searching area for similar textures considering the camera motion and the shape of the scene around the missing region. In experiments, the effectiveness of the proposed method is demonstrated by inpainting missing regions in a real image sequence captured with an omnidirectional camera and generating an omnidirectional video without invisible areas., conference paper

Surface completion is a technique for filling missing regions in 3D models measured by range scanners and videos. Conventionally, although missing regions were filled with the similar shape in a model, the completion process was fairly inefficient because the whole region in the model was searched for the similar shape. In this paper, the completion is efficiently performed using principal curvatures of local shape. In experiments, the effectiveness of the proposed method is successfully verified with subjective evaluation. In addition, the quantitative evaluation which has not been in the literature is newly performed., conference paper

In the field of augmented reality, it is important to solve a geometric registration problem between real and virtual worlds. To solve this problem, many kinds of image based online camera parameter estimation methods have been proposed. As one of these methods, we have been proposed a feature landmark based camera parameter estimation method. In this method, extrinsic camera parameters are estimated from corresponding landmarks and image features. Although the method can work in large and complex environments, our previous method cannot work in real-time due to high computational cost in matching process. Additionally, initial camera parameters for the first frame must be given manually. In this study, we realize real-time and manual-initialization free camera parameter estimation based on feature landmark database. To reduce the computational cost of the matching process, the number of matching candidates is reduced by using priorities of landmarks that are determined from previously captured video sequences. Initial camera parameter for the first frame is determined by a voting scheme for the target space using matching candidates. To demonstrate the effectiveness of the proposed method, applications of landmark based real-time camera parameter estimation are demonstrated in outdoor environments., conference paper

This paper proposes a method to render free viewpoint images from omnidirectional videos using a deformable 3-D mesh model. In the proposed method, a 3-D mesh is placed in front of a virtual viewpoint and deformed by using the pre-estimated omnidirectional depth maps that are selected on the basis of position and posture of the virtual viewpoint. Although our approach is fundamentally based on the model-based rendering approach that renders a geometrically correct virtualized world, in order to avoid the hole problem, we newly employ a viewpoint-dependent deformable 3-D model instead of the use of a unified 3-D model that is generally used in the model based rendering approach. In experiments, free-viewpoint images are generated from the omnidirectional video captured by an omnidirectional multi camera system to show the feasibility of the proposed method for walk-through applications in the virtualized environment., journal article

application/pdf, Several human navigation services are currently available on the cellular phones that uses embedded GPS and 2-D map. However, 2-D map based human navigation is not always easy to understand for users because that is not intuitive. In order to realize more intuitive human navigation, AR (Augmented Reality) based navigation where guiding information is overlaid in the real image is expected to be the next generation navigation system. For AR navigation, the key problem is how to acquire the accurate position and posture of the embedded camera on the cellular phone. Many researchers have intensively tackled to the camera parameter estimation problem for AR in recent years. However, most of these methods cannot be applied to the current mobile devices because they are designed to treat video sequence where temporal information like camera parameter of the previous frame is known. In this research, we propose a novel method that estimates camera parameters of single input image using SIFT features and voting scheme.

application/pdf, 3D mesh models generated with range scanner or video images often have holes due to many occlusions by other objects and the object itself. This paper proposes a novel method to fill the missing parts in the incomplete models. The missing parts are filled by minimizing the energy function, which is defined based on similarity of local shape between the missing region and the rest of the object. The proposed method can generate complex and consistent shapes in the missing region. In the experiment, the effectiveness of the method is successfully demonstrated by applying it to complex shape objects with missing parts.

application/pdf, This paper proposes a new method of diminished reality which removes AR markers from a user's view image in real time. To achieve natural marker hiding, assuming that an area around a marker is locally planar, the marker area in the first frame is inpainted using the rectified image to achieve high-quality inpainting. The unique inpainted texture is overlaid on the marker region in each frame according to camera motion for geometric consistency. Both global and local luminance changes around the marker are separately detected and reflected to the inpainted texture for photometric consistency.

Several human navigation services are currently available on the cellular phones that uses embedded GPS and 2-D map. However, 2-D map based human navigation is not always easy to understand for users because that is not intuitive. In order to realize more intuitive human navigation, AR (Augmented Reality) based navigation where guiding information is overlaid in the real image is expected to be the next generation navigation system. For AR navigation, the key problem is how to acquire the accurate position and posture of the embedded camera on the cellular phone. Many researchers have intensively tackled to the camera parameter estimation problem for AR in recent years. However, most of these methods cannot be applied to the current mobile devices because they are designed to treat video sequence where temporal information like camera parameter of the previous frame is known. In this research, we propose a novel method that estimates camera parameters of single input image using SIFT features and voting scheme., conference paper

This paper proposes a new method of diminished reality which removes AR markers from a user's view image in real time. To achieve natural marker hiding, assuming that an area around a marker is locally planar, the marker area in the first frame is inpainted using the rectified image to achieve high-quality inpainting. The unique inpainted texture is overlaid on the marker region in each frame according to camera motion for geometric consistency. Both global and local luminance changes around the marker are separately detected and reflected to the inpainted texture for photometric consistency., conference paper

3D mesh models generated with range scanner or video images often have holes due to many occlusions by other objects and the object itself. This paper proposes a novel method to fill the missing parts in the incomplete models. The missing parts are filled by minimizing the energy function, which is defined based on similarity of local shape between the missing region and the rest of the object. The proposed method can generate complex and consistent shapes in the missing region. In the experiment, the effectiveness of the method is successfully demonstrated by applying it to complex shape objects with missing parts., conference paper

application/pdf, This paper describes a method for constructing feature landmark database using omnidirectional videos and GPS positions acquired in outdoor environments. The feature landmark database is used to estimate camera positions and postures for various applications such as augmented reality systems and self-localization of robots and automobiles. We have already proposed a camera position and posture estimation method using landmark database that stores 3D positions of sparse feature points with their view-dependent image templates. For large environments, the cost for construction of landmark database is high because conventional 3-D reconstruction methods requires measuring some absolute positions of feature points manually to suppress accumulative estimation errors in structure-from-motion process. To achieve automatic construction of landmark database for large outdoor environments, we newly propose a method that constructs database without manual specification of features using omnidirectional videos and GPS positions.

Image inpainting techniques have been widely used to remove undesired visual objects in images such as damaged portions of photographs and people who have accidentally entered into pictures. Conventionally, the missing parts of an image are completed by optimizing the objective function which is defined based on the sum of SSD (sum of squared differences). However, the naive SSD-based objective function is not robust against intensity change in an image. Thus, unnatural intensity change often appears in the missing parts. In addition, when an image has continuously changing texture patterns, the completed texture in a resultant image sometimes blurs due to inappropriate pattern matching. In this paper, in order to improve the image quality of the completed texture, the conventional objective function is newly extended by considering intensity changes and spatial locality to prevent unnatural intensity changes and blurs in a resultant image. By minimizing the extended energy function, the missing regions can be completed without unnatural intensity changes and blurs. In experiments, the effectiveness of the proposed method is successfully demonstrated by applying our method to various images and comparing the results with those obtained by the conventional method.

application/pdf, This paper proposes a novel method for estimating depth without similarity measures such as SSD and NCC. Our idea for estimating a depth map is very simple; only counting interest points in images is integrated with the framework of multi-baseline stereo. Even by a simple algorithm, depth can be determined without computing similarity measures such as SSD and NCC that have been used for conventional stereo matching. The proposed method realizes robust depth estimation against occlusions with lower computational cost. Though a naive TNIP based method can realize fast and robust depth estimation, the accuracy of estimated depth is lower than one by SSSD based method because TNIP uses sparse data. In this paper, we also show that accuracy of depth estimation can be increased by combining TNIP based method and SSSD based method. In experiments, the validity and feasibility of our algorithm are demonstrated for both synthetic and real outdoor scenes.

Image inpainting techniques have been widely used to remove undesired visual objects in images such as damaged portions of photographs and people who have accidentally entered into pictures. Conventionally, the missing parts of an image are completed by optimizing the objective function which is defined based on the sum of SSD (sum of squared differences). However, the naive SSD-based objective function is not robust against intensity change in an image. Thus, unnatural intensity change often appears in the missing parts. In addition, when an image has continuously changing texture patterns, the completed texture in a resultant image sometimes blurs due to inappropriate pattern matching. In this paper, in order to improve the image quality of the completed texture, the conventional objective function is newly extended by considering intensity changes and spatial locality to prevent unnatural intensity changes and blurs in a resultant image. By minimizing the extended energy function, the missing regions can be completed without unnatural intensity changes and blurs. In experiments, the effectiveness of the proposed method is successfully demonstrated by applying our method to various images and comparing the results with those obtained by the conventional method., conference paper

This paper proposes a novel method for estimating depth without similarity measures such as SSD and NCC. Our idea for estimating a depth map is very simple; only counting interest points in images is integrated with the framework of multi-baseline stereo. Even by a simple algorithm, depth can be determined without computing similarity measures such as SSD and NCC that have been used for conventional stereo matching. The proposed method realizes robust depth estimation against occlusions with lower computational cost. Though a naive TNIP based method can realize fast and robust depth estimation, the accuracy of estimated depth is lower than one by SSSD based method because TNIP uses sparse data. In this paper, we also show that accuracy of depth estimation can be increased by combining TNIP based method and SSSD based method. In experiments, the validity and feasibility of our algorithm are demonstrated for both synthetic and real outdoor scenes., conference paper

application/pdf, SIGGRAPH '06 : The 33rd International Conference and Exhibition on Computer Graphics and Interactive Techniques , Jul 30-Aug 3, 2006 , Boston, MA, USA, conference paper

application/pdf, This paper describes a novel telepresence system which enables users to walk through a photorealistic virtualized environment by actual walking. In this system, a wide-angle high-resolution video is projected on an immersive multi- screen display and a treadmill is controlled according to user’s locomotion. Calibration and motion estimation of a camera system are performed to enhance quality of pre- sented images. The proposed system provides users with rich sense of walking in a remote site., MVA 2005 : IAPR Conference on Machine Vision Applications , May 16-18, 2005 , Tsukuba, Japan, conference paper

This paper proposes a novel method for estimating depth from a long image sequence captured by a moving camera. Our idea for estimating a depth map is very simple; only counting interest points in images is integrated with the framework of multi-baseline stereo. Even by a simple algorithm, depth can be determined without computing similarity measures such as SSD and NCC that have been used for traditional stereo matching. Our method realizes robust depth estimation against image distortions and occlusions that are caused by camera motion. Note that, in this paper, intrinsic and extrinsic parameters of video camera are assumed to be calibrated in advance.

This paper proposes a novel method for estimating depth from a long image sequence captured by a moving camera. Our idea for estimating a depth map is very simple; only counting interest points in images is integrated with the framework of multi-baseline stereo. Even by a simple algorithm, depth can be determined without computing similarity measures such as SSD and NCC that have been used for traditional stereo matching. Our method realizes robust depth estimation against image distortions and occlusions that are caused by camera motion. Note that, in this paper, intrinsic and extrinsic parameters of video camera are assumed to be calibrated in advance., conference paper

application/pdf, Recently, document and photograph digitization from a paper is very important for digital archiving and personal data transmission through the internet. Though many people wish to digitize documents on a paper easily, now heavy and large image scanners are required to obtain high quality digitization. To realize easy and high quality digitization of documents and photographs, we propose a novel digitization method that uses a movie captured by a hand-held camera. In our method, first, 6-DOF(Degree Of Freedom) position and posture parameters of the mobile camera are estimated in each frame by tracking image features automatically. Next, re-appearing feature points in the image sequence are detected and stitched for minimizing accumulated estimation errors. Finally, all the images are merged as a high-resolution mosaic image using the optimized parameters. Experiments have successfully demonstrated the feasibility of the proposed method. Our prototype system can acquire initial estimates of extrinsic camera parameters in real-time with capturing images., Computational Imaging II , Jan 18, 2004 , San Jose, CA, USA, conference paper

Recently, the telepresence which allows us to experience a remote site through a virtualized real world has been investigated. We have already proposed telepresence systems using omni-directional images which enable the user to look around a virtualized dynamic real scene. The system gives the feeling of high presence to the user. However, conventional systems are not networked but are implemented as stand-alone systems that use immersive displays. In this paper we propose a networked telepresence system which enables the user to see a virtualized real world easily in network environments. This paper describes the system implementation using real-time and stored omni-directional video streams as well as its experiments.

Recently, many types of omni-directional cameras have been developed and attracted much attention in a number of different fields. Especially, the multi-camera type of omni-directional camera has advantages of high-resolution and almost uniform resolution for any direction of view. In this paper, an extrinsic camera parameter recovery method for a moving omni-directional multi-camera system (OMS) is proposed. First, we discuss a perspective n-point (PnP) problem for an OMS, and then describe a practical method for estimating extrinsic camera parameters from multiple image sequences obtained by an OMS. The proposed method is based on using the shape-from-motion and the PnP techniques.

Recently, document and photograph digitization from a paper is very important for digital archiving and personal data transmission through the internet. Though many people wish to digitize documents on a paper easily, now heavy and large image scanners are required to obtain high quality digitization. To realize easy and high quality digitization of documents and photographs, we propose a novel digitization method that uses a movie captured by a hand-held camera. In our method, first, 6-DOF(Degree Of Freedom) position and posture parameters of the mobile camera are estimated in each frame by tracking image features automatically. Next, re-appearing feature points in the image sequence are detected and stitched for minimizing accumulated estimation errors. Finally, all the images are merged as a high-resolution mosaic image using the optimized parameters. Experiments have successfully demonstrated the feasibility of the proposed method. Our prototype system can acquire initial estimates of extrinsic camera parameters in real-time with capturing images.

application/pdf, Three-dimensional (3D) models of outdoor scenes can be widely used in a number of fields such as object recognition, navigation, scenic simulation, and mixed reality. Such models are often made manually with high cost, so that automatic 3D reconstruction has been widely investigated. In related works, a dense 3D model is generated by using a stereo method. However, such approaches cannot use several hundred images together or dense depth estimation of large constructs and urban environments because it is difficult to accurately calibrate a large number of cameras. This paper proposes a dense 3D reconstruction method that uses multiple image sequences. First, our method estimates extrinsic camera parameters of each image sequence, then reconstructs a dense 3D model of a scene using an extended multi-baseline stereo and voxel voting technique., MFI 2003 : International Conference on Multisensor Fusion and Integration for Intelligent Systems , Aug 1, 2003 , Tokyo, Japan, conference paper

application/pdf, This paper describes a novel telepresence system that uses high-resolution movies and a reactive display system with a treadmill. In this system, users can walk through a virtualized environment by actually walking on a treadmill. According to walking motion which is detected by using 3-D position sensors put on both legs, the virtualized environment captured by an omnidirectional multi-camera system is projected on a multi-screen display., ISMAR 2003 : IEEE / ACM International Symposium on Mixed and Augmented Reality , Oct 7-10, 2003 , Tokyo, Japan, conference paper

application/pdf, Three-dimensional (3-D) models of outdoor scenes can be widely used in a number of fields such as object recognition, navigation, scenic simulation, and mixed reality. Such models are often made manually with high costs, so that automatic 3-D reconstruction has been widely investigated. In related works, a dense 3-D model is generated by using a stereo method. However, such approaches cannot use several hundred images together for dense depth estimation of large constructs and urban environments because it is difficult to accurately calibrate a large number of cameras. This paper proposes a novel dense 3-D reconstruction method that uses multiple image sequences. First, our method estimates extrinsic camera parameters of each image sequence, and then reconstructs a dense 3-D model of a scene using an extended multi-baseline stereo and voxel voting techniques.