Search

application/pdf, Nowadays, decreasing opportunities for people to have daily conversations due to the increase in the number of withdrawn young people and households living alone. The lack of daily conversation has been pointed out as a risk that can lead to mental problems such as depression, and serious health problems such as dementia for the elderly. Efforts to encourage daily communication by having communication robots that act as talking partners is attracting attention to solve those problems. One of the challenges of communication robots is the difficulty of maintaining users' motivation to continue using robots. In this study, we propose the communication robot that plays a video game together with a user as an approach to keep the user's motivation high to use the robot. The proposed method not only controls the dialogue content of the robot but also controls the video game situation by manipulating the video game character. In this system, we aim to create an atmosphere where users can enjoy playing video games with the robot together, and the proposed game communication robot can keep their motivation high to use the robot., HRI '22: ACM/IEEE International Conference on Human-Robot Interaction,Sapporo Hokkaido Japan,March 7 - 10, 2022, conference paper

The truncated signed distance function (TSDF) fusion is one of the key operations in the 3D reconstruction process. However, existing TSDF fusion methods usually suffer from the inevitable sensor noises. In this paper, we propose a new TSDF fusion network, named DFusion, to minimize the influences from the two most common sensor noises, i.e., depth noises and pose noises. To the best of our knowledge, this is the first depth fusion for resolving both depth noises and pose noises. DFusion consists of a fusion module, which fuses depth maps together and generates a TSDF volume, as well as the following denoising module, which takes the TSDF volume as the input and removes both depth noises and pose noises. To utilize the 3D structural information of the TSDF volume, 3D convolutional layers are used in the encoder and decoder parts of the denoising module. In addition, a specially-designed loss function is adopted to improve the fusion performance in object and surface regions. The experiments are conducted on a synthetic dataset as well as a real-scene dataset. The results prove that our method outperforms existing methods., journal article

A gentle touch is an essential part of human interaction that produces a positive care effect. Previously, robotics studies have shown that robots can reproduce a gentle touch that elicits similar, positive emotional responses in humans. However, whether the positive emotional effects of a robot’s touch combined with speech can be enhanced using a multimodal approach remains unclear. This study supports the hypothesis that a multimodal interaction combining gentle touch and speech by a robot enhances positive emotional responses. Here, we conducted an experiment using a robotic arm to perform a gentle touch combined with speech and compared three conditions: touch alone, speech alone, and touch with speech. We assessed participants’ subjective ratings of valence, arousal, and human likeliness using subjective emotional responses. Furthermore, we recorded facial electromyography (EMG) from the corrugator supercilii and zygomaticus major muscles and measured skin conductance levels (SCLs) as physiological emotional responses. Our results show that touch combined with speech elicited higher subjective valence and arousal ratings, stronger zygomaticus major EMG and SCL activities than touch alone. The results suggest that the positive emotional effects of robotic touch can be boosted by combining elements of speech., journal article

Lenslet array (LA) near-eye displays (NEDs) are a recent technical development that creates a virtual image in the field of view of one or both eyes. A problem occurs when the user’s pupil moves out of the LA-NED eye box (i.e., cross-talk) making the image look doubled or ghosted. It negatively impacts the user experience. Although eye-gaze tracking can mitigate this problem, the effect of the solution has not been studied to understand the impact of pupil size and human perception. In this paper, we redefine the cross-talk region as the practical pupil movable region (PPMR50), which differs from eye box size because it considers pupil size and human visual perception. To evaluate the effect of eye-gaze tracking on subjective image quality, three user studies were conducted. From the results, PPMR50 was found to be consistent with human perception, and cross-talk elimination via eye-gaze tracking was better understood in a static gaze scenario. Although the system latency prevented the complete elimination of cross-talk for fast movements or large pupil changes, the problem was greatly alleviated. We also analyzed system delays based on PPMR50, which we newly defined in this paper and provided an optimization scheme to meet the maximum eyeball rotation speed., journal article

The truncated signed distance function (TSDF) fusion is one of the key operations in the 3D reconstruction process. However, existing TSDF fusion methods usually suffer from the inevitable sensor noises. In this paper, we propose a new TSDF fusion network, named DFusion, to minimize the influences from the two most common sensor noises, i.e., depth noises and pose noises. To the best of our knowledge, this is the first depth fusion for resolving both depth noises and pose noises. DFusion consists of a fusion module, which fuses depth maps together and generates a TSDF volume, as well as the following denoising module, which takes the TSDF volume as the input and removes both depth noises and pose noises. To utilize the 3D structural information of the TSDF volume, 3D convolutional layers are used in the encoder and decoder parts of the denoising module. In addition, a specially-designed loss function is adopted to improve the fusion performance in object and surface regions. The experiments are conducted on a synthetic dataset as well as a real-scene dataset. The results prove that our method outperforms existing methods.

A gentle touch is an essential part of human interaction that produces a positive care effect. Previously, robotics studies have shown that robots can reproduce a gentle touch that elicits similar, positive emotional responses in humans. However, whether the positive emotional effects of a robot’s touch combined with speech can be enhanced using a multimodal approach remains unclear. This study supports the hypothesis that a multimodal interaction combining gentle touch and speech by a robot enhances positive emotional responses. Here, we conducted an experiment using a robotic arm to perform a gentle touch combined with speech and compared three conditions: touch alone, speech alone, and touch with speech. We assessed participants’ subjective ratings of valence, arousal, and human likeliness using subjective emotional responses. Furthermore, we recorded facial electromyography (EMG) from the corrugator supercilii and zygomaticus major muscles and measured skin conductance levels (SCLs) as physiological emotional responses. Our results show that touch combined with speech elicited higher subjective valence and arousal ratings, stronger zygomaticus major EMG and SCL activities than touch alone. The results suggest that the positive emotional effects of robotic touch can be boosted by combining elements of speech.

Lenslet array (LA) near-eye displays (NEDs) are a recent technical development that creates a virtual image in the field of view of one or both eyes. A problem occurs when the user’s pupil moves out of the LA-NED eye box (i.e., cross-talk) making the image look doubled or ghosted. It negatively impacts the user experience. Although eye-gaze tracking can mitigate this problem, the effect of the solution has not been studied to understand the impact of pupil size and human perception. In this paper, we redefine the cross-talk region as the practical pupil movable region (PPMR50), which differs from eye box size because it considers pupil size and human visual perception. To evaluate the effect of eye-gaze tracking on subjective image quality, three user studies were conducted. From the results, PPMR50 was found to be consistent with human perception, and cross-talk elimination via eye-gaze tracking was better understood in a static gaze scenario. Although the system latency prevented the complete elimination of cross-talk for fast movements or large pupil changes, the problem was greatly alleviated. We also analyzed system delays based on PPMR50, which we newly defined in this paper and provided an optimization scheme to meet the maximum eyeball rotation speed.

Nowadays, decreasing opportunities for people to have daily conversations due to the increase in the number of withdrawn young people and households living alone. The lack of daily conversation has been pointed out as a risk that can lead to mental problems such as depression, and serious health problems such as dementia for the elderly. Efforts to encourage daily communication by having communication robots that act as talking partners is attracting attention to solve those problems. One of the challenges of communication robots is the difficulty of maintaining users' motivation to continue using robots. In this study, we propose the communication robot that plays a video game together with a user as an approach to keep the user's motivation high to use the robot. The proposed method not only controls the dialogue content of the robot but also controls the video game situation by manipulating the video game character. In this system, we aim to create an atmosphere where users can enjoy playing video games with the robot together, and the proposed game communication robot can keep their motivation high to use the robot.

One of the main targets of criticism of head-mounted displays (HMDs) is the field of view (FOV) size, whether in virtual or augmented reality. This limitation is prominent with optical see-through head-mounted displays (OST-HMD), as those with narrow overlay FOV (OFOV) sizes only provide a small window to view virtual objects. We investigated if restricting this OFOV negatively affects a user’s ability to memorize spatial locations in a simulation of a work environment, and consequently, long-term memory transfer to an equivalent scenario in the real world two days later. To find empirical evidence, we conducted a within-subjects experiment with 18 participants performing in three phases with an OST-HMD, simulated on an immersive HMD. For each phase, they viewed the training scenario with a different OFOV size of the augmentable area (30°, 70°, 110° diagonal). Results from recall tests showed that smaller OFOV size did not significantly affect user’s performance on both short-term and transfer tests, but HMD data revealed that users rotated their heads less with a 110° OFOV. We also found that proximity of objects to memorize had an interaction effect with smaller OFOV sizes. Our findings could have implications on the design and HMD choices of augmented training., journal article

application/pdf, Accurately placing virtual objects in a scene is a challenging tasks in handheld augmented reality (HAR). To add and arrange virtual objects in HAR, users must manipulate 6 degrees of freedom (DoFs) of the virtual object, namely: position (3) and orientation (3). However, it is difficult to manipulate all DoFs with the two-dimensional display of the handheld device. We present SlidAR + , a method for controlling the position and orientation of objects in HAR. SlidAR + is an extension of SlidAR [1], a technique that allows users to control the position of a virtual object by manipulating only 1 DoF. We use the direction of gravity as a constraint to improve the user’s control and reduce the time it takes to adjust the orientation. Upon comparing it with a state-of-the-art object manipulation method, using SlidAR + , user were able to complete the tasks faster under our expected conditions and were also preferred by most participants., journal article

application/pdf, Video recording is one of the most commonly used techniques for reflection, because video allows people to know what they look like to others and how they could improve their performance, but it is problematic because some people easily fall into negative emotions and worry about their performance, resulting in a low benefit. In this study, the possibility of applying a simple VR-based reflection method was explored. This method uses virtual reality (VR) and a head-mounted display (HMD) to allow presenters to watch their own presentations from the audience’s perspective and uses an avatar, which hides personal appearance, which has low relevance to the quality of presentation, to help reduce self-awareness during reflection. An experimental study was carried out, considering four personal characteristics—gender, personal anxiety, personal confidence and self-bias. The goal of this study is to discuss which populations can benefit more from this system and to assess the impact of the avatar and HMD-based VR. According to the results, the individuals with low self-confidence in their public speaking skills could benefit more on self-evaluation from VR reflection with HMD, while individuals with negative self-bias could reduce more anxiety by using an avatar., journal article

Robots that can talk with humans play increasingly important roles in society. However,current conversation robots remain unskilled at eliciting empathic feelings in humans. To addressthis problem, we used a robot that speaks in a voice synchronized with human vocal prosody. Weconducted an experiment in which human participants held positive conversations with the robotby reading scenarios under conditions with and without vocal synchronization. We assessed sevensubjective responses related to affective empathy (e.g., emotional connection) and measured thephysiological emotional responses using facial electromyography from the corrugator supercilii andzygomatic major muscles as well as the skin conductance level. The subjective ratings consistentlyrevealed heightened empathic responses to the robot in the synchronization condition comparedwith that under the de-synchronizing condition. The physiological signals showed that more positiveand stronger emotional arousal responses to the robot with synchronization. These findings suggestthat robots that are able to vocally synchronize with humans can elicit empathic emotional responses., journal article

Video recording is one of the most commonly used techniques for reflection, because video allows people to know what they look like to others and how they could improve their performance, but it is problematic because some people easily fall into negative emotions and worry about their performance, resulting in a low benefit. In this study, the possibility of applying a simple VR-based reflection method was explored. This method uses virtual reality (VR) and a head-mounted display (HMD) to allow presenters to watch their own presentations from the audience’s perspective and uses an avatar, which hides personal appearance, which has low relevance to the quality of presentation, to help reduce self-awareness during reflection. An experimental study was carried out, considering four personal characteristics?gender, personal anxiety, personal confidence and self-bias. The goal of this study is to discuss which populations can benefit more from this system and to assess the impact of the avatar and HMD-based VR. According to the results, the individuals with low self-confidence in their public speaking skills could benefit more on self-evaluation from VR reflection with HMD, while individuals with negative self-bias could reduce more anxiety by using an avatar.

Accurately placing virtual objects in a scene is a challenging tasks in handheld augmented reality (HAR). To add and arrange virtual objects in HAR, users must manipulate 6 degrees of freedom (DoFs) of the virtual object, namely: position (3) and orientation (3). However, it is difficult to manipulate all DoFs with the two-dimensional display of the handheld device. We present SlidAR + , a method for controlling the position and orientation of objects in HAR. SlidAR + is an extension of SlidAR [1], a technique that allows users to control the position of a virtual object by manipulating only 1 DoF. We use the direction of gravity as a constraint to improve the user’s control and reduce the time it takes to adjust the orientation. Upon comparing it with a state-of-the-art object manipulation method, using SlidAR + , user were able to complete the tasks faster under our expected conditions and were also preferred by most participants.

One of the main targets of criticism of head-mounted displays (HMDs) is the field of view (FOV) size, whether in virtual or augmented reality. This limitation is prominent with optical see-through head-mounted displays (OST-HMD), as those with narrow overlay FOV (OFOV) sizes only provide a small window to view virtual objects. We investigated if restricting this OFOV negatively affects a user’s ability to memorize spatial locations in a simulation of a work environment, and consequently, long-term memory transfer to an equivalent scenario in the real world two days later. To find empirical evidence, we conducted a within-subjects experiment with 18 participants performing in three phases with an OST-HMD, simulated on an immersive HMD. For each phase, they viewed the training scenario with a different OFOV size of the augmentable area (30°, 70°, 110° diagonal). Results from recall tests showed that smaller OFOV size did not significantly affect user’s performance on both short-term and transfer tests, but HMD data revealed that users rotated their heads less with a 110° OFOV. We also found that proximity of objects to memorize had an interaction effect with smaller OFOV sizes. Our findings could have implications on the design and HMD choices of augmented training.

Robots that can talk with humans play increasingly important roles in society. However,current conversation robots remain unskilled at eliciting empathic feelings in humans. To addressthis problem, we used a robot that speaks in a voice synchronized with human vocal prosody. Weconducted an experiment in which human participants held positive conversations with the robotby reading scenarios under conditions with and without vocal synchronization. We assessed sevensubjective responses related to affective empathy (e.g., emotional connection) and measured thephysiological emotional responses using facial electromyography from the corrugator supercilii andzygomatic major muscles as well as the skin conductance level. The subjective ratings consistentlyrevealed heightened empathic responses to the robot in the synchronization condition comparedwith that under the de-synchronizing condition. The physiological signals showed that more positiveand stronger emotional arousal responses to the robot with synchronization. These findings suggestthat robots that are able to vocally synchronize with humans can elicit empathic emotional responses.

This paper proposes emotion amplification for TV chat agents allowing users to get more excited in TV sports programs, and a model that estimates the excitement level of TV programs based on the number of social comment posts. The proposed model extracts the exciting intervals from social comments to the program scenes. By synchronizing recorded video streams and the intervals, the agents may talk with the user dynamically changing the frequency and volume of upbeat utterances, increasing the excitement of the user. To test these agents, participants watched TV content under three conditions: without an agent, with four agents that utter with a flat voice, and with four agents with emotion amplification. Results from 24 young adult Japanese individuals showed that their arousal of participants’ subjective and physiological emotional responses were boosted because of the agents, enhancing their motivation to interact with the agent in the future. With empirical evidence, this paper supports these expectations and demonstrates that these agents can amplify the positive emotions of TV watchers, enhancing their motivation to interact with the agent in the future., journal article

With projection mapping, we can control the appearance of real-world objects by adding illumination. A projector can be used to control light reflected from an object, where the reflected light depends not only on the projection but also on the reflectance and environmental light. Because the resulting colors are affected by the reflectance and environmental light, the presentable color range of a projector is limited. The purpose of this work is to broaden this limited range by focusing on the perceived colors. Although our eyes capture reflected light to perceive the colors of an object, the colors perceived by humans are not always the same as the actual colors, and there are often significant differences between them because of the human visual system. To overcome the limitations of a projector based on human perception, we intentionally generate this difference by inducing a visual illusion, namely, color constancy. In this work, we designed an algorithm to determine the projected colors for presenting the desired colors perceptually by employing a color constancy effect. In addition, we conducted a user study and confirmed that our algorithm can (1) create a misperception regarding the color of illumination, (2) broaden the presentable color range of a projector, and (3) shift the perceptual colors in the desirable direction., journal article

application/pdf, In this paper, factors with positive effects in the playback of virtual reality (VR) presentation in training are discussed. To date, the effectiveness of VR public speaking training in both anxiety reduction and skills improvement has been reported. Though the playback using videotape is an effective way in original public speaking training, very few researchers focused on the effectiveness and possibility of VR playback. In this research, A VR playback system for public speaking training is proposed, and a pilot experiment is carried out, so as to figure out the effects of the virtual agent, immersion and public speaking anxiety level in VR playback., ICMI ’20 Companion, October 25–29, 2020, Virtual event, Netherlands, conference paper

With projection mapping, we can control the appearance of real-world objects by adding illumination. A projector can be used to control light reflected from an object, where the reflected light depends not only on the projection but also on the reflectance and environmental light. Because the resulting colors are affected by the reflectance and environmental light, the presentable color range of a projector is limited. The purpose of this work is to broaden this limited range by focusing on the perceived colors. Although our eyes capture reflected light to perceive the colors of an object, the colors perceived by humans are not always the same as the actual colors, and there are often significant differences between them because of the human visual system. To overcome the limitations of a projector based on human perception, we intentionally generate this difference by inducing a visual illusion, namely, color constancy. In this work, we designed an algorithm to determine the projected colors for presenting the desired colors perceptually by employing a color constancy effect. In addition, we conducted a user study and confirmed that our algorithm can (1) create a misperception regarding the color of illumination, (2) broaden the presentable color range of a projector, and (3) shift the perceptual colors in the desirable direction.

This paper proposes emotion amplification for TV chat agents allowing users to get more excited in TV sports programs, and a model that estimates the excitement level of TV programs based on the number of social comment posts. The proposed model extracts the exciting intervals from social comments to the program scenes. By synchronizing recorded video streams and the intervals, the agents may talk with the user dynamically changing the frequency and volume of upbeat utterances, increasing the excitement of the user. To test these agents, participants watched TV content under three conditions: without an agent, with four agents that utter with a flat voice, and with four agents with emotion amplification. Results from 24 young adult Japanese individuals showed that their arousal of participants’ subjective and physiological emotional responses were boosted because of the agents, enhancing their motivation to interact with the agent in the future. With empirical evidence, this paper supports these expectations and demonstrates that these agents can amplify the positive emotions of TV watchers, enhancing their motivation to interact with the agent in the future.

In this paper, factors with positive effects in the playback of virtual reality (VR) presentation in training are discussed. To date, the effectiveness of VR public speaking training in both anxiety reduction and skills improvement has been reported. Though the playback using videotape is an effective way in original public speaking training, very few researchers focused on the effectiveness and possibility of VR playback. In this research, A VR playback system for public speaking training is proposed, and a pilot experiment is carried out, so as to figure out the effects of the virtual agent, immersion and public speaking anxiety level in VR playback.

application/pdf, Color Vision Deficiency (CVD) is often characterized by the inabilityto distinguish color due to a defective or missing cone in the eye.Although it is possible to modify the observed color to make iteasier for users to distinguish, this can lead to color confusion withunaffected colors. To address this problem we investigate how flickercan assist distinguishing colors for CVD patients. In preliminarystudy, we evaluated the efficiency of color and brightness modulationwith 4 participants with normal vision. Our findings suggests thatwhile brightness modulation was ineffective, color modulation canhelp users distinguish between different colors, conference paper

Color Vision Deficiency (CVD) is often characterized by the inabilityto distinguish color due to a defective or missing cone in the eye.Although it is possible to modify the observed color to make iteasier for users to distinguish, this can lead to color confusion withunaffected colors. To address this problem we investigate how flickercan assist distinguishing colors for CVD patients. In preliminarystudy, we evaluated the efficiency of color and brightness modulationwith 4 participants with normal vision. Our findings suggests thatwhile brightness modulation was ineffective, color modulation canhelp users distinguish between different colors

application/pdf, This article describes a Mixed-Reality (MR) application that superimposes lost buildings of a historical site onto real scenes virtualized using spherical aerial images. The proposed application is set at a UNESCO World Heritage site in Japan, and is based on a novel framework that supports the photorealistic superimposition of virtual objects onto virtualized real scenes. The proposed framework utilizes Image-Based Rendering (IBR), which enables users to freely change their viewpoint in a real-world virtualization constructed using precaptured images. This framework combines the offline rendering of virtual objects and IBR to take advantage of the higher quality of offline rendering without the additional computational cost of online processing; that is, it incurs only the cost of online lightweight IBR, which is simplified through the pregeneration of structured viewpoints (e.g., at grid points)., journal article

This article describes a Mixed-Reality (MR) application that superimposes lost buildings of a historical site onto real scenes virtualized using spherical aerial images. The proposed application is set at a UNESCO World Heritage site in Japan, and is based on a novel framework that supports the photorealistic superimposition of virtual objects onto virtualized real scenes. The proposed framework utilizes Image-Based Rendering (IBR), which enables users to freely change their viewpoint in a real-world virtualization constructed using precaptured images. This framework combines the offline rendering of virtual objects and IBR to take advantage of the higher quality of offline rendering without the additional computational cost of online processing; that is, it incurs only the cost of online lightweight IBR, which is simplified through the pregeneration of structured viewpoints (e.g., at grid points).

application/pdf, This paper describes a framework for aerial imaging of high dynamic range (HDR) scenes for use in virtual reality applications, such as immersive panorama applications and photorealistic superimposition of virtual objects using image-based lighting. We propose a complete and practical system to acquire full spherical HDR images from the sky, using two omnidirectional cameras mounted above and below an unmanned aircraft. The HDR images are generated by combining multiple omnidirectional images captured with different exposures controlled automatically. Our system consists of methods for image completion, alignment, and color correction, as well as a novel approach for automatic exposure control, which selects optimal exposure so as to avoid banding artifacts. Experimental results indicated that our system generated better spherical images compared to an ordinary spherical image completion system in terms of naturalness and accuracy. In addition to proposing an imaging method, we have carried out an experiment about display methods for aerial HDR immersive panoramas utilizing spherical images acquired by the proposed system. The experiment demonstrated HDR imaging is beneficial to immersive panorama using an HMD, in addition to ordinary uses of HDR images., journal article

This paper describes a framework for aerial imaging of high dynamic range (HDR) scenes for use in virtual reality applications, such as immersive panorama applications and photorealistic superimposition of virtual objects using image-based lighting. We propose a complete and practical system to acquire full spherical HDR images from the sky, using two omnidirectional cameras mounted above and below an unmanned aircraft. The HDR images are generated by combining multiple omnidirectional images captured with different exposures controlled automatically. Our system consists of methods for image completion, alignment, and color correction, as well as a novel approach for automatic exposure control, which selects optimal exposure so as to avoid banding artifacts. Experimental results indicated that our system generated better spherical images compared to an ordinary spherical image completion system in terms of naturalness and accuracy. In addition to proposing an imaging method, we have carried out an experiment about display methods for aerial HDR immersive panoramas utilizing spherical images acquired by the proposed system. The experiment demonstrated HDR imaging is beneficial to immersive panorama using an HMD, in addition to ordinary uses of HDR images.

application/pdf, This study proposes a framework to photorealistically synthesize virtual objects and virtualized real-world. We combine the offline rendering of virtual objects and the free-viewpoint image generation to take advantage of the higher quality of offline rendering without the computational cost of online computer graphics (CG) rendering; i.e., it incurs only the cost of the online computation for the free-viewpoint image generation. In addition, the generation of structured viewpoints (e.g., at every grid point) reduces the computational costs required to online process., ISMAR 2013 : IEEE and ACM International Symposium on Mixed and Augmented Reality , Oct 1-4, 2013 , Adelaide, Australia, conference paper

application/pdf, This study proposes a framework to photorealistically synthesize virtual objects and virtualized real-world. We combine the offline rendering of virtual objects and the free-viewpoint image generation to take advantage of the higher quality of offline rendering without the computational cost of online computer graphics (CG) rendering; i.e., it incurs only the cost of the online computation for the free-viewpoint image generation. In addition, the generation of structured viewpoints (e.g., at every grid point) reduces the computational costs required to online process.

application/pdf, We have developed an augmented telepresence system which enables virtual tourism beyond time and space. Augmented telepresence provides a user with both the view of a remote location and related information using augmented reality techniques. This study deals with the geometric and photometric registration problems to generate movie-quality augmented omnidirectional videos automatically. The user can look around the scene from the sky above Heijo palace Site which is an ancient capital in Nara, Japan in the technical demonstration., MM 2012 : 20th ACM international conference on Multimedia , Oct 29-Nov 2, 2012 , Nara, Japan, conference paper

application/pdf, This paper describes a method for acquiring full spherical high dynamic range (HDR) images with no missing areas by using two omni directional cameras mounted on the top and bottom of an unmanned airship. The full spherical HDR images are generated by combining multiple omni directional images that are captured with different shutter speeds. The images generated are intended for uses in telepresence, augmented telepresence, and image-based lighting., ICME 2012 : IEEE International Conference on Multimedia and Expo , Jul 9-13, 2012 , Melbourne, Australia, conference paper

application/pdf, This paper describes a method for acquiring full spherical high dynamic range (HDR) images with no missing areas by using two omni directional cameras mounted on the top and bottom of an unmanned airship. The full spherical HDR images are generated by combining multiple omni directional images that are captured with different shutter speeds. The images generated are intended for uses in telepresence, augmented telepresence, and image-based lighting.

application/pdf, We have developed an augmented telepresence system which enables virtual tourism beyond time and space. Augmented telepresence provides a user with both the view of a remote location and related information using augmented reality techniques. This study deals with the geometric and photometric registration problems to generate movie-quality augmented omnidirectional videos automatically. The user can look around the scene from the sky above Heijo palace Site which is an ancient capital in Nara, Japan in the technical demonstration.

application/pdf, This study developed a virtual tourism system "beyond time and space" using augmented telepresence. Telepresence is a technique that provides a view of a remote site using immersive displays. In this paper, augmented telepresence (AT) refers to a kind of telepresence that provides a user with both a view of a remote site and related information using augmented reality (AR) techniques., SIGGRAPH '11 : The 38rd International Conference and Exhibition on Computer Graphics and Interactive Techniques , Aug 7-11, 2011 , Vancouver, Canada, conference paper

application/pdf, This study developed a virtual tourism system "beyond time and space" using augmented telepresence. Telepresence is a technique that provides a view of a remote site using immersive displays. In this paper, augmented telepresence (AT) refers to a kind of telepresence that provides a user with both a view of a remote site and related information using augmented reality (AR) techniques.

application/pdf, This paper proposes a novel method for generating arbitrary stereoscopic view from multiple omni directional image sequences. Although conventional methods for arbitrary view generation with an image-based rendering approach can create binocular views, positions and directions of viewpoints for stereoscopic vision are limited to a small range. In this research, we attempt to generate arbitrary stereoscopic views from omni directional image sequences that are captured in various multiple paths. To generate a high-quality stereoscopic view from a number of images captured at various viewpoints, appropriate ray information needs to be selected. In this paper, appropriate ray information is selected from a number of omni directional images using a penalty function expressed as ray similarity. In experiments, we show the validity of this penalty function by generating stereoscopic view from multiple real image sequences., ICPR 2010 : 20th International Conference on Pattern Recognition , Aug 23-26, 2010 , Istanbul, Turkey, conference paper

application/pdf, This study is concerned with a large-scale telepresence system based on remote control of mobile robot or aerial vehicle. The proposed system provides a user with not only view of remote site but also related information by AR technique. Such systems are referred to as augmented telepresence in this paper. Aerial imagery can capture a wider area at once than image capturing from the ground. However, it is difficult for a user to change position and direction of viewpoint freely because of the difficulty in remote control and limitation of hardware. To overcome these problems, the proposed system uses an autopilot airship to support changing user's viewpoint and employs an omni-directional camera for changing viewing direction easily. This paper describes hardware configuration for aerial imagery, an approach for overlaying virtual objects, and automatic control of the airship, as well as experimental results using a prototype system., ISMAR 2010 : 9th IEEE International Symposium on Mixed and Augmented Reality , Oct 13-16, 2010 , Seoul, Korea, conference paper

application/pdf, This paper proposes a method for removing image inconsistencies which occur by an existence of moving objects or a change of illumination condition when an omnidirectional image database is generated. The database is used for archiving an outdoor scene in wide areas or generating novel view images with an image-based rendering approach. In related work, it is difficult to remove moving objects in an outdoor environment where illumination condition drastically changes, and to remove inconsistencies of color tone of images which included moving objects. The proposed method iterates the two processes which are the estimation of candidate region of moving objects and the achievement of color consistency to split regions. The color consistency is achieved by estimating linear color transformation parameters which change a histogram of an input image to that of the standard image., ACCV 2010 : 10th Asian Conference on Computer Vision , Nov 8-12, 2010 , Queenstown, New Zealand, conference paper

application/pdf, This paper proposes a novel method for generating arbitrary stereoscopic view from multiple omni directional image sequences. Although conventional methods for arbitrary view generation with an image-based rendering approach can create binocular views, positions and directions of viewpoints for stereoscopic vision are limited to a small range. In this research, we attempt to generate arbitrary stereoscopic views from omni directional image sequences that are captured in various multiple paths. To generate a high-quality stereoscopic view from a number of images captured at various viewpoints, appropriate ray information needs to be selected. In this paper, appropriate ray information is selected from a number of omni directional images using a penalty function expressed as ray similarity. In experiments, we show the validity of this penalty function by generating stereoscopic view from multiple real image sequences.

application/pdf, This paper proposes a method for removing image inconsistencies which occur by an existence of moving objects or a change of illumination condition when an omnidirectional image database is generated. The database is used for archiving an outdoor scene in wide areas or generating novel view images with an image-based rendering approach. In related work, it is difficult to remove moving objects in an outdoor environment where illumination condition drastically changes, and to remove inconsistencies of color tone of images which included moving objects. The proposed method iterates the two processes which are the estimation of candidate region of moving objects and the achievement of color consistency to split regions. The color consistency is achieved by estimating linear color transformation parameters which change a histogram of an input image to that of the standard image.

application/pdf, This study is concerned with a large-scale telepresence system based on remote control of mobile robot or aerial vehicle. The proposed system provides a user with not only view of remote site but also related information by AR technique. Such systems are referred to as augmented telepresence in this paper. Aerial imagery can capture a wider area at once than image capturing from the ground. However, it is difficult for a user to change position and direction of viewpoint freely because of the difficulty in remote control and limitation of hardware. To overcome these problems, the proposed system uses an autopilot airship to support changing user's viewpoint and employs an omni-directional camera for changing viewing direction easily. This paper describes hardware configuration for aerial imagery, an approach for overlaying virtual objects, and automatic control of the airship, as well as experimental results using a prototype system.

application/pdf, This paper presents a telepresence system for a ride, such as on a roller coaster, using a motion platform that can provide a seated user with the sensation of inertial force. Most conventional studies using a motion platform with a few degrees of freedom have not generated an inertial force when a ride accelerates, because a motion platform cannot simulate the same motion of a real roller coaster. We propose a new telepresence system that can provide a user with an inertial force sensation using a motion platform with a few degrees of freedom and an immersive display. In our research, the inertial force sensation is generated by acceleration of gravity produced by inclining the motion platform. The inclination of the seated user is estimated from an image sequence captured using an omnidirectional camera placed on an actual running roller coaster. In our experiments, the inertial force sensation is realized using a motion platform and an immersive display., 3DUI 2009 : IEEE Symposium on 3D User Interfaces , Mar 14-15, 2009 , Lafayette, LA, USA, conference paper

application/pdf, This paper describes a mixed reality (MR) telepresence system for a ride to provide users with a highly realistic sensation. To make a realistic scene in a virtual environment, it is necessary to combine visual information with a reproduction of the forces which a user experiences in the real environment. This paper proposes an MR telepresence system that presents a realistic image and an inertial force sensation using an immersive display and a motion base with limited degrees of freedom. In our approach, the realistic image is acquired with an omnidirectional camera and the inertial force is generated virtually by a combination of the acceleration of gravity and a video effect. In experiments, a prototype system has been proven to produce a highly realistic sensation in various environments., ACE 2009 : International Conference on Advances in Computer Enterntainment Technology , Oct 29-31, 2009 , Athens, Greece, conference paper

application/pdf, In annotation overlay applications using augmented reality (AR), view management is widely used for improving readability and intelligibility of the annotations. In order to recognize the visible portions of objects in the user's view, the positions, orientations, and shapes of the objects should be known in the case of conventional view management methods. However, it is difficult for a wearable AR system to obtain the positions, orientations and shapes of objects because the target object is usually moving or non-rigid. In this paper, we propose a view management method to overlay annotations of moving or non-rigid objects for networked wearable AR. The proposed method obtains positions and shapes of target objects via a network in order to estimate the visible portions of the target objects in the user's view. Annotations are located by minimizing penalties related to the overlap of an annotation, occlusion of target objects, length of a line between the annotation and the target object, and distance of the annotation in sequential frames. Through experiments, we have proven that the prototype system can correctly provide each user with annotations on multiple users of wearable AR systems., ICME 2009 : IEEE International Conference on Multimedia & Expo , Jun 28-Jul 3, 2009 , New York, NY, USA, conference paper

application/pdf, This paper presents a telepresence system for a ride, such as on a roller coaster, using a motion platform that can provide a seated user with the sensation of inertial force. Most conventional studies using a motion platform with a few degrees of freedom have not generated an inertial force when a ride accelerates, because a motion platform cannot simulate the same motion of a real roller coaster. We propose a new telepresence system that can provide a user with an inertial force sensation using a motion platform with a few degrees of freedom and an immersive display. In our research, the inertial force sensation is generated by acceleration of gravity produced by inclining the motion platform. The inclination of the seated user is estimated from an image sequence captured using an omnidirectional camera placed on an actual running roller coaster. In our experiments, the inertial force sensation is realized using a motion platform and an immersive display.

application/pdf, This paper describes a mixed reality (MR) telepresence system for a ride to provide users with a highly realistic sensation. To make a realistic scene in a virtual environment, it is necessary to combine visual information with a reproduction of the forces which a user experiences in the real environment. This paper proposes an MR telepresence system that presents a realistic image and an inertial force sensation using an immersive display and a motion base with limited degrees of freedom. In our approach, the realistic image is acquired with an omnidirectional camera and the inertial force is generated virtually by a combination of the acceleration of gravity and a video effect. In experiments, a prototype system has been proven to produce a highly realistic sensation in various environments.

application/pdf, In annotation overlay applications using augmented reality (AR), view management is widely used for improving readability and intelligibility of the annotations. In order to recognize the visible portions of objects in the user's view, the positions, orientations, and shapes of the objects should be known in the case of conventional view management methods. However, it is difficult for a wearable AR system to obtain the positions, orientations and shapes of objects because the target object is usually moving or non-rigid. In this paper, we propose a view management method to overlay annotations of moving or non-rigid objects for networked wearable AR. The proposed method obtains positions and shapes of target objects via a network in order to estimate the visible portions of the target objects in the user's view. Annotations are located by minimizing penalties related to the overlap of an annotation, occlusion of target objects, length of a line between the annotation and the target object, and distance of the annotation in sequential frames. Through experiments, we have proven that the prototype system can correctly provide each user with annotations on multiple users of wearable AR systems.

application/pdf, We propose a user localization system that uses invisible markers for wearable augmented reality (AR) in large indoor environments. Wearable AR systems have received a great deal of attention as a new method for displaying location-based information in the real world. For using wearable AR systems, it is necessary to measure the position and orientation of a user using a positioning infrastructure without the undesirable visual effects that arise from merging real and virtual worlds. In addition, the infrastructure of the localization environment must be constructed easily and cheaply. The proposed system can estimate the position and orientation of a user precisely by affixing wallpapers containing printed invisible markers on ceilings or walls. The user's position and orientation are estimated by recognizing the markers using an infrared camera with infrared LEDs. To construct environments for the localization system, we developed an initialization tool that calibrates the alignment of the markers from photographs taken by flash illumination using a digital still camera., VRST 2008 : ACM symposium on Virtual reality software and technology , Oct 27-29, 2008 , Bordeaux, France, conference paper

application/pdf, This paper describes a real-time pre-visualization method using augmented reality techniques for videographers. It enables them to test camera work without real actors in a real environment. As a substitute for real actors, virtual ones are superimposed on a live video in real-time according to a real camera motion and an illumination condition. The key technique of this method is real-time motion estimation of a camera, which can be applied to unknown complex environments including natural objects. In our method, geometric and photometric registration problems for such unknown environments are solved to realize the above visualization. A prototype system demonstrates availability of the previsualization method., ICME 2008 : IEEE International Conference on Multimedia & Expo , Jun 23-26, 2008 , Hannover, Germany, conference paper