Search

Robust real-time stereo facial feature tracking is one of the important research topics for a variety multimodal Human-Computer, and human robot Interface applications, including telepresence, face recognition, multimodal voice recognition, and perceptual user interfaces (Moghaddam et al., 1996; Moghaddam et al., 1998; Yehia et al., 1988). Since the motion of a person's facial features and the direction of the gaze is largely related to person's intention and attention, detection of such motions with their 3D real measurement values can be utilized as a natural way of communication for human robot interaction. For example, addition of visual speech information to robot's speech recognizer unit clearly meets at least two practicable criteria: It mimics human visual perception of speech recognition, and it may contain information that is not always present in the acoustic domain (Gurbuz et al., 2001). Another application example is enhancing the social interaction between humans and humanoid agents with robots learning human-like mouth movements from human trainers during speech (Gurbuz et al., 2004; Gurbuz et al., 2005). The motivation of this research is to develop an algorithm to track the facial features using stereo vision system in real world conditions without using prior training data. We also demonstrate the stereo tracking system through a human to humanoid robot mouth mimicking task. Videre stereo vision hardware and SVS software system are used for implementing the algorithm. This work is organized as follows. In section 2, related earlier works are described. Section 3 discusses face RIO localization. Section 4 presents the 2D lip contour tracking and its extention to 3D. Experimental results and discussions are presented in Section 5. Conclusion is given in Section 6. Finally, future extention is described in Section 7.

In this paper we present a stereovision based model free 3D head pose (orientation and position) estimation system suitable for human-machine interface applications. The system works by obtaining a 'face plane' from the 3D reconstructed face data, which is then used for head pose estimation. The key novelty in this work is the utilization of the face plane together with the eye locations on the reconstructed face data to obtain a robust head pose estimate. This approach leads to a model and initialization free head pose estimation system; therefore it is suitable for natural human-machine interfaces. In order to quantitatively asses the accuracy of the system for such applications, several evaluation experiments were conducted using a commercial motion capture system. The evaluation results indicate that this system can be used in human-computer and human-robot applications.

We studied highly realistic communication systems using a super multiview three-dimensional (3-D) video system. We propose the developed glasses-free 200-in. 3-D display using ∼200 high-definition (HD) projector units to reconstruct natural life-size 3-D moving objects, such as cars and humans. We also analyzed the optimal arrangement of the multiview camera system for the 3-D display. In the experiments, we developed a prototype super multiview 3-D camera system using compact HD video cameras with real-time convergence compensation circuits that correct the captured images via image processing optimized for the display. We also performed demonstration experiments using the developed 3-D display and camera system in a public area. As a result, we highlight several possibilities for actual applications of the glasses-free super multiview 3-D video system.

We propose a cubic 3D display named “gCubik” which allows multiple users to simultaneously view and share 3D images. The display uses an auto-stereoscopic technique. Therefore, users can see 3D objects without the need for special 3D glasses. Its cubic shape enables users to view from arbitrary angles of 360 degree by handling the cube as if the 3D objects were in a glazed-showcase. We describe an implementation approach of the gCubik. First, we investigate the necessary parameters for fabricating a cubic auto-stereoscopic display based on the integral photography (IP) technique. Next, according to these parameters, we discuss our design of a wide field of view lens array for the IP technique. Then, we explain the construction of the cubic device using the six IP displays and the application of an OpenGL-based rendering algorithm for each lens. Finally, we discuss the produced 3D images by observing them from several different viewpoints.

Neural correlates of driving and of decision making have been investigated separately, but little is known about the underlying neural mechanisms of decision making in driving. Previous research discusses two types of decision making: reward‐weighted decision making and cost‐weighted decision making. There are many reward‐weighted decision making neuroimaging studies but there are few cost‐weighted studies. Considering that driving involves serious risk, it is assumed that decision making in driving is cost weighted. Therefore, neural substrates of cost‐weighted decision making can be assessed by investigation of driver's decision making. In this study, neural correlates of resolving uncertainty in driver's decision making were investigated. Turning right in left‐hand traffic at a signalized intersection was simulated by computer graphic animation based videos. When the driver's view was occluded by a big truck, the uncertainty of the oncoming traffic was resolved by an in‐car video assist system that presented the driver's occluded view. Resolving the uncertainty reduced activity in a distributed area including the amygdala and anterior cingulate. These results implicate the amygdala and anterior cingulate as serving a role in cost‐weighted decision making. Hum Brain Mapp 2009. © 2008 Wiley‐Liss, Inc.

MGG (Minimal Generation Gap) is one of popular generation alternation models for Genetic Algorithms (GAs). The conventional MGG is effective for single population GAs, but not for multi-population GAs. This paper proposes ``MGG with global selection (MGGGS)'' that is designed for multi-population GAs. In MGGGS, the selection operation is carried out through the whole population, while the crossover operation is restricted in sub-populations. Experiments are carried out to analyze the characteristics of MGGGS with Dynamically Separating Genetic Algorithm (DS-GA). In DS-GA the sub-populations are reconstructed during the evolution, which is suitable for MGGGS. The experimental results show that MGGGS outperforms the conventional MGG especially for multimodal functions, since sub-populations explore various areas by MGGGS.

The Music Table is an augmented reality system for composing music by manipulating objects on a tabletop as a physicalized representation of the music being heard. Educational theory, and the apparent success of related applications in various learning contexts, seems to support this idea. In our experiments with children, all were able to make a musical pattern and made many changes to their pattern over a short period of time. We propose its suitability as an educational tool, particularly in short and intense interactive learning situations such as children's museums. We discuss some future developments of the idea.

SECOND-MVD (Synthesis Error COmpeNsateD Multiview Video plus Depth) is an alternative 3D format that we introduce for representation of multiview video. In this data format, images at some viewpoint remain original, and the others are converted to a novel format. Residual based representation, such as layered depth video and free-viewpoint TV data unit were proposed. We propose hybrid image that not only consists of residual but also remainder pixels. Generation and reconstruction process of a hybrid image uses virtual image synthesized by the images that remained original in SECOND-MVD. In this paper, we investigate the compression performance of SECOND-MVD using hybrid image. Experiments demonstrate reduction in bit rate using hybrid image against residual image in SECOND-MVD framework.

It is important to give useful clues for selecting desiredcontent from a number of retrieval results obtained (usually) from avague search request. Compared with monolingual retrieval, such asupport framework is inevitable and much more significant for filteringgiven translingual retrieval results. This paper describes an attempt toprovide appropriate translation of major keywords in each document in across-language information retrieval (CLIR) result, as a browsingsupport for users. Our idea of determining appropriate translation ofmajor keywords is based on word co-occurrence distribution in thetranslation target language, considering the actual situation of WWWcontent where it is difficult to obtain aligned parallel (multilingual)corpora. The proposed method provides higher quality of keywordtranslation to yield a more effective support in identifying the targetdocuments in the retrieval result. We report the advantage of thisbrowsing support technique through evaluation experiments includingcomparison with conditions of referring to a translated documentsummary, and discuss related issues to be examined towards moreeffective cross-language information extraction.

National Institute of Information and Communications Technology (NICT) is conducting ultra-realistic communication research since 2006. The aim of this research is to create a sense of “being there” for natural and realistic communication. In this research project, NICT has developed some novel glasses-free 3D display systems. Among these systems, a large-size glasses-free 3D display using projector array and an electronic holography system using laser light are introduced in this paper.

National Institute of Information and Communications Technology (NICT) is conducting ultra-realistic communication research since 2006. The aim of this research is to create a sense of “being there” for natural and realistic communication. In this research project, NICT has developed some novel glasses-free 3D display systems. Among these systems, a large-size glasses-free 3D display using super-multi-view 3D video technologies and an electronic holography system using laser light are introduced in this paper.

We investigate highly realistic communication systems using a super-multi-view three-dimensional (3D) video system. We propose and develop a glasses-free 200-view 3D display using almost 200 high-definition (HD) projector units to reconstruct such natural life-size 3D moving objects as cars and humans. We are also developing a 3D capturing system with HD camera units and researching multi-view video transmission techniques for demonstration experiments for the abovementioned highly realistic communication system.

There are several representations for 3D scene that are used for 3DTV and free viewpoint imaging systems. A well-known representation is sparse view-plus-depth. Based on view-plus-depth, layered depth video (LDV), and FTV data Unit (FDU) were proposed. The represented images by them are based on residual values. For view data in view-plus-depth, we propose a novel representation; consist of original and hybrid images that are repeated consecutively. A hybrid image has residual and reminder pixels. A virtual image is synthesized in both generation and reconstruction of the hybrid image. The reconstruction is the compensation of the virtual image using the hybrid image. The reconstructed hybrid image has higher PSNR and fewer visible artifacts than the virtual view compensated by residual or reminder image. The data size of the hybrid images is also less than residual and reminder images.

Using diethyldithiocarbamate (DEDTC), a zinc chelator, we deactivated the mossy fibers that project from the dentate gyrus (DG) to the CA3 during acquisition and testing of a center detection task in mice. The mice were trained to find a food pellet at the center of four objects in a circular area. DEDTC injection just before the training sessions impaired this learning, whereas DEDTC injection before the probe test did not impair recall of the memory. DEDTC injection before a pattern completion test in which only one of the four objects was presented did not cause deficits in this test. DEDTC injection did, however, cause severe deficits in an array shift test in which all four objects were moved to new positions. These results demonstrated that 1) the DG-CA3 system plays a crucial role in the learning of geometric center detection task but not in its recall or pattern completion, and 2) the DG-CA3 system is involved in generalization to a new environment but is not crucial for pattern completion.

We have proposed a glasses-free three-dimensional (3D) display for displaying 3D images on a large screen using multi-projectors and an optical screen consisting of a special diffuser film with a large condenser lens. To achieve high presence communication with natural large-screen 3D images, we numerically analyze the factors responsible for degrading image quality to increase the image size. A major factor that determines the 3D image quality is the arrangement of component units, such as the projector array and condenser lens, as well as the diffuser film characteristics. We design and fabricate a prototype 200-inch glasses-free 3D display system on the basis of the numerical results. We select a suitable diffuser film, and we combine it with an optimally designed condenser lens. We use 57 high-definition projector units to obtain viewing angles of 13.5°. The prototype system can display glasses-free 3D images of a life-size car using natural parallax images.

We have already developed glasses-free three-dimensional (3-D) displays using multi-projectors and a special diffuser screen that results in a highly realistic communication system. The system can display 70-200 inch large-sized 3-D images with full high-definition video image quality. The displayed 3-D images were, however, only computergenerated graphics or still images of actual objects. In this work, we studied a 3-D video capturing method for our multiprojection 3-D display. We analyzed the optimal arrangement of cameras for the display, and the image quality as influenced by calibration error. In the experiments, we developed a prototype multi-camera system using 30 highdefinition video cameras. The captured images were corrected via image processing optimized for the display. We successfully captured and displayed, for the first time, 3-D video of actual moving objects in our glasses-free 3-D video system.

National Institute of Information and Communications Technology (NICT) is conducting ultra-realistic communication research since 2006. The aim of the research is to create a sense of "being there" for natural and realistic communication by developing 3D image display, 3D auditory display, multisensory interaction system with vision, sound, touch and smell, and evaluation technologies of human cognitive dynamism[Inoue 2010]. Digital museum is regarded as a good application of the research, but the potential is not clarified yet. This year, we had a chance to investigate the potential by exhibiting developed systems and allowing users to experience them. A multisensory interaction system and other systems related to digital museum were exhibited for 5 weeks as a prototype of digital museum. In this paper, the exhibited systems and the evaluation results are introduced.

National Institute of Information and Communications Technology (NICT) is conducting ultra-realistic communication research since 2006. The aim of the research is to create a sense of “being there” for natural and realistic communication by developing 3D image/sound technology and multisensory interfaces, and also by investigating human perceptual and cognitive mechanisms in order to develop measurement techniques. For 3D image technology, NICT is focused on developing glasses-free 3D display systems such as with electronic holography, a super multi-view display and a handheld box-shaped 3D display based on the integral photography technique. In this paper, the current status of our research is described.

Previous works that employed an automatic registration of multiple range views resorted to the use of corresponding 3D matching points in order to compute relative orientations. False 3D matches were detected by verifying their compliance with the parameteres of the orientation's mathematical model computed in 3D space. In this research we introduce a new method to detect correct 3D matches. This method employs first the removal of the false 2D matches. Subsequently, 3D matches are detected by using the one to one correspondences provided by the scanning device between the 3D points and their corresponding 2D image projections. Due to its ability to detect the matching points in a fast manner, our method is suited for applications characterized by a 3D fast reconstruction demand. Experimental results with real object demonstrate the effectiveness of our method.

Head-related transfer functions (HRTFs) are essential for creating a virtual sound source when sound waves are transmitted through headphones. The measurement of HRTFs is a complicated and time-consuming task. Therefore, the interpolation of HRTFs is crucial for virtual auditory display systems in which both listeners and sound objects are likely to move in a virtual auditory space. In this study, HRTFs are measured with a high spatial resolution in order to develop an effective interpolation method. An analysis of HRTFs indicates that the functions exhibit periodicity in amplitude along the azimuthal angle. The optimum filter coefficients required to interpolate HRTFs from several functions measured along multiple azimuthal directions are derived. Computer simulations indicate that in comparison to conventional methods, the proposed method yields less estimation error in the interpolation of HRTFs. Listening tests indicate that the proposed method can provide better perception of virtual sound.

Collaborative tasks requiring the sharing of an object can benefit from a compact, group-shared autostereoscopic display. We have analyzed the requirements for a cubic autostereoscopic display employing integral photography (IP). The display should be compact as well as graspable and viewable from any direction; thus suitable for group discussions. This paper describes the implementation of the gCubik display that realizes our conceptual prototype. We envision the application areas to include games and edutainment.

We propose a handheld box-shaped 3D display as a communication tool for realizing natural human-object interactions in face-to-face communications. The display employs an integral photography technique for each surface. However, misalignment of the lens array and the LCD module occurs, and so adjustment is necessary. In this paper, we analyze the error factors caused by this misalignment and propose a convenient calibration method to minimize them. The method estimates the actual positions of each lens using one camera and several stripe pattern images. After brief manual alignment the method computes the calibration parameters. We verify the suitableness of the proposed method by applying the computed parameters into the image-rendering process.

In this paper, we describe an algorithm for the synthesis of natural head motion on a 3D avatar from reconstructed 3D frontal face data with the goal of making 3D video games immersive, engaging and fun when played on 3D displays. The idea is that frame by frame natural head motion of a user is calculated without utilizing any markers from stereo image pair and transmitted to the avatar to generate realistic motion. Thus, the head motion is regenerated with the avatar in the video sequence in real-time. Obviously, the tools developed for these scenarios are applicable to other applications such as virtual space 3D teleconferencing with avatars or human-computer interface applications.

We propose a graspable box-shaped 3D display as a communication tool that allows multiple users to share and naturally interact with 3D images in face-to-face collaborative tasks. We envision an auto-stereoscopic 3D display featuring glasses- free and multi-viewpoint operation. Users should be able to view multiple faces in the box-shaped display simultaneously and from any direction. We employ the integral photography (IP) method for this purpose. In this paper, we first analyze the requirements for an IP lens allowing simultaneous multi-face viewing of 3D images in such a display. Consequently, we find that a mininum 120-degree field of view is necessary. Then, we design and prototype an IP lens that provides such a wide field of view. Visual inspection of the generated 3D images confirm the possibility of simultaneous multiple- face viewing with the proposed display.

Our research focuses on ambient sounds as a key to implementing an ambient atmosphere of people, that is, the feeling of another's proximity or presence. Such knowledge is sought for constructing an ultra-realistic communication system with shared reality. This study examines whether people's impressions and behaviors are affected by (a) the existence or (b) nonexistence of ambient sounds. A pilot study is conducted by using the ambient sounds of a newspaper's pages being turned and a glass being placed on a table in an adjacent room. No difference in the subjects' surface behavior between the two ambient-sound conditions is observed explicitly. However, the preliminary results suggest that participants feel an ambient atmosphere of people in the next room when the ambient sounds are output. They also suggest that the participants become nervous as they feel the presence of other people. This study demonstrates the importance of investigating the fundamentals of an ambient atmosphere in interaction via an ultra-realistic communication system.

Numerous efforts on content-based music information retrieval have been presented in recent years. However, the object of such existing research is to retrieve a specific song from a large music database. In this research, we propose a music retrieval method which retrieves songs based on the user's musical preferences. This enables users to discover new songs which they are expected to like. Since music preferences are expected to be highly ambiguous, we propose the implementation of relevance feedback methods to improve the performance of our music information retrieval method. In order to reduce the burden of users to input learning data to the system, we also propose a method to generate user profiles based on genre preferences, and refinement of such profiles based on relevance feedback. Evaluation experiments are conducted based on a corpus of music data with user ratings. Results of these experiments prove the effectiveness of our method.

Document filtering is a task to retrieve documents relevant to a user's profile from a flow of documents. Generally, filtering systems calculate the similarity between the profile and each incoming document, and retrieve documents with similarity higher than a threshold. However, many systems set a relatively high threshold to reduce retrieval of non-relevant documents, which results in the ignorance of many relevant documents. In this paper, we propose the use of a non-relevant information profile to reduce the mistaken retrieval of non-relevant documents. Results from experiments show that this filter has successfully rejected a sufficient number of non-relevant documents, resulting in an improvement of filtering performance.

This paper describes a method of classifying semantically similar nouns. The approach is based on the "distributional hypothesis". Our approach is characterized by distinguishing among senses of the same word in order to resolve the "polysemy" issue. The classification result demonstrates that our approach is successful.

KDD has developed a telephone operator assisting system, which provides operators with automatic database inquiry using speech recognition. An outline of the system is described and the results of a field trial are evaluated. There have been many speech recognition systems for practical use. A fast recognition method is one of the important subjects for practical systems. For this purpose, various methods are proposed. [Sagayama et al. ‘‘New Approaches Toward Fast Speech Recognition,’’ Proc. Acoust. Soc. Jpn. 25–28 (1996) (in Japanese)]. However, it takes some frames to output recognition results after finishing speech input, because any method deals with a few 100‐ms pause following speech, which is a margin to detect the speech end. Therefore, a new fast recognition method is proposed. The basic idea of this approach is to output results when inquiry keywords are decided, even before detecting the speech end. This approach makes it possible not only to reduce speech recognition time, but also to recogni...

gCubik provides a 3D visual experience, which is interactive and group-shared, by using a graspable cubic display. Users are presented with color, stereo and full motion parallax 3D scenes, viewable from any direction, without the need of special glasses. Newly designed wide field of view integral photography(IP) lens array are employed on each face of the display for its autostereoscopic effect. In our demonstration, real-time rendering of the IP images allows users to interactively manipulate the 3D objects in the scene via simple finger gestures.