Search

In this paper, we propose a new video conferencing system that presents correct gaze directions of a remote user by switching among images obtained from multiple cameras embedded in a screen according to a local user’s position. Our proposed method reproduces a situation like that in which the remote user is in the same space as the local user. The position of the remote user to be displayed on the screen is determined so that the positional relationship between the users is reproduced. The system selects one of the embedded cameras whose viewing direction towards the remote user is the closest to the local user’s viewing direction to the remote user’s image on the screen. As a result of quantitative evaluation, we confirmed that, in comparison with the case using a single camera, the accuracy of gaze estimation was improved by switching among the cameras according to the position of the local user.

In this paper, we propose a method to reproduce the reflection of a real scene on a virtual object using only the images captured by a camera attached to the front of a mobile device. Since it is not possible to acquire the entire scene using only the front camera, the area surrounding the front camera image is extrapolated to obtain a sufficient scene for reflection. Image transformation using deep neural networks is used for extrapolation, and high-quality and stable extrapolation is realized by using the extrapolation results in the previous frame. As a result of experiment that evaluated the quality of extrapolation and reproduction of reflection, we confirmed that both extrapolated images and reproduced reflection looked natural. We also conducted an experiment to evaluate users' impression of the reflection. The results showed that the proposed method was effective for participants who knew about AR in naturalness of reflection and material perception.

In this paper, we propose a method for human detection from low-resolution camera images. The proposed method uses video images as input and uses 3D-CNN for classification, which is an extension of 2D-CNN and that can take into account temporal features such as gait motion. In our experiments, we used Caltech Pedestrian Detection Benchmark to make datasets of low-resolution still and video images and compared the performance between 2D-CNN and 3D-CNN. As a result, 3D-CNN with low-resolution video images achieved 91.8 % accuracy rate, 99.0 % precision rate, and 82.8 % recall rate, and showed higher performance than 2D-CNN with low-resolution images, and comparable performance than 2D-CNN with high-resolution images.

The change in the edema condition is visualized considering the three-dimensional shape. Continuous treatment and observation are indispensable for patients with edema. The measurement and evaluation of the three-dimensional shape of the leg are thus important in evaluating edema of the leg. Such an evaluation can confirm the therapeutic effect and assist in the planning of treatment by confirming the change in local capacity. Additionally, the depth camera of Structure Sensor used this study is feasible for use in home care systems due to its very low cost compared with other depth cameras. We obtain a point cloud of the leg and register shape models. We conducted an experiment to measure legs swathed and not swathed in bandages, with the former representing a leg with edema. In addition, for visualization of the edema condition, the change in shape was color coded according to the change obtained in the proposed analysis of the three-dimensional shape. Our experimental results show that our proposed visualization technique is effective in conveying the change in shape visually and clearly.

In this paper, we propose a method for recognizing the user’s grasping action in VR environment. The proposed method utilizes the fact that the positions and shapes of virtual objects are known in VR environment. The position and shape information of the virtual object in addition to the hand posture is used for classification to improve the recognition accuracy. The result of evaluation showed that the recognition rate was improved by adding the position and shape information of the virtual object.

In this paper, we propose a user interface for large displays that allows pointing operations from a wide area. Two fisheye cameras installed on both sides of the display are used to capture images of a wide area in front of the display, and the system recognizes the user’s skeleton and allows the user to perform pointing operation. Due to the wide viewing angle of the fisheye cameras, the baseline length between the cameras can be long. The experimental result showed that the pointing accuracy was higher with the fisheye cameras than with the standard cameras when operating at the same distance.

In this paper, we propose three methods for removing floating particles from underwater images. The first two methods are based on Generative Adversarial Networks (GANs). The first method uses CycleGAN which can be trained with an unpaired dataset, and the second method uses pix2pixHD that is trained with a paired dataset created by adding artificial particles to underwater images. The third method consists of two-step process – particle detection and image inpainting. For particle detection, an image segmentation neural network U-Net is trained by using underwater images added with artificial particles. Using the output of U-Net, the particle regions are repaired by an image inpainting network Partial Convolutions. The experimental results showed that the methods using GANs were able to remove floating particles, but the resolution became lower than that of the original images. On the other hand, the results of the method using U-Net and Partial Convolutions showed that it is capable of accurate detection and removal of floating particles without loss of resolution.

In this paper, we propose a method for recognizing gestures from videos which implicitly incorporates multimodal data during training, and makes classification by only using RGB modality data. The network is 3d-convolutional, and includes a shared network for implicitly incorporating multiple modalities, a generation branch for estimating motion regions and a classification branch for classifying gestures. We introduce a type of efficient modality data, binarized motion cues, which include information of moving hand regions, and are learned by using the generation network. The binarized motion cues are given as extra supervision for learning motion in the generation branch. Since features of additional motion cues learned by the generation branch are implicitly fused with features learned by the classification branch, the classification performance can be improved. Experimental results showed that the shared network can extract more discriminable intermediate features, and the network with the classification branch can achieve improved performance by only using RGB modality input data.

In this paper, we propose a user interface that allows pointing operation with a user’s palm orientation by measuring the normal directions in the palm region using a dynamic photometric stereo camera based on a multi-tap CMOS image sensor. Our system allows users to control a pointer smoothly with small hand movement. We implemented two types of user interface designs and an interactive game to show the applicability of hand gesture operation using the proposed system.

In this paper, we propose a system that can measure the spatially-varying reflectance of real materials. Our system uses two handheld cameras, a small LED light, a turning table, and a chessboard with markers. The two cameras are used as a view and light cameras respectively to acquire incoming and outgoing light directions simultaneously, and the brightness at each position on the target material. The reflectance is approximated by using the Ward BRDF (Bidirectional Reflectance Distribution Function) model. The normal directions and all model parameters at each position on the material are estimated by non-linear optimization. As the result of experiment, the normal directions for all spatial points were properly estimated, and the correct colors of rendered materials were reproduced. Also, highlight changes on the surfaces were observed when we moved the light source or the rendered materials. It was confirmed that our system was easy to use and was able to measure the spatially-varying reflectance of real materials.

In this paper, we propose a method for recognizing grasping of virtual objects in VR environment. The proposed method utilizes the fact that the position and shape of the virtual object to be grasped are known. A camera acquires an image of the user grasping a virtual object, and the posture of the hand is extracted from that image. The obtained hand posture is used to classify whether it is a grasping action or not. In order to evaluate the proposed method, we created a new dataset that was specialized for grasping virtual objects with a bare hand. There were three shapes and three positions of virtual objects in the dataset. The recognition rate of the classifier that was trained using the dataset with specific shapes of virtual objects was 93.18 %, and that with all the shapes of virtual objects was 87.71 %. This result shows that the recognition rate was improved by training the classifier using the shape-dependent dataset.

In this paper, we propose a method for detecting fall actions using a variational auto-encoder (VAE) with 3D-convolutional residual blocks. The VAE learns a distribution of Activity of Daily Life (ADL) data, and the reconstruction error is used to detect fall actions. The proposed method is a kind of unsupervised learning method with weakly labeled data for solving the problem of imbalance between the amount of fall data and that of ADL data. Furthermore, we propose extracting a human region from an entire image using skeleton information and aligning motions using the same joint point so that the neural network can focus on learning human motions, which can enhance the accuracy of fall detection. The results of experiments showed that our method achieved a competitive level of accuracy and better generalization ability compared to supervised learning with well-labeled data.

In this paper, we propose a method for eliminating image noise (from sand, algae, bubbles, etc.) that is transferred when capturing underwater images. In detail, we separate the input of a moving underwater image into a slowly changing background and foreground moving at a certain speed using Kalman filter and object tracking. In this research, we examined the difference in image quality of the output image by changing the noise variance given to the Kalman filter.

In this paper, we propose a method for measuring the reflectance of anisotropic materials using a simple apparatus consisting of two handheld cameras, a small LED light source, a turning table and a chessboard with markers. The system is configured to obtain the different incoming and outgoing light directions, and the brightness of pixels on the surface of the material. The anisotropic Ward BRDF (Bidirectional Reflectance Distribution Function) model is used to approximate the reflectance, and the model parameters are estimated from the incoming and outgoing angles and the brightness of pixels by using a non-linear optimization method. The initial values of the anisotropic direction are given based on the peak specular lobe on the surface, and the best-fitted one is chosen for the anisotropic direction. The optimized parameters show the well-fitted results between the observed brightness and the BRDF model for each RGB channel. It was confirmed that our system was able to measure the reflectance of different isotropic and anisotropic materials.

In this paper, we propose a system that can reproduce the material appearance of real objects using mobile augmented reality (AR). Our proposed system allows a user to manipulate a virtual object, whose model is generated from the shape and reflectance of a real object, using the user's own hand. The shape of the real object is reconstructed by integrating depth images of the object, which are captured using an RGB-D camera from different directions. The reflectance of the object is obtained by estimating the parameters of a reflectance model from the reconstructed shape and color images, assuming that a single light source is attached to the camera. We measured the shape and reflectance of some real objects and presented the material appearance of the objects using mobile AR. It was confirmed that users were able to obtain the perception of materials from changes in gloss and burnish of the objects by rotating the objects using their own hand.

In our previous work, we proposed a user interface in which a user wears a projector and a depth camera on his or her head and performs touch operations on an image projected on a flat surface. By using the head-mounted projector, images are always projected in front of the user in the direction of the user&rsquo, s gaze. The image to be projected is changed according to the user&rsquo, s head pose so as to fix the superimposed image on the surface, which realizes a large effective screen size. In this paper, we conducted an experiment for evaluating the accuracy of registration by measuring the positional and rotational errors between the real world and the superimposed image using our experimental system. As a result, the mean absolute errors of translation were about 10 mm when the user stopped his head, and the delay was estimated to be about 0.2 s. We also discuss the limitations of our prototype and show the direction of future development.

In this paper, we propose a system that combines a mouse with a projector and that allows users to perform peephole interaction, which enables stable operation in a wide workspace. In our prototype system, a mobile projector is attached above a mouse and projects part of the workspace on the plane in front of the mouse. The user can change the area he/she wishes to see by moving the mouse and can perform operations in a virtually extended workspace. We conducted an experiment to compare the methods of changing the displayed region and the cursor position when the mouse moves. Our results show that the method with the highest ratio of view transition to mouse movement, in which the workspace is not fixed to the real space, outperformed the other methods and was also highly evaluated by participants. Furthermore, the task completion time decreased with repeating trials with fixed target positions, which suggests that spatial memory is helpful even when the workspace is not fixed to the real space.

In this paper, we propose an AR system using a mobile display that allows the user to manipulate a virtual object using the user's own hand to provide the user with perception of materials. To realize more natural interaction with a virtual object, our proposed system provides visual information that is consistent with the user's somatic sensation by displaying user-perspective images on the mobile display. The system allows the user to manipulate the virtual object with six degrees of freedom (three degrees of freedom for translation and three degrees of freedom for rotation) by mapping the hand pose to the object's translation and rotation. Material appearance of objects is presented by rendering the virtual object using Unity 3D. We showed the effectiveness of our proposed system by showing how the user manipulates the virtual object and obtains the gloss and burnish of the object using the proposed system.

We have proposed Augmented Reality (AR) Typing Interface as a new input interface for mobile devices. By using this interface, users can utilize a wide space in the air in inputting texts. This interface uses a camera which is attached to the back of the device. A virtual keyboard is overlaid on the image captured by a camera using AR technology. In this paper, we propose a new method to recognize typing motions more precisely. In this method, feature vectors are generated by using information of time-series optical flows. Typing motions are recognized by using support vector machine (SVM). We confirmed that the proposed method was able to recognize typing motions with about 90% accuracy for the recorded video of typing in the air.

In this paper, we compared text entry methods for mobile devices such as smartphones, in which part of a software keyboard around the finger position is enlarged by using a hover function. We examined four methods: fixing the center of enlargement, not fixing the center of enlargement, adding a scrolling function, and using a standard software keyboard for reference. We recruited 12 participants and evaluated the text entry speed and error rates. The method in which the center of enlargement was fixed had the slowest text entry speed, but showed an error rate that was significantly lower than the method in which the center of enlargement was not fixed. This result shows that fixing the center of enlargement enables users to select keys stably and enter target letters accurately.

In this paper, we propose a system that presents correct gaze directions of each user in a group-to-group video conference. It is known that gaze direction plays a significant role in face-to-face communication. However, in ordinary video conference systems, there is a problem that gaze directions cannot be correctly transmitted to a user in the other side since the camera is placed on the top or bottom of the display. We propose a method to generate images of each person from an appropriate viewpoint position using Kinect and screen-embedded cameras and to present the images to the user. By estimating each user's gaze direction and rendering the image of the person who the user is looking at from the users' viewpoint position, it is possible to present an image suitable for each user even in a group-to-group situation. We show demonstration of a group-to-group video conference using the proposed system and show that the system can present images suitable for each user.

In this paper, we propose a system that realizes remote control of a computer with small hand gestures. Distant pointing is realized using a 3D hand pointing recognition algorithm that obtains position and direction of the pointing hand. We show the effectiveness of the system by constructing three types of user interfaces considering the accuracy of distant pointing in the current system. We created a tile layout interface for rough selection operation, a pie menu interface for detailed operation, and a viewer interface for document browsing.

In this paper, we propose a user interface that combines the existing touch interface with 3D view navigation by aerial hand gestures and that realizes simultaneous zooming and scrolling. While the touch interface enables only planar operation, aerial hand gestures enable 3D operation with an increased number of axes, and enables smooth navigation that integrates zoom and scroll operations by assigning the direction that is normal to the screen to zooming and the direction that is parallel to the screen to scrolling. We implemented a map application based on the proposed interface, and confirmed the effectiveness of the proposed interface.

We propose a method for obtaining clear underwater images by tracking the motion of suspended matter from video images captured in water and by separating the images into foreground and background. We assume that input images are the superposition of a foreground and a background, and constructed a transition model and the observation model. An input image is divided into patches and tracking of the foreground in each patch is performed while applying Kalman filter to separate the input images into the foreground and the background. From the result of the experiment using simulated images, we confirmed that the background images were successfully estimated and a region that was moving slowly was also recognized as a part of the background.

In this paper, we propose a method for estimating the 3D poses of a front-pointing hand from camera images to realize freehand pointing interaction from a distance. Our method uses a Random Regression Forest (RRF) to realize robust estimation against environmental and individual variations. In order to improve the estimation accuracy, our method supports the use of two cameras and integrates the distributions of the hand poses for these cameras, which are modeled by the Gaussian mixture model. Moreover, tracking of the hand poses further improves the estimation accuracy and stability. The results of performance evaluation showed that the root mean square error of the angle estimation was 4.10\(^{\circ }\), which is accurate enough to expect that our proposed method can be applied to user interface systems.

We propose a user interface that superimposes a virtual touch panel on a flat surface using an optical see-through head-mounted display and an RGB-D camera. The user can use the interface in a hands-free state, and can perform the operation with both hands. The interface performs markerless superimposition of virtual objects on a real scene. In addition, the interface can recognize three-dimensional information of the user's fingers, allowing the user to operate with the virtual touch panel. We developed some applications in which the user can perform various operations on the virtual touch panel.

In this paper, we propose Predictive Zooming Keyboard (PZBoard) which predicts a target position from the finger movement above the touch-screen of a mobile device and that enlarges a part of the keyboard around the predicted position. We use a hover function of the mobile device to obtain the finger position above the touch-screen. While a finger is detected, a part of the keyboard around the finger position are enlarged. When the user moves his/her finger fast to enter a distant key, the target position is predicted and the center of enlargement moves to the predicted position. Using prediction, the system can start drawing the screen before the finger reaches the target position, which reduces the system's latency and quickens the user's response. The proposed interface does not force the user to perform additional operation for enlarging keys, and enables stable selection of both near and distant keys by fixing or changing the center of enlargement based on the velocity of the finger.

In this paper, we propose On-mouse projector, an interface that combines a mouse and a mobile projector. This system satisfies both portability and large information space and enables stable operation. The mobile projector is placed on the mouse and projects images on a surface in front of the mouse. The projected image presents a part of large information space and users can change the area to see by moving the mouse. The system is assumed to be used on a flat surface such as a desk and users can stably perform the same operation as the ordinary mouse operation. We created a prototype in which a projector is fixed above a mouse using acrylic plates. This prototype works in a standalone configuration by using a stick PC and realizes various operations such as object selection, object moving and image zooming.

In this paper, we propose a tabletop interface in which a user wears a projector with a depth camera on his or her head and can perform touch operations on an image projected on a flat surface. By using the head-mounted projector, images are always projected in front of the user in the direction of the user's gaze. By changing the image to be projected based on the user's head movement, this interface realizes a large effective screen size. The system superimposes an image on the flat surface by performing plane detection, placing the image on the detected plane, performing perspective projection to obtain a 2D image, and projecting the 2D image using the projector. Registration between the real world and the image is performed by estimating the user's head pose using the detected plane information. Furthermore, touch input is recognized by detecting the user's finger on the plane using the depth camera. We implemented some application examples into the system to demonstrate the usefulness of the proposed interface.

In this paper, we propose a markerless augmented reality (AR) system that works on multiple mobile devices. The relative positions and orientations of the devices and their individual motions are estimated from 3D information in real space obtained by depth cameras attached to the devices. The system allows multiple users to share the AR space and to interact with the same virtual object. To estimate the relative positions and orientations of the devices, the system generates 2D images by looking down from above at the 3D scene obtained by the depth cameras, performs 2D registration using template matching, and obtains a transformation matrix that transforms the coordinate system of one camera to that of another camera. The motion of a camera is estimated using the ICP algorithm to realize markerless AR. Using the proposed system, we created an application that enables multiple users to interact with the same virtual object.

A probabilistic method to remove the random telegraph signal (RTS) noise and to increase the signal level is proposed, and was verified by simulation based on measured real sensor noise. Although semi-photon-counting-level (SPCL) ultra-low noise complementary-metal-oxide-semiconductor (CMOS) image sensors (CISs) with high conversion gain pixels have emerged, they still suffer from huge RTS noise, which is inherent to the CISs. The proposed method utilizes a multi-aperture (MA) camera that is composed of multiple sets of an SPCL CIS and a moderately fast and compact imaging lens to emulate a very fast single lens. Due to the redundancy of the MA camera, the RTS noise is removed by the maximum likelihood estimation where noise characteristics are modeled by the probability density distribution. In the proposed method, the photon shot noise is also relatively reduced because of the averaging effect, where the pixel values of all the multiple apertures are considered. An extremely low-light condition that the maximum number of electrons per aperture was the only 2 e − was simulated. PSNRs of a test image for simple averaging, selective averaging (our previous method), and the proposed method were 11.92 dB, 11.61 dB, and 13.14 dB, respectively. The selective averaging, which can remove RTS noise, was worse than the simple averaging because it ignores the pixels with RTS noise and photon shot noise was less improved. The simulation results showed that the proposed method provided the best noise reduction performance.

In this paper, we proposed an architecture of embedded systems for high-frame-rate real-time vision on the order of 1000 f/s, which achieved both hardware reconfigurability and easy algorithm implementation while fulfilling performance demands. The proposed system consisted of an embedded microprocessor and field programmable gate arrays (FPGAs). A coprocessor consisting of memory units, direct memory access controller units, and image processing units were implemented in each FPGA. While the number of units and functions are reconfigurable by reprogramming the FPGAs, users can implement algorithms without hardware knowledge. A descriptor method in which the central processing unit gave instructions to each coprocessor through a register array enabled task-level parallel processing as well as pixel-level parallel processing in the processing units. The specifications of an evaluation system developed based on the proposed architecture, the results of performance evaluation, and application examples using the system were shown.

We demonstrate surface-image synthesis of moving, spinning cylindrical objects using a commercially available high-speed area scan camera. The frame rate used in the demonstration experiment was 1,000 fps, which is sufficient to achieve surface-image synthesis of cylinders spun at up to 36 rps. We successfully demonstrated a technique based on an algorithm similar to image mosaicing at 1,000 fps, for the first time to the best of our knowledge. In this paper, we discuss techniques to overcome the potential problems faced when applying surface-image synthesis to cylindrical objects, such as image distortion, quantization errors due to superimposing images, and intensity variations due to the surface curvature. An FPGA-based parallel image processing board, PB-1, that we developed was used to implement these demonstrations. We introduce this application of PB-1 as a potential practical solution to the long-standing problem of industrial visual inspection using real-time high-speed vision.

Massively parallel image processing with as many processing elements as pixels can achieve real-time vision with rates as high as 1,000 frames per second. The authors implemented a QVGA-size pixel-parallel image processor for object identification and pose estimation on a single chip. Bit-serial operation and dynamic logic reduce the circuit area, and pipelining enables high processing speed.

A 128 x 128 pixel functional image sensor was implemented. The sensor was able to capture images at 1,000 frame/s and extract the sizes and positions of 10 objects/frame when clocked at 8 MHz. The size of each pixel was 18 μm x 18 μm and the fill factor was 28%. The chip, 3.24 mm x 3.48 mm in size, was implemented with a 0.35 μm CMOS sensor process; the power consumption was 29.7 mW at 8 MHz.

This paper describes a real-time three-dimensional sensing system for applications based on feedback automation. The proposed system enables to observe a moving/deforming object at high frame rate and can acquire data in realtime. These are provided by three distinctive features, three-dimensional sensing by a single frame, high-frame-rate imaging and high-speed image processing. We also present some results of evaluation experiments. The experimental results show the advantages of our system compared with conventional approaches. Our system is expected to achieve improvements in a wide range of three-dimensional-sensing applications.

We developed a smart image-sensor especially for extracting statistical values that contain areas,positions,orientations and shapes of the objects in an image.These values are called moments.The sensor has a small processing circuit in each pixel in order to achieve high resolution,but it outputs various information at real-time.We describe the implementation and performance evaluation of a 128 128 pixel image-moment sensor.

In this paper, we propose an interaction system in which the appearance of the image displayed on a mobile display is consistent with that of the real space and that enables a user to interact with virtual objects overlaid on the image using the user's hand. The three-dimensional scene obtained by a depth camera is projected according to the user's viewpoint position obtained by face tracking, and the see-through image whose appearance is consistent with that outside the mobile display is generated. Interaction with virtual objects is realized by using the depth information obtained by the depth camera. To move virtual objects as if they were in real space, virtual objects are rendered in the world coordinate system that is fixed to a real scene even if the mobile display moves, and the direction of gravitational force added to virtual objects is made consistent with that of the world coordinate system. The former is realized by using the ICP (Iterative Closest Point) algorithm and the latter is realized by using the information obtained by an accelerometer. Thus, natural interaction with virtual objects using the user's hand is realized.

In this paper, we propose a method for overlaying navigation signs on a road surface and displaying them on a head-up display (HUD). Accurate overlaying is realized by measuring 3D data of the surface in real time using a depth camera. In addition, the effect of head movement is reduced by performing face tracking with a camera that is placed in front of the HUD, and by performing distortion correction of projection images according to the driver's viewpoint position. Using an experimental system, we conducted an experiment to display a navigation sign and confirmed that the sign is overlaid on a surface. We also confirmed that the sign looks to be fixed on the surface in real space.

In this paper, we propose a pointing interface that can be easily installed and that enables remote and nearby operations with high recognition accuracy. We use a system put on a table consisting of multiple projectors and depth cameras. The system projects a shared screen and a personal screen per each user on a projection screen and on a table respectively and enables interaction according to the user's hand motion. A personal screen can be operated by touch operations and a shared screen can be operated by pointing with fingers. Touch and pointing direction are recognized from the depth image and the confidence map obtained by a depth camera. We developed an early prototype system for verification and implemented a presentation application that supports collaborative work to the system. This application allows a user to move a window between the shared screen and the personal screen and to perform various operations on the slides.

In this paper, we propose a new user interface designed to improve the usability of mobile devices. The interface overlays a virtual keyboard on real images captured by a camera attached to the back of a mobile device and recognizes user's hand motions by using optical flow information. The user can operate the virtual keyboard with his/her multiple fingers in the space behind the mobile device. The interface recognizes only hand motions and does not recognize poses of a hand or fingers, which enables high-speed stable recognition without complicated processing. The experimental system recognized hand motions with a processing time of about 8.33 [ms/frame] (about 120 [fps]). We implemented a keyboard application using our interface on the experimental system and evaluated the performance. The result of a simple user study showed that typing speed became faster as a trial is repeated.

We propose a system that enables dynamic 3D interaction with real and virtual objects using an optical see-through head-mounted display and an RGB-D camera. The virtual objects move according to physical laws. The system uses a physics engine for calculation of the motion of virtual objects and collision detection. In addition, the system performs collision detection between virtual objects and real objects in the three-dimensional scene obtained from the camera which is dynamically updated. A user wears the device and interacts with virtual objects in a seated position. The system gives users a great sense of reality through an interaction with virtual objects.

In this study, we propose a VR system for allowing various types of interaction with virtual objects using an autostereoscopic mobile display and an accelerometer. The system obtains the orientation and motion information from the accelerometer attached to the mobile display and reflects them to the motion of virtual objects. It can present 3D images with motion parallax by estimating the position of the user's viewpoint and by displaying properly projected images. Furthermore, our method enables to connect the real space and the virtual space seamlessly through the mobile display by determining the coordinate system so that one of the horizontal surfaces in the virtual space coincides with the display surface. To show the effectiveness of this concept, we implemented an application to simulate food cooking by regarding the mobile display as a frying pan.

In this paper, we try to compare standard and low latency mid-air 3D interactions in a tabletop user interface. For this purpose, we developed a 3D tabletop user interface that allows direct manipulation of 3D virtual objects presented on an autostereoscopic display. The system uses a high-speed stereo camera to synchronize real hands and virtual objects, both temporally and spatially, with a high degree of accuracy. In the developed system, we realized a processing speed resulting in a frame rate of 200 fps and a latency of 28.33 ms. We conducted a user study for evaluating the usability and sense of reality and the results showed that there was a significant difference in the time required to complete an object moving operation between the 200 fps and 30 fps cases.