Search

We propose a system for real-time six degrees of freedom (6DoF) tracking of a passive stylus that achieves sub-millimeter accuracy, which is suitable for writing or drawing in mixed reality applications. Our system is particularly easy to implement, requiring only a monocular camera, a 3D printed dodecahedron, and hand-glued binary square markers. The accuracy and performance we achieve are due to model-based tracking using a calibrated model and a combination of sparse pose estimation and dense alignment. We demonstrate the system performance in terms of speed and accuracy on a number of synthetic and real datasets, showing that it can be competitive with state-of-the-art multi-camera motion capture systems. We also demonstrate several applications of the technology ranging from 2D and 3D drawing in VR to general object manipulation and board games.

Image color editing techniques such as color transfer, HDR tone mapping, dehazing, and white balance have been widely used and investigated in recent decades. However, naively employing them to videos frame-by-frame often leads to flickering or color inconsistency. To solve it generally, earlier methods rely on temporal filtering or warping from the previous frame, but they still fail in the cases of occlusion and produce blurry results. We introduce a new framework for these challenges: (1) We develop an online keyframe strategy to keep track of the dynamic objects, where more temporal information can be acquired than a single previous frame. (2) To preserve image details, local color affine model is employed. The main concept of this post-processing step is to capture the color transformation from editing algorithms and maintain the detail structures of the raw image simultaneously. Practically, our approach takes a raw video and its per-frame processed version, and generates a temporally consistent output. In addition, we propose a video quality metric to evaluate temporal coherence. Extensive experiments and subjective test are done to show the superiority of the proposed framework with respect to color fidelity, detail preservation, and temporal consistency.

Surfaces are now where the augmented reality comes true. In this paper, we propose an efficient, learning-free and reliable way to detect for not only planar but also curved surfaces. Furthermore, our approach combines both advantages of local region growing and top-down refinements to retrieve wholesome surfaces in both organized point clouds and prebuilt 3D models. First, we obtain a down-sampled graph by supervoxel segmentation. Afterward, a recursive bottom-up agglomerative hierarchical clustering approach will iteratively merge the supervoxels into surfaces. Finally, top-down refinements on noisy and occluded planes will correct over-segmentations. To sum up, this is a model- and learning-free approach with experimental efficiency --- around 80 seeds and 3000 supervoxels are calculated for a 640x480 test point cloud, costing less than 0.2 sec on an ordinary laptop with 2.6GHz Intel core i5, 8GB RAM and without GPU aided.

In this paper, we proposed a 3D background modeling system for multi-view 3D video. We first reconstructed a 3D model, and we updated the subsequent frames into it using our proposed updating strategy. The results show that dynamic objects in the model can be excluded, leaving behind a compact 3D background model.

In this paper, an aesthetic modeling method for scenic photographs is proposed. A bottom-up approach is developed to construct an aesthetic library with bag-of-aesthetics preserving features instead of top-down methods that implement the heuristic guidelines (rule-specific features) listed in the photography literature, which is employed in previous works. The proposed method can cover both implicit and explicit aesthetic features with a learning process. The experimental results show that the proposed features in the library (92.06% in accuracy) outperform the state-of-the-art rule-specific features (83.63% in accuracy) significantly in the aesthetic quality assessment for scenic photos, and the rule-specific features are also proved to be encompassed by the proposed features. Meanwhile, it is observed from experiments that the features extracted for contrast information are more effective than those for absolute information, which is consistent with the properties of human visual systems.

Tennis Real Play (TRP) is an interactive tennis game system constructed with models extracted from videos of real matches. The key techniques proposed for TRP include player modeling and video-based player/court rendering. For player model creation, we propose a database normalization process and a behavioral transition model of tennis players, which might be a good alternative for motion capture in the conventional video games. For player/court rendering, we propose a framework for rendering vivid game characters and providing the real-time ability. We can say that image-based rendering leads to a more interactive and realistic rendering. Experiments show that video games with vivid viewing effects and characteristic players can be generated from match videos without much user intervention. Because the player model can adequately record the ability and condition of a player in the real world, it can then be used to roughly predict the results of real tennis matches in the next days. The results of a user study reveal that subjects like the increased interaction, immersive experience, and enjoyment from playing TRP.

In order to increase the capability of mobile GPUs in image/video processing, a multi-purpose configurable filtering unit (CFU), which is a new configurable unit for image filtering on stream processing architecture, is proposed in this paper. CFU is located in the texture unit of a GPU and can efficiently execute many kinds of filtering operations by directly accessing multi-bank texture cache and specially-designed data-paths. The following programmabilities are supported in our proposed CFU. First, different sampling point windows can be selected by programmers. Besides, the arithmetic type of the filter can be chosen. Not only original texture filtering functions and finite impulse response (FIR) filters, morphological operations in computer vision are also embedded in CFU. Furthermore, the weighting coefficients of FIR filters and morphological operations can be defined by programmers. Simulation results show that in average, compared with conventional texture unit, 25.35% of processing time in H.264/AVC motion compensation and 58.6% of processing time in video segmentation can be reduced with the assistance of CFU.

This video demo presents a new framework of sport video applications called as Tennis Video 2.0. The proposed information extraction scheme retrieves the temporal structure of a video and separates the video foreground and background objects into different layers. With the structure and layer information, the new multimedia is generated. Contrary to the conventional video contents, the proposed new multimedia enables users to generate their own contents and feedback requests to the video players for more interaction. Users even can share their created contents with friends in different transmission bandwidth with considering the semantic.