Search

Template-based 3D object tracking still lacks a high-precision benchmark of real scenes due to the difficulty of annotating the accurate 3D poses of real moving video objects without using markers. In this paper, we present a multi-view approach to estimate the accurate 3D poses of real moving objects, and then use binocular data to construct a new benchmark for monocular textureless 3D object tracking. The proposed method requires no markers, and the cameras only need to be synchronous, relatively fixed as cross-view and calibrated. Based on our object-centered model, we jointly optimize the object pose by minimizing shape re-projection constraints in all views, which greatly improves the accuracy compared with the single-view approach, and is even more accurate than the depth-based method. Our new benchmark dataset contains 20 textureless objects, 22 scenes, 404 video sequences and 126K images captured in real scenes. The annotation error is guaranteed to be less than 2mm, according to both theoretical analysis and validation experiments. We re-evaluate the state-of-the-art 3D object tracking methods with our dataset, reporting their performance ranking in real scenes. Our BCOT benchmark and code can be found at https://ar3dv.github.io/BCOT-Benchmark/.

3D object tracking with monocular RGB images faces many challenges in real environments. The popular color- and edge-based methods, although have been well studied, are still known to be limited in handling specific cases. We observed that the color and edge features are complementary for different cases, and thus propose to fuse them for improving tracking robustness. To optimize the combination and to cope with inconsistency between color and edge features, we propose to fuse different energy terms with respect to a set of local bundles. Each bundle represents a local region containing a set of pixel locations for computing color and edge energies, in which two energy terms are adaptively weighted to play advantages of them. Experiments show that the proposed method can improve the accuracy in challenging cases, especially in light changing and similar color condition.

This paper considers the AC/DC power grid as three parts of the external grid, the internal main grid and the internal low voltage grid separately according to the demand of harmonic conduction research. The external power grid with far electric distance is equivalent to simple model. The internal main grid which is main harmonic conduction channel is kept intact. The internal low voltage grid, which would have influence on harmonic conduction, is equivalent to accurate model. The accuracy of the model is verified by comparing the PSCAD simulation results with the measured harmonic data. In this paper, the multi-input interaction factor is applied to the harmonic frequency, and the factor is improved considering the current inflow through the self-harmonic impedance of two nodes. This improved multi-infeed interaction factor under harmonic frequency is used to analyse the AC/DC power grid models. Compared with the measured harmonic data, it is verified that this factor can reflect the actual harmonic conduction of the power grid.

Detecting moving objects in the real world with reliability, robustness and efficiency is an essential but difficult task in AR applications, especially for interactions between virtual agents and real pedestrians, motorcycles and more, where the spatial occupancy of non-rigid objects should be perceived. In this paper, a novel object tracking method using visual cues with pre-training is proposed to track dynamic objects in 2D online videos robustly and reliably. The object's area in images can be transformed to 3D spatial area in the physical world with some simple, well-defined constraints and priors, thus spatial collision of agents and pedestrians can be avoided in AR environments. To achieve robust tracking in a markerless AR environment, we first create a novel representation of non-rigid objects, which is actually the manifold of normalized sub-images of all the possible appearances of the target object. These sub-images, captured from multiple views and under varying lighting conditions, are free from any occlusion and can be obtained from both video sequences and synthetic image generation. Then, from the instance pool made up of these sub-images, a compact set of templates which can well represent the manifold is learned by our proposed iterative method using sparse dictionary learning. We ensure that this template set is complete by using an SVM-based sparsity detection method. This compact, complete set of templates is then used to track the target trajectory online in video and augmented reality (AR) systems. Experiments demonstrate the robustness and efficiency of our method.

In this paper, we introduce an online video-based system that actively assists users in assembly tasks. The system guides and monitors the assembly process by providing instructions and feedback on possibly erroneous operations, enabling easy and effective guidance in AR/MR applications. The core of our system is an online video-based assembly parsing method that can understand the assembly process, which is known to be extremely hard previously. Our method exploits the availability of the participating parts to significantly alleviate the problem, reducing the recognition task to an identification problem, within a constrained search space. To further constrain the search space, and understand the observed assembly activity, we introduce a tree-based global-inference technique. Our key idea is to incorporate part-interaction rules as powerful constraints which significantly regularize the search space and correctly parse the assembly video at interactive rates. Complex examples demonstrate the effectiveness of our method.

Realistic image composite requires the appearance of foreground and background layers to be consistent. This is difficult to achieve because the foreground and the background may be taken from very different environments. This paper proposes a novel composite adjustment method that can harmonize appearance of different composite layers. We introduce the Best-Buddy Prior (BBP), which is a novel compact representations of the joint co-occurrence distribution of natural image patches. BBP can be learned from unlabelled images given only the unsupervised regional segmentation. The most-probable adjustment of foreground can be estimated efficiently in the BBP space as the shift vector to the local maximum of density function. Both qualitative and quantitative evaluations show that our method outperforms previous composite adjustment methods.

Low-cost depth sensing devices, such as Microsoft Kinect, can only produce noisy depth maps that are mis-aligned with color images, and even contain many holes. Even though the coupled high quality color images contain rich information which can be exploited to enhance the depth maps, the redundant color edges often introduce incorrect depth edges in the result depth map, since color images contain more textures than depth maps. To solve this problem, we propose a novel approach which generates accurate color-consistent depth edges by employing both color and depth images. First, Edges of raw depth maps are extracted using image pyramid strategy. Then, the redundant edges in color images are removed according to the raw depth edges, and, accurate color-consistent depth edges are generated by combining raw depth edges with current color edges. Finally, constraints extracted from both raw depth and color images and the generated depth edges are fused in a MRF optimization framework to obtain the enhanced depth map, which is accurately aligned with coupled color image. As experimentally demonstrated, the proposed method achieves outstanding performance when compared with previous approaches.

Optical spin-Hall effect (OSHE) is a spin-dependent transport phenomenon of light. Here, the spin of light is associated with the circular polarization. Although being observed one decade ago [1], OSHEs are usually extremely weak and in uncontrollable manner owning to the inability to fully control the spin-orbit coupling. Recently, with the development of metasurfaces, the spin-dependent splitting of light beam can be significantly enhanced by using metamaterials [2]. Many attentions have been paid to the spin-orbit interaction (SOI) of light using geometric-phase enabled optical and plasmonic systems. However, these previous works on OSHE only demonstrated simple and symmetric splitting of the two spins [1–4], far from the capability to generate independent OSHEs, which can then work coherently together for more flexible and tunable functionalities. Here, we present a plasmonic metasurface consisting of an array of nano-slots with tailor-made orientation profiles to generate independent surface plasmon polariton (SPP) patterns on the metasurface with the two incident spins [5]. For example a “cross” pattern for one spin and a “triangle” pattern for another as incidence. With such flexible control of OSHEs, we also demonstrate that it is possible to make the two spins work coherently together by controlling the relative phase between the two polarizations. It allows us to play motion pictures, for example, to write a letter “b” with a series of picture frames by varying the phase difference between the two spins. Our work may stimulate more applications in near-field optics, such as optical integrated circuits, tip-free near-field scanning optical microscopy, and plasmonic tweezers to trap and move micron size particles.

Tensor field design plays an essential role in various computer graphics applications. One of the main challenges in field design is how to obtain a smooth field preserving meaningful singularities presented in the original scene. Compared with well-studied point singularities, line singularities, commonly occur on object boundaries and occluding contours, are not exploited enough for field design in previous work. In this paper, we discuss the definition of line singularities, and introduce a line-singularity-based interactive tensor field design method, allowing the user to design feature-preserving tensor fields with less effort and to preserve both input singularities and user-specified stroke directions. To avoid introducing extra interaction burdens to the user, our method automatically locates line singularities using a geodesic-based segmentation. We demonstrate the capabilities of our method on tensor field design with various nonphotorealistic rendering applications and the real-time performance accelerated on GPU.

With the strong demand for human machine interaction, action recognition has attracted more and more attention in recent years. Traditional video-based approaches are very sensitive to background activity, and also lack the ability to discriminate complex 3D motion. With the emergence and development of commercial depth cameras, action recognition based on 3D skeleton joints is becoming more and more popular. However, a skeleton-based approach is still very challenging because of the large variation in human actions and temporal dynamics. In this paper, we propose a hierarchical model for action recognition. To handle confusing motions in a large feature space, a motion-based grouping method is first proposed, which can efficiently assign each video a group label, and then for each group, a pre-trained classifier is used for frame-labeling. Unlike previous methods, we adopt a bottom-up approach that first performs action recognition for each frame. The final action label is obtained by fusing the classification to its frames, with the effect of each frame being adaptively adjusted based on its local properties. The proposed method is evaluated using two challenge datasets captured by a Kinect. Experiments show that our method can perform more robustly than state-of-the-art approaches.

Online foreground segmentation usually requires to be initialized in order to specify the segmentation model. In real environments, re-initialization is also often required because of unexpected events that can break the segmentation process (e.g. unintentional movements of the camera). Traditional approaches for the initialization can not deal with common cases conveniently, especially in the situations that the segmentation process is frequently broken (e.g. network meeting with laptop). In this paper we propose a method that enables online segmentation to be initialized easily. The core of our approach is a novel motion segmentation method, that is, the Segmentation from Small Motion (SfSM), which can accurately extract the foreground object based on its slight motion between two frames. To initialize the segmentation process, we first use SfSM to segment a series of frames, and then select the one with the highest confidence as the initialization frame. Unexpected events can be handled easily by detecting changes in the background, and then re-initialize the segmentation model in the changed scene. Experiments demonstrate that our method can effectively deal with common cases in a convenient way.

We propose a visual tele-AR system, which is a combination of tele-existence and AR techniques. By wearing a video see-through HMD and moving his head, a user in this system is able to tele-operate a remote pan-tilt platform to observe the remote scene. Unlike traditional tele-existence systems which use mechanical or magnetic sensors for tele-operation, we propose a novel tele-operation method based on real-time tracking of master and slave cameras. The master camera is a component of the video see-through HMD mounted in front of user's eyes, while the slave camera is carried by a remote platform. We estimate use's head motion by tracking the master camera in real time, and translate head motions into motion commands to drive the remote pan-tilt platform. Ideally, motion of the remote platform will be exactly the same as user's head. Unfortunately, motion speed of the slave platform is usually limited by their mechanical capability, which may cause pose inconsistence between master and slave cameras, especially when user's head moves fast. We solve this problem by also tracking slave camera in real time, and warping video images according to difference between master and slave camera pose. Since slave camera is tracked, we can integrate virtual objects into remote scene, giving the user an augmented visual sense. To reduce system delay, our implementation is accelerated GPU parallel computation. Experimental results show the efficiency and robustness of our method.

One of the key challenges for cooperative relaying multi-rate wireless networks is the integration and interaction of relaying with link adaptation algorithms, in particular under incomplete and/or imprecise channel state information conditions. In this paper we propose a practical cooperative relaying link adaptation medium access protocol that considers the historical link quality to enable the relay terminal to adjust its transmission relaying rate by estimating the error probability based on past packet transmission history. Simulation results show that this protocol achieves significant performance improvement in terms of end-to-end throughput and energy efficiency for different network conditions compared with other link adaptation algorithms. It also has the best performance of the direct transmission without perfect knowledge of current channel condition. One of the key challenges for cooperative relaying multi-rate wireless networks is the integration and interaction of relaying with link adaptation algorithms, in particular under incomplete and/or imprecise channel state information conditions. In this paper we propose a practical cooperative relaying link adaptation medium access protocol that considers the historical link quality to enable the relay terminal to adjust its transmission relaying rate by estimating the error probability based on past packet transmission history. Simulation results show that this protocol achieves significant performance improvement in terms of end-to-end throughput and energy efficiency for different network conditions compared with other link adaptation algorithms. It also has the best performance of the direct transmission without perfect knowledge of current channel condition.

In geological engineering, stratum structure detection is a fundamental problem in project planning and implementation. One of the most commonly employed detection technologies is to take videos of borehole using a forward moving camera. Following this approach, the problem of stratum structure detection is transformed into the problem of constructing a panoramic image from the taken video sequences, which are typically in low quality. In this paper, we propose a novel method to create a panoramic image of the borehole from the video sequence without camera calibration and tracking. To stitch together pixels of neighboring frame images, our camera model is designed with a focal length changing feature, along with a small rotation freedom in the two-dimensional image space. Essentially, our camera model assumes target objects lie on a cylindrical wall and the camera moves forward along the central axis of the cylindrical wall. Our method robustly resolves these two degrees-of-freedoms through KLT feature tracking and constructs a panoramic image by stitching strips. Experiment results show that our method could efficiently generate high-quality panoramas for very long video sequences.

Automatic segmentation without any user interaction is very difficult due to potentially high complexity of the scene. No wonder, most existing segmentation algorithms are based on user interactions. However, automatic segmentation in some special situations has great significance. In this paper, we introduce an automatic segmentation algorithm for frontal head-and-shoulder images. Our algorithm combines edge feature and shape prior to extract the foreground silhouette automatically. The novelty of our approach lies in two aspects, namely, the Cost Path Segmentation $(CPS)$ algorithm to extract the initial foreground silhouette, and a general active prior shape model, to extract the final foreground segmentation. We demonstrate the high quality and performance of the proposed approach with a variety of head-and-shoulder images. Compared with previous methods, our approach is much more robust for images with complex color distributions in foreground and background.

The cognitive radio is growing an important interest in wireless communication study. A cognitive radio ad hoc network may take a master-slave tree-based structure in some special applications. For a master node with limited communication capability, slave nodes usually use the same frequency to access a subnet managed by the master. Each slave node can acquire many frequencies for communication by a local spectrum sensing process. However, there may be no common set of frequencies available for every slave node. In this case, we should delete some slave nodes and keep other nodes staying in the subnet as many as possible. By mapping the node set and frequency set to be both parts of a bipartite graph respectively, the problem can be turned into a special case of searching for maximal bicliques. Based on a well-known LCM (Linear time Closed itemset Miner) algorithm which enumerates frequent item sets (maximal bicliques), and using a new technique in terms of dynamic thresholds, we have solved this problem in real time to meet requirements of most cases from our application. And we also improved the LCM algorithm by pruning more rows and columns of vertices in a bipartite graph and by mining more heuristic information about what vertices make others unclosed to achieve much better performance.

Online foreground extraction is very difficult due to the complexity of real scenes. Almost all the previous methods assume that the background is stationary, which not only incur unreliable result due to background activities like dynamic shadow, moving background objects etc., but also makes them hard to be extended to the case of non-stationary background. In this paper we assume that the background is continuous instead of stationary, and present a transductive video segmentation method that can handle dynamic scenes captured by a hand-held moving camera. The segmentation is propagated based on local color models and temporal prior, as well as a dynamic global color model (DGKDE) in the case of occlusion. A novel local color modeling method, FLKDE, is proposed to model both local color distribution and temporal prior at each pixel. FLKDE can be learned additively to reach real-time speed. Finally, a very fast geodesic-based method is adopted to solve for the segmentation. Experiments show that our method can generate good quality segmentation for wide variety of scenes, and can reach 15∼25 fps for 640×480 size of input image sequences.

By analyzing the aeronautic channel, it is known the channel has the ability of sparseness. Using the common channel estimation will bring out high-order matrix computation. To improve the speed of the convergence and convenience of implementation, a new algorithm on OFDM aeronautic channel estimation using GAIC (generalized Akaike information criterion) and RPC (phase-rotated Chu sequences) is presented in this paper. The algorithm is to gain the order of channel and the place of tap by GAIC. In this algorithm RPC is used as train sequence to reduce the complexity and computation. The simulation proves the algorithm needs less resources and calculation and the error of estimation is remarkably reduced. So the algorithm can trace channel variations efficiently.

Pointing to dimension limit of serial feature fusion method, and quantity limit of parallel complex vector feature fusion method, an evolution of parallel vector feature fusion method based on quaternion has been proposed. The feasibility and the rationality of its application is proved in details theoretically. The feasibility of Fisher classification method in real domain and complex domain has been extended to quaternion space. Then the detail usage and process of the generalized Fisher method used in image classification has been given in this paper. Finally, the new algorithm is used in face detection and good result has been obtained