Search

Distinguishing between dynamic foreground objects and a mostly static background is a fundamental problem in many computer vision and computer graphics tasks. This paper presents a novel online video background identification method with the assistance of inertial measurement unit (IMU). Based on the fact that the background motion of a video essentially reflects the 3D camera motion, we leverage IMU data to realize a robust camera motion estimation for identifying background feature points by only investigating a few historical frames. We observe that the displacement of the 2D projection of a scene point caused by camera rotation is depth-invariant, and the rotation estimation by using IMU data can be quite accurate. We thus propose to analyze 2D feature points by decomposing the 2D motion into two components: rotation projection and translation projection. In our method, after establishing the 3D camera rotations, we generate the depth-relevant 2D feature point movement induced by the camera 3D translation. Then, by examining the disparity between inter-frame offset and the projection of estimated 3D camera motion, we can identify the background feature points. In the experiments, our online method is able to run at 30FPS with only 1 frame latency and outperforms state-of-the-art background identification and other relevant methods. Our method directly leads to a better camera motion estimation, which is beneficial to many applications like online video stabilization, SLAM, image stitching, etc. [ABSTRACT FROM AUTHOR]

Inferring the scene illumination from a single image is an essential yet challenging task in computer vision and computer graphics. Existing works estimate lighting by regressing representative illumination parameters or generating illumination maps directly. However, these methods often suffer from poor accuracy and generalization. This paper presents Geometric Mover’s Light (GMLight), a lighting estimation framework that employs a regression network and a generative projector for effective illumination estimation. We parameterize illumination scenes in terms of the geometric light distribution, light intensity, ambient term, and auxiliary depth, which can be estimated by a regression network. Inspired by the earth mover’s distance, we design a novel geometric mover’s loss to guide the accurate regression of light distribution parameters. With the estimated light parameters, the generative projector synthesizes panoramic illumination maps with realistic appearance and high-frequency details. Extensive experiments show that GMLight achieves accurate illumination estimation and superior fidelity in relighting for 3D object insertion. The codes are available at https://github.com/fnzhan/Illumination-Estimation [ABSTRACT FROM AUTHOR]

Superpixel segmentation, as a central image processing task, has many applications in computer vision and computer graphics. Boundary alignment and shape compactness are leading indicators to evaluate a superpixel segmentation algorithm. Furthermore, convexity can make superpixels reflect more geometric structures in images and provide a more concise over-segmentation result. In this paper, we consider generating convex and compact superpixels while satisfying the constraints of adhering to the boundary as far as possible. We formulate the new superpixel segmentation into an edge-constrained centroidal power diagram (ECCPD) optimization problem. In the implementation, we optimize the superpixel configurations by repeatedly performing two alternative operations, which include site location updating and weight updating through a weight function defined by image features. Compared with existing superpixel methods, our method can partition an image into fully convex and compact superpixels with better boundary adherence. Extensive experimental results show that our approach outperforms existing superpixel segmentation methods in boundary alignment and compactness for generating convex superpixels. [ABSTRACT FROM AUTHOR]

Recent emerging technologies such AR/VR and HCI are drawing high demand on more comprehensive hand shape understanding, requiring not only 3D hand skeleton pose but also hand shape geometry. In this paper, we propose a deep learning framework to produce 3D hand shape from a single depth image. To address the challenge that capturing ground truth 3D hand shape in the training dataset is non-trivial, we leverage synthetic data to construct a statistical hand shape model and adopt weak supervision from widely accessible hand skeleton pose annotation. To bridge the gap due to the different hand skeleton definitions in the existing public datasets, we propose a joint regression network for hand pose adaptation. To reconstruct the hand shape, we use Chamfer loss between the predicted hand shape and the point cloud from the input depth to learn the shape reconstruction model in a weakly-supervised manner. Experiments demonstrate that our model adapts well to the real data and produces accurate hand shapes that outperform the state-of-the-art methods both qualitatively and quantitatively. [ABSTRACT FROM AUTHOR]

Generating text to have a consistent style with only a few observed highly-stylized text samples is a difficult task for image processing. The text style involving the typography, i.e., font, stroke, color, decoration, effects, etc., should be considered for transfer. In this paper, we propose a novel approach to stylize target text by decoding weighted deep features from only a few referenced samples. The deep features, including content and style features of each referenced text, are extracted from a Convolutional Neural Network (CNN) that is optimized for character recognition. Then, we calculate the similarity scores of the target text and the referenced samples by measuring the distance along the corresponding channels from the content features of the CNN when considering only the content, and assign them as the weights for aggregating the deep features. To enforce the stylized text to be realistic, a discriminative network with adversarial loss is employed. We demonstrate the effectiveness of our network by conducting experiments on three different datasets which have various styles, fonts, languages, etc. Additionally, the coefficients for character style transfer, including the character content, the effect of similarity matrix, the number of referenced characters, the similarity between characters, and performance evaluation by a new protocol are analyzed for better understanding our proposed framework. [ABSTRACT FROM AUTHOR]

Measuring the similarity between the patches in images is a fundamental building block in various tasks. Naturally, the patch size has a major impact on the matching quality and on the consequent application performance. Under the assumption that our patch database is sufficiently sampled, using large patches (e.g., $21 \,\, \times \,\, 21$ ) should be preferred over small ones (e.g., $7 \,\, \times \,\, 7$ ). However, this dense-sampling assumption is rarely true; in most cases, large patches cannot find relevant nearby examples. This phenomenon is a consequence of the curse of dimensionality, stating that the database size should grow exponentially with the patch size to ensure proper matches. This explains the favored choice of small patch size in most applications. Is there a way to keep the simplicity and work with small patches while getting some of the benefits that large patches provide? In this paper, we offer such an approach. We propose to concatenate the regular content of a conventional (small) patch with a compact representation of its (large) surroundings—its context. Therefore, with a minor increase of the dimensions (e.g., with additional ten values to the patch representation), we implicitly/softly describe the information of a large patch. The additional descriptors are computed based on a self-similarity behavior of the patch surrounding. We show that this approach achieves better matches, compared with the use of conventional-size patches, without the need to increase the database-size. Also, the effectiveness of the proposed method is tested on three distinct problems: 1) external natural image denoising; 2) depth image super-resolution; and 3) motion-compensated frame-rate up conversion. [ABSTRACT FROM PUBLISHER]

Automatic sketch colorization is a challenging task in both computer graphics and computer vision since all the color, texture, shading generation have to be created based on the abstract sketch. Besides, it is a subjective task in painting process, which needs illustrators to comprehend drawing priori (DP), such as hue variation, saturation contrast and gray contrast and utilize them in the HSV color space which is closer to human visual cognition system. As such, incorporating supplementary supervision in the HSV color space may be beneficial to sketch colorization. However, previous methods improve the colorization quality only in the RGB color space without considering the HSV color space, often causing results with dull color, inappropriate saturation contrast, and artifacts. To address this issue, we propose a novel sketch colorization method, dual color space guided generative adversarial network (DCSGAN), that considers the complementary information contained in both the RGB and HSV color space. Specifically, we incorporate the HSV color space to construct dual color spaces for supervising our method with a color space transformation (CST) network that learns transformation from the RGB to HSV color space. Then, we propose a DP loss that enables the DCSGAN to generate vivid color images with pixel level supervision. Additionally, a novel dual color space adversarial (DCSA) loss is designed to guide the generator at global level to reduce the artifacts to meet audiences’ aesthetic expectations. Extensive experiments and ablation studies demonstrate the superiority of the proposed method over previous state-of-the-art (SOTA) methods. [ABSTRACT FROM AUTHOR]

Occlusions will reduce the performance of systems in many computer vision applications with discontinuous surfaces of 3D scenes. We explore a signal-processing framework of occlusions based on the light ray visibility to improve the rendering quality of views. An occlusion field (OCF) theory is derived by calculating the relationship between the occluded light rays and the nonoccluded light rays to quantify the occlusion degree (OCD). The OCF framework can describe the various in-scene information captured by the changes in the camera configuration (i.e., position and direction) through a quantitative description of the occlusion information. From a spectral analysis of the OCF, we mathematically derive analytical functions to determine the changing relationship between the scene and the camera configuration. A reconstruction filter can be designed to achieve interference cancellation and compensate for the missing information caused by the occlusions. Our measurements of different occlusions using this OCF framework included both synthetic and actual scenes. The experimental results show that the proposed OCF framework can improves the rendering quality of views and outperforms other known occlusion quantization schemes in a complex scene. [ABSTRACT FROM AUTHOR]

We present a unified framework to track multiple people, as well localize, and label their activities, in complex long-duration video sequences. To do this, we focus on two aspects: 1) the influence of tracks on the activities performed by the corresponding actors and 2) the structural relationships across activities. We propose a two-level hierarchical graphical model, which learns the relationship between tracks, relationship between tracks, and their corresponding activity segments, as well as the spatiotemporal relationships across activity segments. Such contextual relationships between tracks and activity segments are exploited at both the levels in the hierarchy for increased robustness. An L1-regularized structure learning approach is proposed for this purpose. While it is well known that availability of the labels and locations of activities can help in determining tracks more accurately and vice-versa, most current approaches have dealt with these problems separately. Inspired by research in the area of biological vision, we propose a bidirectional approach that integrates both bottom-up and top-down processing, i.e., bottom-up recognition of activities using computed tracks and top-down computation of tracks using the obtained recognition. We demonstrate our results on the recent and publicly available UCLA and VIRAT data sets consisting of realistic indoor and outdoor surveillance sequences. [ABSTRACT FROM AUTHOR]