Search

Directly benefiting from the rapid advancement of deep learning methods, person re-identification (Re-ID) applications have been widespread with remarkable successes in recent years. Nevertheless, cross-scene Re-ID is still hindered by large view variation, since it is challenging to effectively exploit and leverage the temporal clues due to heavy computational burden and the difficulty in flexibly incorporating discriminative features. To alleviate, we articulate a long-short temporal–spatial clues excited network (LSTS-NET) for robust person Re-ID across different scenes. In essence, our LSTS-NET comprises a motion appearance model and a motion-refinement aggregating scheme. Of which, the former abstracts temporal clues based on multi-range low-rank analysis both in consecutive frames and in cross-camera videos, which can augment the person-related features with details while suppressing the clutter background across different scenes. In addition, to aggregate the temporal clues with spatial features, the latter is proposed to automatically activate the person-specific features by incorporating personalized motion-refinement layers and several motion-excitation CNN blocks into deep networks, which expedites the extraction and learning of discriminative features from different temporal clues. As a result, our LSTS-NET can robustly distinguish persons across different scenes. To verify the improvement of our LSTS-NET, we conduct extensive experiments and make comprehensive evaluations on 8 widely-recognized public benchmarks. All the experiments confirm that, our LSTS-NET can significantly boost the Re-ID performance of existing deep learning methods, and outperforms the state-of-the-art methods in terms of robustness and accuracy.

Different types of cloth show distinctive appearances owing to their unique yarn-level geometrical details. Despite its importance in applications such as cloth rendering and simulation, capturing yarn-level geometry is nontrivial and requires special hardware, e.g., computed tomography scanners, for conventional methods. In this paper, we propose a novel method that can produce the yarn-level geometry of real cloth using a single micro-image, captured by a consumer digital camera with a macro lens. Given a single input image, our method estimates the large-scale yarn geometry by image shading, and the fine-scale fiber details can be recovered via the proposed fiber tracing and generation algorithms. Experimental results indicate that our method can capture the detailed yarn-level geometry of a wide range of cloth and reproduce plausible cloth appearances.

Despite the growing number of unmanned aerial vehicles (UAVs) and aerial videos, there is a paucity of studies focusing on the aesthetics of aerial videos that can provide valuable information for improving the aesthetic quality of aerial photography. In this article, we present a method of deep multimodality learning for UAV video aesthetic quality assessment. More specifically, a multistream framework is designed to exploit aesthetic attributes from multiple modalities, including spatial appearance, drone camera motion, and scene structure. A novel specially designed motion stream network is proposed for this new multistream framework. We construct a dataset with 6,000 UAV video shots captured by drone cameras. Our model can judge whether a UAV video was shot by professional photographers or amateurs together with the scene type classification. The experimental results reveal that our method outperforms the video classification methods and traditional SVM-based methods for video aesthetics. In addition, we present three application examples of UAV video grading, professional segment detection and aesthetic-based UAV path planning using the proposed method., Comment: IEEE Trans. on Multimedia, 2020

This paper presents a new method for 3D model labeling guided by weakly tagged 2D color images. Many previous methods on 3D model labeling achieve impressive performances using large training data sets. However, it is difficult and time-consuming to build such a carefully annotated data set. In order to solve this problem, we take advantage of the large number of weakly tagged color images to label the 3D models. In our approach, we first collect and tag the web color images with semantic annotations. Then we project the input 3D model into multiview projections. Through the multiview alignment, we transfer the semantic labels onto the model projections via a color-weighting process. Combining pre-segment information, we back-project the labels and get final labeling results. Experimental results between two benchmarks show that our approach could get comparable labeling accuracy compared to other two state-of-art methods without expensive training cost.

Despite growing progresses in recent years, cross-scenario person re-identification remains challenging, mainly due to the pedestrians commonly surrounded by highly-complex environment contexts. In reality, the human perception mechanism could adaptively find proper contextualized spatial-temporal clues towards pedestrian recognition. However, conventional methods fall short in adaptively leveraging the long-term spatial-temporal information due to ever-increasing computational cost. Moreover, CNN-based deep learning methods are hard to conduct optimization due to the non-differentiable property of the built-in context search operation. To ameliorate, this paper proposes a novel Context-Interactive CNN (CI-CNN) to dynamically find both spatial and temporal contexts by embedding multi-task Reinforcement Learning (MTRL). The CI-CNN streamlines the multi-task reinforcement learning by using an actor-critic agent to capture the temporal-spatial context simultaneously, which comprises a context-policy network and a context-critic network. The former network learns policies to determine the optimal spatial context region and temporal sequence range. Based on the inferred temporal-spatial cues, the latter one focuses on the identification task and provides feedback for the policy network. Thus, CI-CNN can simultaneously zoom in/out the perception field in spatial and temporal domain for the context interaction with the environment. By fostering the collaborative interaction between the person and context, our method could achieve outstanding performance on various public benchmarks, which confirms the rationality of our hypothesis, and verifies the effectiveness of our CI-CNN framework.

For the task of mobility analysis of 3D shapes, we propose joint analysis for simultaneous motion part segmentation and motion attribute estimation, taking a single 3D model as input. The problem is significantly different from those tackled in the existing works which assume the availability of either a pre-existing shape segmentation or multiple 3D models in different motion states. To that end, we develop Shape2Motion which takes a single 3D point cloud as input, and jointly computes a mobility-oriented segmentation and the associated motion attributes. Shape2Motion is comprised of two deep neural networks designed for mobility proposal generation and mobility optimization, respectively. The key contribution of these networks is the novel motion-driven features and losses used in both motion part segmentation and motion attribute estimation. This is based on the observation that the movement of a functional part preserves the shape structure. We evaluate Shape2Motion with a newly proposed benchmark for mobility analysis of 3D shapes. Results demonstrate that our method achieves the state-of-the-art performance both in terms of motion part segmentation and motion attribute estimation., Comment: CVPR 2019 (oral presentation); Corresponding author: Kai Xu (kevin.kai.xu@gmail.com); Project page: www.kevinkaixu.net/projects/shape2motion.html

With the increasing availability of high-resolution images, videos, and 3D models, the demand for scalable large data processing techniques increases. We introduce a method of sparse dictionary learning for edit propagation of large input data. Previous approaches for edit propagation typically employ a global optimization over the whole set of pixels (or vertexes), incurring a prohibitively high memory and time-consumption for large input data. Rather than propagating an edit pixel by pixel, we follow the principle of sparse representation to obtain a representative and compact dictionary and perform edit propagation on the dictionary instead. The sparse dictionary provides an intrinsic basis for input data, and the coding coefficients capture the linear relationship between all pixels and the dictionary atoms. The learned dictionary is then optimized by a novel scheme, which maximizes the Kullback-Leibler divergence between each atom pair to remove redundant atoms. To enable local edit propagation for images or videos with similar appearance, a dictionary learning strategy is proposed by considering range constraint to better account for the global distribution of pixels in their feature space. We show several applications of the sparsity-based edit propagation, including video recoloring, theme editing, and seamless cloning, operating on both color and texture features. Our approach can also be applied to computer graphics tasks, such as 3D surface deformation. We demonstrate that with an atom-to-pixel ratio in the order of 0.01% signifying a significant reduction on memory consumption, our method still maintains a high degree of visual fidelity.

A majority of stock 3D models in modern shape repositories are assembled with many fine-grained components. The main cause of such data form is the component-wise modeling process widely practiced by human modelers. These modeling components thus inherently reflect some function-based shape decomposition the artist had in mind during modeling. On the other hand, modeling components represent an over-segmentation since a functional part is usually modeled as a multi-component assembly. Based on these observations, we advocate that labeled segmentation of stock 3D models should not overlook the modeling components and propose a learning solution to grouping and labeling of the fine-grained components. However, directly characterizing the shape of individual components for the purpose of labeling is unreliable, since they can be arbitrarily tiny and semantically meaningless. We propose to generate part hypotheses from the components based on a hierarchical grouping strategy, and perform labeling on those part groups instead of directly on the components. Part hypotheses are mid-level elements which are more probable to carry semantic information. A multiscale 3D convolutional neural network is trained to extract context-aware features for the hypotheses. To accomplish a labeled segmentation of the whole shape, we formulate higher-order conditional random fields (CRFs) to infer an optimal label assignment for all components. Extensive experiments demonstrate that our method achieves significantly robust labeling results on raw 3D models from public shape repositories. Our work also contributes the first benchmark for component-wise labeling., Accepted to SIGGRAPH Asia 2018. Corresponding Author: Kai Xu (kevin.kai.xu@gmail.com)

Lighting is a key factor in creating impressive artistic portraits. In this paper, we propose to analyze portrait lighting by learning templates of lighting styles. Inspired by the experience of artists, we first define several novel features that describe the local contrasts in various face regions. The most informative features are then selected with a stepwise feature pursuit algorithm to derive the templates of various lighting styles. After that, the matching scores that measure the similarity between a testing portrait and those templates are calculated for lighting style classification. Furthermore, we train a regression model by the subjective scores and the feature responses of a template to predict the score of a portrait lighting quality. Based on the templates, a novel face illumination descriptor is defined to measure the difference between two portrait lightings. Experimental results show that the learned templates can well describe the lighting styles, whereas the proposed approach can assess the lighting quality of artistic portraits as human being does.

We resolve the canopy-frame interactions for umbrella simulation in this paper. An umbrella is a heterogeneous articulated system that consists of a frame composed of elastic ribs and other rigid components, and a fabric-made canopy bound to the frame. The canopy interacts with the frame through the binding constraint and collisions. Since the flexible canopy and elastic ribs have different stiffness, we employ a bi-time-step scheme to obtain tractable performance. It evolves the frame with a smaller time step but the canopy with a larger time step. We resolve the canopy-frame interactions by changing the canopy positions and velocities at an interval of the larger time step, which is limited to be an integer multiple of the smaller time step. These displacements and velocity changes are transferred into opposite impulses and forces, and applied to the frame in the following smaller time steps. With these transformations, the solution of the canopy-frame interactions is independent of the difference of the two time steps. Moreover, we adopt a projection-based algorithm to accurately detect the canopy-frame collisions. We demonstrate the effectiveness of our method with several umbrella animations.

Skin synthesis is important for the actual appearance of digital human models. However, it is difficult to design a general algorithm to efficiently produce high quality results. This paper proposes a parallel texture synthesis method for large scale skin of digital human models. Two major procedures are included in this method, a parallel matching procedure and a multi-pass optimizing procedure. Compared with other methods, this algorithm is easy to use, requires only a small size of skin image as input, and generates an arbitrary size of skin texture with high quality. As demonstrated by experiments, the effectiveness of this skin texture synthesis method is confirmed.

This paper proposes a novel image-based framework to manipulate the illumination of human face through adaptive layer decomposition. According to our framework, only a single reference image, without any knowledge of the 3D geometry or material information of the input face, is needed. To transfer the illumination effects of a reference face image to a normal lighting face, we first decompose the lightness layers of the reference and the input images into large-scale and detail layers through weighted least squares (WLS) filter with adaptive smoothing parameters according to the gradient values of the face images. The large-scale layer of the reference image is filtered with the guidance of the input image by guided filter with adaptive smoothing parameters according to the face structures. The relit result is obtained by replacing the largescale layer of the input image with that of the reference image. To normalize the illumination effects of a non-normal lighting face (i.e., face delighting), we introduce similar reflectance prior to the layer decomposition stage by WLS filter, which make the normalized result less affected by the high contrast light and shadow effects of the input face. Through these two procedures, we can change the illumination effects of a non-normal lighting face by first normalizing the illumination and then transferring the illumination of another reference face to it. We acquire convincing relit results of both face relighting and delighting on numerous input and reference face images with various illumination effects and genders. Comparisons with previous papers show that our framework is less affected by geometry differences and can preserve better the identification structure and skin color of the input face.

To bring computer vision closer to human vision, we attempt to enable computer to understand the occlusion relationship in an image. In this paper, we propose five low dimensional region-based occlusion cues inspired by the human perception of occlusion. These cues are semantic cue, position cue, compactness cue, shared boundary cue and junction cue. We apply these cues to predict the region-wise occlusion relationship in an image and infer the layer sequence of the image scene. A preference function, trained with samples consisting of these cues, is defined to predict the occlusion relationship in an image. Then we put all the occlusion predictions into the layering algorithm to infer the layer sequence of the image scene. The experiments on rural, artificial and outdoor scene datasets show the effectiveness of our method for occlusion relationship prediction and image scene layering.

In this paper, we present a novel automatic image completion solution in a greedy manner inspired by a primal sketch representation model. Firstly, an image is divided into structure (sketchable) components and texture (non-sketchable) components, and the missing structures, such as curves and corners, are predicted by tensor voting. Secondly, the textures along structural sketches are synthesized with the sampled patches of some known structure components. Then, using the texture completion priorities decided by the confidence term, data term and distance term, the similar image patches of some known texture components are found by selecting a point with the maximum priority on the boundary of hole region. Finally, these image patches inpaint the missing textures of hole region seamlessly through graph cuts. The characteristics of this solution include: (1) introducing the primal sketch representation model to guide completion for visual consistency; (2) achieving fully automatic completion. The experiments on natural images illustrate satisfying image completion results.

Virtual Reality was one of the 14 Grand Challenges identified as awaiting engineering solutions for the 21st century announced in 2008 by the U.S. National Academy of Engineering (NAE http://www.nae.edu/). Here, I explore 10 related open VR challenges, with hoped-for potential breakthroughs promising to advance VR techniques and applications. VR is today being applied in multiple contexts, including training, exercise, engineering design, and entertainment, while also serving as a research tool in such fields as neuroscience and psychology, as explored in Michael Heim's pioneering 1993 book Metaphysics of Virtual Reality (http://www.mheim.com/books.html). More recently, scholars have described the Internet itself as representing a virtual world modeling its real-world counterpart. The relationship between VR and its application fields is, in terms of expression and validation, like the relationship between mathematics and physics, while VR is attracting attention from a growing number of governments and science/engineering communities. Along with the NAE Committee on Engineering 14 Grand Challenges (http://www.engineeringchallenges.org/), the Chinese government's 2006 report Development Plan Outline for Medium- and Long-Term Science and Technology Development (2006--2020) (http://www.gov.cn/jrzg/2006-02/09/content_183787.htm) and the Japanese government's 2007 report Innovation 2025 (http://www.cao.go.jp/innovation/index.html) both included VR as a priority technology worthy of development. VR has likewise emerged as an important research area for many Chinese universities and research institutes, ranging from theoretical foundations to technology innovation, system development, and practical applications. For example, Zhejiang University in Hangzhou and Tsinghua University in Beijing are known for realistic modeling and rendering; Peking University in Beijing focuses on computer vision and human-machine interaction; the Beijing Institute of Technology in Beijing emphasizes head-mounted displays; and the Institute of Computing Technology of Chinese Academy of Sciences in Beijing has made significant progress in crowd simulation. Since 1995, I have directed the State Key Laboratory of Virtual Reality Technology and Systems of China (http://www.skvrlab.net) in Beijing, focusing on VR. Over that time, we have proposed many new methods and technologies in distributed virtual environments, realistic modeling and rendering, and augmented reality, including two major VR software infrastructures: BH_RTI for distributed virtual environments (http://www.hlarti.com) and BH_GRAPH for real-time 3D rendering (http://www.skvrlab.net:8080/soft). This work has produced several influential VR application systems, including the 2008 Beijing Olympic Games Opening Ceremony Simulation Platform and the Beijing 08 Digital Museum. We also designed a training-and-scenario-simulation system for a military review of China's 2009 60th anniversary celebration, with more than 8,000 soldiers, based on visualized immersion technologies to deliver scenario preview, modification, and decision-making functions. Meanwhile, our work advancing theory in related fundamental disciplines may help produce further breakthroughs in VR; for example, electrorheological fluids could change the existing method of haptic rendering.

This paper presents a semantic labeling framework with geodesic propagation (GP). Under the same framework, three algorithms are proposed, including GP, supervised GP (SGP) for image, and hybrid GP (HGP) for video. In these algorithms, we resort to the recognition proposal map and select confident pixels with maximum probability as the initial propagation seeds. From these seeds, the GP algorithm iteratively updates the weights of geodesic distances until the semantic labels are propagated to all pixels. On the contrary, the SGP algorithm further exploits the contextual information to guide the direction of propagation, leading to better performance but higher computational complexity than the GP. For video labeling, we further propose the HGP algorithm, in which the geodesic metric is used in both spatial and temporal spaces. Experiments on four public data sets show that our algorithms outperform several state-of-the-art methods. With the GP framework, convincing results for both image and video semantic labeling can be obtained.

We introduce a method of sparse dictionary learning for edit propagation of high-resolution images or video. Previous approaches for edit propagation typically employ a global optimization over the whole set of image pixels, incurring a prohibitively high memory and time consumption for high-resolution images. Rather than propagating an edit pixel by pixel, we follow the principle of sparse representation to obtain a compact set of representative samples (or features) and perform edit propagation on the samples instead. The sparse set of samples provides an intrinsic basis for an input image, and the coding coefficients capture the linear relationship between all pixels and the samples. The representative set of samples is then optimized by a novel scheme which maximizes the KL-divergence between each sample pair to remove redundant samples. We show several applications of sparsity-based edit propagation including video recoloring, theme editing, and seamless cloning, operating on both color and texture features. We demonstrate that with a sample-to-pixel ratio in the order of 0.01%, signifying a significant reduction on memory consumption, our method still maintains a high-degree of visual fidelity.

Pedestrian detection and human pose estimation are instructive for reconstructing a three-dimensional scenario and for robot navigation, particularly when large amounts of vision data are captured using various data-recording techniques. Using an unrestricted capture scheme, which produces occlusions or breezing, the information describing each part of a human body and the relationship between each part or even different pedestrians must be present in a still image. Using this framework, a multi-layered, spatial, virtual, human pose reconstruction framework is presented in this study to recover any deficient information in planar images. In this framework, a hierarchical parts-based deep model is used to detect body parts by using the available restricted information in a still image and is then combined with spatial Markov random fields to re-estimate the accurate joint positions in the deep network. Then, the planar estimation results are mapped onto a virtual three-dimensional space using multiple constraints to recover any deficient spatial information. The proposed approach can be viewed as a general pre-processing method to guide the generation of continuous, three-dimensional motion data. The experiment results of this study are used to describe the effectiveness and usability of the proposed approach.

A system of multivalued logical equations and its solution algorithm are put forward in this paper. Based on this work we generalize SLD-resolution into multivalued logic and establish the corresponding truth value calculus. As a result, M, an approximate reasoning system, is built. We present the language and inference rules of M. Furthermore, we analyse inconsistency of assignments to truth degrees and give the solving strategies of M.

We propose an approach that parses registered images captured at ground level into architectural units for large-scale city modeling. Each parsed unit has a regularized shape, which can be used for further modeling purposes. In our approach, we first parse the environment into buildings, the ground, and the sky using a joint 2D-3D segmentation method. Then, we partition buildings into individual facades. The partition problem is formulated as a dynamic programming optimization for a sequence of natural vertical separating lines. Each facade is regularized by a floor line and a roof line. The floor line is the intersection line of the vertical plane of buildings and the horizontal plane of the ground. The roof line links edge points of roof region. The parsed results provide a first geometric approximation to the city environment, and can be further analyzed if necessary. The approach is demonstrated and validated on several large-scale city datasets.

Existing light field-based lighting follows the representation on plane, such as the commonly used light slab. A novel light field representation on scene surface is proposed. The illumination is captured by a panorama camera on translation stage with a nonuniform sampling and calibration strategy. Sampled light rays are reprojected on the reconstructed scene to generate the scene surface light field representations for both direct light source and indirect light. This representation is more consistent with the actual emission and reflection of light transport than the existing ones. An accurate light source estimation and a practical indirect light resampling approach are presented based on the representation. The rendering results illustrate the realistic lighting effect, especially in spatially varying illumination.