Search

With the availability of easy access to the Internet, and the proliferation of various mobile devices, people can get access to mobile multimedia ubiquitously in order to have a desktop experience. Mobile multimedia refers to different types of multimedia content (e.g., text, images, audios, and videos), which are accessed via heterogeneous mobile devices, such as mobile phones, portable devices, and smartphones. Mobile multimedia services and applications have tremendous potential in the healthcare sector. Currently, as the healthcare sector is increasingly dependent on mobile multimedia services and applications, many challenges remain unsolved for the successful deployment of mobile multimedia services in order to have a balance between power-limited mobile devices and resource hungry medical multimedia content. This Special Section aims to theme innovative research achievements in the field of related techniques, applications, services, and systems for mobile multimedia healthcare.

The text rendered by special effects can give a rich visual experience. Text stylization can help users migrate their favorite styles to specified texts, improving production efficiency and saving design cost. This paper proposes a novel text stylization framework, which can transfer mixed text styles, including font glyph and fine decorations, to user-specified texts. The transfer of decorative elements is difficult due to the text is obscured by decorative elements to a certain extent. Our method is divided into three stages: firstly, the position of decorative elements in the image is extracted and retained; secondly, the effects of font glyph and textures other than decorative elements are migrated; finally, a structure-aware strategy is used to reorganize the decorative elements to complete the entire stylization process. Experiments on open source text data sets demonstrated the advantages of our approach over other state- of-the-art style migration methods.

Existing dominant approaches for video moment retrieval task are to learn semantic correlation between a given query and the video. However, these methods rarely explore the fine-grained semantic structure and comprehensive visual structure, leading to insufficient utilization of textual and visual relations. In this paper, we propose a unified framework for video moment retrieval, which considers to simultaneously encode semantic and visual structures. Specifically, a semantic role tree is built to reveal the fine-grained semantic information by generating hierarchical textual embeddings. Then the semantic structure is adopted to facilitate the visual structure learning with a contextual attention-based proposal interaction module. Finally, we adaptively aggregate and obtain the visual-semantic matching information through a multi-level fusion strategy to select the best matching moment proposal. Extensive experiments on two popular benchmarks (Charades-STA and ActivityNet Captions) show that our proposed method achieves state-of-the-art performance. Codes are available in the Supplementary Material.

Weakly supervised object localization(WSOL) remains an open problem given the deficiency of finding object extent information using a classification network. Although prior works struggled to localize objects through various spatial regularization strategies, we argue that how to extract object structural information from the trained classification network is neglected. In this paper, we propose a two-stage approach, termed structure-preserving activation (SPA), toward fully leveraging the structure information incorporated in convolutional features for WSOL. First, a restricted activation module (RAM) is designed to alleviate the structure-missing issue caused by the classification network on the basis of the observation that the unbounded classification map and global average pooling layer drive the network to focus only on object parts. Second, we designed a post-process approach, termed self-correlation map generating (SCG) module to obtain structure-preserving localization maps on the basis of the activation maps acquired from the first stage. Specifically, we utilize the high-order self-correlation (HSC) to extract the inherent structural information retained in the learned model and then aggregate HSC of multiple points for precise object localization. Extensive experiments on two publicly available benchmarks including CUB-200-2011 and ILSVRC show that the proposed SPA achieves substantial and consistent performance gains compared with baseline approaches.Code and models are available at https://github.com/Panxjia/SPA_CVPR2021, Accepted by CVPR2021

Controllable painting generation plays a pivotal role in image stylization. Currently, the control way of style transfer is subject to exemplar-based reference or a random one-hot vector guidance. Few works focus on decoupling the intrinsic properties of painting as control conditions, e.g., artist, genre and period. Under this circumstance, we propose a novel framework adopting multiple attributes from the painting to control the stylized results. An asymmetrical cycle structure is equipped to preserve the fidelity, associating with style preserving and attribute regression loss to keep the unique distinction of colors and textures between domains. Several qualitative and quantitative results demonstrate the effect of the combinations of multiple attributes and achieve satisfactory performance.

Domain adaptation aims at learning from a source data distribution a well-performing model on a different target data distribution. Recently, the self-ensembling-based methods have been proved to be effective for unsupervised domain adaptation. However, they still have two shortcomings: 1) no explicitly constraint about the distributions between the source and target domains; 2) the Euclidean distance fails to measure the similarity between two distributions with no overlap. To solve those shortcomings, we propose a novel Category-level Adversarial Self-ensembling (CAS) model for domain adaptation, which contains two types of consistency constraints. The first one is how to constrain the descriptions for source and target domains to be aligned. Therefore, we adopt a minimax game with a discrepancy loss between the category information generated by two classifiers. As the self-ensembling consists of two sub-networks: student and teacher networks, the second one is the consistency between those two networks for the target samples. Aiming to overcome the disadvantage of the Euclidean metric, we employ the Wasserstein distance to measure the difference between two probabilistic distributions. Experiments on several benchmarks demonstrate that our proposed CAS is superior to existing methods.

Low-cost live broadcasting is highly anticipated for soccer matches that involve heavy expenses in human and equipment currently. This demo showcases the autoSoccer system to approach the target. It takes a fix-view panoramic soccer video as the input and automatically generates its live broadcasting, which continuously focuses on the most interesting area in the soccer field. To this end, a novel pipeline is developed by leveraging both low-level motion and visual features, and high-level soccer-related semantics. By appropriately utilizing these clues, autoSoccer is capable of delivering a soccer match video analogous to human directed. Demo on school soccer show that autoSoccer produces videos with satisfactory watching experience. Moreover, it executes in real-time on a PC with Intel i7 CPU and one Nvidia Titan XP GPU.

Facial expression recognition (FER) is a challenging task due to the lack of sufficient training data. Most conventional approaches usually rotate or flip the images for data augmentation. More recently, numerous methods synthesize images automatically by using Generative Adversarial Network (GAN). However, paired images are always required in these methods. Different from existing methods, in this paper, we propose an end-to-end deep learning model for simultaneous facial expression synthesis and facial expression recognition. In our method, paired images are not required, which makes the proposed model much more flexible and general. Furthermore, different expressions are encoded in a disentangled manner in a latent space, which enables us to generate facial images with arbitrary expressions by exchanging certain parts of their latent identity features. Finally, the facial expression synthesis and facial expression recognition tasks can further boost their performance for each other via our model. Quantitative and qualitative evaluations on both controlled and in-the-wild datasets demonstrate that the proposed method performs favorably against state-of-the-art methods.

Nowadays, visual relationship detection has shown an important utility in scene understanding. Predicate detection, which aims to detect the predicate between entities in an image, is an important part of visual relationship detection. In this paper, we propose Multimodal Latent Factor Model with Language Constraint (MMLFM-LC) for predicate detection with the novelty of integrating knowledge learned from multiple modalities, valid relationships and semantical similarities. Representations of visual and textual modalities are firstly input into the constructed model. Secondly, a bilinear structure is introduced to model the relationships using valid relationships, while a language constraint is also built utilizing semantical similarities. Lastly, visual and textual representations are fused in an embedded subspace for predicate detection. Experiments on both Visual Relationship and Visual Genome datasets show that our method outperforms other methods on predicate detection.

To bridge source and target domains for domain adaptation, there are three important types of information including data structure, domain label, and class label. Most existing domain adaptation approaches exploit only one or two types of this information and cannot make them complement and enhance each other. Different from existing methods, we propose an end-to-end Graph Convolutional Adversarial Network (GCAN) for unsupervised domain adaptation by jointly modeling data structure, domain label, and class label in a unified deep framework. The proposed GCAN model enjoys several merits. First, to the best of our knowledge, this is the first work to model the three kinds of information jointly in a deep model for unsupervised domain adaptation. Second, the proposed model has designed three effective alignment mechanisms including structure-aware alignment, domain alignment, and class centroid alignment, which can learn domain-invariant and semantic representations effectively to reduce the domain discrepancy for domain adaptation. Extensive experimental results on five standard benchmarks demonstrate that the proposed GCAN algorithm performs favorably against state-of-the-art unsupervised domain adaptation methods.

Tracking by siamese networks has achieved favorable performance in recent years. However, most of existing siamese methods do not take full advantage of spatial-temporal target appearance modeling under different contextual situations. In fact, the spatial-temporal information can provide diverse features to enhance the target representation, and the context information is important for online adaption of target localization. To comprehensively leverage the spatial-temporal structure of historical target exemplars and get benefit from the context information, in this work, we present a novel Graph Convolutional Tracking (GCT) method for high-performance visual tracking. Specifically, the GCT jointly incorporates two types of Graph Convolutional Networks (GCNs) into a siamese framework for target appearance modeling. Here, we adopt a spatial-temporal GCN to model the structured representation of historical target exemplars. Furthermore, a context GCN is designed to utilize the context of the current frame to learn adaptive features for target localization. Extensive results on 4 challenging benchmarks show that our GCT method performs favorably against state-of-the-art trackers while running around 50 frames per second.

Representation learning of knowledge graphs encodes both entities and relations into a continuous low-dimensional space. Most existing methods focus on learning representations with structured fact triples indicating relations between entities, ignoring rich additional information of entities including entity attributes and associated multimodal content descriptions. In this paper, we propose a new model to learn knowledge representations with entity attributes and multimedia descriptions (KR-AMD). Specifically, we construct three triple encoders to obtain structure-based entity representation, attribute-based entity representation and multimedia content-based entity representation, and finally generate the knowledge representations for knowledge graphs in KR-AMD. The experimental results show that, by special modeling of entity attributes and text-image descriptions, KR-AMD can significantly outperform state-of-the-art KR models in prediction of entities, attributes and relations, which validates the effectiveness of KR-AMD.

Feature aggregation focuses on the problem of how to fuse the extracted frame-level features into video-level features, which is crucial to the video classification task. The common aggregation networks directly deploy a pooling strategy over time to generate video-level feature. Such approaches treat inputs equally, therefore the spatio-temporal information cannot be fully utilized for varied inputs. To tackle this issue, we introduce a learnable non-linear network called Entity Competition Network (ECN) which harnesses the relationships among entities from adjacent frames. By doing this, we can reassess the importance of each entity and distinguish the entities related to target event from redundant visual information, thus better video-level features can be generated. Our approach achieves the state-of-the-art results on the action recognition datasets of HMDB51 and UCF101. Improvement is also obtained with RGB modality on the untrimmed video classification dataset ActivityNet 200.

Facial expression recognition (FER) is a challenging task due to different expressions under arbitrary poses. Most conventional approaches either perform face frontalization on a non-frontal facial image or learn separate classifiers for each pose. Different from existing methods, in this paper, we propose an end-to-end deep learning model by exploiting different poses and expressions jointly for simultaneous facial image synthesis and pose-invariant facial expression recognition. The proposed model is based on generative adversarial network (GAN) and enjoys several merits. First, the encoder-decoder structure of the generator can learn a generative and discriminative identity representation for face images. Second, the identity representation is explicitly disentangled from both expression and pose variations through the expression and pose codes. Third, our model can automatically generate face images with different expressions under arbitrary poses to enlarge and enrich the training set for FER. Quantitative and qualitative evaluations on both controlled and in-the-wild datasets demonstrate that the proposed algorithm performs favorably against state-of-the-art methods.

The popular quiz game "Describe and Guess" can only be played by at least two players which limits its utilization. In this paper, we propose a standalone demo with a multimodal interface which is much more convenient for a single player. To deduce the target object based on user-provided attributes and sketches, we propose attributeimproved neural networks (AINN) which are joint learned for both the single-label object classification and the multilabel attribute prediction. Finally, we build a real-world system which can be effectively played on mobile devices.

Data-inefficiency greatly hinders sketch-related applications and studies. Transferring knowledge from the realimage domain to the sketch domain may eliminate this obstacle. However, a huge domain gap exists between images and sketches. To reduce the domain shift between the image and the sketch, we propose an attribute-assisted domain transfer method. By separating the shared geometric features from the private semantic features of the real images, the proposed attribute-assisted networks (ASN) can learn more effective domain-invariant features for unsupervised domain adaptation. Extensive results on the Imageto- Sketch task demonstrate the effectiveness of the proposed method.

The Visual Object Tracking challenge VOT2017 is the fifth annual tracker benchmarking activity organized by the VOT initiative. Results of 51 trackers are presented; many are state-of-the-art published at major computer vision conferences or journals in recent years. The evaluation included the standard VOT and other popular methodologies and a new "real-time" experiment simulating a situation where a tracker processes images as if provided by a continuously running sensor. Performance of the tested trackers typically by far exceeds standard baselines. The source code for most of the trackers is publicly available from the VOT page. The VOT2017 goes beyond its predecessors by (i) improving the VOT public dataset and introducing a separate VOT2017 sequestered dataset, (ii) introducing a realtime tracking experiment and (iii) releasing a redesigned toolkit that supports complex experiments. The dataset, the evaluation kit and the results are publicly available at the challenge website(1). Funding Agencies|Slovenian research agency research programs [P2-0214, P2-0094]; Slovenian research agency project [J2-8175]; Czech Science Foundation Project [GACR P103/12/G084]; WASP; VR (EMC2); SSF (SymbiCloud); SNIC; AIT Strategic Research Programme Visual Surveillance and Insight; Faculty of Computer Science, University of Ljubljana, Slovenia

In this paper, we propose a multi-task correlation particle filter (MCPF) for robust visual tracking. We first present the multi-task correlation filter (MCF) that takes the interdependencies among different features into account to learn correlation filters jointly. The proposed MCPF is designed to exploit and complement the strength of a MCF and a particle filter. Compared with existing tracking methods based on correlation filters and particle filters, the proposed tracker has several advantages. First, it can shepherd the sampled particles toward the modes of the target state distribution via the MCF, thereby resulting in robust tracking performance. Second, it can effectively handle large-scale variation via a particle sampling strategy. Third, it can effectively maintain multiple modes in the posterior density using fewer particles than conventional particle filters, thereby lowering the computational cost. Extensive experimental results on three benchmark datasets demonstrate that the proposed MCPF performs favorably against the state-of-the-art methods.

Social multimedia is the hybrid of social media and multimedia. Social multimedia mining employs data mining techniques to understand the association among content, user and interaction in social multimedia. It has three basic problems at basis, micro and macro association levels. This paper extends the current interpretation of social multimedia mining as processing multi-modal individual objects, to a generalized definition beyond modality and beyond object. Cross-OSN (Online Social Networking) data mining, an instantiation to general social multimedia mining, is then positioned with main challenges, review of current studies and future directions.

In this paper, we propose a novel structural correlation filter (SCF) model for robust visual tracking. The proposed SCF model takes part-based tracking strategies into account in a correlation filter tracker, and exploits circular shifts of all parts for their motion modeling to preserve target object structure. Compared with existing correlation filter trackers, our proposed tracker has several advantages: (1) Due to the part strategy, the learned structural correlation filters are less sensitive to partial occlusion, and have computational efficiency and robustness. (2) The learned filters are able to not only distinguish the parts from the background as the traditional correlation filters, but also exploit the intrinsic relationship among local parts via spatial constraints to preserve object structure. (3) The learned correlation filters not only make most parts share similar motion, but also tolerate outlier parts that have different motion. Both qualitative and quantitative evaluations on challenging benchmark image sequences demonstrate that the proposed SCF tracking algorithm performs favorably against several state-of-the-art methods.

Clustering on multi-type relational data has attracted increasing interest due to its great practical and theoretical importance. One of the most popular solutions is nonnegative matrix factorization. However, previous work on nonnegative matrix factorization typically copes with multi-type relations individually, and ignores the underlying semantics conveyed by the relation propagation. Additionally, these approaches may suffer from data sparsity as most of the relations between object pairs are unknown. In this paper we propose a novel Meta-Path based Nonnegative Matrix Factorization (MPNMF) framework, which enriches potentially useful similarity semantics for the improved clustering performance. We begin with constructing meta-paths, i.e., paths that connects object types via a sequence of relations, which are appropriately weighted according to certain propagation decay rules. Based on the weighted meta-paths, we are promised to characterize the strength of pairwise interactions among the objects. Together with the attributes in the bag-of-word form, we cluster the objects of target type by collective nonnegative matrix factorization. Experiments on real world datasets demonstrate the effectiveness of our method.

Sparse representation has been applied to visual tracking by finding the best target candidate with minimal reconstruction error by use of target templates. However, most sparse representation based trackers only consider holistic or local representations and do not make full use of the intrinsic structure among and inside target candidates, thereby making the representation less effective when similar objects appear or under occlusion. In this paper, we propose a novel Structural Sparse Tracking (SST) algorithm, which not only exploits the intrinsic relationship among target candidates and their local patches to learn their sparse representations jointly, but also preserves the spatial layout structure among the local patches inside each target candidate. We show that our SST algorithm accommodates most existing sparse trackers with the respective merits. Both qualitative and quantitative evaluations on challenging benchmark image sequences demonstrate that the proposed SST algorithm performs favorably against several state-of-the-art methods.

Social media contributes significantly to the arrival of the Big Data era. It is interesting to re-examine "multimedia" in the context of social media. This positioning paper proposes to analyzing into Variety of big social multimedia from the perspective of various sources, which is the study of heterogeneous data generated and consumed in various social media networks that receives little attention in the literature. Since social multimedia is essentially "user-centric", which is generated from user and customized for user, a research schema is introduced to exploit the overlapped users as bridges to address challenges and exploit social multimedia data from various sources.

Visual summarization of landmarks is an important task for applications, such as landmark organization, search and browsing. In this work, we make the first attempt towards landmark summarization by simultaneously considering both the scenes (e.g., sunny view and night view) and viewpoints (e.g., front-side and close-distant viewpoint). In the proposed framework of landmark summarization, we first group images into different clusters by viewpoints, then the distinctive scenes for each viewpoint cluster are discovered by the proposed scene-viewpoint based theme modeling. Compared with the existing topic models, our model is capable of mining scene-viewpoint themes directly from all viewpoint clusters and meanwhile differentiating among these themes by viewpoints. The landmark summary is generated by the discovered scene-viewpoint themes, where each theme is represented by the selected images with one certain scene and viewpoint. The experimental results validate the proposed method and demonstrate its advantage in improving user experience.

With the rapidly increasing popularity of Social Media sites (e.g., Flickr, YouTube, and Facebook), it is convenient for users to share their own comments on many social events, which successfully facilitates social event generation, sharing and propagation and results in a large amount of user-contributed media data (e.g., images, videos, and texts) for a wide variety of real-world events of different types and scales. As a consequence, it has become more and more difficult to find exactly the interesting events from massive social media data, which is useful to browse, search and monitor social events by users or governments. To deal with these issues, we propose a novel boosted multi-modal supervised Latent Dirichlet Allocation (BMM-SLDA) for social event classification. Our BMM-SLDA has a number of advantages. (1) It can effectively exploit the multi-modality and the supervised information of social events jointly. (2) It is suitable to large-scale data analysis by utilizing boosting weighted sampling strategy to iteratively select a small subset data to efficiently train the corresponding topic models. (3) It effectively exploits boosting document weight distribution by classification error, and can iteratively learn new topic model to correct the previously misclassified documents. We evaluate our BMM-SLDA on a real-world dataset and show extensive results, which show that our model outperforms state-of-the-art methods.

Part-based visual tracking is advantageous due to its robustness against partial occlusion. However, how to effectively exploit the confidence scores of individual parts to construct a robust tracker is still a challenging problem. In this paper, we address this problem by simultaneously matching parts in each of multiple frames, which is realized by a locality-constrained low-rank sparse learning method that establishes multi-frame part correspondences through optimization of partial permutation matrices. The proposed part matching tracker (PMT) has a number of attractive properties. (1) It exploits the spatial-temporal localityconstrained property for robust part matching. (2) It matches local parts from multiple frames jointly by considering their low-rank and sparse structure information, which can effectively handle part appearance variations due to occlusion or noise. (3) The proposed PMT model has the inbuilt mechanism of leveraging multi-mode target templates, so that the dilemma of template updating when encountering occlusion in tracking can be better handled. This contrasts with existing methods that only do part matching between a pair of frames. We evaluate PMT and compare with 10 popular state-of-the-art methods on challenging benchmarks. Experimental results show that PMT consistently outperform these existing trackers.

In this paper, we propose a low-rank sparse coding (LRSC) method that exploits local structure information among features in an image for the purpose of image-level classification. LRSC represents densely sampled SIFT descriptors, in a spatial neighborhood, collectively as low-rank, sparse linear combinations of code words. As such, it casts the feature coding problem as a low-rank matrix learning problem, which is different from previous methods that encode features independently. This LRSC has a number of attractive properties. (1) It encourages sparsity in feature codes, locality in codebook construction, and low-rankness for spatial consistency. (2) LRSC encodes local features jointly by considering their low-rank structure information, and is computationally attractive. We evaluate the LRSC by comparing its performance on a set of challenging benchmarks with that of 7 popular coding and other state-of-the-art methods. Our experiments show that by representing local features jointly, LRSC not only outperforms the state-of-the-art in classification accuracy but also improves the time complexity of methods that use a similar sparse linear representation model for feature coding.

In this paper, we propose a novel algorithm to model and recognize sign language with Kinect sensor. We assume that in a sign language video, some frames are expected to be both discriminative and representative. Under this assumption, each frame in training videos is assigned a binary latent variable indicating its discriminative capability. A Latent Support Vector Machine model is then developed to classify the signs, as well as localize the discriminative and representative frames in videos. In addition, we utilize the depth map together with color image captured by Kinect sensor to obtain more effective and accurate feature to enhance the recognition accuracy. To evaluate our approach, we collected an American Sign Language (ASL) dataset which included approximately 2000 phrases, while each phrase was captured by Kinect sensor and hence included color, depth and skeleton information. Experiments on our dataset demonstrate the effectiveness of the proposed method for sign language recognition.

Automatically localizing the image labels to the corresponding regions is a challenging but valuable task, which provides detailed semantic information for better image understanding and image retrieval. In this paper, we propose a novel appearance guided graph inferring (AGI) framework for label localization. The framework iterates with two stages: graph inferring and appearance learning. Given the image set, each image is over-segmented into a bag of small patches. In the first step, we adopt graph propagation based method to infer the patch labels collaboratively on the whole image set. A multi-cue graph is constructed for more consistent spatial layout and image label constraints are imposed in propagation. In the second step, SVM classifiers are trained as appearance models by gradually exploiting the inferring results. And then the patch labels are reevaluated by the learned appearance model and feedback to the first step. The global graph propagation and local appearance model complement each other by iteration. Extensive experiments on three public datasets, MSRC-v1, MSRC-v2 and SAIAPR TC-12, demonstrate the encouraging performance of our method in comparison with other baselines.

With the rising of internet photos-sharing web sites, the rich aware text information surrounding images on the sites are proved helpful to improve the image classification. This paper presents a novel nested deep learning model called Nested Deep Belief Network(NDBN) for tag-aware image classification. A multi-layer structure of Deep Belief Network(DBN) is established to learn a unified representation of visual feature and tag feature for an image, and an additional Gaussian Restricted Boltzmann Machine is built to capture the tag-tag dependency. Compared with conventional methods, the proposed model can not only find correlations across modalities, but mine the importance for different tags, and also bring about low-rank tag feature representation. We conduct experiments over the MIR Flickr dataset and the results show that the proposed NDBN model outperforms the existing image classification techniques.

Online propagation of videos has surged up to an unparalleled level. Most personalized video recommendation methods are based on single-platform user modeling, which suffer from data sparsity and cold-start issues. In this paper, we introduce cross-platform user modeling as a solution by smartly aggregating user information from different platforms. Unlike traditional recommendation methods where sufficient user information is assumed available in the target platform, this proposed method works well when there is little knowledge about users' interests in the target platform. While considering the difference of user behaviors in different platforms, on one hand, we enrich user profile in the target platform with related information in the auxiliary platform. On the other hand, we transfer the collaborative relationship defined in behaviors from the auxiliary platform to the target platform. Carefully designed experiments have demonstrated the effectiveness of the proposed method.

The emergence of various and disparate social media platforms has opened opportunities for the research on cross-platform media analysis. This provides huge potentials to solve many challenging problems which cannot be well explored in one single platform. In this paper, we investigate into cross-platform social relation and behavior information to address the cold-start friend recommendation problem. In particular, we conduct an in-depth data analysis to examine what information can better transfer from one platform to another and the result demonstrates a strong correlation for the bidirectional relation and common contact behavior between our test platforms. Inspired by the observations, we design a random walk-based method to employ and integrate these convinced social information to boost friend recommendation performance. To validate the effectiveness of our cross-platform social transfer learning, we have collected a cross-platform dataset including 3,000 users with recognized accounts in both Flickr and Twitter. We demonstrate the effectiveness of the proposed friend transfer methods by promising results.

Properly utilizing the spatial correlation of regions benefits for improving the performance of label localization task. However, we could not obtain this information directly since we do not have the region level ground truth. In this paper, we propose a weakly spatial constrained graph propagation by mining the spatial correlation from unlabeled regions and integrating it into the graph propagation framework. Our main framework contains two steps: the spatial constrained graph (SCG) construction and label propagation. Firstly, images are over-segmented and each patch is considered as a node. We deem the relatively stable patch combination as a spatial context to construct the SCG, and encourage label propagations where those patches are visually similar as well as spatially consistent. In the second step, we add the dissimilarity constraints and image level label constraints to the label propagation. The propagation procedure is formulated as a constrained optimization problem and it can be efficiently solved by an iteration method. Experiments on three benchmark datasets demonstrate that the spatial correlation mined by our method is effective to the label localization task.

Sparse representation based methods have recently drawn much attention in visual tracking due to good performance against illumination variation and occlusion. They assume the errors caused by image variations can be modeled as pixel-wise sparse. However, in many practical scenarios these errors are not truly pixel-wise sparse but rather sparsely distributed in a structured way. In fact, pixels in error constitute contiguous regions within the object's track. This is the case when significant occlusion occurs. To accommodate for non-sparse occlusion in a given frame, we assume that occlusion detected in previous frames can be propagated to the current one. This propagated information determines which pixels will contribute to the sparse representation of the current track. In other words, pixels that were detected as part of an occlusion in the previous frame will be removed from the target representation process. As such, this paper proposes a novel tracking algorithm that models and detects occlusion through structured sparse learning. We test our tracker on challenging benchmark sequences, such as sports videos, which involve heavy occlusion, drastic illumination changes, and large pose variations. Experimental results show that our tracker consistently outperforms the state-of-the-art.

The Bag-of-Features (BOF) model is widely used for image classification. Most BOF models incorporate a step of maximum pooling to generate the raw image representation, where salient atoms with maximum response are reserved for final representation. However, recent locality-preserving coding schemes do not account for the saliency characteristic during the process of generating the raw image representations. In this paper, we propose a saliency aware locality-preserving coding scheme by explicitly considering saliency into the dictionary creation and feature coding stages. The novel coding scheme guarantees strong response in the pooling operation and thus contributes to a discriminative image representation. Experiments on three benchmark datasets validate the effectiveness of the proposed method.

The bag-of-visual-words (BoW) representation has received wide application and public acceptance for visual categorization. However, the histogram based image representation ignores the spatial information and correlations among visual words. To tackle these problems, in this paper, we propose to use some image regions called ‘components’, as the higher-level visual elements to represent an image associating with the lower-level elements of ‘visual words’. Then we formulate the task of visual categorization into two progressive relationships among a given concept and the two-level visual elements of images, i.e., visual-words-to-components and components-to-concept. Firstly, component level linear SVM classifiers are learned to model the relationship between visual words and components, then the output of these SVM classifiers are linearly combined to model the relationships between components and concept. Experiments on the Scene-15 dataset and the Oxford Flowers dataset demonstrate the effectiveness of the proposed method.

In large-scale video database, lots of different videos frequently share the similar content copied from the same source. Generally, those videos have certain semantic correlations, such as being of similar events and sharing the same topic. Mining these semantic correlations can greatly facilitate video search. However, as a preprocessing step, detecting and localizing the copy pair among videos, i.e. copy detection problem, plays a key role for precise semantic mining. To meet the requirements in semantic mining scenario, we propose a frame fusion based copy detection scheme. In this scheme, the copy detection problem is converted to HMM decoding problem with three relaxed constraints, where Viterbi algorithm is employed to automatically detect the copy pair. The experimental results show that the proposed approach achieves high localization accuracy even when the copied clip undergoes some complex transformations, while achieving comparable performance compared with state-of-the-art copy detection methods.

Movie shot classification is vital but challenging task due to various movie genres, different movie shooting techniques and much more shot types than other video domain. Variety of shot types are used in movies in order to attract audiences attention and enhance their watching experience. In this paper, we introduce context saliency to measure visual attention distributed in keyframes for movie shot classification. Different from traditional saliency maps, context saliency map is generated by removing redundancy from contrast saliency and incorporating geometry constrains. Context saliency is later combined with color and texture features to generate feature vectors. Support Vector Machine (SVM) is used to classify keyframes into pre-defined shot classes. Different from the existing works of either performing in a certain movie genre or classifying movie shot into limited directing semantic classes, the proposed method has three unique features: 1) context saliency significantly improves movie shot classification; 2) our method works for all movie genres; 3) our method deals with the most common types of video shots in movies. The experimental results indicate that the proposed method is effective and efficient for movie shot classification. © 2011 IEEE.

In the conventional bag of visual words (BoW) based image representation, single visual word is not discriminative enough and the spatial contextual information among local image features is ignored. In this paper, descriptive local feature groups are proposed to address these two problems. First, local image features are refined by slightly transforming the original image. Then they are clustered and represented by visual words. Second, the candidate local feature groups are generated by searching the neighbors of every local image features. This kind of grouping shows more discriminative power than a single feature and the local spatial contexts can be catched. Third, we obtain the groups more descriptive to the object category by defining a significance score and the groups with high score are selected. Finally, the high order descriptive local feature groups are integrated to the vector based object categorization framework by a feature reweighting strategy. Experimental results on Scene-15 and Caltech 101 demonstrate the superior performance of our method.

Automatic face identification of characters in movies has drawn significant research interests and led to various applications. It is a challenging problem due to the huge variation in the appearance of each character. Although existing methods demonstrate promising results in clean environment, the performances are limited in complex movie scenes due to the noises generated during the face tracking and face clustering process. In this paper we present a robust character identification approach by incorporating a noise insensitive relationship representation and a graph matching algorithm. Beyond existing character identification approaches, we further perform explicit sensitivity analysis on character identification by introducing two types of simulated noises. Experiments validate the advantage of the proposed method.

We propose an image classification framework by leveraging the non-negative sparse coding, low-rank and sparse matrix decomposition techniques (LR-Sc+ SPM). First, we propose a new non-negative sparse coding along with max pooling and spatial pyramid matching method (Sc+ SPM) to extract local features' information in order to represent images, where non-negative sparse coding is used to encode local features. Max pooling along with spatial pyramid matching (SPM) is then utilized to get the feature vectors to represent images. Second, motivated by the observation that images of the same class often contain correlated (or common) items and specific (or noisy) items, we propose to leverage the low-rank and sparse matrix recovery technique to decompose the feature vectors of images per class into a low-rank matrix and a sparse error matrix. To incorporate the common and specific attributes into the image representation, we still adopt the idea of sparse coding to recode the Sc+ SPM representation of each image. In particular, we collect the columns of the both matrixes as the bases and use the coding parameters as the updated image representation by learning them through the locality-constrained linear coding (LLC). Finally, linear SVM classifier is leveraged for the final classification. Experimental results show that the proposed method achieves or outperforms the state-of-the-art results on several benchmarks.