Search

The vast amount of user-generated data in various and disparate social media sites contains rich and diverse information about what is happening around the world. Digging into such user-generated data distributed in different social media sites helps us better understand what people are interested in and how they feel about certain topics. In this paper, we investigate into users' behavior data in Twitter and YouTube to figure out whether people's attention on certain topics has some sort of temporal order between Twitter and YouTube on user level. We collected a real world dataset of 8,518 users with account associations between Twitter and YouTube as well as all their behavior data with timestamp since Jan. 2012. The results demonstrate that more users tend to get access to certain events earlier in Twitter than in YouTube and the ratio is somewhat topic-sensitive.

Most existing image classification methods focus on handling images with only “object” concepts. At the same time, in real-world cases, there exists a great variety of images which contain “verb-object” concepts, rather than only “object” ones. The hierarchical structure embedded in these “verb-object” concepts can help to enhance classification. However, traditional feature representing methods cannot utilize it. To tackle this defect, we present in this paper a novel approach, called Hierarchical Nonnegative Graph Embedding (HNGE). By assuming that those “verb-object” concept images which share the same “object” part but different “verb” part have a specific hierarchical structure, we make use of this hierarchical structure and employ an effective technique, named nonnegative graph embedding, to perform feature extraction as well as image classification. Extensive experiments compared with the state-of-the-art algorithms on nonnegative data factorization demonstrate the feasibility, convergency and classification power of proposed approach on “verb-object” concept images classification.

The prevalence of digital photo capturing devices has gen- erated large-scale photos with geographical information, leading to in- teresting tasks like geographically organizing photos and location visual- ization. In this work, we propose to organize photos both geographically and thematically, and investigate the problem of location visualization from multiple themes. The novel visualization scheme provides a rich dis- play landscape for location exploration from all-round views. A two-level solution is presented, where we first identify the highly photographed places (POI) and discover their distributed themes, and then aggregate the lower-level themes to generate the higher-level themes for location visualization. We have conducted experiments on a Flickr dataset and ex- hibited the visualization for the Singapore city. The experimental results have validated the proposed method and demonstrated the potentials of location visualization from multiple themes.

Landmark image mining and detection have been studied for many years, however, most of the existing work focuses on their spatial attributes, while largely ignoring the temporal information in specific ones, which are taken during historical moments. This kind of images are more valuable than the normal ones as they not only contain more comprehensive information to illustrate the landmarks in different moments, but also are useful in many real world applications, such as tour recommendation and history education. In this paper, we present a novel framework named Landmark History Visualization (LHV) to mine relevant and diverse images for each landmark’s historic moments. There are two steps in LHV. The first one is to extract the event list of each landmark from Wikipedia. The event keywords are extracted, and some of them are automatically labeled as 3W (What, Who, When). In the second step, images searched by the landmark name are firstly collected from Flickr and Google images. Secondly, we employ manifold ranking with detected 3W to retrieve the relevant images, and lastly, an outlier detection and diversification based re-ranking approach is introduced to provide users with various returned images. We implemented our approach on 6 landmarks and the results demonstrate the effectiveness of LHV.

In relevance feedback of content-based image retrieval, active learning is an effective method to alleviate the burden of labeling by selecting the most informative examples to label. Traditionally, batch mode is adopted in selective sampling, which suffers from two problems. On the one hand, the existing classification boundary which dominates example selection is usually unreliable for lack of labeled examples; on the other hand, the previously labeled examples in the batch cannot offer instructive information for further selection. In this paper, we propose a novel dynamic batch sampling mode for SVM active learning which addresses the above problems. We select a batch of examples dynamically, using the previously labeled examples as guidance. Experimental results demonstrate the advantage of the proposed scheme in comparison with the traditional ones.

The fast development of Internet and personal electronic devices has led to a dramatic growth of landmark images available on social media sites such as Flickr. With their informative and attractive content, landmark images have received considerable attention in multimedia research community. Most of existing methods focus on exploring landmark images by their spatial distribution, while largely ignoring the temporal aspect. Through digging the temporal information in the shared images, we can reveal the series of historical moments of landmarks, which could be useful for social studies and attractive to tourists. In this paper, we propose a scheme named Landmark Timeline Construction(LTC), which automatically mines and selects diverse images on historical events of a landmark simply from Flickr. In our scheme, Time Interval Selection Algorithm is proposed to handle the limitation of the existing methods on the fixed time interval by assigning the optimal ones to different tags. The experimental results demonstrate the effectiveness of our methods.

Online videos are becoming popular these days. Personalized search has been recognized as effective solution for user accessing desired information when facing a daunting volume of videos. Personalized query understanding serves as one of the most challenges in personalized search, which indicates that unique query has distributed meanings and produce different semantics for different users. Take query of celebrity as example, many celebrities are engaged in multiple fields and certain user may be just interested in the field of videos related to his/her own preference. In this paper, we address the challenge of personalized query understanding by focusing on the problem of personalized celebrity video search. An interest-popularity cross-space mining based method is proposed for solution. Specifically, celebrity popularity and user interest distributions are first learned by topic modeling from heterogeneous data of expert knowledge and user online activities, respectively. We then exploit topic-word distribution refinement to correlate the two heterogeneous topic spaces. Finally the candidate videos are re-ranked based on the derived interest-popularity correlations. Carefully designed experiments have demonstrated the effectiveness of the proposed method. The obtained ranking list is highly consistent with the test users' preferences.

The ignorance on spatial information and semantics of visual words becomes main obstacles in the bag-of-visual-words (BoW) method for image classification. To address the obstacles, we present an improved BoW representation using spatial pyramid coding (SPC) and visual word reweighting. In SPC procedure, we adopt the sparse coding technique to encode visual features with the spatial constraint. Visual features from the same spatial sub-region of images are collected to generate the visual vocabulary. Additionally, a relaxed but simple solution for semantic embedding into visual words is proposed. We relax the semantic embedding from ideal semantic correspondence to naive semantic purity of visual words, and reweight each visual word according to its semantic purity. Higher weights are given to semantically distinctive visual words, and lower weights to semantically general ones. Experiments on a public dataset demonstrate the effectiveness of the proposed method.

Can you imagine that two people who have different native languages and cannot understand other's language are able to communicate with each other without professional interpreter? In this paper, a visualized communication system is designed to facilitate such people chatting with each other via visual information. Differing from the online instant message tools such as MSN, Google talk and ICQ, which are mostly based on textual information, the visualized communication system resorts to the vivid images which are relevant to the conversation context aside from text to jump the language obstacle. The multi-phase visual concept detection strategy is applied to associate the text with the corresponding web images. Then, a re-ranking algorithm attempts to return the most related and highest quality images at top positions. In addition, sentiment analysis is performed to help people understand the emotion of each other to further reduce the language obstacle. A number of daily conversation scenes are implemented in the experiments and the performance is evaluated by user study. The experimental results show that the visualized communication system is able to effectively help people with language obstacle to better understand each other.

In this paper, we have designed and implement a mobile personalized sports video customization system, which aims to provide mobile users with interesting video clips according to their personalized preferences. With the B/S architecture, the whole system includes an intelligent multimedia content server and a client interface on smart phones. For the content server, the web casting text is utilized to detect live events from sports video, which can generate both accurate event location and rich content description. The annotation results are stored in the MPEG-7 format and then the server can provide personalized video retrieval and summarization services based on both game content and user preference. For the client interface, a friend UI is designed for mobile users to customize their favorite video clips. With a new ‘4C' evaluation criterion, our proposed system is proved effective by both quantitative and qualitative experiments conducted on five sports matches.

This demo presents a web image search engine via learning semantics of query image. Unlike traditional CBIR systems which search images according to visual similarities, our system implements an extended CBIR (ExCBIR) which returns both visually and semantically relevant images. Given a query image, we first automatically learn its semantic representation from those visual similar images, and then combine the semantic representation and their visual properties to output the searching result. Considering that different visual features have variously discriminative power under a certain semantic context, we give more confidence to the feature whose result images are more consistent on semantics. Experiments on a large-scale web images demonstrate the effectiveness of our system.

In image annotation, the annotation words are expected to represent image content at both visual level and semantic level. However, a single word sometimes is ambiguous in annotation, for example, "apple" may refer to a fruit or a company. However, when "apple" combines with "phone" or "fruit", it will be more semantically and visually consistent. In this paper, we attempt to find this kind of combination and construct a less ambiguous phrase-level lexicon for annotation. First, concept-based image search is conducted to obtain a semantically consistent image set (SC-IS). Then, a hierarchical clustering algorithm is adopted to visually cluster the images in SC-IS to obtain a semantically and visually specific image set (SVC-IS). Finally, we apply a frequent itemset mining in SVC-IS to construct the phrase-level lexicon and associate the lexicon into a probabilistic annotation framework to estimate annotation words of any untagged images. Our experimental results show that the discovered phrase-level lexicon is able to improve the annotation performance.

Text facilitated sports video analysis has achieved extensive success in video indexing, retrieval and summarization. A commonly adopted basis in previous work is the separate alignment of timestamps between sports video and game text, which isn't a robust method for generic cross-media analysis. In this paper, we propose a hierarchical semantics-matching approach to annotate sports video. Our key idea is to link video and text with high-level semantics rather than low-level features and find the optimal video-text alignment based on the integral structure rather than individual conditions. For accurate event location, the whole algorithm is implemented in a hierarchical way to generate both refined and accurate video annotation result. Experiments conducted on both basketball and football matches demonstrate that our proposed approach is effective for text facilitated sports video annotation.

Semantic scene segmentation is a crucial step in movie video analysis and extensive research efforts have been devoted to this area. However, previous methods are heavily relying on video content itself, which are lack of objective evaluation criterion and necessary semantic link due to the semantic gap. In this paper, we propose a novel role-based approach for movie scene segmentation using script. Script is a text description of movie content that contains the scene structure information and related character names, which can be regarded as an objective evaluation criterion and useful external reference. The main novelty of our approach is that we convert the movie scene segmentation into a movie-script alignment problem and propose a HMM alignment algorithm to map the script scene structure to the movie content. The promising results obtained from three Hollywood movies demonstrate the effectiveness of our proposed approach.

In this paper we report our work using visual feature fusion for the tasks of medical image retrieval and annotation in the benchmark of ImageCLEF 2005. In the retrieval task, we use visual features without text information, having no relevance feedback. Both local and global features in terms of both structural and statistical nature are captured. We first identify visually similar images manually and form templates for each query topic. A pre-filtering process is utilized for a coarse retrieval. In the fine retrieval, two similarity measuring channels with different visual features are used in parallel and then combined in the decision level to produce a final score for image ranking. Our approach is evaluated over all 25 query topics with each containing example image(s) and topic textual statements. Over 50,000 images we achieved a mean average precision of 14.6%, as one of the best performed runs. In the annotation task, visual features are fused in an early stage by concatenation with normalization. We use support vector machines (SVM) with RBF kernels for the classification. Our approach is trained over a 9,000 image training set and tested over the given test set with 1000 images and on 57 classes with a correct classification rate of about 80%.

TV logo detection, tracking and removal play an important role in the applications of claiming video content ownership, logo-based broadcasting surveillance, commercial skipping, and program rebroadcasting with new logos. In this paper, we present a novel and robust framework using tensor method for these three tasks. First, we use tensor based generalized gradient and the OTSU binarization algorithm to logo detection, and propose a two level framework from coarse to fine to tracking the TV logos. Finally, we extend the regularization PDEs by incorporation of temporal information to inpaint the logo region. Due to the introduction of the structure tensor, the generalized gradient based method can detect the logo region by tracking the change rate of pixels in spatio-temporal domain, and the region of logo removal is well filled in a structure-preserving way. Since temporal correlation of multiple consecutive frames is considered, the proposed method can deal with opaque, semi-transparent, and animated logos. The experiments and comparison with previous methods are conducted on the part of TRECVID 2005 news corpus and several Chinese TV channels with challenging TV logos, and the experimental results are promising.

Motion analysis in broadcast sports video is a challenging problem especially for player action recognition due to the low resolution of players in the frames. In this paper, we present a novel approach to recognize the basic player actions in broadcast tennis video where the player is about 30 pixels tall. Two research challenges, motion representation and action recognition, are addressed. A new motion descriptor, which is a group of histograms based on optical flow, is proposed for motion representation. The optical flow here is treated as spatial pattern of noisy measurement instead of precise pixel displacement. To recognize the action performed by the player, support vector machine is employed to train the classifier where the concatenation of histograms is formed as the input features. Experimental results demonstrate that our method is promising by integrating with the framework of multimodal analysis in sports video.

Query by video clip (QVC) has attracted wide research interests in multimedia information retrieval. In general, QVC may include feature extraction, similarity measure, database organization, and search or query scheme. Towards an effective and efficient solution, diverse applications have different considerations and challenges on the abovementioned phases. In this paper, we firstly attempt to broadly categorize most existing QVC work into 3 levels: concept based video retrieval, video title identification, and video copy detection. This 3-level categorization is expected to explicitly identify typical applications, robust requirements, likely features, and main challenges existing between mature techniques and hard performance requirements. A brief survey is presented to concretize the QVC categorization. Under this categorization, in this paper we focus on the copy detection task, wherein the challenges are mainly due to the design of compact and robust low level features (i.e. an effective signature) and a kind of fast searching mechanism. In order to effectively and robustly characterize the video segments of variable lengths, we design a novel global visual feature (a fixed-size 144-d signature) combining the spatial-temporal and the color range information. Different from previous key frame based shot representation, the ambiguity of key frame selection and the difficulty of detecting gradual shot transition could be avoided. Experiments have shown the signature is also insensitive to color shifting and variations from video compression. As our feature can be extracted directly from MPEG compressed domain, lower computational cost is required. In terms of fast searching, we employ the active search algorithm. Combining the proposed signature and the active search, we have achieved an efficient and robust solution for video copy detection. For example, we can search for a short video clip among the 10.5 hours MPEG-1 video database in merely 2 seconds in the case of unknown query length, and in 0.011 second when fixing the query length as 10 seconds.

With the exponential growth in the production creation of multimedia data, there is an increasing need for video semantic analysis. Audio, as a significant part of video, provides important cues to human perception when humans are browsing and understanding video contents. To detect semantic content by useful audio information, we introduce audio keywords which are sets of specific audio sounds related to semantic events. In our previous work, we designed a hierarchical Support Vector Machine (SVM) classifier for audio keyword identification. However, a weakness of our previous work is that audio signals are artificially segmented into 20 ms frames for frame-based SVM identification without any contextual information. In this paper, we propose a classification method based on Hidden Markov Modal (HMM) for audio keyword identification as an improved work instead of using hierarchical SVM classifier. Choosing HMM is motivated by the successful story of HMM in speech recognition. Unlike the frame-based SVM classification followed by major voting, our proposed HMM-based classifiers treat specific sound as a continuous time series data and employ hidden states transition to capture context information. In particular, we study how to find an effective HMM, i.e., determining topology, observation vectors and statistical parameters of HMM. We also compare different HMM structures with different hidden states, and adjust time series data with variable length. Experimental data includes 40 minutes basketball au-dio which comes from real-time sports games. Experimental results show that, for audio keyword generation, the proposed HMM-based method outperforms the previous hierarchical SVM.

Video processing has found many applications in sports such as slow motion replay, pattern analysis, statistics collection, video archiving, etc. This paper presents an automatic archiving system for tennis games. It extracts ball trajectory using calibrated cameras. The trajectory is then used to classify tennis video clips into 58 tactic patterns. Semantic annotation is then attached to each clip so future query can be made easily. The annotated video clips can be used for training, sports analysis, broadcasting, etc. It can also be used for online browsing and streaming.

We propose a novel approach to detect semantic regions (pure vocals, pure instrumental and instrumental mixed vocals) in acoustic music signals. The acoustic music signal is first segmented at the beat level based on our proposed rhythm tracking algorithm. Then for each segment Cepstral coefficients are extracted from the Octave Scale to characterize music content. Finally, a hierarchical classification method is proposed to detect semantic regions. Different from previous methods, our proposed approach fully considers the music knowledge in segmenting and detecting the semantic regions in music signals. Experimental results illustrate that over 80% accuracy is achieved for semantic region detection.

In this paper, we propose a novel method to implant virtual advertisements into broadcast soccer video without disturbing audience's view experience. The salient objects are first detected from broadcast soccer video. These objects include static regions, central ellipse, goal mouth, and field boundary. Based on these objects, we identify suitable locations in the video for virtual advertisement implantation. The advertisement implantation can be done in real time. The proposed method allows a non-intrusive means to incorporate additional virtual content into a video presentation, facilitating an additional channel of communications to enhance greater video interactivity. It also creates a new business model for broadcast and advertising industry.

We describe novel methods to automatically augment content into video highlights detected from soccer and tennis video. First, we extract generic and domain-specific features from the video to isolate key audio-visual events that we have empirically found to correlate well with the ground-truth highlights. Next, based on a set of heuristics-driven rules to minimize view disruption, spatial regions in the image frames of these video highlight segments are segmented for content augmentation. Preliminary trials from subjective viewing indicate a high level of acceptance for the content insertions.

In this chapter, we summarize the research achievements in the area of content-based audio and music retrieval. This chapter covers the research aspects of audio feature extraction, generic audio classification and retrieval, music content analysis, and content-based music retrieval, providing an overview of current research in the area. In addition, two typical systems for content-based audio and music retrieval are discussed in detail. Finally, based on the current technology used in content-based audio/ music retrieval and the demand from real-world applications, future promising directions are identified.

One important field in the research of computer music concerns the modeling of sounds. In order to design computational models mirroring as closely as possible a real sound and permitting in addition transformation by altering the synthesis parameters, we look for a signal model based on additive synthesis, whose parameters are estimated by the analysis of real sound. In this paper we present model-based analysis of musical notes generated by the electric guitar. Both time domain and frequency domain feature analysis experiments have been performed to select the appropriate parameters and the features for the musical signal analysis. Finally, non-parametric classification technique i.e. Nearest Neighbor Rule has been utilized to classify musical notes with this best set of parameters of the musical features.

This paper presents an object-based stream authentication scheme for practical end-to-end applications on present network infrastructure and protocols. In this proposal, a stream is divided into objects that are delivered sequentially. Each object consists of a unique identifier, content and operations. At the sender side, a locked object identifier is produced by encrypting the object identifier with a key generated from MAC of the object content. The object content and locked identifier are transmitted to the receiver. The receiver can recover the object identifier and check it to authenticate the object. The scheme tolerates packet loss, and incurs no additional overhead if an identifier is ingeniously chosen. In the experiment, we overload some particular fields in the headers of standard protocols as object identifier to implement a real time video stream authentication scheme.

In this paper, we propose an effective algorithm to automatically identify and discriminate music content. Linear prediction coefficients, zero crossing rates and mel-frequency cepstral coefficients are calculated to characterize music content. Based on calculated features, support vector machines are applied to obtain the optimal class boundaries between vocal music and pure music by learning from training data. Experimental results of support vector machine learning show good performance in music discrimination and are more advantageous than traditional Euclidean distance based method.

There have been several query-by-humming techniques developed for music retrivel. The techniques either are error-prone due to the inaccuracy of the hummed query or force the users to hum according to a metronome.This paper presesnts a new slope-based query-by-humming technique, in which the retrivel is robust to the inaccuracy in query and the use of metronome is eliminated. A pitch tracking method is developed to construct the melody curve from a user's humming. The method is robust to the noise in the humming. users are even allowed to sing the melody with lyrics. we use melody curve to represent the melodies of the original songs and the hummed query. And curve features like melody slope pitch range, time duration and note changes in the slopes are extrated from the melody curves. A melody slope matching method has been developed to do music retrivel, in which the melody slopes from a query is matched with the melody slopes for the original songs in the database. Results have shown the pitch tracking method and slope-based feature matching algoriyhms are robust to humming inaccuracy.

In this paper, we describe a 2D and a 3D practical visual processing system for facial prediction in surgical correction of dento-maxillafacial deformities. For 2D system, we use adaptive filtering and edge detecting techniques and some artificial intelligent approaches to locate landmarks and make diagnostic decision on facial cephalogram and photo. For 3D system, we employ laser active triangulation principle to acquire range data of human face and reconstruct 3D high quality face image. The 2D system can realize the automation from landmark location to parameter measurement on cephalogram while the 3D system can make the diagnostic design of preoperation of orthognathic surgery more visual and accurate.