Search

This study explores a simple but strong baseline for person re-identification (ReID). Person ReID with deep neural networks has progressed and achieved high performance in recent years. However, many state-of-the-art methods design complex network structures and concatenate multi-branch features. In the literature, some effective training tricks briefly appear in several papers or source codes. The present study collects and evaluates these effective training tricks in person ReID. By combining these tricks, the model achieves 94.5% rank-1 and 85.9% mean average precision on Market1501 with only using the global features of ResNet50. The performance surpasses all existing global- and part-based baselines in person ReID. We propose a novel neck structure named as batch normalization neck (BNNeck). BNNeck adds a batch normalization layer after global pooling layer to separate metric and classification losses into two different feature spaces because we observe they are inconsistent in one embedding space. Extended experiments show that BNNeck can boost the baseline, and our baseline can improve the performance of existing state-of-the-art methods. Our codes and models are available at: https://github.com/michuanhaohao/reid-strong-baseline., Comment: Accepted by IEEE Transactions on Multimedia. This is the submitted journal version of the oral paper [arXiv:1903.07071] in CVPRW'19. Code are avaliable at: https://github.com/michuanhaohao/reid-strong-baseline

The attractiveness of a property is one of the most interesting, yet challenging, categories to model. Image characteristics are used to describe certain attributes, and to examine the influence of visual factors on the price or timeframe of the listing. In this paper, we propose a set of techniques for the extraction of visual features for efficient numerical inclusion in modern-day predictive algorithms. We discuss techniques such as Shannon's entropy, calculating the center of gravity, employing image segmentation, and using Convolutional Neural Networks. After comparing these techniques as applied to a set of property-related images (indoor, outdoor, and satellite), we conclude the following: (i) the entropy is the most efficient single-digit visual measure for housing price prediction; (ii) image segmentation is the most important visual feature for the prediction of housing lifespan; and (iii) deep image features can be used to quantify interior characteristics and contribute to captivation modeling. The set of 40 image features selected here carries a significant amount of predictive power and outperforms some of the strongest metadata predictors. Without any need to replace a human expert in a real-estate appraisal process, we conclude that the techniques presented in this paper can efficiently describe visible characteristics, thus introducing perceived attractiveness as a quantitative measure into the predictive modeling of housing.

This paper presents a system and methodology for the analysis of quality of experience factors for dense multiview (MV) video using a head-mounted device (HMD). An MV-HMD player has been designed and implemented to immerse the users in a virtual environment, where they are placed in front of a virtual lightfield display that shows a different viewpoint depending on the position of their head. This paper describes a methodology for the analysis of the subjective perception of the transition among views (motion parallax), which is specific to the visualization of MV content. While previous works simulated the user movement by predefined view paths or used complex devices to track them, this system allows the observer to move freely, varying the perspective of the scene while easily tracking the observer's position. This paper is, up to our knowledge, the first providing a complete framework for the assessment of this subjective factor using an HMD. The subjective results obtained using this framework are used to 1) assess the influence of the user movement, display settings, and content characteristics in the perception of smoothness in the view transition, and 2) analyze the performance and limitations of a prediction model for subjective smoothness scores.

This paper focuses on semantic task planning, that is, predicting a sequence of actions toward accomplishing a specific task under a certain scene, which is a new problem in computer vision research. The primary challenges are how to model the task-specific knowledge and how to integrate this knowledge into the learning procedure. In this paper, we propose training a recurrent long short-term memory (LSTM) network to address this problem, that is, taking a scene image (including prelocated objects) and the specified task as input and recurrently predicting action sequences. However, training such a network generally requires large numbers of annotated samples to cover the semantic space (e.g., diverse action decomposition and ordering). To overcome this issue, we introduce a knowledge and – or graph (AOG) for task description, which hierarchically represents a task as atomic actions. With this AOG representation, we can produce many valid samples (i.e., action sequences according to common sense) by training another auxiliary LSTM network with a small set of annotated samples. Furthermore, these generated samples (i.e., task-oriented action sequences) effectively facilitate training of the model for semantic task planning. In our experiments, we create a new dataset that contains diverse daily tasks and extensively evaluates the effectiveness of our approach.

The interactive robot is expected to support emotion analysis and utilize the deep learning and machine learning to provide users with continuous emotional care. However, it is a great challenge to securely acquire sufficient data for emotion analysis such that the privacy of emotional data is adequately protected. To address the security issue, this paper proposes a security policy based on identity authentication and access control to ensure the security certificate through an interactive robot or edge devices while the access control of private data stored in the edge cloud is adequately protected. Specifically, this paper adopts a polynomial-based access control policy and designs a secure and effective access control scheme. At the same time, this paper puts forward an identity authentication mechanism in view of edge cloud systems, which can reduce the computational overhead and authentication delay in a collaborative authentication of multiple edge clouds. The effectiveness of the proposed access control policy and identity authentication mechanism is verified by an actual testbed platform.

Pedestrian detection has achieved great improve-ments in recent years, while complex occlusion handling and high-accurate localization are still the most important problems. To take advantage of the body part semantic information and the contextual information for pedestrian detection, we propose the part and context network (PCN) in this paper. A PCN is composed of three branches: the basic branch; the part branch; and the context branch. It specially utilizes two branches to detect the pedestrians through the body part semantic information and the contextual information, respectively. In the part branch, the semantic information of body parts can communicate with each other via long short-term memory (LSTM). In the context branch, we adopt a local competition mechanism (maxout) for adaptive context scale selection. By combining the outputs of all branches, we develop a strong complementary pedestrian detector with a lower miss rate and higher localization accuracy, especially for the occlusion pedestrian. The combination of the body part semantic information and the contextual information in pedestrian detection is fully explored in this paper. Comprehensive evaluations on three challenging pedestrian detection datasets (i.e., Caltech, INRIA and KITTI) well demonstrate the effectiveness of our proposed PCN. Code for PCN is publicly available on GitHub https://github.com/sunnyxiaohu/pcn_pedestrian .

© 1999-2012 IEEE. It is desired (especially for young people) to shop for the same or similar products shown in the multimedia contents (such as online TV programs). This indicates an urgent demand for improving the experience of TV-to-Online (T2O). In this paper, a transfer learning approach as well as a prototype system for effortless T2O experience is developed. In the system, a key component is high-precision product search, which is to fulfill exact matching between a query item and the database ones. The matching performance primarily relies on distance estimation, but the data characteristics cannot be well modeled and exploited by a simple Euclidean distance. This motivates us to introduce distance metric learning (DML) for improving the distance estimation. However, in traditional DML methods, the side information (such as the similar/dissimilar constraints or relevance/irrelevance judgements) in the target domain is leveraged. These methods may fail due to limited side information. Fortunately, this issue can be alleviated by utilizing transfer metric learning (TML) to exploit information from other related domains. In this paper, a novel manifold regularized heterogeneous multitask metric learning framework is proposed, in which each domain is treated equally. The proposed approach allows us to simultaneously exploit the information from other domains and the unlabeled information. Furthermore, the ranking-based loss is adopted to make our model more appropriate for search. Experiments on two challenging real-world datasets demonstrate the effectiveness of the proposed method. This TML approach is expected to impact the transformation of the emerging T2O trend in both TV and online video domains.

Different views represent different aspects of images. It is more effective to combine them for visual classifications. This paper proposes a novel multiview label sharing method to combine the discriminative power of different views for classifications. Especially, we linearly transfer different views into a shared space for representations. The inter-view similarities are kept in the shared space for each view. We also ensure the intra-view similarities of the same class between different views are preserved in the shared space. We jointly learn the classifiers and transformation matrices by minimizing the summed classification loss along with the inter-view and intra-view similarity constraints. In this paper, the inter-view constraints refer to the similarities between images of the corresponding view, whereas the intra-view constraints refer to the similarities between different views of images with the same semantics. Experimental results and analysis on several public datasets show the effectiveness of the proposed multiview label sharing method for visual classifications.

Every day, vast amounts of data are uploaded to various social-sharing websites. Each social-sharing website has its own media dataset. Recently, mining media datasets has shown great potential for our daily lives, e.g., earthquake detection. Generally, different datasets have different characteristics. Combining different datasets is capable of achieving better performance than using any dataset independently, particularly if the datasets can compensate for each other. The resulting performance, however, depends on the fusion method. Effectively combining different datasets is challenging. As a solution to this challenge, this paper presents a generic two-stage framework for events of interest. Specifically, the first stage normalizes the contents of different datasets to make them comparable; then, the second stage combines the normalized contents for a ranked event list using graph-based algorithms. Practically, this paper unifies a flow-based media dataset and a check-in-based media dataset. Based on the precision for the top n events, the experimental results demonstrate that the proposed framework can achieve better performance in finding events associated with sports, local festivals, concerts, and exhibitions compared with a state-of-the-art approach that uses one dataset alone.

This paper proposes the concept of perceptual pruning to improve the scalability of immersive video conferencing (IVC) systems. The aim of perceptual pruning is to enable servers to dynamically reduce the bit rate of videos on-the-fly in response to changes in virtual distance and orientation of videos with respect to the receiving client. It is shown that the video of each participant will have to undergo varying degrees of texture to pixel mapping before it is rendered by the client. By reducing the resolution of the video at the scale of each transform block in response to nuances of this mapping, the video bit rate could be reduced without any noticeable degradation of quality. This paper presents the results of experimental tests and also real-world scalability tests to quantify the impact of perceptual pruning. It is concluded that perceptual pruning will be of significant benefit, especially in those situations when the number of videos in the client's viewpoint is large.

This paper addresses the problem of recognizing human skeletal interactions using multiview data captured from depth sensors. The interactions among people are important cues for group and crowd human behavior analysis. In this paper, we focus on modeling the person–person skeletal interactions for human activity recognition. First, we propose a novel graph model in each single-view case to encode class-specific person–person interaction patterns. Particularly, we model each person–person interaction by an attributed graph, which is designed to preserve the complex spatial structure among skeletal joints according to their activity levels as well as the spatio-temporal joint features. Then, combining the graph models for each single-view case, we propose the multigraph model to characterize each multiview interaction. Finally, we apply a general multiple kernel learning method to determine the optimal kernel weights for the proposed multigraph model while the optimal classifier is jointly learned. We evaluate the proposed approach on the M $^2$ I dataset, the SBU Kinect interaction dataset, and our interaction dataset. The experimental results show that our proposed approach outperforms several existing interaction recognition methods.

Crowd understanding has drawn increasing attention from the computer vision community, and its progress is driven by the availability of public crowd datasets. In this paper, we contribute a large-scale benchmark dataset collected from the Shanghai 2010 World Expo. It includes $2630$ annotated video sequences captured by $245$ surveillance cameras, far larger than any public dataset. It covers a large number of different scenes and is suitable for evaluating the performance of crowd segmentation and estimation of crowd density, collectiveness, and cohesiveness, all of which are universal properties of crowd systems. In total, $53\,637$ crowd segments are manually annotated with the three crowd properties. This dataset is released to the public to advance research on crowd understanding. The large-scale annotated dataset enables using data-driven approaches for crowd understanding. In this paper, a data-driven approach is proposed as a baseline of crowd segmentation and estimation of crowd properties for the proposed dataset. Novel global and local crowd features are designed to retrieve similar training scenes and to match spatio-temporal crowd patches so that the labels of the training scenes can be accurately transferred to the query image. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art approaches for crowd understanding.

In the context of social media services, data shortage has severally hindered accurate user modeling and practical personalized applications. This paper is motivated to leverage the user data distributed in disparate online social networks (OSN) to make up for the data shortage in user modeling, which we refer to as “cross-OSN user modeling.” Generally, the data that the same user distributes in different OSNs consist of both behavior data (i.e., interaction with multimedia items) and social data (i.e., interaction between users). This paper focuses on the following two challenges: 1) how to aggregate the users’ cross-OSN interactions with multimedia items of the same modality, which we call cross-OSN homogeneous behaviors, and 2) how to integrate users’ cross-OSN social data with behavior data. Our proposed solution to address the challenges consist of two corresponding components as follows. 1) Homogeneous behavior quantification, where homogeneous user behaviors are quantified based on their importance in reflecting user preferences. After quantification, the examined cross-OSN user behaviors are aggregated to construct a unified user-item interaction matrix. 2) Local social regularization, where the cross-OSN social data is integrated as regularization in matrix factorization-based user modeling at local topic level. The proposed cross-OSN user modeling solution is evaluated in the application of personalized video recommendation. Carefully designed experiments on self-collected Google+ and YouTube datasets have validated its effectiveness and the advantage over single-OSN-based methods.

How accurately are people able to use the absolute category rating (ACR) 5-level scale? Put another way, how repeatable are an individual subject’s scores? Several subjective experiments have asked subjects to rate the same sequences a couple of times. Analyses indicate that none of the subjects exactly repeated their prior scores for these sequences. We would like to better understand this imperfection. This paper uses ACR subjective video quality tests to explore the precision of subjective ratings. To make formal measurements possible, we propose a theoretical subject model that is the main contribution of this paper. The proposed subject model indicates three major factors that influence accuracy: subject bias, subject inaccuracy, and stimulus scoring difficulty. These appear to be separate random effects and their existence is a reason why none of the subjects were able to perfectly repeat scores. There are three key consequences. First, subject scoring behavior includes a random component that spans approximately half of the rating scale. Second, the sensitivity and accuracy of most subjective analyses can be improved if the subject scores are normalized by removing subject bias. Third, to some extent, multiple subjects can be replaced with a single subject who rates each sequence multiple times.

Image retagging is significant and essential for tag-based applications, such as search and browsing. However, most existing image retagging approaches are typically based on enriching-and-removing and/or reranking strategies, which lead to two drawbacks: 1) since the object and/or human appeared in the images are tagged as individuals, the meanings represented by the mutual context of object and human are ignored and not tagged, and 2) some images which are visually dissimilar but semantically similar could be filtered incorrectly, as they are conflict with the content consistency rule. These two defects are distinct especially when images with human-object interactions are retagged. To tackle these defects, in this paper we propose a Bayesian approach to jointly consider the human and object in an image and retag it properly. In our approach, human and objects in images are detected and their interrelationships are taken into account. Tags which represent the mutual context of human and objects are then mapped to those interrelationships by a probabilistic graphical model. For a new image which lacks the tag representing the interaction between human and object, our model can correctly retag it for the interaction. In this paper, those images involving human-object interactions are called verb-object concept images, and experiments on a 60-class dataset demonstrate the capacity of our Bayesian retagging approach of verb-object concept images (BRVOI).

In this paper, we propose a Hierarchical Multimodal Variational Encoder-Decoder (HMMVED) to predict the popularity of micro-videos by comprehensively leveraging the user information and micro-video content in a hierarchical fashion. In particular, the multimodal variational encoder-decoder framework encodes the input modalities to a lower dimensional stochastic variable, from which the popularity of micro-videos can be decoded. Considering the important role of users social influence in social media for information dissemination, a user encoderdecoder is designed to learn the user Gaussian embedding from the user information, which is informative about the coarsegrained popularity for the micro-videos uploaded by the user. The learned user embedding then acts as the prior of the micro-video latent embeddings in the micro-video encoder-decoder, where the refined posterior embedding of the micro-video is obtained by taking the multimodal content into consideration. The finegrained popularity of each micro-video is generated from the posterior embedding of the micro-video. Based on the multimodal extension of variational information bottleneck theory, we show that the learned latent embeddings of micro-videos are maximally expressive about the popularity whilst maximally compressing the information from input modalities. Extensive experiments conducted on two real-world datasets demonstrate the effectiveness of the proposed method. Codes and datasets are available at: https://github.com/JennyXieJiayi/HMMVED.

Weakly supervised object detection (WSOD aims to train object detectors by using only image-level annotations. Many recent works on WSOD adopt multiple instance detection networks (MIDN, which usually generate a certain number of proposals and regard proposal classification as a latent model learning within image classification. However, these methods tend to detect salient object, salient object parts and clustered objects due to lack of instance-level annotations during training. Thus a core issue is how to guarantee that the network learn as many objects with precise bounding boxes as possible. In this paper, we address this issue by exploiting the potential of proposal scores during training. We propose an adaptive instance refinement (AIR framework with three novel designs, which can be integrated with MIDN into a single network. Specifically, adaptive instance mining attempts to discover all positive instances according to the score distribution of proposals and their spatial similarity. Adaptive score modulation dynamically adjusts proposal scores to make the network focus more on instances with different difficulties in different training iterations. Adaptive knowledge refinement distills important information from all previous stages by the weighted average of proposal scores. The experimental results on the PASCAL VOC 2007 and 2012 benchmarks and the MS COCO benchmark demonstrate that AIR significantly improves the performance of the original MIDN and achieves the state-of-the-art results.

Fusion and interaction of multimodal features are essential for video question answering. Structural information composed of the relationships between different objects in videos is very complex, which restricts understanding and reasoning. In this paper, we propose a quaternion hypergraph network (QHGN) for multimodal video question answering, to simultaneously involve multimodal features and structural information. Since quaternion operations are suitable for multimodal interactions, four components of the quaternion vectors are applied to represent the multimodal features. Furthermore, we construct a hypergraph based on the visual objects detected in the video. Most importantly, the quaternion hypergraph convolution operator is theoretically derived to realize multimodal and relational reasoning. Question and candidate answers are embedded in quaternion space, and a Q&A reasoning module is creatively designed for selecting the answer accurately. Moreover, the unified framework can be extended to other video-text tasks with different quaternion decoders. Experimental evaluations on the TVQA dataset and DramaQA dataset show that our method achieves state-of-the-art performance.

Currently, with the rapid development of mobile Internet, micro-video has become a prevailing format of user-generated contents (UGCs) on various social media platforms. Several studies have been conducted towards to understanding high-level micro-video semantics, such as venue categorization, memorability analysis, and popularity prediction. However, these approaches supported tasks with only a single output, which exhibited limitations when attempting to use them to resolve tasks with multiple outputs, especially the multi-label micro-video classification. To tackle this problem, in this paper, we propose a dual multi-modal low-rank decomposition (DMLRD) method for multi-label micro-video classification tasks. To learn more comprehensive micro-video representations, we first learn the low-rank-regularized modality-specific and modality-shared components by considering the consistency and the complementarity among modalities simultaneously. Meanwhile, the less descriptive power of each modality aroused by inherent properties can be solved. To obtain unseen label representations, we next construct a sparsity-regularized multi-matrix normal estimation term to jointly encode the latent relationship structures among labels and dimensions. Experiments on a publicly available large-scale micro-video dataset demonstrate the effectiveness of our proposed model over the state-of-art methods.

Recently, we have observed an exponential increase of user-generated content (UGC) videos. The distinguished characteristic of UGC videos originates from the video production and delivery chain, as they are usually acquired and processed by non-professional users before uploading to the hosting platforms for sharing. As such, these videos usually undergo multiple distortion stages that may affect visual quality before ultimately being viewed. Inspired by the increasing consensus that the optimization of the video coding and processing shall be fully driven by the perceptual quality, in this paper, we propose to study the quality of the UGC videos from both objective and subjective perspectives. We first construct a UGC video quality assessment (VQA) database, aiming to provide useful guidance for the UGC video coding and processing in the hosting platform. The database contains source UGC videos uploaded to the platform and their transcoded versions that are ultimately enjoyed by end-users, along with their subjective scores. Furthermore, we develop an objective quality assessment algorithm that automatically evaluates the quality of the transcoded videos based on the corrupted reference, which is in accordance with the application scenarios of UGC video sharing in the hosting platforms. The information from the corrupted reference is well leveraged and the quality is predicted based on the inferred quality maps with deep neural networks (DNN). Experimental results show that the proposed method yields superior performance. Both subjective and objective evaluations of the UGC videos also shed lights on the design of perceptual UGC video coding.

Multimedia synchronization involves a temporal relationship between audio and visual media components. The presentation of “in-sync” data streams is essential to achieve a natural impression, as “out-of-sync” effects are often associated with user quality of experience (QoE) decrease . Recently , multi-sensory media (mulsemedia) has been demonstrated to provide a highly immersive experience for its users. Unlike traditional multimedia, mulsemedia consists of other media types (i.e., haptic, olfaction, taste, etc.) in addition to audio and visual content. Therefore, the goal of achieving high quality mulsemedia transmission is to present no or little synchronization errors between the multiple media components. In order to achieve this ideal synchronization, there is a need for comprehensive knowledge of the synchronization requirements at the user interface. This paper presents the results of a subjective study carried out to explore the temporal boundaries within which haptic and air-flow media objects can be successfully synchronized with video media. Results show that skews between sensorial media and multimedia might still give the effect that the mulsemedia sequence is “in-sync” and provide certain constraints under which synchronization errors might be tolerated. The outcomes of the paper are used to provide recommendations for mulsemedia service providers in order for their services to be associated with acceptable user experience levels, e.g. haptic media could be presented with a delay of up to 1 s behind video content, while air-flow media could be released either 5 s ahead of or 3 s behind video content.

Large-scale image search has been attracting lots of attention from both academic and commercial fields. The conventional bag-of-visual-words (BoVW) model with inverted index is verified efficient at retrieving near-duplicate images, but it is less capable of discovering fine-grained concepts in the query and returning semantically matched search results. In this paper, we suggest that instance search should return not only near- duplicate images, but also fine-grained results, which is usually the actual intention of a user. We propose a new and interesting problem named fine-grained image search, which means that we prefer those images containing the same fine-grained concept with the query. We formulate the problem by constructing a hierarchical database and defining an evaluation method. We thereafter introduce a baseline system using fine-grained classification scores to represent and co-index images so that the semantic attributes are better incorporated in the online querying stage. Large-scale experiments reveal that promising search results are achieved with reasonable time and memory consumption. We hope this paper will be the foundation for future work on image search. We also expect more follow-up efforts along this research topic and look forward to commercial fine-grained image search engines.

Automatic audio-visual speech recognition currently lags behind its audio-only counterpart in terms of major progress. One of the reasons commonly cited by researchers is the scarcity of suitable research corpora. This paper details the creation of a new corpus designed for continuous audio-visual speech recognition research . TCD-TIMIT consists of high-quality audio and video footage of 62 speakers reading a total of 6913 phonetically rich sentences. Three of the speakers are professionally-trained lipspeakers, recorded to test the hypothesis that lipspeakers may have an advantage over regular speakers in automatic visual speech recognition systems. Video footage was recorded from two angles: straight on, and at $30^{\circ}$ . The paper outlines the recording of footage, and the required post-processing to yield video and audio clips for each sentence. Audio, visual, and joint audio-visual baseline experiments are reported. Separate experiments were run on the lipspeaker and non-lipspeaker data, and the results compared. Visual and audio-visual baseline results on the non-lipspeakers were low overall. Results on the lipspeakers were found to be significantly higher. It is hoped that as a publicly available database, TCD-TIMIT will now help further state of the art in audio-visual speech recognition research.

Image search reranking and image research result summarization are two effective approaches which enhance text-based image search results using visual information. Since the existing approaches optimize search relevance in terms of average performance, they usually cannot achieve satisfactory results for some particular classes of queries, like “object queries,” which is defined as the queries with the intent of searching for some kinds of objects. One possible reason is that the generic approaches such as , , are mostly built based on the global statistics of images as features while ignoring the fact that the relevance between the image and the query sometimes depends on an image patch instead of the whole image. In this paper, we therefore design a novel bag-of-object retrieval model to predict image relevance, which is particularly effective for object queries. First, we construct an object vocabulary containing query-relative objects by mining frequent object patches from the result image collection of the expanded query set. After representing each image as a bag of objects, our retrieval model can be derived from a risk-minimization framework for language modeling. To demonstrate the effectiveness of the proposed model, this paper also present two related applications: for image search reranking, we adopt a supervised framework to combine multiple ranking features from different assumptions; for image search result summarization, we propose a two-step ranking process which optimizes not only representativeness but also image attractiveness. The experimental results show that the proposed methods can significantly outperform the existing approaches.

In this paper we present an introduction to Variational Bayesian (VB) methods in the context of probabilistic graphical models, and discuss their application in multimedia related problems. VB is a family of deterministic probability distribution approximation procedures that offer distinct advantages over alternative approaches based on stochastic sampling and those providing only point estimates. VB inference is flexible to be applied in different practical problems, yet is broad enough to subsume as its special cases several alternative inference approaches including Maximum A Posteriori (MAP) and the Expectation-Maximization (EM) algorithm. In this paper we also show the connections between VB and other posterior approximation methods such as the marginalization-based Loopy Belief Propagation (LBP) and the Expectation Propagation (EP) algorithms. Specifically, both VB and EP are variational methods that minimize functionals based on the Kullback-Leibler (KL) divergence. LBP, traditionally developed using graphical models, can also be viewed as a VB inference procedure. We present several multimedia related applications illustrating the use and effectiveness of the VB algorithms discussed herein. We hope that by reading this tutorial the readers will obtain a general understanding of Bayesian methods and establish connections among popular algorithms used in practice.

Effective media server placement strategies are critical for the quality and cost of multimedia services. Existing studies have primarily focused on optimization-based algorithms to select server locations from a small pool of candidates based on the entire topological information and thus these algorithms are not scalable due to unavailability of the small pool of candidates and low-efficiency of gathering the topological information in large-scale networks. To overcome this limitation, a novel scalable framework called NetClust is proposed in this paper. NetClust takes advantage of the latest network coordinate technique to reduce the workloads when obtaining the global network information for server placement, adopts a new K-means-clustering-based algorithm to select server locations and identify the optimal matching between clients and servers. The key contribution of this paper is that the proposed framework optimizes the trade-off between the service delay performance and the deployment cost under the constraints of client location distribution and the computing/storage/bandwidth capacity of each server simultaneously. To evaluate the performance of the proposed framework, a prototype system is developed and deployed in a real-world large-scale Internet. Experimental results demonstrate that 1) NetClust achieves the lower deployment cost and lower delay compared to the traditional server selection method; and 2) NetClust offers a practical and feasible solution for multimedia service providers.

Virtual try-on of clothes has received much attention recently due to its commercial potential. It can be used for online shopping or intelligent recommendation to narrow down the selections to a few designs and sizes. In this paper, we present a mixed reality system for 3D virtual clothes try-on that enables a user to see herself wearing virtual clothes while looking at a mirror display, without taking off her actual clothes. The user can select various virtual clothes for trying-on. The system physically simulates the selected virtual clothes on the user's body in real-time and the user can see virtual clothes fitting on the her mirror image from various angles as she moves. The major contribution of this paper is that we automatically customize an invisible (or partially visible) avatar based on the user's body size and the skin color and use it for proper clothes fitting, alignment and clothes simulation in our virtual try-on system. We present three scenarios: i) virtual clothes on the avatar, ii) virtual clothes on the user's image and iii) virtual clothes on the avatar blended with the user's face image. We have conducted a user study to evaluate the effectiveness of these three solutions from the end user's perception of quality attributes, cognitive attributes and attitude towards using. The user study shows that among these three scenarios, the second one is most preferred by the users and for 50% of them the experience they had with our system was sufficient to make the purchase decision for the outfits they virtually tried-on.

In this paper, we present a novel approach for automatic visual summarization of a geographic area that exploits user-contributed images and related explicit and implicit metadata collected from popular content-sharing websites. By means of this approach, we search for a limited number of representative but diverse images to represent the area within a certain radius around a specific location. Our approach is based on the random walk with restarts over a graph that models relations between images, visual features extracted from them, associated text, as well as the information on the uploader and commentators. In addition to introducing a novel edge weighting mechanism, we propose in this paper a simple but effective scheme for selecting the most representative and diverse set of images based on the information derived from the graph. We also present a novel evaluation protocol, which does not require input of human annotators, but only exploits the geographical coordinates accompanying the images in order to reflect conditions on image sets that must necessarily be fulfilled in order for users to find them representative and diverse. Experiments performed on a collection of Flickr images, captured around 207 locations in Paris, demonstrate the effectiveness of our approach.

This paper presents a novel Multi-message Ciphertext Policy Attribute-Based Encryption (MCP-ABE) technique, and employs the MCP-ABE to design an access control scheme for sharing scalable media based on data consumers' attributes (e.g., age, nationality, or gender) rather than an explicit list of the consumers' names. The scheme is efficient and flexible because MCP-ABE allows a content provider to specify an access policy and encrypt multiple messages within one ciphertext such that only the users whose attributes satisfy the access policy can decrypt the ciphertext. Moreover, the paper shows how to support resource-limited mobile devices by offloading computational intensive operations to cloud servers while without compromising data privacy.

Users in multimedia social networks actively interact with each other. It is crucial to study the complex user dynamics and analyze its impact on the performance of multimedia social networks. This paper uses multimedia fingerprinting as an example and studies user dynamics in colluder social networks. During collusion, a group of attackers collectively attack multimedia fingerprinting system and use multimedia content illegally. This paper analyzes the incentives of cooperation among attackers and investigates how colluders form their coalitions to maximize their payoffs. We present a game-theoretic framework to model the complex dynamics among colluders, analyze when attackers cooperate with each other, and investigate how a colluder selects his/her fellow attackers to maximize his/her own payoff. We analyze multiuser collusion in two scenarios: when all attackers receive fingerprinted copies of the same resolution, and when they have copies of different resolutions. The proposed framework considers both the colluders' risk of being detected by the digital rights enforcer and the reward received from illegal usage of multimedia content. Our analysis shows that in both scenarios, colluding with more attackers does not always increase an attacker's utility, and attackers may not always want to cooperate with each other. We first examine the necessary conditions for attackers to collude together, and study how they select the collusion parameters such that cooperation benefits all colluders. We then study how the number of colluders affects each attacker's utility, and investigate the optimum strategy that an attacker should use to select fellow attackers and to form a coalition in order to maximize his or her own payoff.

In music information retrieval (MIR) an important research topic, which has attracted much attention recently, is the utilization of user-assigned tags, artist-related style, and mood labels, which can be extracted from music listening web sites, such as Last.fm (http://www.last.fm/) and All Music Guide (http://www.allmusic.com/). A fundamental research problem in the area is how to understand the relationships among artists/songs and these different pieces of information. Co-clustering is the problem of simultaneously clustering two types of data (e.g., documents and words, and webpages and urls). We can naturally bring this idea to the situation at hand and consider clustering artists and tags together, artists and styles together, or artists and mood labels together. Once such co-clustering has been successfully completed, one can identify co-existing clusters of artists and tags, styles, or mood labels (T/S/M). For simplicity, we use the acronym T/S/M to refer to tag(s), style(s), or mood(s) for the rest of the paper. When dealing with tags it is worth noticing that some tags are more specific versions of others. This naturally suggests that the tags could be organized in hierarchical clusters. Such hierarchical organizations exist for styles and mood labels, so we will consider hierarchical co-clustering of artists and T/S/M. In this paper, we systematically study the application of hierarchical co-clustering (HCC) methods for organizing the music data. There are two standard strategies for hierarchical clustering. One is the divisive strategy, in which we attempt to divide the input data set into smaller groups recursively, and the other is the agglomerative strategy, in which we attempt to combine initially individually separated data points into larger groups by finding the most closely related pair at each iteration. We will compare these two strategies against each other. We apply a previously known divisive hierarchical co-clustering method and a novel agglomerative hierarchical co-clustering. In addition, we demonstrate that these two methods have the capability of incorporating instance-level constraints to achieve better performance. We perform experiments to show that these two hierarchical co-clustering methods can be effectively deployed for organizing the music data and they present reasonable clustering performance comparing with the other clustering methods. A case study is also conducted to show that HCC provides us a new method to quantify the artist similarity.

Images that have a different visual appearance may be semantically related using a higher level conceptualization. However, image classification and retrieval systems tend to rely only on the low-level visual structure within images. This paper presents a framework to deal with this semantic gap limitation by exploiting the well-known bag-of-visual words (BVW) to represent visual content. The novelty of this paper is threefold. First, the quality of visual words is improved by constructing visual words from representative keypoints. Second, domain specific “non-informative visual words” are detected which are useless to represent the content of visual data but which can degrade the categorization capability. Distinct from existing frameworks, two main characteristics for non-informative visual words are defined: a high document frequency (DF) and a small statistical association with all the concepts in the collection. The third contribution in this paper is that a novel method is used to restructure the vector space model of visual words with respect to a structural ontology model in order to resolve visual synonym and polysemy problems. The experimental results show that our method can disambiguate visual word senses effectively and can significantly improve classification, interpretation, and retrieval performance for the athletics images.

This paper describes a videoconferencing system that meets performance constraints and functional requirements for use in consumer homes. Our system improves existing home technologies (such as video chat) by providing high-quality audiovisual communication, efficient encoding mechanisms, and low end-to-end delay. Moreover, the system includes a control interface that is capable of dynamically manipulating and compositing audiovisual content streams. This innovative architectural component is required for a domestic setting, where the television acts as the main screen and multiple people gather around it. Apart from the requirements and architecture, this paper analyses the performance of our system. The results validate our architectural decisions and provide a valuable input for further research in domestic videoconferencing.

Scalable video coding is a powerful solution for content delivery in many interactive multimedia services due to its adaptability to varying terminal and network constraints. In order to successfully exploit such adaptability, it is necessary to understand users' preference among various scalability options and consequently develop an optimal bit rate adaptation strategy. In this paper, we present a study of subjective quality assessment of scalable video coding, which investigates the influence of the combination of scalability options on perceived quality with the goal of providing guidelines for an adaptive strategy that selects the optimal combination for a given bandwidth constraint. In particular, the study is based on paired comparison of stimuli that is suitable for our goal due to its simplicity and easiness. We propose a new method, called Paired Evaluation via Analysis of Reliability (PEAR), which analyzes paired comparison results and produces not only quality scores but also intuitive measures of confidence of the scores for significance analysis. Results and analysis of extensive subjective tests for two different scalable video codecs and high definition contents are described, from which general consistent conclusions are drawn. The video and subjective data used in the paper are publicly available to the research community. (http://mmspg.epfl.ch/svd)

The 15 papers in this special issue are extended versions of papers presented at the 2010 IEEE International Conference on Multimedia and Expo (ICME), held in Singapore on July 19-23, 2010. These papers cover a wide range of topics in multimedia including user interface, content understanding, mobility, 3-D processing, storage, and forensics.

Image annotation based on visual features has been a difficult problem due to the diverse associations that exist between visual features and human concepts. In this paper, we propose a novel approach called Annotation by Image-to-Concept Distribution Model (AICDM) for image annotation by discovering the associations between visual features and human concepts from image-to-concept distribution. Through the proposed image-to-concept distribution model, visual features and concepts can be bridged to achieve high-quality image annotation. In this paper, we propose to use “visual features”, “models”, and “visual genes” which represent analogous functions to the biological chromosome, DNA, and gene. Based on the proposed models using entropy, tf-idf, rules, and SVM, the goal of high-quality image annotation can be achieved effectively. Our empirical evaluation results reveal that the AICDM method can effectively alleviate the problem of visual-to-concept diversity and achieve better annotation results than many existing state-of-the-art approaches in terms of precision and recall.

Music information retrieval (MIR) is an emerging research area that receives growing attention from both the research community and music industry. It addresses the problem of querying and retrieving certain types of music from large music data set. Classification is a fundamental problem in MIR. Many tasks in MIR can be naturally cast in a classification setting, such as genre classification, mood classification, artist recognition, instrument recognition, etc. Music annotation, a new research area in MIR that has attracted much attention in recent years, is also a classification problem in the general sense. Due to the importance of music classification in MIR research, rapid development of new methods, and lack of review papers on recent progress of the field, we provide a comprehensive review on audio-based classification in this paper and systematically summarize the state-of-the-art techniques for music classification. Specifically, we have stressed the difference in the features and the types of classifiers used for different classification tasks. This survey emphasizes on recent development of the techniques and discusses several open issues for future research.

Multimedia social network is a network infrastructure in which the social network users share multimedia contents with all different purposes. Analyzing user behavior in multimedia social networks helps design more secured and efficient multimedia and networking systems. Multimedia fingerprinting protects multimedia from illegal alterations and multiuser collusion is a cost-effective attack. The colluder social network is naturally formed during multiuser collusion with which colluders gain reward by redistributing the colluded multimedia contents. Since the colluders have conflicting interest, the maximal-payoff collusion for one colluder may not be the maximal-payoff collusion for others. Hence, before a collusion being successful, the colluders must bargain with each other to reach agreements. We first model the bargaining behavior among colluders as a noncooperative game and study four different bargaining solutions of this game. Moreover, the market value of the redistributed multimedia content is often time-sensitive. The earlier the colluded copy being released, the more the people are willing to pay for it. Thus, the colluders have to reach agreements on how to distribute reward and risk among themselves as soon as possible. This paper further incorporates this time-sensitiveness of the colluders' reward and studies the time-sensitive bargaining equilibrium. The study in this paper reveals the strategies that are optimal for the colluders; thus, all the colluders have no inventive to disagree. Such understanding reduces the possible types of collusion into a small finite set.

This paper has designed a cross-layer framework for voice over Internet protocol (VoIP) services in IEEE 802.16 systems. It uses the application session information of the session description protocol to generate the quality of service parameters in IEEE 802.16 systems. This feature allows the system to efficiently allocate the radio resource because it can exactly estimate the properties of VoIP services such as packet-size and packet-generation-interval. In other words, the cross-layer framework is expected to achieve a novel resource request scheme for a VoIP service that dynamically assigns the radio resource. This paper has analyzed the maximum number of supportable VoIP users for the resource request schemes in terms of the packet-generation-interval in the silent-period, the duration of the silent-period, and the major VoIP speech codec. The numerical results show that the proposed scheme can efficiently support the VoIP services for the various communication environments. Particularly, it can improve the maximum number of supportable VoIP users by 14 ~ 93% compared to an extended real-time polling service.

In this paper, we study the statistical scheduling of offline subjective tests for evaluating alternative control schemes in real time multimedia applications. These applications are characterized by multiple counteracting objective quality metrics (such as delay and signal quality) that can be affected by various control schemes. However, the trade-offs among these metrics with respect to the subjective preferences of users are not defined. As a result, it is difficult to select the proper control schemes that lead to the best subjective quality at run time. Since subjective tests are expensive to conduct and the number of possible control schemes and run-time conditions is prohibitively large, it is important that a minimum number of such tests be conducted offline, and that the results learned can be generalized to unseen conditions with statistical confidence. To this end, we study in this paper efficient algorithms for scheduling a sequence of subjective tests, while leaving the generalization of limited offline subjective tests to guide the operation of the control schemes at run time to a future paper. Using Monte Carlo simulations, we verify the robustness of our algorithms in terms of their accuracy and efficiency.

In this paper, we consider a hybrid P2P video on-demand architecture that utilizes both the server and the peer resources for efficient transmission of popular videos. In our system architecture, each peer dedicates some cache space to store a particular segment of a video file as well as some of its upload bandwidth to serve the cached segment to other peers. Peers join the system and issue a streaming request to a control server. Control server directs the peers to streaming servers or to other peers who have the desired video segments. Control server also decides which peer should cache which video segment. Our main contribution in this paper is to determine the proper caching strategies at peers such that we minimize the average load on the streaming servers. To minimize the server load, we pose the caching problem as a supply-demand-based utility optimization problem. By exploiting the inherent structure of a typical on-demand streaming application as well as the availability of a global view on the current supply-demand at the control server, we demonstrate how the system performance can be significantly improved over the brute-force caching decisions. In our analysis, we mainly consider three caching mechanisms. In the first mechanism (cache prefetching), a segment is prefetched to a given peer for caching purposes upon peer's arrival to the system regardless of whether that segment is currently demanded by that peer or not. In the second mechanism (opportunistic cache update), a peer has the option of replacing the segment that is currently in its cache with the last segment that it finished streaming. In the third mechanism, we combine both mechanisms as a hybrid caching strategy. In particular, we find that a dynamic-programming (DP)-based utility maximization solution using only the cache update method performs significantly better in reducing the server load. Furthermore, our findings suggest that even less sophisticated cache update solutions can perform almost as good as prefetching strategies in interesting regions of operation.

Recent developments have given birth to H.264/AVC: a video coding standard offering better bandwidth to video quality ratios than previous standards (such as H.263, MPEG-2, MPEG-4, etc.), due to its improved inter- and intraprediction modes at the expense of higher computation complexity. It is expected that H.264/AVC will take over the digital video market, replacing the use of previous standards in most digital video applications. This creates an important need for heterogeneous video transcoding technologies from older standards to H.264. In this paper, we focus our attention on the interframe prediction, the most computationally intensive task involved in the heterogeneous video transcoding process. This paper presents a novel macroblock (MB) mode decision algorithm for interframe prediction based on data mining techniques to be used as part of a very low complexity heterogeneous video transcoder. The proposed approach is based on the hypothesis that MB coding mode decisions in H.264 video have a correlation with the distribution of the motion compensated residual in the decoded video. We use data mining tools to exploit the correlation and derive decision trees to classify the incoming decoded MBs into one of the several coding modes in H.264. The proposed approach reduces the H.264 MB mode computation process into a decision tree lookup with very low complexity. For general validation purposes, we apply our algorithm to two of the most important heterogeneous video transcoders: MPEG-2 to H.264 and H.263 to H.264. Our results show that the our data-mining based transcoding algorithm is able to maintain a good video quality while considerably reducing the computational complexity by 72% on average when applied in MPEG-2 to H.264 transcoders, and by 62% on average when applied in H.263 to H.264 transcoders. Finally, we conduct a comparative study with some of the most prominent fast interprediction methods for H.264 presented in the literature. Our results show that the proposed data mining-based approach achieves the best results for video transcoding applications.

Peer-to-peer (P2P) technology has found much success in applications like file distributions and VoIP yet, its adoption in live video streaming remains as an elusive goal. Our recent success in Coolstreaming system brings promises in this direction; however, it also reveals that there exist many practical engineering problems in real live streaming systems over the Internet. Our focus in this paper is on a nonoptimal real working system, in which we illustrate a set of existing practical problems and how they could be handled. We believe this is essential in providing the basic understanding of P2P streaming systems. This paper uses a set of real traces and attempts to develop some theoretical basis to demonstrate that a random peer partnership selection with a hybrid pull-push scheme has the potentially to scale. Specifically, first, we describe the fundamental system design tradeoffs and key changes in the design of a Coolstreaming system including substreaming, buffer management, scheduling and the adopt of a hybrid pull-push mechanism over the original pull-based content delivery approach; second, we examine the overlay topology and its convergence; third, using a combination of real traces and analysis, we quantitatively provide the insights on how the buffering technique resolves the problems associated with dynamics and heterogeneity; fourth, we show how substream and path diversity can help to alleviate the impact from congestion and churns; fifth, we discuss the system scalability and limitations.

Increased speeds of PCs and networks have made media communications possible on the Internet. Today, the need for desktop videoconferencing is experiencing robust growth in both business and consumer markets. However, the synchronous delivery of high-volume media content is still a big challenge under a current heterogeneous Internet environment. In this paper, we present a multiparty videoconferencing system based on a peer-to-peer (P2P) solution. The contribution of our paper is twofold. On the one hand, we design an application-level multicast scheme which intends to tolerate the heterogeneity in videoconferencing applications. Design tradeoffs are analyzed and our decisions are made based on extensive experimentation. On the other, we design a five-layer architecture for implementing a multiparty videoconferencing system. This architecture makes a clear-cut distinction between different functional modules and therefore provides rich flexibility in feature adaptation. We believe that our work can be a helpful reference in other efforts on building desktop videoconferencing systems.

For pt.1see ibid., vol. 9, p. 3 (2007). In this paper, the task and user interface modules of a multimodal dialogue system development platform are presented. The main goal of this work is to provide a simple, application-independent solution to the problem of multimodal dialogue design for information seeking applications. The proposed system architecture clearly separates the task and interface components of the system. A task manager is designed and implemented that consists of two main submodules: the electronic form module that handles the list of attributes that have to be instantiated by the user, and the agenda module that contains the sequence of user and system tasks. Both the electronic forms and the agenda can be dynamically updated by the user. Next a spoken dialogue module is designed that implements the speech interface for the task manager. The dialogue manager can handle complex error correction and clarification user input, building on the semantics and pragmatic modules presented in Part I of this paper. The spoken dialogue system is evaluated for a travel reservation task of the DARPA Communicator research program and shown to yield over 90% task completion and good performance for both objective and subjective evaluation metrics. Finally, a multimodal dialogue system which combines graphical and speech interfaces, is designed, implemented and evaluated. Minor modifications to the unimodal semantic and pragmatic modules were required to build the multimodal system. It is shown that the multimodal system significantly outperforms the unimodal speech-only system both in terms of efficiency (task success and time to completion) and user satisfaction for a travel reservation task

Due to the prevalence of digital video camcorders, home videos have become an important part of life-logs of personal experiences. To enable efficient video parsing, a critical step is to automatically extract objects, events and scene characteristics present in videos. This paper addresses the problem of extracting objects from home videos. Automatic detection of objects is a classical yet difficult vision problem, particularly for videos with complex scenes and unrestricted domains. Compared with edited and surveillant videos, home videos captured in uncontrolled environment are usually coupled with several notable features such as shaking artifacts, irregular motions, and arbitrary settings. These characteristics have actually prohibited the effective parsing of semantic video content using conventional vision analysis. In this paper, we propose a new approach to automatically locate multiple objects in home videos, by taking into account of how and when to initialize objects. Previous approaches mostly consider the problem of how but not when due to the efficiency or real-time requirements. In home-video indexing, online processing is optional. By considering when, some difficult problems can be alleviated, and most importantly, enlightens the possibility of parsing semantic video objects. In our proposed approach, the how part is formulated as an object detection and association problem, while the when part is a saliency measurement to determine the best few locations to start multiple object initialization

Many recently proposed cross-layer protocols for wireless video, have advocated the relay of corrupted packet to higher layers. Such protocols lead to both errors and erasures at the compressed video application layer. We generically refer to such schemes as hybrid erasure-error protocols (HEEPs). In this paper, we analyze the utility of HEEPs for efficient transmission of video over wireless channels. In order to maintain the generic nature of the deductions in this paper, we base our analysis on two (rather abstract) communication schemes for wireless video: hybrid error-erasure cross-layer design (CLD) and hybrid error-erasure cross-layer design with side-information (CLDS). We make a comparative analysis of the channel capacities of these schemes over single and multi-hop wireless channels to identify the conditions under which the HEEPs can provide improved performance over conventional (CON) protocols. In addition, we employ Reed Solomon (RS) and low-density parity check (LDPC)-code-based forward-error correction (FEC) schemes to illustrate that the improvement in capacity can easily enable an FEC scheme employed in conjunction with a HEEP to provide improved throughput. Finally we compare the performance of CON, CLD, and CLDS in terms of video quality using the H.264 video standard. The simulation results show a significant advantage for the HEEPs

The primary aim of this paper is to present new models for objective, nonintrusive, prediction of voice quality for IP networks and to illustrate their application to voice quality monitoring and playout buffer control in VoIP networks. The contributions of the paper are threefold. First, we present a new methodology for developing perceptually accurate models for nonintrusive prediction of voice quality which avoids time-consuming subjective tests. The methodology is generic and as such it has wide applicability in multimedia applications. Second, based on the new methodology, we present efficient regression models for predicting conversational voice quality nonintrusively for four modern codecs (G.729, G.723.1, AMR and iLBC). Third, we illustrate the usefulness of the models in two main applications - voice quality prediction for real Internet VoIP traces and perceived quality-driven playout buffer optimization. For voice quality prediction, the results show that the models have accuracy close to the combined ITU PESQ/E-model method using real Internet traces (correlation coefficient over 0.98). For playout buffer optimization, the proposed buffer algorithm provides an optimum voice quality when compared to five other buffer algorithms for all the traces considered

Efficient and intelligent music information retrieval is a very important topic of the 21st century. With the ultimate goal of building personal music information retrieval systems, this paper studies the problem of intelligent music information retrieval. Huron points out that since the preeminent functions of music are social and psychological, the most useful characterization would be based on four types of information: genre, emotion, style,and similarity. This paper introduces Daubechies Wavelet Coefficient Histograms (DWCH)for music feature extraction for music information retrieval. The histograms are computed from the coefficients of the db/sub 8/ Daubechies wavelet filter applied to 3 s of music. A comparative study of sound features and classification algorithms on a dataset compiled by Tzanetakis shows that combining DWCH with timbral features (MFCC and FFT), with the use of multiclass extensions of support vector machine,achieves approximately 80% of accuracy, which is a significant improvement over the previously known result on this dataset. On another dataset the combination achieves 75% of accuracy. The paper also studies the issue of detecting emotion in music. Rating of two subjects in the three bipolar adjective pairs are used. The accuracy of around 70% was achieved in predicting emotional labeling in these adjective pairs. The paper also studies the problem of identifying groups of artists based on their lyrics and sound using a semi-supervised classification algorithm. Identification of artist groups based on the Similar Artist lists at All Music Guide is attempted. The semi-supervised learning algorithm resulted in nontrivial increases in the accuracy to more than 70%. Finally, the paper conducts a proof-of-concept experiment on similarity search using the feature set.

Sign Language Translation (SLT) aims to generate spoken language translations from sign language videos. Currently, the available sign language datasets are relatively too small to learn the linguistic properties of spoken language. In this paper, towards effective SLT, we propose a novel framework which takes the advantage of the spoken language grammar learnt from a large corpus of text sentences. Our framework consists of three key modules: word existence verification, conditional sentence generation and cross-modal re-ranking. We first check the existence of words in the vocabulary by a series of binary classification in parallel. After that, the appearing words are assembled and guided by a pretrained spoken language generator to produce multiple candidate sentences in spoken language manner. Last but not least, we select the sentence most semantically similar to the input sign video as the translation result with a cross-modal re-ranking model. We evaluate our framework on two large scale continuous SLT benchmarks, \emph{i.e.}, CSL and RWTH-PHOENIX-Weather 2014T. Experimental results demonstrate that the proposed framework achieves promising performance on both datasets.