Search

With Multimedia Information Retrieval frustrated by the seemingly intractable semantic gap, we turn to the related field of linguistics for fresh inspiration and ideas for old problems and new opportunities. An explosion in the amount and ease with which multimedia items are created and shared, courtesy of new devices and Web 2.0, prompts us to consider what happens when those items are viewed not as artefacts, or "built things", but as utterances. These conversations occur in a mixture of mediums, including text, images, audio, and video, and channels, including Twitter, Facebook, Youtube and blogs, and range in scope from one-to-one exchanges to loosely-bounded meta-conversations that cross cultures and span the globe via remixing of shared meanings like memes. We propose that MIR add to its toolbox a linguistic perspective, and highlight three useful emphases of research: genre, emergence, and effect.

A convergence of emotions among people in social networks is potentially resulted by the occurrence of an unprecedented event in real world. E.g., a majority of bloggers would react angrily at the September 11 terrorist attacks. Based on this observation, we introduce a sentiment index, computed from the current mood tags in a collection of blog posts utilizing an affective lexicon, potentially revealing subtle events discussed in the blogosphere. We then develop a method for extracting events based on this index and its distribution. Our second contribution is establishment of a new bursty structure in text streams termed a sentiment burst. We employ a stochastic model to detect bursty periods of moods and the events associated. Our results on a dataset of more than 12 million mood-tagged blog posts over a 4-year period have shown that our sentiment-based bursty events are indeed meaningful, in several ways.

Almost every aspect of how we create, transmit, and consume video has changed, but video interfaces still mimic those from video's inception. We extend Temporal Semantic Compression for interactive video browsing, which uses an arbitrary frame-by-frame interest measure to sub-sample video in real time, with user interface elements that visualize these measures and the effect of compressing on them. We experiment with a novel interest measure for popularity, and design novel visualizations for expressing interest measures and the compression interaction. We conduct the first formative evaluation of the TSC paradigm, with 8 subjects, and report design implications arising from it.

There is a growing gap between the number of children with autism requiring early intervention and available therapy. We present a portable platform for pervasive delivery of early intervention therapy using multi-touch interfaces and principled ways to deliver stimuli of increasing complexity and adapt to a child’s performance. Our implementation weaves Natural Environment Tasks with iPad tasks, facilitating a learning platform that integrates early intervention in the child’s daily life. The system’s construction of stimulus complexity relative to task is evaluated by therapists, together with field trials for evaluating both the integrity of the instructional design and goal of stimulus presentation and adjustment relative to performance for learning tasks. We show positive results across all our stakeholders–children, parents and therapists. Our results have implications for other early learning fields that require principled ways to construct lessons across skills and adjust stimuli relative to performance.

Nonnegative matrix factorization based methods provide one of the simplest and most effective approaches to text mining. However, their applicability is mainly limited to analyzing a single data source. In this paper, we propose a novel joint matrix factorization framework which can jointly analyze multiple data sources by exploiting their shared and individual structures. The proposed framework is flexible to handle any arbitrary sharing configurations encountered in real world data. We derive an efficient algorithm for learning the factorization and show that its convergence is theoretically guaranteed. We demonstrate the utility and effectiveness of the proposed framework in two real-world applications–improving social media retrieval using auxiliary sources and cross-social media retrieval. Representing each social media source using their textual tags, for both applications, we show that retrieval performance exceeds the existing state-of-the-art techniques. The proposed solution provides a generic framework and can be applicable to a wider context in data mining wherever one needs to exploit mutual and individual knowledge present across multiple data sources.

Estimation of a person's influence and personality traits from social media data has many applications. We use social linkage criteria, such as number of followers and friends, as proxies to form corpora, from popular blogging site Livejournal, for examining two two-class classification problems: influential vs. non-influential, and extraversion vs. introversion. Classification is performed using automatically-derived psycholinguistic and mood-based features of a user's textual messages. We experiment with three sub-corpora of 10000 users each, and present the most effective predictors for each category. The best classification result, at 80%, is achieved using psycholinguistic features; e.g., influentials are found to use more complex language, than non-influentials, and use more leisure-related terms.

Joint modeling of related data sources has the potential to improve various data mining tasks such as transfer learning, multitask clustering, information retrieval etc. However, diversity among various data sources might outweigh the advantages of the joint modeling, and thus may result in performance degradations. To this end, we propose a regularized shared subspace learning framework, which can exploit the mutual strengths of related data sources while being immune to the effects of the variabilities of each source. This is achieved by further imposing a mutual orthogonality constraint on the constituent subspaces which segregates the common patterns from the source specific patterns, and thus, avoids performance degradations. Our approach is rooted in nonnegative matrix factorization and extends it further to enable joint analysis of related data sources. Experiments performed using three real world data sets for both retrieval and clustering applications demonstrate the benefits of regularization and validate the effectiveness of the model. Our proposed solution provides a formal framework appropriate for jointly analyzing related data sources and therefore, it is applicable to a wider context in data mining.

This paper presents a novel Bayesian formulation to exploit shared structures across multiple data sources, constructing foundations for effective mining and retrieval across disparate domains. We jointly analyze diverse data sources using a unifying piece of metadata (textual tags). We propose a method based on Bayesian Probabilistic Matrix Factorization (BPMF) which is able to explicitly model the partial knowledge common to the datasets using shared subspaces and the knowledge specific to each dataset using individual subspaces. For the proposed model, we derive an efficient algorithm for learning the joint factorization based on Gibbs sampling. The effectiveness of the model is demonstrated by social media retrieval tasks across single and multiple media. The proposed solution is applicable to a wider context, providing a formal framework suitable for exploiting individual as well as mutual knowledge present across heterogeneous data sources of many kinds.

Most existing work on learning community structure in social network is graph-based whose links among the members are often represented as an adjacency matrix, encoding direct pairwise associations between members. In this paper, we propose a method to group online communities in blogosphere based on the topicslearnt from the content blogged. We then consider a different type of online community formulation - the sentiment-based grouping of online communities. The problem of sentiment-based clustering for community structure discovery is rich with many interesting open aspects to be explored. We propose a novel approach foraddressing hyper-community detection based on users' sentiment. We employ a nonparametric clustering to automatically discover hidden hyper-communities and present the results obtained from a large dataset.

Although tagging has become increasingly popular in online image and video sharing systems, tags are known to be noisy, ambiguous, incomplete and subjective. These factors can seriously affect the precision of a social tag-based web retrieval system. Therefore improving the precision performance of these social tag-based web retrieval systems has become an increasingly important research topic. To this end, we propose a shared subspace learning framework to leverage a secondary source to improve retrieval performance from a primary dataset.This is achieved by learning a shared subspace between the two sources under a joint Nonnegative Matrix Factorization in which the level of subspace sharing can be explicitly controlled. We derive an efficient algorithm for learning the factorization, analyze its complexity, and provide proof of convergence. We validate the framework on image and video retrieval tasks in which tags from the LabelMe dataset are used to improve image retrieval performance from a Flickr dataset and video retrieval performance from a YouTube dataset. This has implications for how to exploit and transfer knowledge from readily available auxiliary tagging resources to improve another social web retrieval system. Our shared subspace learning framework is applicable to a range of problems where one needs to exploit the strengths existing among multiple and heterogeneous datasets.

The blogosphere has grown to be a mainstream forum of social interaction as well as a commercially attractive source of information and influence. Tools are needed to better understand how communities that adhere to individual blogs are constituted in order to facilitate new personal, socially-focused browsing paradigms, and understand how blog content is consumed, which is of interest to blog authors, big media, and search.We present a novel approach to blog subcommunity characterization by modeling individual blog readers using mixtures of an extension to the LDA family that jointly models phrases and time, Ngram Topic over Time (NTOT), and cluster with a number of similarity measures using Affinity Propagation. We experiment with two datasets: a small set of blogs whose authors provide feedback, and a set of popular, highly commented blogs, which provide indicators of algorithm scalability and interpretability without prior knowledge of agiven blog. The results offer useful insight to the blog authors about their commenting community, and are observed to offer an integrated perspective on the topics of discussion and members engaged in those discussions for unfamiliar blogs. Our approach also holds promise as a component of solutions to related problems, such as online entity resolution and role discovery.

Automatic data-driven analysis of mood from text is anemerging problem with many potential applications. Unlike generic text categorization, mood classification based on textual features is complicated by various factors, including its context- and user-sensitive nature. We present a comprehensive study of different feature selection schemes in machine learning for the problem of mood classification in weblogs. Notably, we introduce the novel use of a feature set based on the affective norms for English words (ANEW) lexicon studied in psychology. This feature set has the advantage of being computationally efficient while maintaining accuracy comparable to other state-of-the-art feature sets experimented with. In addition, we present results of data-driven clustering on a dataset of over 17 million blog posts with mood groundtruth. Our analysis reveals an interesting, and readily interpreted, structure to the linguistic expression of emotion, one that comprises valuable empirical evidence in support of existing psychological models of emotion, and in particular the dipoles pleasure-displeasure and activation-deactivation.

We describe a novel video player that uses Temporal Semantic Compression (TSC) to present a compressed summary of a movie. Compression is based on tempo which is derived from film rhythms. The technique identifies periods of action, drama, foreshadowing and resolution, which can be mixed in different amounts to vary the kind of summary presented. The compression algorithm is embedded in a video player, so that the summary can be interactively recomputed during playback.

This paper examines the recovery of user context in indoor environmnents with existing wireless infrastructures to enable assistive systems. We present a novel approach to the extraction of user context, casting the problem of context recovery as an unsupervised, clustering problem. A well known density-based clustering technique, DBSCAN, is adapted to recover user context that includes user motion state, and significant places the user visits from WiFi observations consisting of access point id and signal strength. Furthermore, user rhythms or sequences of places the user visits periodically are derived from the above low level contexts by employing state-of-the-art probabilistic clustering technique, the Latent Dirichiet Allocation (LDA), to enable a variety of application services. Experimental results with real data are presented to validate the proposed unsupervised learning approach and demonstrate its applicability.

The amount of multimedia content available online constantly increases, and this leads to problems for users who search for content or similar communities. Users in Flickr often self-organize in user communities through Flickr Groups. These groups are particularly interesting as they are a natural instantiation of the content + relations social media paradigm. We propose a novel approach to group searching through hypergroup discovery. Starting from roughly 11,000 Flickr groups' content and membership information, we create three different bag-of-word representations for groups, on which we learn probabilistic topic models. Finally, we cast the hypergroup discovery as a clustering problem that is solved via probabilistic affinity propagation. We show that hypergroups so found are generally consistent and canbe described through topic-based and similarity-based measures. Our proposed solution could be relatively easily implemented as an application to enrich Flickr's traditional group search.

The social interactions manifest in blogs by the network of comments left by owners and readers are an under-used resource, both for blog pundits and industry. We present a web-based feed reader that renders these relationships with a graph representation, and enables exploration by displaying people and blogs who are proximate to a user's network. Social Reader is an example of Casual Information Visualization, and aims to help the user understand and explore blog-based social networks in a daily, real-life setting. Asix week study of the software involving 20 users confirmed the usefulness of the novel visual display, via a quantitative analysis of use logs, and an exit survey

This paper examines the recovery of user context in indoor environmnents with existing wireless infrastructures to enable assistive systems. We present a novel approach to the extraction of user context, casting the problem of context recovery as an unsupervised, clustering problem. A well known density-based clustering technique, DBSCAN, is adapted to recover user context that includes user motion state, and significant places the user visits from WiFi observations consisting of access point id and signal strength. Furthermore, user rhythms or sequences of places the user visits periodically are derived from the above low level contexts by employing a state-of-the-art probabilistic clustering technique, the Latent Dirichlet Allocation (LDA), to enable a variety of application services. Experimental results with real data are presented to validate the proposed unsupervised learning approach and demonstrate its applicability.

This paper presents a framework for indoor location prediction system using multiple wireless signals available freely in public or office spaces. We first propose an abstract architectural design for the system, outlining its key components and their functionalities. Different from existing works, such as robot indoor localization which requires as precise localization as possible, our work focuses on a higher grain: location prediction. Such a problem has a great implication in context-aware systems such as indoor navigation or smart self-managed mobile devices (e.g., battery management). Central to these systems is an effective method to perform location prediction under different constraints such as dealing with multiple wireless sources, effects of human body heats or mobility of the users. To this end, the second part of this pa- per presents a comparative and comprehensive study on different choices for modeling signals strengths and prediction methods under different condition settings. The results show that with simple, but effective modeling method, almost perfect prediction accuracy can be achieved in the static environment, and up to 85% in the presence of human movements. Finally, adopting the proposed framework we outline a fully developed system, named Marauder, that support user interface interaction and real-time voice-enabled location prediction.

We present a video browsing approach, termed Temporal Semantic Compression (TSC) that uses automated measures of interest to support today's foraging behaviours. Conventional browsers 'compress' a video stream using simple 2x or 8x fast-forward. TSC browsers dynamically filter video based on a single user gesture to leave out more or less of the boring bits. We demonstrate a browser with an example interest measure, derived from an automated estimate of movie tempo, to forage in terms of narrative structures such as crises, climaxes, and action sequence book-ends. Media understanding algorithms facilitate browsing, and interactivity enables the human-in-the-loop to cope when those algorithms fail to cross the semantic gap.

This paper describes a novel suggestive interface embedded in a smart camera prototype aimed at aiding home movie makers. We focus on the problem of generating shot capture suggestions suitable to the user's filming context, intended audience and style, and formulate a novel aesthetic measure by which to judge proposed suggestions. Tight coupling between media and software allows the aesthetic measure to be sensitive to previous footage captures, including those taken without the system's prompting, in a manner allowing flexible, end-to-end migration of the authoring task from user to machine. An approximate method is used to find timely, near-optimal solutions to the aesthetic measure. Qualitative evaluation in the form of a user study shows it to be a promising approach to the flexible home movie authoring context.

Personal media collections are often viewed and managed along the social dimension, the places we spend time at and the people we see, thus tools for extracting and using this information are required. We present novel algorithms for identifying socially signicant places termed social spheres unobtrusively from GPS traces of daily life, and label them as one of Home, Work, or Other, with quantitative evaluation of 9 months taken from 5 users. We extract locational co-presence of these users and formulate a novel measure of social tie strength based on frequency of interaction, and the nature of spheres it occurs within. Comparative user studies of a multimedia browser designed to demonstrate the utility of social metadata indicate the usefulness of a simple interface allowing navigation and ltering in these terms. We note the application of social context is potentially much broader than personal media management, including context-aware device behaviour, life logs, social networks, and location-aware information services.

This paper presents novel techniques for using panoramas as spatial context to enhance browsing of personal media archives. This context, scenes where frequent media capture takes place, is present in the disparate photos and videos, but not leveraged by traditional browsing techniques (e.g. thumbnails or zoomable interfaces). Coarse geo-position is often an insufficient index at such media capture hotspots. We experiment with panoramic video, which presents archive video organically blended with panoramas of media capture hotspots; Immersive browsing and filtering with media items projected onto spherical panoramas; and Detection and representation of links between panoramas to enable browsing of situated media in quasi-3D. We present proof-of-concept implementations and observations of their effectiveness, limitations, and open problems. Experiments confirm the intuition that each holds promise for augmenting traditional browsing environments.

This paper details the design of an algorithm for automatically manipulating the important aesthetic element of video, visual tempo. Automatic injection, detection and repair of such aesthetic elements, it is argued, is vital to the next generation of amateur multimedia authoring tools. We evaluate the performance of the algorithm on a battery of synthetic data and demonstrate its ability to return the visual tempo of the final media a considerable degree closer to the target signal. The novelty of this work lies chiefly in the systematic manipulation of this high level aesthetic element of video.

This paper details the design of an algorithm for automatically manipulating the important aesthetic element of video, visual tempo. Automatic injection, detection and repair of such aesthetic elements, it is argued, is vital to the next generation of amateur multimedia authoring tools. We evaluate the performance of the algorithm on a battery of synthetic data and demonstrate its ability to return the visual tempo of the final media a considerable degree closer to the target signal. The novelty of this work lies chiefly in the systematic manipulation of this high level aesthetic element of video.

This paper describes a novel interactive media authoring framework, MediaTE, that enables amateurs to create videos of higher narrative or aesthetic quality with a completely mobile lifecycle. A novel event bootstrapping dialog is used to derive shot suggestions that yield both targeted footage and annotation enabling an automatic Computational Media Aesthetics-aware editing phase, the manual performance of which is typically a barrier to the amateur. This facilitates a move away from requiring a prior-conception of the events or locale being filmed, in the form of a template, to at-capture bootstrapping of this information. Metadata gathered as part of the critical path of media creation also has implications for the longevity and reuse of captured media assets. Results of an evaluation performed on both the usability and delivered media aspects of the system are discussed, which highlight the tenability of the proposed framework and the quality of the produced media.

This paper presents novel additions to our existing amateur media creation framework. The framework provides at-capture guidance to enable the home movie maker to realize their aesthetic and narrative goals and automation of post-production editing. A common problem with the amateur filming context is its contingent nature, which often results in the failure to gain footage vital to the user's goals, even with at-capture software embedding. Accordingly, we have modelled minimizing the difference between target and captured footage at a given time during filming as a probability distribution divergence problem. We apply two policies of feedback to the user on their performance, passive communication via a suggestion desirability measure, and active filtering of undesirable suggestions. We demonstrate the framework using each policy with a simulation of various user and filming situations with promising results.