Search

A human-centric emotion estimation method based on correlation maximization with consideration of changes with time in visual attention and brain activity when viewing images is proposed in this paper. Owing to the recent developments of many kinds of biological sensors, many researchers have focused on multimodal emotion estimation using both eye gaze data and brain activity data for improving the quality of emotion estimation. In this paper, a novel method that focuses on the following two points is introduced. First, in order to reduce the burden on users, we obtain brain activity data from users only in the training phase by using a projection matrix calculated by canonical correlation analysis (CCA) between gaze-based visual features and brain activity-based features. Next, for considering the changes with time in both visual attention and brain activity, we obtain novel features based on CCA-based projection in each time unit. In order to include these two points, the proposed method analyzes a fourth-order gaze and image tensor for which modes are pixel location, color channel and the changes with time in visual attention. Moreover, in each time unit, the proposed method performs CCA between gaze-based visual features and brain activity-based features to realize human-centric emotion estimation with a high level of accuracy. Experimental results show that accurate human emotion estimation is achieved by using our new human-centric image representation.

A new approach that drastically improves cross-modal retrieval performance in vision and language (hereinafter referred to as & x201C;vision and language retrieval & x201D;) is proposed in this paper. Vision and language retrieval takes data of one modality as a query to retrieve relevant data of another modality, and it enables flexible retrieval across different modalities. Most of the existing methods learn optimal embeddings of visual and lingual information to a single common representation space. However, we argue that the forced embedding optimization results in loss of key information for sentences and images. In this paper, we propose an effective utilization of representation spaces in a simple but robust vision and language retrieval method. The proposed method makes use of multiple individual representation spaces through text-to-image and image-to-text models. Experimental results showed that the proposed approach enhances the performance of existing methods that embed visual and lingual information to a single common representation space.

Sentiment in multimedia contents has an influence on their topics, since multimedia contents are tools for social media users to convey their sentiment. Performance of applications such as retrieval and recommendation will be improved if sentiment in multimedia contents can be estimated; however, there have been few works in which such applications were realized by utilizing sentiment analysis. In this paper, a novel method for extracting the hierarchical structure of Web video groups based on sentiment-aware signed network analysis is presented to realize Web video retrieval. First, the proposed method estimates latent links between Web videos by using multimodal features of contents and sentiment features obtained from texts attached to Web videos. Thus, our method enables construction of a signed network that reflects not only similarities but also positive and negative relations between topics of Web videos. Moreover, an algorithm to optimize a modularity-based measure, which can adaptively adjust the balance between positive and negative edges, was newly developed. This algorithm detects Web video groups with similar topics at multiple abstraction levels; thus, successful extraction of the hierarchical structure becomes feasible. By providing the hierarchical structure, users can obtain an overview of many Web videos and it becomes feasible to successfully retrieve the desired Web videos. Results of experiments using a new benchmark dataset, YouTube-8M, validate the contributions of this paper, i.e. , 1) the first attempt to utilize sentiment analysis for Web video grouping and 2) a novel algorithm for analyzing a weighted signed network derived from sentiment and multimodal features.

Eigendecomposition is the factorization of a matrix into its canonical form, whereby the matrix is represented in terms of its eigenvalues and eigenvectors. A common step is the reduction of the data to a kernel matrix also known as a Gram matrix which is used for machine learning tasks. A significant drawback of kernel methods is the computational complexity associated with manipulating kernel matrices. This paper demonstrates that leading eigenvectors derived from singular value decomposition (SVD) and Nyström approximation methods can be utilized for classification tasks without the need to construct Gram matrices. Experiments were conducted with 14 biomedical datasets to compare classifier performance when taking as input into a classifier matrices containing: 1) leading eigenvectors which result from each approximation method, and 2) matrices which result from constructing the patient-by-patient Gram matrix. The results provide evidence to support the main hypothesis of this paper that using the leading eigenvectors as input into a classifier significantly (p