Search

The analysis of large-scale multimodal data has become very popular recently. Image captioning, whose goal is to describe the content of image with natural language automatically, is an essential and challenging task in artificial intelligence. Commonly, most existing image caption methods utilize the mixture of Convolutional Neural Network and Recurrent Neural Network framework. These methods either pay attention to global representation at the image level or only focus on the specific concepts, such as regions and objects. To make the most of characteristics about a given image, in this study, we present a novel model named Multilevel Attention Networks and Policy Reinforcement Learning for image caption generation. Specifically, our model is composed of a multilevel attention network module and a policy reinforcement learning module. In the multilevel attention network, the object-attention network aims to capture global and local details about objects, whereas the region-attention network obtains global and local features about regions. After that, a policy reinforcement learning algorithm is adopted to overcome the exposure bias problem in the training phase and solve the loss-evaluation mismatching problem at the caption generation stage. With the attention network and policy algorithm, our model can automatically generate accurate and natural sentences for any particular image. We carry out extensive experiments on the MSCOCO and Flickr30k data sets, demonstrating that our model is superior to other competitive methods.

These days, social media users tend to express their feelings through sharing images online. Capturing the emotions embedded in these social images involves great research challenges and practical values. Most existing works concentrate on extracting the visual feature from a global view, while ignoring the fact that visual objects are also rich in emotion. How to leverage the multilevel visual features to improve the sentiment analysis performance is important yet challenging. Besides, existing works view each social image as an independent sample while ignoring the rich correlations among social images, which may be helpful in detecting visual emotion. In this article, we propose a novel model called social relations-guided multiattention networks (SRGMANs) to incorporate both the multilevel (region-level and object-level) visual features of a single image and the correlations among multiple social images to conduct visual sentiment analysis. Specifically, we first construct a heterogeneous network consisting of various types of social relations and introduce a heterogeneous network embedding method to learn the network representation for each image. Then, two visual attention branches (region attention network and object attention network) are devised to extract emotional and discriminative visual features. For each branch, we design a self-attention module to capture the emotional dependencies among visual parts. Besides, a network-guided attention module is also designed in each branch to focus on more network-related emotional visual parts with the guidance of the topology information. Finally, the attended visual features from the two attention models, together with network representation features, are combined within a holistic framework to predict the sentiment of social images. Extensive experiments demonstrate the superiority of our model on three benchmark datasets.

Vehicular networks have tremendous potential to improve the road safety and traffic efficiency, and the adoption of the Space-Air-Ground Integrated Network (SAGIN) architecture in vehicular networks can greatly improve the performance of vehicular networks by leveraging the respective advantages of the space, air, and ground segments on coverage, flexibility, reliability, and availability, which results in Space-Air-Ground Integrated Vehicular Networks (SAGIVNs). Trust management is an important tool for constructing trustworthy SAGIVNs, and privacy preservation is also a primary concern in SAGIVNs. They have conflicting requirements and a satisfactory balance between them is urgently required. In this article, we propose a novel Privacy-Preserving Trust Management (PPTM) scheme for the emergency message dissemination in SAGIVNs. The proposed scheme can realize precise trust management and strong conditional privacy preservation simultaneously with low communication overhead, and can provide strong applicability, strong robustness, and multiple other attractive features. Furthermore, the exhaustive theoretical analysis and simulation evaluation are detailed. The results reveal that the proposed scheme is significantly superior to the existing schemes in several aspects.

Due to the rich spatio-temporal visual content and complex multimodal relations, Video Question Answering (VideoQA) has become a challenging task and attracted increasing attention. Current methods usually leverage visual attention, linguistic attention, or self-attention to uncover latent correlations between video content and question semantics. Although these methods exploit interactive information between different modalities to improve comprehension ability, inter- and intra-modality correlations cannot be effectively integrated in a uniform model. To address this problem, we propose a novel VideoQA model called Cross-Attentional Spatio-Temporal Semantic Graph Networks (CASSG). Specifically, a multi-head multi-hop attention module with diversity and progressivity is first proposed to explore fine-grained interactions between different modalities in a crossing manner. Then, heterogeneous graphs are constructed from the cross-attended video frames, clips, and question words, in which the multi-stream spatio-temporal semantic graphs are designed to synchronously reasoning inter- and intra-modality correlations. Last, the global and local information fusion method is proposed to coalesce the local reasoning vector learned from multi-stream spatio-temporal semantic graphs and the global vector learned from another branch to infer the answer. Experimental results on three public VideoQA datasets confirm the effectiveness and superiority of our model compared with state-of-the-art methods.

Multimodal representation learning is beneficial for many multimedia-oriented applications, such as social image recognition and visual question answering. The different modalities of the same instance (e.g., a social image and its corresponding description) are usually correlational and complementary. Most existing approaches for multimodal representation learning are not effective to model the deep correlation between different modalities. Moreover, it is difficult for these approaches to deal with the noise within social images. In this article, we propose a deep learning-based approach named evolutionary adversarial attention networks (EAANs), which combines the attention mechanism with adversarial networks through evolutionary training, for robust multimodal representation learning. Specifically, a two-branch visual-textual attention model is proposed to correlate visual and textual content for joint representation. Then adversarial networks are employed to impose regularization upon the representation by matching its posterior distribution to the given priors. Finally, the attention model and adversarial networks are integrated into an evolutionary training framework for robust multimodal representation learning. Extensive experiments have been conducted on four real-world datasets, including PASCAL, MIR, CLEF, and NUS-WIDE. Substantial performance improvements on the tasks of image classification and tag recommendation demonstrate the superiority of the proposed approach.

Vehicular networks have tremendous potential to improve road safety, traffic efficiency, and driving comfort, where cooperative vehicular safety applications are a significant branch. In cooperative vehicular safety applications, through the distributed data fusion for large amounts of data from multiple nearby vehicles, each vehicle can intelligently perceive the surrounding conditions beyond the capability of its own onboard sensors. Trust evaluation and privacy preservation are two primary concerns for facilitating the distributed data fusion in cooperative vehicular safety applications. They have conflicting requirements and a good balance between them is urgently needed. Meanwhile, the computation, communication, and storage overheads will all influence the applicability of a candidate scheme. In this paper, we propose a Lightweight Privacy-Preserving Trust Evaluation (LPPTE) scheme which can primely balance the trust evaluation and privacy preservation with low overheads for facilitating the distributed data fusion in cooperative vehicular safety applications. Furthermore, we provide exhaustive theoretical analysis and simulation evaluation for the LPPTE scheme, and the results demonstrate that the LPPTE scheme can obviously improve the accuracy of fusion results and is significantly superior to the state-of-the-art schemes in multiple aspects.

Travel route planning aims to map out a feasible sightseeing itinerary for a traveler covering famous attractions and meeting the tourist’s desire. It is very useful for tourists to plan their travel routes when they want to travel at unfamiliar scenic cities. Existing methods for travel route planning mainly concentrate on a single planning problem for a special task, but is not capable of being applied to other tasks. For example, previous must-visit planning methods cannot be applied to the next-point recommendation, despite these two tasks are closely related to each other in travel route planning. Besides, most of the existing work do not consider the important auxiliary information such as Point of Interests (POI) attributes, user preference, and historical route data in their approaches. In this paper, we propose a flexible Multi-task Deep Travel Route Planning framework named MDTRP to integrate rich auxiliary information for more effective planning. Specifically, we first construct a heterogeneous network through the relations between users and POIs and employ a heterogeneous network embedding method to learn the features of users and POIs. Then we present an attention-based deep model to integrate the auxiliary information and focus on important visited points for the prediction of next POIs. Finally, a beam search algorithm is introduced to flexibly generate multiple feasible route candidates for three types of planning tasks (next-point recommendation, general route planning, and must-visit planning). We introduce six public datasets to conduct extensive experiments, of which the results demonstrate the flexibility and superiority of the proposed approach in travel route planning.

The analysis for social networks, such as the socially connected Internet of Things, has shown a deep influence of intelligent information processing technology on industrial systems for Smart Cities. The goal of social media representation learning is to learn dense, low-dimensional, and continuous representations for multimodal data within social networks, facilitating many real-world applications. Since social media images are usually accompanied by rich metadata (e.g., textual descriptions, tags, groups, and submitted users), simply modeling the image is not effective to learn the comprehensive information from social media images. In this work, we treat the image and its textual description as multimodal content, and transform other metainformation into the links between contents (such as two images marked by the same tag or submitted by the same user). Based on the multimodal content and social links, we propose a Deep Attentive Multimodal Graph Embedding model named DAMGE for more effective social image representation learning. We introduce both small- and large-scale datasets to conduct extensive experiments, of which the results confirm the superiority of the proposal on the tasks of social image classification and link prediction.

Fake news is a longstanding issue that has existed on the social network, whose negative impact has been increasingly recognized since the 2016 US presidential election. During the election, numerous fake news about the candidates distributes vastly in the online social networks. Identifying inauthentic news quickly is an essential purpose for this research to enhance the trustworthiness of news in online social networks, which will be the task studied in this paper. The fake news stance detection can contribute to detect a startling amount of fake news, which aims at evaluating the relevance between the headline and text bodies. There exists a significant difference between news article headline and text body, since headlines with several key phrases are usually much shorter than the text bodies. Such an information imbalance challenge may cause serious problems for the stance detection task. What’s more, news article data in online social networks is usually exposed to various types of noise and can be contaminated, which poses more challenges for the stance detection task. In this paper, we propose a novel fake news stance detection model, namely Adversarial PseudoSiamese Network model (APSN), to solve these challenges. With coupled input components with imbalanced parameters, APSN can learn and compute feature vectors and similarity score of news article headlines and text bodies simultaneously. In addition, by adopting adversarial setting, besides the regular training set, a set of noisy training instances will be generated and fed to APSN in the learning process, which can significantly enhance the robustness of the model. Extensive experiments have been done on real-world fake news dataset, and the experimental results reveal that the presented model exceeds compared suspicious information detection models with significant advantages.