Search

We propose a deep attention-based alignment network, which aims to automatically predict lyrics and melody with given incomplete lyrics as input in a way similar to the music creation of humans. Most importantly, a deep neural lyrics-to-melody net is trained in an encoder-decoder way to predict possible pairs of lyrics-melody when given incomplete lyrics (few keywords). The attention mechanism is exploited to align the predicted lyrics with the melody during the lyrics-to-melody generation. The qualitative and quantitative evaluation metrics reveal that the proposed method is indeed capable of generating proper lyrics and corresponding melody for composing new songs given a piece of incomplete seed lyrics., Comment: arXiv admin note: substantial text overlap with arXiv:2011.06380

Cross-modal retrieval is to utilize one modality as a query to retrieve data from another modality, which has become a popular topic in information retrieval, machine learning, and database. How to effectively measure the similarity between different modality data is the major challenge of cross-modal retrieval. Although several reasearch works have calculated the correlation between different modality data via learning a common subspace representation, the encoder's ability to extract features from multi-modal information is not satisfactory. In this paper, we present a novel variational autoencoder (VAE) architecture for audio-visual cross-modal retrieval, by learning paired audio-visual correlation embedding and category correlation embedding as constraints to reinforce the mutuality of audio-visual information. On the one hand, audio encoder and visual encoder separately encode audio data and visual data into two different latent spaces. Further, two mutual latent spaces are respectively constructed by canonical correlation analysis (CCA). On the other hand, probabilistic modeling methods is used to deal with possible noise and missing information in the data. Additionally, in this way, the cross-modal discrepancy from intra-modal and inter-modal information are simultaneously eliminated in the joint embedding subspace. We conduct extensive experiments over two benchmark datasets. The experimental outcomes exhibit that the proposed architecture is effective in learning audio-visual correlation and is appreciably better than the existing cross-modal retrieval methods.

In this paper, we propose a technique to address the most challenging aspect of algorithmic songwriting process, which enables the human community to discover original lyrics, and melodies suitable for the generated lyrics. The proposed songwriting system, Automatic Neural Lyrics and Melody Composition (AutoNLMC) is an attempt to make the whole process of songwriting automatic using artificial neural networks. Our lyric to vector (lyric2vec) model trained on a large set of lyric-melody pairs dataset parsed at syllable, word and sentence levels are large scale embedding models enable us to train data driven model such as recurrent neural networks for popular English songs. AutoNLMC is a encoder-decoder sequential recurrent neural network model consisting of a lyric generator, a lyric encoder and melody decoder trained end-to-end. AutoNLMC is designed to generate both lyrics and corresponding melody automatically for an amateur or a person without music knowledge. It can also take lyrics from professional lyric writer to generate matching melodies. The qualitative and quantitative evaluation measures revealed that the proposed method is indeed capable of generating original lyrics and corresponding melody for composing new songs., Comment: 15 pages

IoT systems typically involve separate data collection and processing, and the former faces the scalability issue when the number of nodes increases. For some tasks, only the result of data fusion is needed. Then, the whole process can be realized in an efficient way, integrating the data collection and fusion in one step by over-the-air computation (AirComp). Its shortcoming, however, is signal distortion when channel gains of nodes are different, which cannot be well solved by transmission power control alone in times of deep fading. To address this issue, in this paper, we propose a multi-slot over-the-air computation (MS-AirComp) framework for the sum estimation in fading channels. Compared with conventional data collection (one slot for each node) and AirComp (one slot for all nodes), MS-AirComp is an alternative policy that lies between them, exploiting multiple slots to improve channel gains so as to facilitate power control. Specifically, the transmissions are distributed over multiple slots and a threshold of channel gain is set for distributed transmission scheduling. Each node transmits its signal only once, in the slot when its channel gain first gets above the threshold, or in the last slot when its channel gain remains below the threshold. Theoretical analysis gives the closed-form of the computation error in fading channels, based on which the optimal parameters are found. Noticing that computation error tends to be reduced at the cost of more transmission power, a method is suggested to control the increase of transmission power. Simulations confirm that the proposed method can effectively reduce computation error, compared with state-of-the-art methods.

In typical sensor networks, data collection and processing are separated. A sink collects data from all nodes sequentially, which is very time consuming. Over-the-air computation, as a new diagram of sensor networks, integrates data collection and processing in one slot: all nodes transmit their signals simultaneously in the analog wave and the processing is done in the air. This method, although efficient, requires that signals from all nodes arrive at the sink, aligned in signal magnitude so as to enable an unbiased estimation. For nodes far away from the sink with a low channel gain, misalignment in signal magnitude is unavoidable. To solve this problem, in this paper, we investigate the amplify-and-forward based relay, in which a relay node amplifies signals from many nodes at the same time. We first discuss the general relay model and a simple relay policy. Then, a coherent relay policy is proposed to reduce relay transmission power. Directly minimizing the computation error tends to over-increase node transmission power. Therefore, the two relay policies are further refined with a new metric, and the transmission power is reduced while the computation error is kept low. In addition, the coherent relay policy helps to reduce the relay transmission power by half, to below the limit, which makes it one step ahead towards practical applications.

In this paper, the impact of imperfect beam alignment (IBA) on millimeter wave (mmWave) wireless power transfer (WPT) is investigated. We consider a mmWave WPT network, where the location of the energy transmitters follows a Poisson point process. Instead of the mostly used flat-top antenna model, we adopt the Gaussian antenna model suggested by the 3rd Generation Partnership Project (3GPP) for better accuracy. Two beam alignment error (BAE) models, i.e., truncated Gaussian and uniform models, are introduced to represent different BAE sources. We derive the probability density functions (PDFs) of the cascaded antenna gain with both BAE models and then provide the approximated PDFs for tractability. With the help of Fox's H function, the analytic expression for the energy coverage probability with nonlinear energy harvesting model is derived. Besides, we deduce a closed-form expression of the average harvested radio frequency (RF) energy. The simulation results verify our theoretical results and demonstrate the performance degradation incurred by BAE. It also shows that the flat-top antenna model cannot always provide accurate performance evaluation in the presence of BAE., Comment: Published by IEEE TVT. Copyright (c) 2015 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieee.org

By developing sophisticated image priors or designing deep(er) architectures, a variety of image Super-Resolution (SR) approaches have been proposed recently and achieved very promising performance. A natural question that arises is whether these methods can be reformulated into a unifying framework and whether this framework assists in SR reconstruction? In this paper, we present a simple but effective single image SR method based on ensemble learning, which can produce a better performance than that could be obtained from any of SR methods to be ensembled (or called component super-resolvers). Based on the assumption that better component super-resolver should have larger ensemble weight when performing SR reconstruction, we present a Maximum A Posteriori (MAP) estimation framework for the inference of optimal ensemble weights. Specially, we introduce a reference dataset, which is composed of High-Resolution (HR) and Low-Resolution (LR) image pairs, to measure the super-resolution abilities (prior knowledge) of different component super-resolvers. To obtain the optimal ensemble weights, we propose to incorporate the reconstruction constraint, which states that the degenerated HR image should be equal to the LR observation one, as well as the prior knowledge of ensemble weights into the MAP estimation framework. Moreover, the proposed optimization problem can be solved by an analytical solution. We study the performance of the proposed method by comparing with different competitive approaches, including four state-of-the-art non-deep learning based methods, four latest deep learning based methods and one ensemble learning based method, and prove its effectiveness and superiority on three public datasets., Comment: 14 pages, 11 figures

Face hallucination is a technique that reconstruct high-resolution (HR) faces from low-resolution (LR) faces, by using the prior knowledge learned from HR/LR face pairs. Most state-of-the-arts leverage position-patch prior knowledge of human face to estimate the optimal representation coefficients for each image patch. However, they focus only the position information and usually ignore the context information of image patch. In addition, when they are confronted with misalignment or the Small Sample Size (SSS) problem, the hallucination performance is very poor. To this end, this study incorporates the contextual information of image patch and proposes a powerful and efficient context-patch based face hallucination approach, namely Thresholding Locality-constrained Representation and Reproducing learning (TLcR-RL). Under the context-patch based framework, we advance a thresholding based representation method to enhance the reconstruction accuracy and reduce the computational complexity. To further improve the performance of the proposed algorithm, we propose a promotion strategy called reproducing learning. By adding the estimated HR face to the training set, which can simulates the case that the HR version of the input LR face is present in the training set, thus iteratively enhancing the final hallucination result. Experiments demonstrate that the proposed TLcR-RL method achieves a substantial increase in the hallucinated results, both subjectively and objectively. Additionally, the proposed framework is more robust to face misalignment and the SSS problem, and its hallucinated HR face is still very good when the LR test face is from the real-world. The MATLAB source code is available at https://github.com/junjun-jiang/TLcR-RL, Comment: 13 pages, 15 figures, Accepted by IEEE TCYB

Most of the current face hallucination methods, whether they are shallow learning-based or deep learning-based, all try to learn a relationship model between Low-Resolution (LR) and High-Resolution (HR) spaces with the help of a training set. They mainly focus on modeling image prior through either model-based optimization or discriminative inference learning. However, when the input LR face is tiny, the learned prior knowledge is no longer effective and their performance will drop sharply. To solve this problem, in this paper we propose a general face hallucination method that can integrate model-based optimization and discriminative inference. In particular, to exploit the model based prior, the Deep Convolutional Neural Networks (CNN) denoiser prior is plugged into the super-resolution optimization model with the aid of image-adaptive Laplacian regularization. Additionally, we further develop a high-frequency details compensation method by dividing the face image to facial components and performing face hallucination in a multi-layer neighbor embedding manner. Experiments demonstrate that the proposed method can achieve promising super-resolution results for tiny input LR faces., Comment: Accepted by IJCAI 2018

In this work, travel destination and business location are taken as venues. Discovering a venue by a photo is very important for context-aware applications. Unfortunately, few efforts paid attention to complicated real images such as venue photos generated by users. Our goal is fine-grained venue discovery from heterogeneous social multimodal data. To this end, we propose a novel deep learning model, Category-based Deep Canonical Correlation Analysis (C-DCCA). Given a photo as input, this model performs (i) exact venue search (find the venue where the photo was taken), and (ii) group venue search (find relevant venues with the same category as that of the photo), by the cross-modal correlation between the input photo and textual description of venues. In this model, data in different modalities are projected to a same space via deep networks. Pairwise correlation (between different modal data from the same venue) for exact venue search and category-based correlation (between different modal data from different venues with the same category) for group venue search are jointly optimized. Because a photo cannot fully reflect rich text description of a venue, the number of photos per venue in the training phase is increased to capture more aspects of a venue. We build a new venue-aware multimodal dataset by integrating Wikipedia featured articles and Foursquare venue photos. Experimental results on this dataset confirm the feasibility of the proposed method. Moreover, the evaluation over another publicly available dataset confirms that the proposed method outperforms state-of-the-arts for cross-modal retrieval between image and text.

Modeling of music audio semantics has been previously tackled through learning of mappings from audio data to high-level tags or latent unsupervised spaces. The resulting semantic spaces are theoretically limited, either because the chosen high-level tags do not cover all of music semantics or because audio data itself is not enough to determine music semantics. In this paper, we propose a generic framework for semantics modeling that focuses on the perception of the listener, through EEG data, in addition to audio data. We implement this framework using a novel end-to-end 2-view Neural Network (NN) architecture and a Deep Canonical Correlation Analysis (DCCA) loss function that forces the semantic embedding spaces of both views to be maximally correlated. We also detail how the EEG dataset was collected and use it to train our proposed model. We evaluate the learned semantic space in a transfer learning context, by using it as an audio feature extractor in an independent dataset and proxy task: music audio-lyrics cross-modal retrieval. We show that our embedding model outperforms Spotify features and performs comparably to a state-of-the-art embedding model that was trained on 700 times more data. We further discuss improvements to the model that are likely to improve its performance., Comment: 5 pages, 2 figures

Little research focuses on cross-modal correlation learning where temporal structures of different data modalities such as audio and lyrics are taken into account. Stemming from the characteristic of temporal structures of music in nature, we are motivated to learn the deep sequential correlation between audio and lyrics. In this work, we propose a deep cross-modal correlation learning architecture involving two-branch deep neural networks for audio modality and text modality (lyrics). Different modality data are converted to the same canonical space where inter modal canonical correlation analysis is utilized as an objective function to calculate the similarity of temporal structures. This is the first study on understanding the correlation between language and music audio through deep architectures for learning the paired temporal correlation of audio and lyrics. Pre-trained Doc2vec model followed by fully-connected layers (fully-connected deep neural network) is used to represent lyrics. Two significant contributions are made in the audio branch, as follows: i) pre-trained CNN followed by fully-connected layers is investigated for representing music audio. ii) We further suggest an end-to-end architecture that simultaneously trains convolutional layers and fully-connected layers to better learn temporal structures of music audio. Particularly, our end-to-end deep architecture contains two properties: simultaneously implementing feature learning and cross-modal correlation learning, and learning joint representation by considering temporal structures. Experimental results, using audio to retrieve lyrics or using lyrics to retrieve audio, verify the effectiveness of the proposed deep correlation learning architectures in cross-modal music retrieval.

In this paper, we design a simple, low-cost, and low-power wake-up receiver which can be used for an IEEE 802.11-compliant device to remotely wake up the other devices by utilizing its own wireless LAN (WLAN) signals. The employed wake-up mechanism utilizes the length of 802.11 data frame generated by a WiFi transmitter to differentiate the information conveyed to the wake-up receiver. The wake-up receiver is designed to reliably detect the length of transmitted data frame only with simple envelope detection and limited signal processing. We develop a prototype of the wake-up receiver and investigate the detection performance of the envelope of 802.11 signals. Our numerical results show that the proposed wake-up receiver achieves much larger detection range than the off-the-shelf, commercial receiver having the similar functionality., Comment: 6 pages, 10 figures, to be presented at IEEE Globecom 2012