Search

We propose a new approximation method for Gaussian process (GP) learning for large data sets that combines inline active set selection with hyperparameter optimization. The predictive probability of the label is used for ranking the data points. We use the leave-one-out predictive probability available in GPs to make a common ranking for both active and inactive points, allowing points to be removed again from the active set. This is important for keeping the complexity down and at the same time focusing on points close to the decision boundary. We lend both theoretical and empirical support to the active set selection strategy and marginal likelihood optimization on the active set. We make extensive tests on the USPS and MNIST digit classification databases with and without incorporating invariances, demonstrating that we can get state-of-the-art results (e.g.0.86% error on MNIST) with reasonable time complexity.

The molecular pathways leading to thyroid follicular neoplasia are incompletely understood, and the diagnosis of follicular tumors is a clinical challenge. To provide leads to the pathogenesis and diagnosis of the tumors, we examined the global transcriptome signatures of follicular thyroid carcinoma (FC) and normofollicular adenoma (FA) as well as fetal/microFA (fetal adenoma). Carcinomas were strongly enriched in transcripts encoding proteins involved in DNA replication and mitosis corresponding to increased number of proliferating cells and depleted number of transcripts encoding factors involved in growth arrest and apoptosis. In the latter group, the combined loss of transcripts encoding the nuclear orphan receptors NR4A1 and NR4A3, which were recently shown to play a causal role in hematopoetic neoplasia, was noteworthy. The analysis of differentially expressed transcripts provided a mechanism for cancer progression, which is why we exploited the results in order to generate a molecular classifier that could identify 95% of all carcinomas. Validation employing public domain and cross-platform data demonstrated that the signature was robust and could diagnose follicular nodules originating from different geographical locations and platforms with similar accuracy. We came to the conclusion that down-regulation of factors involved in growth arrest and apoptosis may represent a decisive step in the pathogenesis of FC. Moreover, the described molecular pathways provide an accurate and robust genetic signature for the diagnosis of FA and FC.

In this paper we consider sparse and identifiable linear latent variable (factor) and linear Bayesian network models for parsimonious analysis of multivariate data. We propose a computationally efficient method for joint parameter and model inference, and model comparison. It consists of a fully Bayesian hierarchy for sparse models using slab and spike priors (two-component delta-function and continuous mixtures), non-Gaussian latent factors and a stochastic search over the ordering of the variables. The framework, which we call SLIM (Sparse Linear Identifiable Multivariate modeling), is validated and bench-marked on artificial and real biological data sets. SLIM is closest in spirit to LiNGAM (Shimizu et al., 2006), but differs substantially in inference, Bayesian network structure learning and model comparison. Experimentally, SLIM performs equally well or better than LiNGAM with comparable computational complexity. We attribute this mainly to the stochastic search strategy used, and to parsimony (sparsity and identifiability), which is an explicit part of the model. We propose two extensions to the basic i.i.d. linear framework: non-linear dependence on observed variables, called SNIM (Sparse Non-linear Identifiable Multivariate modeling) and allowing for correlations between latent variables, called CSLIM (Correlated SLIM), for the temporal and/or spatial data. The source code and scripts are available from http://cogsys.imm.dtu.dk/slim/., 45 pages, 17 figures

This paper introduces a new temporal version of Principal Component Analysis by using a Hidden Markov Model in order to obtain optimized representations of observed data through time. The novelty of the proposed method consists mainly in the way in which a static dimensionality reduction technique has been combined with a classic mixture model in time, to enhance the capabilities of dimensionality reduction and classification of myocardial ischemia data. Experimental results show improvements in classification accuracies even with highly reduced representations.

This paper introduces a temporal version of Probabilistic Kernel Principal Component Analysis by using a hidden Markov model in order to obtain optimized representations of observed data through time. Recently introduced, Probabilistic Kernel Principal Component Analysis overcomes the two main disadvantages of standard Principal Component Analysis, namely, absence of probability density model and lack of high-order statistical information due to its linear structure. We extend this probabilistic approach of KPCA to mixture models in time, to enhance the capabilities of transformation and reduction of time series vectors. Results over voice disorder databases show improvements in classification accuracies even with highly reduced representations.

We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching. Within a framework of conditional entropy, we propose both adversarial and non-adversarial approaches to learn desirable matched joint distributions for unsupervised and supervised tasks. We unify a broad family of adversarial models as joint distribution matching problems. Our approach stabilizes learning of unsupervised bidirectional adversarial learning methods. Further, we introduce an extension for semi-supervised learning tasks. Theoretical results are validated in synthetic data and real-world applications., NIPS 2017 (22 pages); short version (9 pages): http://people.duke.edu/~cl319/doc/papers/nips_2017_alice.pdf

Kernel Principal Component analysis is a non- linear generalization of the popular linear multivariate analysis method. However, this method assumes that the observed data is independent, a disadvantage for many practical applications. In order to overcome this difficulty, the authors propose a com- bination of Kernel Principal Component analysis and hidden Markov models. The novelty of the proposed method consists mainly in the way in which a static dimensionality reduction technique has been combined with a classic mixture model in time, to enhance the capabilities of transformation, reduction and classification of voice disorder data. Experimental results show improvements in classification accuracies even with highly reduced representations of the two databases used. In this paper a temporal version of KPCA is introduced by using a hidden Markov model (HMM) as a way to obtain an optimized representation of the observed data through time. With this scheme, every data point has an associated local representation corresponding to the most probable state produced by a trained HMM. The integration between HMM and KPCA was done by incorporating a factorization of the training data and the responsibility weights in the covariance of the observation model in the HMM in such way that KPCA can be readily calculated. This paper is organized as follows: sections II and III contain reviews of KPCA and HMM respectively. In section IV an extension for KPCA through time is presented. In section V experimental results for voice disorder databases are reported and finally in section VI the conclusions of the entire work.

A novel variational autoencoder is developed to model images, as well as associated labels or captions. The Deep Generative Deconvolutional Network (DGDN) is used as a decoder of the latent image features, and a deep Convolutional Neural Network (CNN) is used as an image encoder; the CNN is used to approximate a distribution for the latent DGDN features/code. The latent code is also linked to generative models for labels (Bayesian support vector machine) or captions (recurrent neural network). When predicting a label/caption for a new image at test, averaging is performed across the distribution of latent codes; this is computationally efficient as a consequence of the learned CNN-based encoder. Since the framework is capable of modeling the image in the presence/absence of associated labels/captions, a new semi-supervised setting is manifested for CNN learning with images; the framework even allows unsupervised CNN learning, based on images alone., Comment: NIPS 2016 (To appear)

Background Cancers of unknown primary (CUPs) constitute ~5% of all cancers. The tumors have an aggressive biological and clinical behavior. The aim of the present study has been to uncover whether CUPs exhibit distinct molecular features compared to metastases of known origin. Methods Employing genome wide transcriptome analysis, Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA), we defined the putative origins of a large series of CUP and how closely related a particular CUP was to corresponding metastases of known origin. LDA predictions were subsequently used to define a universal CUP core set of differentially expressed genes, that by means of gene set enrichment analysis was exploited to depict molecular pathways characterizing CUP. Results The analyses show that CUPs are distinct from metastases of known origin. CUPs exhibit inconsistent expression of conventional cancer biomarkers and QDA derived outlier scores show that CUPs are more distantly related to their primary tumor class than corresponding metastases of known origin. Gene set enrichment analysis showed that CUPs display increased expression of genes involved in DNA damage repair and mRNA signatures of chromosome instability (CIN), indicating that CUPs are chromosome unstable compared to metastases of known origin. Conclusions CIN may account for the uncommon clinical presentation, chemoresistance and poor outcome in patients with CUP and warrant selective diagnostic strategies and treatment. Electronic supplementary material The online version of this article (doi:10.1186/s12885-015-1128-x) contains supplementary material, which is available to authorized users.

The Generative Adversarial Network (GAN) has achieved great success in generating realistic (real-valued) synthetic data. However, convergence issues and difficulties dealing with discrete data hinder the applicability of GAN to text. We propose a framework for generating realistic text via adversarial training. We employ a long short-term memory network as generator, and a convolutional network as discriminator. Instead of using the standard objective of GAN, we propose matching the high-dimensional latent feature distributions of real and synthetic sentences, via a kernelized discrepancy metric. This eases adversarial training by alleviating the mode-collapsing problem. Our experiments show superior performance in quantitative evaluation, and demonstrate that our model can generate realistic-looking sentences., Comment: Accepted by ICML 2017

We unify slice sampling and Hamiltonian Monte Carlo (HMC) sampling, demonstrating their connection via the Hamiltonian-Jacobi equation from Hamiltonian mechanics. This insight enables extension of HMC and slice sampling to a broader family of samplers, called Monomial Gamma Samplers (MGS). We provide a theoretical analysis of the mixing performance of such samplers, proving that in the limit of a single parameter, the MGS draws decorrelated samples from the desired target distribution. We further show that as this parameter tends toward this limit, performance gains are achieved at a cost of increasing numerical difficulty and some practical convergence issues. Our theoretical results are validated with synthetic data and real-world applications., Comment: updated version

A latent-variable model is introduced for text matching, inferring sentence representations by jointly optimizing generative and discriminative objectives. To alleviate typical optimization challenges in latent-variable models for text, we employ deconvolutional networks as the sequence decoder (generator), providing learned latent codes with more semantic information and better generalization. Our model, trained in an unsupervised manner, yields stronger empirical predictive performance than a decoder based on Long Short-Term Memory (LSTM), with less parameters and considerably faster training. Further, we apply it to text sequence-matching problems. The proposed model significantly outperforms several strong sentence-encoding baselines, especially in the semi-supervised setting., Comment: Accepted by AAAI-2018

Recent advances in stochastic gradient techniques have made it possible to estimate posterior distributions from large datasets via Markov Chain Monte Carlo (MCMC). However, when the target posterior is multimodal, mixing performance is often poor. This results in inadequate exploration of the posterior distribution. A framework is proposed to improve the sampling efficiency of stochastic gradient MCMC, based on Hamiltonian Monte Carlo. A generalized kinetic function is leveraged, delivering superior stationary mixing, especially for multimodal distributions. Techniques are also discussed to overcome the practical issues introduced by this generalization. It is shown that the proposed approach is better at exploring complex multimodal posterior distributions, as demonstrated on multiple applications and in comparison with other stochastic gradient MCMC methods., Published on ICML 2017