Search

We consider the problem of reconstructing a vector x 0 ∊ Rn from noisy linear observations y = Ax o + w, where A ∊ Rm×n is a known operator and w is a noise vector, using the elastic net method. Assuming that A is random with independent and identically distributed entries, and under suitable moment conditions, we prove the following universality result. In the high-dimensional asymptotics n→∞ and m/n → δ > 0, the normalized error of the elastic net minimizer converges in probability to a limit, that does not depend on the exact distribution that the entries are drawn from. We also provide an explicit formula for the limit.

We develop an information-theoretic view of the stochastic block model, a popular statistical model for the large-scale structure of complex networks. A graph G from such a model is generated by first assigning vertex labels at random from a finite alphabet, and then connecting vertices with edge probabilities depending on the labels of the endpoints. In the case of the symmetric two-group model, we establish an explicit ‘single-letter’ characterization of the per-vertex mutual information between the vertex labels and the graph, when the graph average degree diverges. The explicit expression of the mutual information is intimately related to estimation-theoretic quantities, and -in particular- reveals a phase transition at the critical point for community detection. Below the critical point the per-vertex mutual information is asymptotically the same as if edges were independent of the vertex labels. Correspondingly, no algorithm can estimate the partition better than random guessing. Conversely, above the threshold, the per-vertex mutual information is strictly smaller than the independent-edges upper bound. In this regime there exists a procedure that estimates the vertex labels better than random guessing.

This paper studies a class of channels obtained by combining a graph and a memoryless channel. A particular example is a memoryless channel pre-coded with a graph-based codes. Other examples are planted constrained satisfaction problems, and probabilistic network models with communities. In an earlier work by the authors, concentration results are obtained for such channels when the graphs is a sparse Erdos-Renyi graph. We overview in this note the approach in an information-theoretic framework, describe how it establishes a conjecture on parity-check codes, and connect the coding and community detection problems.

We study a simple modification to the conventional time of flight mass spectrometry (TOFMS) where a \emph{variable} and (pseudo)-\emph{random} pulsing rate is used which allows for traces from different pulses to overlap. This modification requires little alteration to the currently employed hardware. However, it requires a reconstruction method to recover the spectrum from highly aliased traces. We propose and demonstrate an efficient algorithm that can process massive TOFMS data using computational resources that can be considered modest with today's standards. This approach can be used to improve duty cycle, speed, and mass resolving power of TOFMS at the same time. We expect this to extend the applicability of TOFMS to new domains., Comment: 14 pages, 18 figures. This paper is submitted to IEEE Transaction on Signal Processing

We consider a class of nonlinear mappings F A, N in RN indexed by symmetric random matrices A ∊ RN×N with independent entries. Within spin glass theory, special cases of these mappings correspond to iterating the TAP equations and were studied by Erwin Bolthausen. Within information theory, they are known as ‘approximate message passing’ algorithms. We study the high-dimensional (large N) behavior of the iterates of F for polynomial functions F, and prove that it is universal, i.e. it depends only on the first two moments of the entries of A. As an application, we prove the universality of a certain phase transition arising in polytope geometry and compressed sensing. This solves a conjecture by David Donoho and Jared Tanner.

We consider the compressed sensing problem where the object x 0 ∈ ℝN is to be recovered from incomplete measurements y = Ax 0 +z. Here the sensing matrix A is an n×N random matrix with Gaussian entries and n < N. A popular method of sparsity-promoting reconstruction is l1-penalized least-squares reconstruction (aka LASSO, Basis Pursuit).

The basic question in matrix completion is to infer a large low-rank matrix from a small subset of its entries. Positioning refers to the task of inferring the locations of n points from a subset of their distance. It turns out that positioning can be viewed as a matrix completion problem, although of a peculiar type. This paper discusses the applicability of distributed matrix completion algorithms to the positioning problem.

We consider the problem of reconstructing a low rank matrix from noisy observations of a subset of its entries. This task has applications in statistical learning, computer vision, and signal processing. In these contexts, "noise" generically refers to any contribution to the data that is not captured by the low-rank model. In most applications, the noise level is large compared to the underlying signal and it is important to avoid overfitting. In order to tackle this problem, we define a regularized cost function well suited for spectral reconstruction methods. Within a random noise model, and in the large system limit, we prove that the resulting accuracy undergoes a phase transition depending on the noise level and on the fraction of observed entries. The cost function can be minimized using OPTSPACE (a manifold gradient descent algorithm). Numerical simulations show that this approach is competitive with state-of-the-art alternatives., Comment: 5 pages, 3 figures, Conference Version

Finding fast recovery algorithms is a problem of significant interest in compressed sensing. In many applications, extremely large problem sizes are envisioned, with at least tens of thousands of equations and hundreds of thousands of unknowns. The interior point methods for solving the best studied algorithm -l 1 minimization- are very slow and hence they are ineffective for such problem sizes. Faster methods such as LARS or homotopy [1] have been proposed for solving l 1 minimization but there is still a need for algorithms with less computational complexity. Therefore there is a fast growing literature on greedy methods and iterative thresholding for finding the sparsest solution. It was shown by [2] that most of these algorithms perform worse than `1 when it comes to the sparsity-measurement tradeoff [2]. In a recent paper the authors introduced a new algorithm called AMP which is as fast as iterative soft thresholding algorithms proposed before and performs exactly the same as l 1 minimization[3]. This algorithm is inspired from the message passing algorithms on bipartite graphs. The statistical properties of AMP let the authors propose a theoretical framework to analyze the asymptotic performance of the algorithm. This results in very sharp predictions of different observables in the algorithm. In this paper we address several questions regarding the thresholding policy. We consider a very general thresholding policy λ(σ) for the algorithm and will use the maxmin framework to tune it optimally. It is shown that when formulated in this general form, the maxmin thresholding policy is not unique and many different thresholding policies may lead to the same phase transition. We will then propose two very simple thresholding policies that can be implemented easily in practice and prove that they are both maxmin. This analysis will also shed some light on several other aspects of the algorithm, such as the least favorable distribution and similarity of all maxmin optimal thresholding policies. We will also show how one can derive the AMP algorithm from the full message passing algorithm.

We present a new lossy compression algorithm for discrete sources. The encoder assigns a certain cost to each reconstruction sequence, finds the sequence that minimizes the cost, and describes it losslessly to the decoder via a universal lossless compressor. The cost of a sequence is defined as a linear combination of its empirical probabilities of some order k + 1 and its distortion relative to the source sequence. The linear structure of the cost in the empirical count matrix allows the encoder to employ a Viterbi-like algorithm for obtaining the minimizing reconstruction sequence simply. We identify a choice of coefficients for the linear combination in the cost function which ensures that the algorithm universally achieves the optimum rate-distortion performance of any Markov source in the limit of large n, provided k is increased as o(log n). Finding the optimal coefficients is complex and requires solving a non-convex optimization problem. As a detour, we propose a simple heuristic iterative procedure, and demonstrate its efficiency through our experimental results.

We present a new lossy compressor for discrete sources. For coding a source sequence $x^n$, the encoder starts by assigning a certain cost to each reconstruction sequence. It then finds the reconstruction that minimizes this cost and describes it losslessly to the decoder via a universal lossless compressor. The cost of a sequence is given by a linear combination of its empirical probabilities of some order $k+1$ and its distortion relative to the source sequence. The linear structure of the cost in the empirical count matrix allows the encoder to employ a Viterbi-like algorithm for obtaining the minimizing reconstruction sequence simply. We identify a choice of coefficients for the linear combination in the cost function which ensures that the algorithm universally achieves the optimum rate-distortion performance of any Markov source in the limit of large $n$, provided $k$ is increased as $o(\log n)$., Comment: 10 pages, 2 figures, Data Compression Conference (DCC) 2009

We present a simple and efficient algorithm for randomly generating Tanner-graphs with given symbol-node and check-node degrees and without small cycles. These graphs can be used to design high performance Low-Density Parity-Check (LDPC) codes. Our algorithm generates a graph by sequentially adding the edges to an empty graph. Recently, these types of sequential methods for counting and random generation have been very successful [35], [18], [11], [7], [4], [6].

A novel counter architecture, called Counter braids, has recently been proposed for per-flow counting on high-speed links. Counter braids has a layered structure and compresses the flow sizes as it counts. It has been shown that with a maximum likelihood (ML) decoding algorithm, the number of bits needed to store the size of a flow matches the entropy lower bound. As ML decoding is too complex to implement, an efficient message passing decoding algorithm has been proposed for practical purposes. The layers of Counter Braids are decoded sequentially, from the most significant to the least significant bits. In each layer, the message passing decoder solves a sparse signal recovery problem. In this paper we analyze the threshold dimensionality reduction rate (d-rate) of the message passing algorithm, and prove that it is correctly predicted by density evolution. Given a signal in R+ n with ne non-vanishing entries, we prove that one layer of Counter Braids can reduce its dimensionality by a factor 2.08 epsi log(1/epsi) + O(epsi). This essentially matches the rate for sparse signal recovery via L1 minimization, while keeping the overall complexity linear in n.

A data compression scheme is defined to be smooth if its image (the codeword) depends gracefully on the source (the data). Smoothness is a desirable property in many practical contexts, and widely used source coding schemes lack of it. We introduce a family of smooth source codes based on sparse graph constructions, and prove them to achieve the (information theoretic) optimal compression rate for a dense set of iid sources. As a byproduct, we show how Gallager bound on sparsity can be overcome using non-linear function nodes.

Scaling laws are a powerful way to analyze the performance of moderately sized iteratively decoded sparse graph codes. Our aim is to provide an easily usable finite-length optimization tool that is applicable to the wide variety of channels, blocklengths, error probability requirements, and decoders that one encounters for practical systems. The tool is aimed at non-experts in the field, who need to quickly find code designs that are comparable with the best known codes available today but do not have the luxury of spending months in doing so. In previous work we have shown how to compute scaling parameters for transmission over the binary erasure channel, as well as general channels and general quantized message-passing decoders when applied to regular ensembles. In this paper we show how to compute the message variance for a fixed number of iterations for irregular low-density parity-check ensembles. From these calculations the basic scaling parameter alpha can be deduced by determining the leading term of the limiting expression when the number of iterations tends to infinity and the channel parameter approaches the density evolution threshold.

We investigate the finite-length scaling methodology for irregular LDPC code ensembles when transmission takes place over the binary erasure channel (BEC). We first show how the necessary computations, namely the covariance evolution and the computation of the finite-length shift, can be accomplished in the irregular case. We then investigate how the obtained approximation can be used to predict the performance of irregular code ensembles and to optimize the degree distributions for finite-length codes.

We present a new approach to the analysis of codes on graphs under symbol-maximum a posteriori probability (MAP) decoding. The approach is based on Guerra's interpolation technique for spin glasses. The basic idea is to consider a smooth interpolation between the original decoding problem, and a much simpler system in which each bit is retransmitted without coding through a (different) "effective" channel. Quite interestingly, the resulting expressions are strictly related to the density-evolution analysis of belief propagation decoding. The fundamental quantities involved in the interpolation procedure are, for instance, the same as in density evolution, i.e. densities of messages. Motivated by heuristic statistical mechanics results, we conjecture our bounds to be tight.

In this paper, we address the decision problem for a system of monadic second-order logic interpreted over an /spl omega/-layered temporal structure devoid of both a finest layer and a coarsest one (we call such a structure totally unbounded). We propose an automaton-theoretic method that solves the problem in two steps: first, we reduce the considered problem to the problem of determining, for any given Rabin tree automaton, whether it accepts a fixed vertex-colored tree; then, we exploit a suitable notion of tree equivalence to reduce the latter problem to the decidable case of regular trees.

Using scaling laws, we obtain estimates of the block error probability of polar codes under successive cancellation decoding. For the binary erasure channel we present an upper and a lower bound for the scaling parameter. Numerically these two bounds match. We also present a scaling law for general binary discrete memoryless channels.