Search

In this paper, we study a hypothesis test to determine the underlying directed graph structure of nodes in a network, where the nodes represent random processes and the direction of the links indicate a causal relationship between said processes. Specifically, a k-th order Markov structure is considered for them, and the chosen metric to determine a connection between nodes is the directed information. The hypothesis test is based on the empirically calculated transition probabilities which are used to estimate the directed information. For a single edge, it is proven that the detection probability can be chosen arbitrarily close to one, while the false alarm probability remains negligible. When the test is performed on the whole graph, we derive bounds for the false alarm and detection probabilities, which show that the test is asymptotically optimal by properly setting the threshold test and using a large number of samples. Furthermore, we study how the convergence of the measures relies on the existence of links in the true graph., Comment: 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2017

This work presents several expected generalization error bounds based on the Wasserstein distance. More specifically, it introduces full-dataset, single-letter, and random-subset bounds, and their analogues in the randomized subsample setting from Steinke and Zakynthinou [1]. Moreover, when the loss function is bounded and the geometry of the space is ignored by the choice of the metric in the Wasserstein distance, these bounds recover from below (and thus, are tighter than) current bounds based on the relative entropy. In particular, they generate new, non-vacuous bounds based on the relative entropy. Therefore, these results can be seen as a bridge between works that account for the geometry of the hypothesis space and those based on the relative entropy, which is agnostic to such geometry. Furthermore, it is shown how to produce various new bounds based on different information measures (e.g., the lautum information or several $f$-divergences) based on these bounds and how to derive similar bounds with respect to the backward channel using the presented proof techniques., Comment: 29 pages: 9 of the main text, 3 of references, and 17 of appendices. Presented at ITR3 at ICML 2021. Accepted at NeurIPS 2021

In an intelligent transportation system, the effects and relations of traffic flow at different points in a network are valuable features which can be exploited for control system design and traffic forecasting. In this paper, we define the notion of causality based on the directed information, a well-established data-driven measure, to represent the effective connectivity among nodes of a vehicular traffic network. This notion indicates whether the traffic flow at any given point affects another point's flow in the future and, more importantly, reveals the extent of this effect. In contrast with conventional methods to express connections in a network, it is not limited to linear models and normality conditions. In this work, directed information is used to determine the underlying graph structure of a network, denoted directed information graph, which expresses the causal relations among nodes in the network. We devise an algorithm to estimate the extent of the effects in each link and build the graph. The performance of the algorithm is then analyzed with synthetic data and real aggregated data of vehicular traffic.

In this work, we unify several expected generalization error bounds based on random subsets using the framework developed by Hellstr\"om and Durisi [1]. First, we recover the bounds based on the individual sample mutual information from Bu et al. [2] and on a random subset of the dataset from Negrea et al. [3]. Then, we introduce their new, analogous bounds in the randomized subsample setting from Steinke and Zakynthinou [4], and we identify some limitations of the framework. Finally, we extend the bounds from Haghifam et al. [5] for Langevin dynamics to stochastic gradient Langevin dynamics and we refine them for loss functions with potentially large gradient norms., Comment: To appear in the Information Theory Workshop (ITW 2020) conference. 10 pages, 5 of the main text, and 5 of appendices

We consider a network of two nodes separated by a noisy channel, in which the source and its reconstruction have to be strongly coordinated, while simultaneously satisfying the strong secrecy condition with respect to an outside observer of the noisy channel. In the case of non-causal encoding and decoding, we propose a joint source-channel coding scheme for the secure strong coordination region. Furthermore, we provide a complete characterization of the secure strong coordination region when the decoder has to reliably reconstruct the source sequence and the legitimate channel is more capable than the channel of the eavesdropper.

The estimation of mutual information (MI) or conditional mutual information (CMI) from a set of samples is a long-standing problem. A recent line of work in this area has leveraged the approximation power of artificial neural networks and has shown improvements over conventional methods. One important challenge in this new approach is the need to obtain, given the original dataset, a different set where the samples are distributed according to a specific product density function. This is particularly challenging when estimating CMI. In this paper, we introduce a new technique, based on k nearest neighbors (k-NN), to perform the resampling and derive high-confidence concentration bounds for the sample average. Then the technique is employed to train a neural network classifier and the CMI is estimated accordingly. We propose three estimators using this technique and prove their consistency, make a comparison between them and similar approaches in the literature, and experimentally show improvements in estimating the CMI in terms of accuracy and variance of the estimators.

In this work, we study the generalization capability of algorithms from an information-theoretic perspective. It has been shown that the expected generalization error of an algorithm is bounded from above by a function of the relative entropy between the conditional probability distribution of the algorithm's output hypothesis, given the dataset with which it was trained, and its marginal probability distribution. We build upon this fact and introduce a mathematical formulation to obtain upper bounds on this relative entropy. Assuming that the data is discrete, we then develop a strategy using this formulation, based on the method of types and typicality, to find explicit upper bounds on the generalization error of stable algorithms, i.e., algorithms that produce similar output hypotheses given similar input datasets. In particular, we show the bounds obtained with this strategy for the case of $\epsilon$-DP and $\mu$-GDP algorithms., Comment: 18 pages (double column), 4 figures, accepted at IEEE Transactions on Information Theory

Several recent works in communication systems have proposed to leverage the power of neural networks in the design of encoders and decoders. In this approach, these blocks can be tailored to maximize the transmission rate based on aggregated samples from the channel. Motivated by the fact that, in many communication schemes, the achievable transmission rate is determined by a conditional mutual information term, this paper focuses on neural-based estimators for this information-theoretic quantity. Our results are based on variational bounds for the KL-divergence and, in contrast to some previous works, we provide a mathematically rigorous lower bound. However, additional challenges with respect to the unconditional mutual information emerge due to the presence of a conditional density function which we address here., Comment: To be presented at ICASSP 2020

In this paper, we study the maximum likelihood estimate of the probability mass function (pmf) of $n$ independent and identically distributed (i.i.d.) random variables, in the non-asymptotic regime. We are interested in characterizing the Neyman--Pearson criterion, i.e., the log-likelihood ratio for testing a true hypothesis within a larger hypothesis. Wilks' theorem states that this ratio behaves like a $\chi^2$ random variable in the asymptotic case; however, less is known about the precise behavior of the ratio when the number of samples is finite. In this work, we find an explicit bound for the difference between the cumulative distribution function (cdf) of the log-likelihood ratio and the cdf of a $\chi^2$ random variable. Furthermore, we show that this difference vanishes with a rate of order $1/\sqrt{n}$ in accordance with Wilks' theorem., Comment: 30 pages, 1 figure, submitted

This paper investigates the problem of secret key generation over a wiretap channel when the terminals observe correlated sources. These sources are independent of the main channel and the users overhear them before the transmission takes place. A novel achievable scheme is proposed, and its optimality is shown under certain less-noisy conditions. This result improves upon the existing literature where the more stringent condition of degradedness is required. Furthermore, numerical evaluation of the proposed scheme and previously reported results for a binary model are presented; a comparison of the numerical bounds provides insights on the benefit of the novel scheme., Comment: 16 pages, 5 figures. Submitted to IEEE Transactions on Information Theory (revised version)

It is a well-known fact that feedback does not increase the capacity of point-to-point memoryless channels, however, its effect in secure communications is not fully understood yet. In this work, an achievable scheme for the wiretap channel with generalized feedback is presented. This scheme, which uses the feedback signal to generate a shared secret key between the legitimate users, encrypts the message to be sent at the bit level. New capacity results for a class of channels are provided, as well as some new insights into the secret key agreement problem. Moreover, this scheme recovers previously reported rate regions from the literature, and thus it can be seen as a generalization that unifies several results in the field., Comment: 20 pages, 3 figures. Submitted to IEEE Transactions on Information Theory (accepted version)

The capacity of a class of Interference Relay Channels (IRC) -the Injective Semideterministic IRC where the relay can only observe one of the sources- is investigated. We first derive a novel outer bound and two inner bounds which are based on a careful use of each of the available cooperative strategies together with the adequate interference decoding technique. The outer bound extends Telatar and Tse's work while the inner bounds contain several known results in the literature as special cases. Our main result is the characterization of the capacity region of the Gaussian class of IRCs studied within a fixed number of bits per dimension -constant gap. The proof relies on the use of the different cooperative strategies in specific SNR regimes due to the complexity of the schemes. As a matter of fact, this issue reveals the complex nature of the Gaussian IRC where the combination of a single coding scheme for the Gaussian relay and interference channel may not lead to a good coding scheme for this problem, even when the focus is only on capacity to within a constant gap over all possible fading statistics., Comment: 23 pages, 6 figures. Submitted to IEEE Transactions on Information Theory (revised version)