Search

In the robust secure aggregation problem, a server wishes to learn and only learn the sum of the inputs of a number of users while some users may drop out (i.e., may not respond). The identity of the dropped users is not known a priori and the server needs to securely recover the sum of the remaining surviving users. We consider the following minimal two-round model of secure aggregation. Over the first round, any set of no fewer than $U$ users out of $K$ users respond to the server and the server wants to learn the sum of the inputs of all responding users. The remaining users are viewed as dropped. Over the second round, any set of no fewer than $U$ users of the surviving users respond (i.e., dropouts are still possible over the second round) and from the information obtained from the surviving users over the two rounds, the server can decode the desired sum. The security constraint is that even if the server colludes with any $T$ users and the messages from the dropped users are received by the server (e.g., delayed packets), the server is not able to infer any additional information beyond the sum in the information theoretic sense. For this information theoretic secure aggregation problem, we characterize the optimal communication cost. When $U\leq T$ , secure aggregation is not feasible, and when $U > T$ , to securely compute one symbol of the sum, the minimum number of symbols sent from each user to the server is 1 over the first round, and $1/(U-T)$ over the second round.

This article is concerned with a distributed filtering problem for Markov jump systems subject to the measurement loss with unknown probabilities. A centralized robust Kalman filter is designed by using variational Bayesian methods and a modified interacting multiple model method based on information theory (IT-IMM). Then, a distributed robust Kalman filter based on the centralized filter and a hybrid consensus method called hybrid consensus on measurement and information (HCMCI) is designed. Moreover, boundedness of the estimation errors and the estimation error covariances are studied for the distributed robust Kalman filter.

Wide-area measurement systems (WAMS) are efficient tools to estimate the fault location in the large-scale power systems. In this study, a fast and accurate deep learning based structure is proposed to detect the faulty lines and locate the fault based on the measured data of WAMS including phasor measurement units. To this end, a modulated gated recurrent unit deep learning based network named improved gated recurrent neural network has been proposed. The designed network, which has a recurrent neural network based structure, benefits from one update gate that can capture sudden changes in transient voltage and current signals and also enhance computational efficiency. Furthermore, to enhance robustness of the designed network under noisy conditions, a new informative-based loss function is formulated. The proposed loss function utilizes generalized form of mutual information to enhance robustness of designed network in the presence of Gaussian and non-Gaussian noises. The numerical results on IEEE 68-bus system verifies the effectiveness and superiority of the proposed method considering several operational conditions and comparison by two combined shallow-based networks and one deep learning based networks. Post-print / Final draft

Nature has always inspired the human spirit and scientists frequently developed new methods based on observations from nature. Recent advances in imaging and sensing technology allow fascinating insights into biological neural processes. With the objective of finding new strategies to enhance the learning capabilities of neural networks, we focus on a phenomenon that is closely related to learning tasks and neural stability in biological neural networks, called homeostatic plasticity. Among the theories that have been developed to describe homeostatic plasticity, synaptic scaling has been found to be the most mature and applicable. We systematically discuss previous studies on the synaptic scaling theory and how they could be applied to artificial neural networks. Therefore, we utilize information theory to analytically evaluate how mutual information is affected by synaptic scaling. Based on these analytic findings, we propose two flavors in which synaptic scaling can be applied in the training process of simple and complex, feedforward, and recurrent neural networks. We compare our approach with state-of-the-art regularization techniques on standard benchmarks. We found that the proposed method yields the lowest error in both regression and classification tasks compared to previous regularization approaches in our experiments across a wide range of network feedforward and recurrent topologies and data sets.

In the free online game industry, churn prediction is an important research topic. Reducing the churn rate of a game significantly helps with the success of the game. Churn prediction helps a game operator identify possible churning players and keep them engaged in the game via appropriate operational strategies, marketing strategies, and/or incentives. Playtime related features are some of the widely used universal features for most churn prediction models. In this paper, we consider developing new universal features for churn predictions for long-term players based on players' playtime. In particular, we measure playtime regularity using the notion of entropy and cross-entropy from information theory. After we calculate the playtime regularity of players from data sets of six free online games of different types. We leverage information from players' playtime regularity in the form of universal features for churn prediction. Experiments show that our developed features are better at predicting churners compared to baseline features. Thus, the experiment results imply that our proposed features could utilize the information extracted from players' playtime more effectively than related baseline playtime features.

Reed-Muller codes are error-correcting codes used in many areas related to coding theory, such as electrical engineering and computer science. The binary rth-order Reed-Muller code RM(r, n) can be viewed as the set of all n-variable Boolean functions of algebraic degree at most r. Despite the intense work on these codes, many problems are known to be hard (notably, determining their covering radius) and remain open to this day. Fourteen years ago, Carlet and Mesnager improved in [IEEE Transactions on Information Theory, “Improving the Upper Bounds on the Covering Radii of Binary Reed-Muller Codes”, 53(1), 2007] the upper bound on the covering radius of the Reed-Muller code of order 2, and they deduced improved upper bounds on the covering radii of the Reed-Muller codes of higher orders. Until 2021, these upper bounds remain the best ones in the literature. The Reed-Muller code RM(n -3, n), which corresponds to the dual of the Reed-Muller code RM(2, n), has attracted much attention. One of the main reasons is that it is precisely the code that has been considered to get the upper bounds derived by Carlet and Mesnager. Those upper bounds have been obtained thanks to the characterization of the codewords of the Reed-Muller code, whose Hamming weights are strictly less than 2.5 times the minimum distance 2n-r due to Kasami, Tokura, and Azumi. Despite their impressive work in the seventieth, a more refined study and profound description of those codewords of RM(n -3, n) whose Hamming weight equals 16, and especially 18, seem necessary, as it could help us significantly in improving the covering radius of Reed-Muller codes. In this paper, we push further the known results on the Reed-Muller codes by focusing on the Reed-Muller code RM(n -3, n). We provide a classification of the codewords of weight 16 and 18 of the Reed-Muller code RM(n -3, n). Our algebraic descriptions allow us to count the number of such codewords and to enumerate all of them explicitly.

We study the role of interactivity in distributed statistical inference under information constraints, e.g., communication constraints and local differential privacy. We focus on the tasks of goodness-of-fit testing and estimation of discrete distributions. From prior work, these tasks are well understood under noninteractive protocols. Extending these approaches directly for interactive protocols is difficult due to correlations that can build due to interactivity; in fact, gaps can be found in prior claims of tight bounds of distribution estimation using interactive protocols. We propose a new approach to handle this correlation and establish a unified method to establish lower bounds for both tasks. As an application, we obtain optimal bounds for both estimation and testing under local differential privacy and communication constraints. We also provide an example of a natural testing problem where interactivity helps., Comment: Modifying the proof and statement of Proposition 24 to address an issue in the variance analysis

This article introduces structural information potential (SIP), a measure of information based on the potential of structures to be informative about their content. An example of this concept is the clustered appearance that typically characterizes the first page of scientific articles, which summarizes the article’s contents and provides additional data, yielding potentially the largest and most diverse amount of information from a single page in the shortest time with the least effort. This characteristic makes SIP particularly well-adapted to triage tasks (i.e., rapid decision-making under conditions of uncertainty and limited resources), an application illustrated by means of a case study on classifying document images. The SIP method consists in unifying the Shannon entropy, the Fourier transform, the fractal dimension, and the golden ratio into a single equation and several algorithmic components. While the application domain is document images, the concept has generic character. The method results in a mathematically and perceptually coherent pattern space, characterized by continuous transition between uniform, clustered, and regular configurations, and corresponding to a structural information potential with a well-defined maximum. The maximum SIP leads to the identification of shapes and patterns with minimal structural redundancy, termed “fluorescent objects” as a complement to regular graphs and the Platonic solids.

Band sampling (BSam) is an innovative concept for hyperspectral imaging, which is derived from signal sampling in communications/signal processing as well as sampling theory in information theory. It is quite different from band selection (BSel) in several aspects. First of all, BSam ``samples'' bands with a given fixed BSam rate (BSamR) as opposed to BSel, which ``selects'' appropriate bands according to the number of bands to be selected. Second, BSam requires no specific means of sampling bands compared to BSel, which requires a specific rule to select bands such as band prioritization (BP) criteria or band search strategies. Third, BSam samples bands without prior band knowledge or specific applications in contrast to BSel, which considers certain bands more significant than other bands according to various applications. Two strategies are developed for BSam. One is from information theory. Under a completely blind and unknown environment, the maximum entropy is achieved by the uniform distribution. This suggests that one best strategy for BSam is uniform band sampling (UBSam). Another strategy is random BSam (RBSam) analogous to random signal sampling in compressive sensing (CS) where the bands sampled by RBSam are random but are not deterministic like the bands selected by BSel. Interestingly, UBSam and RBSam generally perform better than custom-designed BSel methods. To illustrate its potential utility of BSam in various applications, target detection, anomaly detection, and image classification are studied through extensive experiments for demonstration.

In hyperspectral anomaly detection, anomalies are rare targets that exhibit distinct spectral signatures from the background. Thus, anomalies are with low probabilities of occurrence in hyperspectral images. In this article, we develop a new technique for hyperspectral anomaly detection that adopts a new information theory perspective, to fully utilize the aforementioned concepts. Our goal is to transform system entropy into quantitative metrics of anomaly conspicuousness of pixels. To do so, two tasks are first completed: first, the construction of occurrence probability of pixels based on the density peak clustering algorithm, and second, the valid system definitions for pixels in specific anomaly detection problems with multiviews. Specifically, three types of systems are separately established by pixel pairs to conform to the definitions of three entropy definitions in information theory, i.e., Shannon entropy, joint entropy, and relative entropy. Then, three individual entropy-based metrics that assess the anomaly conspicuousness are defined. In addition, we design a standard deviation-based ensemble strategy for the integrated representation of the three individual metrics, which considers both logic ``or'' and ``and'' operations to simultaneously improve the detection rate and reduce the false alarm rate. Our experimental results obtained on two publicly available datasets with anomalies of different sizes and shapes demonstrate the superiority of our newly proposed anomaly detection method.

Feature selection plays an important role in many areas that are relevant to machine learning. Information theory is widely applied to feature selection methods because it can measure linear and nonlinear correlations among variables. However, previous information theory-based feature selection methods ignore two critical problems. First, features are collected or annotated manually, it is inevitable to make some incorrect features or outliers. Second, previous information theory-based methods pay much attention to the correlations among features while ignoring the correlations between samples. To address these problems, this paper proposes an information theory-based feature selection method: Preserving Similarity and Staring Decisis. The proposed method firstly employs the similarity between samples to obtain a new transformed data that avoids outliers or incorrect features in original data. Meanwhile, the new transformed data enhances the separability of features, which is beneficial to feature selection. Additionally, different from previous methods that consider all the already-selected features in the feature selection process, the proposed method stares the last already-selected feature only, which reduces the computational cost. Extensive experiments demonstrate that the proposed method outperforms nine state-of-the-art methods in terms of multiple criteria significantly.

Under the present circumstances, when we are still under the threat of different strains of coronavirus, and since the most widely used method for COVID-19 detection, RT-PCR is a tedious and time-consuming manual procedure with poor precision, the application of Artificial Intelligence (AI) and Computer-Aided Diagnosis (CAD) is inevitable. Though, some vaccines have now been authorized worldwide, it will take huge time to reach everyone, especially in developing countries. In this work, we have analyzed Chest X-ray (CXR) images for the detection of the coronavirus. The primary agenda of this proposed research study is to leverage the classification performance of the deep learning models using ensemble learning. Many papers have proposed different ensemble learning techniques in this field, some methods using aggregation functions like Weighted Arithmetic Mean (WAM) among others. However, none of these methods take into consideration the decisions that subsets of the classifiers take. In this paper, we have applied Choquet integral for ensemble and propose a novel method for the evaluation of fuzzy measures using coalition game theory, information theory, and Lambda fuzzy approximation. Three different sets of fuzzy measures are calculated using three different weighting schemes along with information theory and coalition game theory. Using these three sets of fuzzy measures, three Choquet integrals are calculated and their decisions are finally combined. Besides, we have created a database by combining several image repositories developed recently. Impressive results on the newly developed dataset and the challenging COVIDx dataset support the efficacy and robustness of the proposed method. Our experimental results outperform many recently proposed methods.

We consider the estimation of the entropy of a discretely-supported time series through a plug-in estimator. We provide a correction of the bias and we study the asymptotic properties of the estimator. We show that the widely-used correction proposed by Roulston (1999) is incorrect as it does not remove the $O\left ({N^{-1}}\right)$ part of the bias while ours does. We provide the asymptotic distribution and we show that it differs when the values taken by the marginal distribution of the process are equiprobable (a situation that we call degeneracy ) and when they are not. We introduce estimators of the bias, the variance and the distribution under degeneracy and we study the estimation error. Finally, we propose a goodness-of-fit test based on entropy and give two motivations for it. The theoretical results are supported by specific numerical examples.

Index coding and coded caching are two active research topics in information theory with strong ties to each other. Motivated by the multi-access coded caching problem, we study a new class of structured index coding problems (ICPs) which are formed by the union of several symmetric ICPs. We derive upper and lower bounds on the optimal server transmission rate for this class of ICPs and demonstrate that they differ by at most a factor of two. Finally, we apply these results to the multi-access coded caching problem to derive better bounds than the state of the art., Comment: 44 pages, single column, ITW and JSAIT

Efficient estimation of line spectral from quantized samples is of significant importance in information theory and signal processing, e.g., channel estimation in energy efficient massive MIMO systems and direction of arrival estimation. The goal of this paper is to recover the line spectral as well as its corresponding parameters including the model order, frequencies and amplitudes from heavily quantized samples. To this end, we propose an efficient grid-less Bayesian algorithm named VALSE-EP, which is a combination of the high resolution and low complexity gridless variational line spectral estimation (VALSE) and expectation propagation (EP). The basic idea of VALSE-EP is to iteratively approximate the challenging quantized model of line spectral estimation as a sequence of simple pseudo unquantized models, where VALSE is applied. Moreover, to obtain a benchmark of the performance of the proposed algorithm, the Cramer Rao bound (CRB) is derived. Finally, numerical experiments on both synthetic and real data are performed, demonstrating the near CRB performance of the proposed VALSE-EP for line spectral estimation from quantized samples.

As an attempt to bridge the gap between the probabilistic world of classical information theory and the combinatorial world of zero-error information theory, this paper studies the performance of randomly generated codebooks over discrete memoryless channels under a zero-error list-decoding constraint. This study allows the application of tools from one area to the other. Furthermore, it leads to an information-theoretic formulation of the birthday problem, which is concerned with the probability that in a given population, a fixed number of people have the same birthday. Due to the lack of a closed-form expression for this probability when the distribution of birthdays is not uniform, the resulting expression is not simple to analyze; in the information-theoretic formulation, however, the asymptotic behavior of this probability can be characterized exactly for all distributions., Comment: Extended version of paper presented at ISIT 2017 in Aachen

Two parties observe independent copies of a $d$-dimensional vector and a scalar. They seek to test if their data is correlated or not, namely they seek to test if the norm $\|\rho\|_2$ of the correlation vector $\rho$ between their observations exceeds $\tau$ or is it $0$. To that end, they communicate interactively and declare the output of the test. We show that roughly order $d/\tau^2$ bits of communication are sufficient and necessary for resolving the distributed correlation testing problem above. Furthermore, we establish a lower bound of roughly $d^2/\tau^2$ bits for communication needed for distributed correlation estimation, rendering the estimate-and-test approach suboptimal in communication required for distributed correlation testing. For the one-dimensional case with one-way communication, our bounds are tight even in the constant and provide a precise dependence of communication complexity on the probabilities of error of two types., Comment: 30 pages, 1 figure

We develop a communication-efficient distributed learning algorithm that is robust against Byzantine worker machines. We propose and analyze a distributed gradient-descent algorithm that performs a simple thresholding based on gradient norms to mitigate Byzantine failures. We show the (statistical) error-rate of our algorithm matches that of Yin et al. (2018), which uses more complicated schemes (coordinate-wise median, trimmed mean). Furthermore, for communication efficiency, we consider a generic class of $\delta $ -approximate compressors from Karimireddi et al. (2019) that encompasses sign-based compressors and top- $k$ sparsification. Our algorithm uses compressed gradients and gradient norms for aggregation and Byzantine removal respectively. We establish the statistical error rate for non-convex smooth loss functions. We show that, in certain range of the compression factor $\delta $ , the (order-wise) rate of convergence is not affected by the compression operation. Moreover, we analyze the compressed gradient descent algorithm with error feedback (proposed in Karimireddi et al. 2019) in a distributed setting and in the presence of Byzantine worker machines. We show that exploiting error feedback improves the statistical error rate. Finally, we experimentally validate our results and show good performance in convergence for convex (least-square regression) and non-convex (neural network training) problems.

The privacy risk has become an emerging challenge in both information theory and computer science due to the massive (centralized) collection of user data. In this paper, we overview privacy-preserving mechanisms and metrics from the lenses of information theory, and unify different privacy metrics, including f-divergences, R\'enyi divergences, and differential privacy, by the probability likelihood ratio (and the logarithm of it). We introduce recent progresses on designing privacy-preserving mechanisms according to the privacy metrics in (i) computer science, where differential privacy is the standard privacy notion that controls the output shift given small input perturbation, and (ii) information theory, where the privacy is guaranteed by minimizing information leakage. In particular, for differential privacy, we include its important variants (e.g., R\'enyi differential privacy, Pufferfish privacy) and properties, discuss its connections with information-theoretic quantities, and provide the operational interpretations of its additive noise mechanisms. For information-theoretic privacy, we cover notable frameworks from privacy funnel, originated from rate-distortion theory and information bottleneck, to privacy guarantee against statistical inference/guessing, and information obfuscation on samples and features. Finally, we discuss the implementations of these privacy-preserving mechanisms in current data-driven machine learning scenarios, including deep learning, information obfuscation, federated learning, and dataset sharing.

In this paper we study stopping sets of AG codes over a family of curves, denoted by $\mathcal {X}_{f,g}$ , that includes several important curves with applications in coding theory. We present results concerning stopping sets of one-point and $m$ -point codes over $\mathcal {X}_{f,g}$ , generalizing some results for Hermitian codes presented by Anderson and Matthews.

We construct and analyze active learning algorithms for the problem of binary classification with abstention, in which the learner has an additional option to withhold its decision on certain points in the input space. We consider this problem in the fixed-cost setting, where the learner incurs a cost $\lambda \in (0, 1/2)$ every time the abstain option is invoked. Our proposed algorithm can work with the three most commonly used active learning query models, namely, membership-query , pool-based , and stream-based models. We obtain a high probability upper-bound on the excess risk of our algorithm, and establish its minimax near-optimality by deriving matching lower-bound (modulo polylogarithmic factors). Since our algorithm relies on the knowledge of the smoothness parameters of the regression function, we also describe a new strategy to adapt to these unknown parameters in a data-driven manner under an additional quality assumption. We show that using this strategy our algorithm achieves the same performance in terms of excess risk as their counterparts with the knowledge of the smoothness parameters. We end the paper with a discussion about the extension of our results to the setting of bounded rate of abstention.

Spectrum sensing is used to detect whether primary user are using authorized spectrums, which can be regarded as a key and core issue for opportunistic spectrum access in cognitive radio networks. In traditional information theory and clustering algorithm-based spectrum sensing methods, they need to evaluate noise environment for constructing a reference point. However, the reference point is fixed, which is unreasonable in dynamic cognitive radio environment. Moreover, these methods convert signal feature from manifold onto Euclidean space, which will cause to overall performance degradation, since sensing information losses. To address this problem, an information geometry (IG)-based K-means clustering algorithm, namely IGK, is developed, it clusters samples on manifold instead of Euclidean space with an unsupervised way to train a classifier for spectrum sensing. Specifically, secondary users observe and collect data from a selected authorized spectrum, which needs to be detected, and send these sensing data to a fusion center (FC). Then, the FC transforms these data into samples on the manifold to obtained a classifier by using the proposed IGK algorithm. According to the trained classifier, we can get related result of the authorized spectrum. Finally, in simulation section, the effectiveness of the proposed scheme is verified under different conditions.

Centralized coded caching schemes have been widely studied in wireless networks. Placement delivery array (PDA) can be used to generate a coded caching scheme with low subpacketization when all the requested files are different. However, all existing schemes based on PDA design treat the request of each user as an independent file, without leveraging the multicast opportunities when one file is requested by several users. In this letter, for any existing scheme based on PDA design, after its coded multicast messages generation, we proposed an approach to remove the redundant multicast messages.

We present a new non-parametric statistic, called the weighed $\ell _{2}$ divergence, based on empirical distributions for sequential change detection. We start by constructing the weighed $\ell _{2}$ divergence as a fundamental building block for two-sample tests and change detection. The proposed statistic is proved to attain the optimal sample complexity in the offline setting. We then study the sequential change detection using the weighed $\ell _{2}$ divergence and characterize the fundamental performance metrics, including the average run length (ARL) and the expected detection delay (EDD). We also present practical algorithms to find the optimal projection to handle high-dimensional data and the optimal weights, which is critical to quick detection since, in such settings, there are not many post-change samples. Simulation results and real data examples are provided to validate the good performance of the proposed method.

We investigate a $K$ -user parallel packet-erasure broadcast channel. There is an ongoing effort to harness millimeter-wave bands, which are known to be unstable having high outage probabilities, by combining them with stable legacy bands. Motivated by this effort, we consider a heterogeneous scenario in which the parallel subchannels are categorized into two classes having different outage probabilities. For the two-user case, we characterize the sum capacity by developing an explicit achievable scheme and deriving a matching upper bound. In contrast to suboptimal schemes that apply coding on a per-subchannel basis only, our scheme applies coding across subchannels to exploit coding opportunities that arise from asymmetric outage probabilities more efficiently, thereby achieving optimality. By extending our scheme systematically to be applicable for the $K$ -user case, we show that it can provide significant gains over existing schemes. Compared to the $K$ -user scheme currently employed in practice, which allocates chunks of subchannels to users exclusively, we demonstrate the performance improvement attainable by our scheme to be substantial, as the multiplicative gain scales with $K$ . Moreover, we find that our scheme outperforms a per-subchannel extension of state-of-the-art $K$ -user schemes by large margins, further reducing the optimality gap. Our results suggest a potential coding scheme that can be employed in future wireless systems to meet ever-growing mobile data demands.

The bandwidth of a bandlimited signal is an important number that is relevant in many applications and concepts. For example, according to the Shannon sampling theorem, the bandwidth determines the minimum sampling rate that is required for a perfect reconstruction. In this paper we consider bandlimited signals with finite energy and bandlimited signals that are absolutely integrable and analyze whether the bandwidth of these signals can be determined algorithmically. We employ the concept of Turing computability, a theoretical model that describes the fundamental limits of what can be solved algorithmically on a digital hardware, and ask if, for a given computable bandlimited signal, it is possible to compute its bandwidth on a Turing machine. We show that this is not possible in general, because there exist computable bandlimited signals for which the bandwidth is a non-computable real number. Even the weaker question if the bandwidth of a given signal is smaller than a predefined value cannot be always answered algorithmically. Further, we prove that in the case where the bandwidth in not computable, it is even impossible to algorithmically determine a sequence of upper bounds that converges to the actual bandwidth of the signal. As a positive result, we show that the set of signals whose bandwidth is larger than some given value is semi-decidable.

We study the relationship between randomized low influence functions and differentially private mechanisms. Our main aim is to formally determine whether differentially private mechanisms are low influence and whether low influence randomized functions can be differentially private. We show that differential privacy does not necessarily imply low influence in a formal sense. However, low influence implies approximate differential privacy. These results hold for both independent and non-independent randomized mechanisms, where an important instance of the former is the widely-used additive noise techniques in the differential privacy literature. Our study also reveals the interesting dynamics between utility, low influence, and independence of a differentially private mechanism. As the name of this article suggests, we show that any two such features are simultaneously possible. However, in order to have a differentially private mechanism that has both utility and low influence, even under a very mild utility condition, one has to employ non-independent mechanisms.

An equiangular tight frame (ETF) yields a type of optimal packing of lines in a Euclidean space. ETFs seem to be rare, and all known infinite families of them arise from some type of combinatorial design. In this paper, we introduce a new method for constructing ETFs. We begin by showing that it is sometimes possible to construct multiple ETFs for the same space that are "mutually unbiased" in a way that is analogous to the quantum-information-theoretic concept of mutually unbiased bases. We then show that taking certain tensor products of these mutually unbiased ETFs with other ETFs sometimes yields infinite families of new complex ETFs.

Model-based incoherent polarimetric decomposition is a frequently used technique to analyze multilook data of polarimetric synthetic aperture radars (POLSARs). The purpose of this study is to analyze and compare different model-based incoherent polarimetric decomposition algorithms from the polarimetric information change aspect. For the input of a model-based incoherent polarimetric decomposition algorithm, polarimetric entropy was used to represent the polarimetric information of a coherency matrix. For the output of a model-based incoherent polarimetric decomposition algorithm, there are usually several decomposed components. To quantitatively represent their total polarimetric information, a new concept, generalized polarimetric entropy, was proposed which generalized the concept of polarimetric entropy based on the information entropy additivity of information theory. Generalized polarimetric entropy consists of two parts named as polarimetric power entropy and polarimetric residual entropy, respectively. Polarimetric power entropy describes the distribution status of the Span values of all decomposed components. Polarimetric residual entropy represents the residual randomness of all decomposed components. With the three new concepts, eight model-based incoherent polarimetric decomposition algorithms were compared and analyzed. Two real POLSAR images, respectively, derived by the E-SAR airborne system of Germany and the GF-3 satellite of China were used for the experiments. Experimental results had illustrated several useful conclusions.

There are different inequivalent ways to define the R\'enyi capacity of a channel for a fixed input distribution $P$. In a 1995 paper Csisz\'ar has shown that for classical discrete memoryless channels there is a distinguished such quantity that has an operational interpretation as a generalized cutoff rate for constant composition channel coding. We show that the analogous notion of R\'enyi capacity, defined in terms of the sandwiched quantum R\'enyi divergences, has the same operational interpretation in the strong converse problem of classical-quantum channel coding. Denoting the constant composition strong converse exponent for a memoryless classical-quantum channel $W$ with composition $P$ and rate $R$ as $sc(W,R,P)$, our main result is that \[ sc(W,R,P)=\sup_{\alpha>1}\frac{\alpha-1}{\alpha}\left[R-\chi_{\alpha}^*(W,P)\right], \] where $\chi_{\alpha}^*(W,P)$ is the $P$-weighted sandwiched R\'enyi divergence radius of the image of the channel., Comment: 46 pages. V7: Added the strong converse exponent with cost constraint

How to train a machine learning model while keeping the data private and secure? We present CodedPrivateML, a fast and scalable approach to this critical problem. CodedPrivateML keeps both the data and the model information-theoretically private, while allowing efficient parallelization of training across distributed workers. We characterize CodedPrivateML's privacy threshold and prove its convergence for logistic (and linear) regression. Furthermore, via extensive experiments on Amazon EC2, we demonstrate that CodedPrivateML provides significant speedup over cryptographic approaches based on multi-party computing (MPC).

In this letter, we propose soft compression, an lossless compression approach to shape coding for images using location index and codebook of designed shapes with various sizes. This method is different from traditional image compression methods, as it aims at finding the basic shape blocks of pictures to improve the compression ratio from the perspective of information theory and frequency. In particular, the triplets including location and codeword are stored when encoding an image instead of the entire array. It is shown that soft compression can greatly reduce the bandwidth and storage space needed in the process of transmitting and storing the same kind of images.

Source coding is the canonical problem of data compression in information theory. In a locally encodable source coding, each compressed bit depends on only few bits of the input. In this paper, we show that a recently popular model of semi-supervised clustering is equivalent to locally encodable source coding. In this model, the task is to perform multiclass labeling of unlabeled elements. At the beginning, we can ask in parallel a set of simple queries to an oracle who provides (possibly erroneous) binary answers to the queries. The queries cannot involve more than two (or a fixed constant number of) elements. Now the labeling of all the elements (or clustering) must be performed based on the noisy query answers. The goal is to recover all the correct labelings while minimizing the number of such queries. The equivalence to locally encodable source codes leads us to find lower bounds on the number of queries required in a variety of scenarios. We provide querying schemes based on pairwise `same cluster' queries - and pairwise AND queries and show provable performance guarantees for each of the schemes., 16 pages, 11 figures. Some of the results of this paper have appeared in the proceedings of the 2017 Conference on Neural Information Processing Systems (NeurIPS 2017)

In the Shannon lecture at the 2019 International Symposium on Information Theory (ISIT), Ar{\i}kan proposed to employ a one-to-one convolutional transform as a pre-coding step before the polar transform. The resulting codes of this concatenation are called polarization-adjusted convolutional (PAC) codes. In this scheme, a pair of polar mapper and demapper as pre- and postprocessing devices are deployed around a memoryless channel, which provides polarized information to an outer decoder leading to improved error correction performance of the outer code. In this paper, the list decoding and sequential decoding (including Fano decoding and stack decoding) are first adapted for use to decode PAC codes. Then, to reduce the complexity of sequential decoding of PAC/polar codes, we propose (i) an adaptive heuristic metric, (ii) tree search constraints for backtracking to avoid exploration of unlikely sub-paths, and (iii) tree search strategies consistent with the pattern of error occurrence in polar codes. These contribute to the reduction of the average decoding time complexity from 50% to 80%, trading with 0.05 to 0.3 dB degradation in error correction performance within FER=10^-3 range, respectively, relative to not applying the corresponding search strategies. Additionally, as an important ingredient in Fano decoding of PAC/polar codes, an efficient computation method for the intermediate LLRs and partial sums is provided. This method is effective in backtracking and avoids storing the intermediate information or restarting the decoding process. Eventually, all three decoding algorithms are compared in terms of performance, complexity, and resource requirements., Comment: To appear in IEEE Transactions on Vehicular Technology

Shannon’s information theory states that there is a maximum rate for transmitting data reliably, implying that data almost always inevitably leak. Although there has been advancement in data protection technologies, a lack of understanding about data leaks and human error persists.

It is of central interest in information theory to determine whether a given vector in the entropy space is an almost entropic vector. This problem can be answered if all the information inequalities are known, but this is an extremely challenging problem. On the other hand, we can establish that a given vector is an entropy vector if we can show the existence of distribution such that the corresponding entropy vector is the same as the given vector. However, there is no known algorithm to solve this problem. Only for the simplest case of binary entropy vectors, an algorithm is known to solve this problem. In this paper, we present a recursive algorithm to determine whether a given vector is a quasi-uniform entropy vector and, if it is, to return a consistent quasi-uniform distribution. We also present two applications of the recursive procedure: (i) to generate all quasi-uniform distributions motivated by the problem of finding the smallest quasi-uniform distribution such that its entropy vector violates the well known Ingleton inequality and (ii) to obtain an entropy vector (not necessarily quasi-uniform) near to a target vector in the entropy space for random variables with given alphabet size.

The advanced notebooks, mobile phones, and internet applications in today's world that we use are all entrenched in classical communication bits of zeros and ones. Classical internet has laid its foundation originating from the amalgamation of mathematics and Claude Shannon's theory of information. But today's internet technology is a playground for eavesdroppers. This poses a serious challenge to various applications that relies on classical internet technology. This has motivated the researchers to switch to new technologies that are fundamentally more secure. Exploring the quantum effects, researchers paved the way into quantum networks that provide security, privacy and range of capabilities such as quantum computation, communication and metrology. The realization of quantum internet requires quantum communication between various remote nodes through quantum channels guarded by quantum cryptographic protocols. Such networks rely upon quantum bits (qubits) that can simultaneously take the value of zeros and ones. Due to extraordinary properties of qubits such as entanglement, teleportation and superposition, it gives an edge to quantum networks over traditional networks in many ways. But at the same time transmitting qubits over long distances is a formidable task and extensive research is going on quantum teleportation over such distances, which will become a breakthrough in physically realizing quantum internet in near future. In this paper, quantum internet functionalities, technologies, applications and open challenges have been extensively surveyed to help readers gain a basic understanding of infrastructure required for the development of global quantum internet., Comment: This survey paper is submitted in IEEE Communications Surveys and Tutorials and revised on 27th May 2021. It includes 31 pages, 14 figures, and 5 tables

Caching has been recognized as a viable solution to surmount the limited capability of backhaul links in handling abundant network traffic. Although optimal approaches for minimizing the average delivery load do exist, current caching strategies fail to avert intelligent adversaries from obtaining invaluable contextual information by inspecting the wireless communication links and thus, violating users’ privacy. Grounded in information theory, in this letter, we propose a mathematical model for preserving privacy in a network caching system involving a server and a cache-aided end user. We then present an efficient content caching method that maximizes the degree of privacy preservation while maintaining the average delivery load at a given level. Given the Pareto optimal nature of the proposed $\epsilon $ -constraint optimization approach, we also obtain the maximum privacy degree achievable under any given average delivery load. Numerical results and comparisons validate the correctness of our context-oriented privacy model.

The use of Bandt–Pompe probability distributions and descriptors of information theory has been presenting satisfactory results with low computational cost in the time series analysis literature [1]–[3]. However, these tools have limitations when applied to data without time dependency. Given this context, we present a newly proposed technique for texture analysis and classification based on the Bandt–Pompe symbolization for SAR data. It consists of linearizing a 2-D patch of the image using the Hilbert–Peano curve, build an ordinal pattern transition graph that considers the data amplitude encoded into the weight of the edges, obtain a probability distribution function derived from this graph, and compute information theory descriptors (permutation entropy and statistical complexity) from this distribution and use them as features to feed a classifier. The ordinal pattern graph we propose considers that the edges’ weight is related to the absolute difference of observations, which encodes the information about the data amplitude. This modification considers the unfavorable signal-to-noise ratio of SAR images and leads to the characterization of several types of textures. Experiments with data from Munich urban areas, Guatemala forest regions, and Cape Canaveral ocean samples show the effectiveness of our technique in homogeneous areas, achieving satisfactory separability levels. The two descriptors chosen in this work are easy and quick to calculate and are used as input for a k-nearest neighbor classifier. Experiments show that this technique presents results similar to state-of-the-art techniques that employ a much larger number of features and, consequently, impose a higher computational cost.

In this article, we consider the minimum data rate problem for linear system stabilization under noiseless communication channels. Previous results indicated that having a data rate very approaching the entropy bound leads to large delays and data buffer sizes. In analogy, the entropy bound in Shannon's source coding theorem in traditional information theory displays this behavior, where the data rate can be arbitrarily close to the entropy bound but only at the cost of boundlessly enlarging the blocklength. However, in this article, we show the analogy is not strict. We prove that it is possible to stabilize a linear system at a rate equal to the entropy bound with zero delay, i.e., where each system state is encoded and decoded within one time unit. We establish a set of sufficient conditions for guaranteeing zero-delay entropy-achieving codes. Following this, we design an entropy-achieving code with finite blocklength satisfying the set of sufficient conditions, where the codeword length is uniformly bounded.

We address the requirement of image coding for joint human-machine vision, i.e., the decoded image serves both human observation and machine analysis/understanding. Previously, human vision and machine vision have been extensively studied by image (signal) compression and (image) feature compression, respectively. Recently, for joint human-machine vision, several studies have been devoted to joint compression of images and features, but the correlation between images and features is still unclear. We identify the deep network as a powerful toolkit for generating structural image representations. From the perspective of information theory, the deep features of an image naturally form an entropy decreasing series: a scalable bitstream is achieved by compressing the features backward from a deeper layer to a shallower layer until culminating with the image signal. Moreover, we can obtain learned representations by training the deep network for a given semantic analysis task or multiple tasks and acquire deep features that are related to semantics. With the learned structural representations, we propose SSSIC, a framework to obtain an embedded bitstream that can be either partially decoded for semantic analysis or fully decoded for human vision. We implement an exemplar SSSIC scheme using coarse-to-fine image classification as the driven semantic analysis task. We also extend the scheme for object detection and instance segmentation tasks. The experimental results demonstrate the effectiveness of the proposed SSSIC framework and establish that the exemplar scheme achieves higher compression efficiency than separate compression of images and features.

In the fields of information theory and network coding, a complete characterization of the almost entropic region is a fundamental but inherently difficult open problem. This paper focuses on the characterization of this region via explicit inner bounds, optimization of functions over this region, and network code construction. One approach to study the entropic region is to put a constraint on the alphabets for involved random variables, enabling us to focus on a specific set of joint distributions. We present the notion of alphabet constrained entropic set and describe its properties, namely, its a closed and path-connected set. Motivated by the properties, a random local search algorithm is designed to find an entropic vector and associated distribution near to a given target vector and extended to optimize functions of joint entropies. We present significantly improved inner bound compared to known inner bounds for the almost entropic region involving four random variables using a refinement of our algorithm. Finally, another application of the algorithm is shown to construct network codes and index codes for a given rate vector in the capacity region. This approach to constructing network codes is novel and has an entirely different flavor than classical coding theoretic construction.

To realize the unmanned automation of the full mechanized caving, the bottleneck problem of coal-gangue interface detection in top coal caving must be solved first. Targeting coal-gangue interface detection on fully mechanized mining face, an alternative scheme to detect coal-gangue interface based on vibration signal analysis of the tail boom support of the longwall mining machine. It is found that when coal and gangue fall, the characteristics of vibration signals generated by coal and gangue shocking the tail boom are different. First, EEMD algorithm is used to decompose the original vibration signals into intrinsic mode functions (IMFs). Each IMF represents the distribution of energy from high to low. EEMD algorithm can restrain the mode mixing phenomenon caused by empirical mode decomposition (EMD). The energy of vibration signals will change in different frequency bands when the top-coal fall down or the coal-gangue fall down. According the information theory, we define EEMD energy entropy to describe this change. Experimental results show that EEMD energy entropy of top-coal caving is always greater than that of coal-gangue caving. Thus, the Mahalanobis distance metric method based on EEMD energy entropy is proposed for coal-gangue interface detection. The results show the proposed method can be used as a robust empirical method for coal-gangue interface detection.

A high dimensional low sample size (HDLSS) dataset typically contains many features but a limited number of samples. It is commonly found in domains such as microarray data and medical imaging. When sample size is small, the population probability density function (PDF) of a HDLSS dataset may not be well represented, causing difficulties of applying feature selection or feature extraction methods for HDLSS data classification. In this paper, we explore the possibility of designing feature selection and feature extraction methods for HDLSS data classification by making loose assumption on the underlying PDF of a HDLSS dataset. Specifically, we propose to leverage on Correlation Explanation (CorEx), a recent unsupervised probabilistic graphical model that assumes (hierarchical) hidden structure for generating subsets of features that are conditionally independent. We benchmark the proposed method against frequently cited Information Theory based feature extraction and feature selection methods, including Conditional Infomax Feature Extraction (CIFE), Maximum Relevance Minimum Redundancy (MRMR), Maximization of Mutual Information (MMI), Infomax Independent Component Analysis (Infomax ICA),and Kernel Entropy Component Analysis (KECA). The HDLSS datasets used in this study are Breast Cancer Dataset by Gravier et. al and West et. al, Colon Cancer dataset by Alon et. al., Leukemia Dataset by Golub et.al and the Gisette Dataset used by Guyon et. al. Experimental results demonstrate that the proposed method shows some improvement in classification performance over MMI, and Infomax ICA and is competitive with MRMR and CIFE.

International audience; Methods for information-theoretic control for networks of sensors are of interest for enabling the development of autonomous sensor systems. In this paper we revisit the fundamentals of information theoretic-based control for multi-target systems and present a systematic approach for determining information-theoretic situational awareness based on mutual information for point processes. The extension to multi-sensor systems is developed using the concept of a broadcast channel from information theory. Analytic results are presented for linear-Gaussian systems which enable low complexity solutions for determining information from multiple sensors and we consider a large number of potential sensor configurations. We consider extensions of single-target methods to multi-target scenarios and present results in simulations.

The problem of distributed representation learning is one in which multiple sources of information $X_1,\ldots,X_K$ are processed separately so as to learn as much information as possible about some ground truth $Y$. We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, $K$ encoders, $K \geq 2$, compress their observations $X_1,\ldots,X_K$ separately in a manner such that, collectively, the produced representations preserve as much information as possible about $Y$. We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations $X_1,\ldots,X_K$ being conditionally independent given $Y$). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper., Comment: 35 pages, 10 figures, submitted for possible publication

This paper aims to estimate the information between two random phenomena by using consolidated second-order statistics tools. The squared-loss mutual information, a surrogate of the Shannon mutual information, is chosen due to its property of being expressed as a second-order moment. We first review the rationale for i.i.d. discrete sources, which involves mapping the data onto the simplex space, and we highlight the links with other well-known related concepts in the literature based on local approximations of information-theoretic measures. Then, the problem is translated to analog sources by mapping the data onto the characteristic space, focusing on the adaptability between the discrete and the analog case and its limitations. The proposed approach gains interpretability and scalability for its use on large data sets, providing a unified rationale for the free regularization parameters. Moreover, the structure of the proposed mapping allows resorting to Szegö’s theorem to reduce the complexity for high dimensional mappings, exhibiting a strong duality with spectral analysis. The performance of the developed estimators is analyzed using Gaussian mixtures. This work has been supported by the Spanish Ministry of Science and Innovation through project RODIN (PID2019-105717RB- C22/MCIN/AEI/10.13039/501100011033), by the grant 2021 SGR 01033 (AGAUR, Generalitat de Catalunya), and fellowship FI 2019 by the Secretary for University and Research of the Generalitat de Catalunya and the European Social Fund.