Search

The concept of entropy and information gain of BE-algebras in scientific disciplines such as information theory, data science, supply chain and machine learning assists us to calculate the uncertanity of the scientific processes of phenomena. In this respect the notion of filter entropy for a transitive BE-algebra is introduced and its properties are investigated. The notion of a dynamical system on a transitive BE-algebra is introduced. The concept of the entropy for a transitive BE-algebra dynamical system is developed and, its characteristics are considered. The notion of equivalent transitive BE-algebra dynamical systems is defined, and it is proved the fact that two equivalent BE-algebra dynamical systems have the same entropy. Theorems to help calculate the entropy are given. Specifically, a new version of Kolmogorov– Sinai Theorem has been proved. The study introduces the concept of information gain of a transitive BE-algebra with respect to its filters and investigates its properties. This study proposes the use of filter entropy to approximate the level of risk introduced by a BE-algebra dynamical system. This aim is reached by defining the information gain with respect to the filters of a BE-algebra. This methodology is well developed for use in engineering, especially in industrial networks. This paper proposes a novel approach to assess the quantity of uncertainty, and the impact of information gain of a BE-algebra dynamical system. [ABSTRACT FROM AUTHOR]

In the theory of special relativity, energy can be found in two forms: kinetic energy and rest mass. The potential energy of a body is actually stored in the form of rest mass, the interaction energy too, but temperature is not. Information acquired about a dynamical system can be potentially used to extract useful work from it. Hence, the "mass–energy–information equivalence principle" that has been recently proposed. In this paper, it is first recalled that for a thermodynamic system made of non-interacting entities at constant temperature, the internal energy is also constant. So, the energy involved in a variation in entropy ( T Δ S ) differs from a change in the potential energy stored or released and cannot be associated to a corresponding variation in mass of the system, even if it is expressed in terms of the quantity of information. This debate gives us the opportunity to deepen the notion of entropy seen as a quantity of information, to highlight the difference between logical irreversibility (a state-dependent property) and thermodynamical irreversibility (a path-dependent property), and to return to the nature of the link between energy and information that is dynamical. [ABSTRACT FROM AUTHOR]

In this paper, we provide geometric insights with visualization into the multivariate Gaussian distribution and its entropy and mutual information. In order to develop the multivariate Gaussian distribution with entropy and mutual information, several significant methodologies are presented through the discussion, supported by illustrations, both technically and statistically. The paper examines broad measurements of structure for the Gaussian distributions, which show that they can be described in terms of the information theory between the given covariance matrix and correlated random variables (in terms of relative entropy). The content obtained allows readers to better perceive concepts, comprehend techniques, and properly execute software programs for future study on the topic's science and implementations. It also helps readers grasp the themes' fundamental concepts to study the application of multivariate sets of data in Gaussian distribution. The simulation results also convey the behavior of different elliptical interpretations based on the multivariate Gaussian distribution with entropy for real-world applications in our daily lives, including information coding, nonlinear signal detection, etc. Involving the relative entropy and mutual information as well as the potential correlated covariance analysis, a wide range of information is addressed, including basic application concerns as well as clinical diagnostics to detect the multi-disease effects. [ABSTRACT FROM AUTHOR]

In modern realities, an important place for the effective management of the cattle breeding industry is a properly implemented selection and breeding process with the sampling of cows that are marked by the highest indicators of a set of productive qualities. In this case, the application of entropy and information analysis is one of the most expedient, as it makes it possible to assess the economic and useful qualities of animals as fully as possible. The purpose of the study is to evaluate and analyze the degree of organization of the biological system of dairy cattle productive traits under the influence of hereditary factors such as the age and origin of cows. During the research, methods generally accepted in zootechnics and methods using information and entropy analysis, which are adapted and modified in animal husbandry, were used. The data obtained from the entropy and information analysis show that for cattle of the Black-and-White breed of different lineal affiliations, the level of organization of systems varies - R from 0.009 to 1.341 bits. Moreover, the most stable trait from the point of view of variability was the fat content in milk, the level of unconditional entropy in the section of lactations was equal to 3.333-4.550 bits, which indicates a smaller influence of disorganized factors on the level of manifestation of this trait, and a greater dependence on hereditary factors, i.e. origin. Although in general, a reliable influence of the lineal affiliation of cows on indicators of unconditional entropy and organization of the system was not established, the influence of the age factor on indicators of entropy was observed. Thus, the researched livestock is not characterized by uniformity and consolidation in terms of the main selection characteristics, except for the fat content in milk, which indicates a wide range of variability and serves as a flexible material both for selection and breeding work and for increasing the level of milk productivity. Therefore, the use of empirical data of information theory can be a kind of marker when predicting hereditary traits of a particular productivity, since entropy and information analysis provides wider and deeper values of trait variability. [ABSTRACT FROM AUTHOR]

This article is devoted to the study of the entropic orderliness of road safety systems of various dimensionalities. The author's methodology for quantitative assessment of the quality of the road safety systems is based on the use of information entropy analysis, the essence of which is to assess the significance (or "weights") of various information-technological stages of the road traffic accident rate formation process. The main emphasis in this paper is on the philosophical interpretation of the results of entropic evaluation of the orderliness of urban road safety systems. The article aimed to philosophically understand the reasons for the diversity in the results of assessing the entropy of road safety (RS) in Russian cities. Within the framework of this goal, the results of the analysis of the state of the issue, ideological approaches and methods for assessing the relative entropy of urban road safety systems were presented. The study was based on analyzing statistics that characterize the processes of the formation of road traffic accidents in Russian cities classified into three groups based on population size. The experimental results obtained were explained from the point of view of human psychology. Rather, results were explained from the perspective of human psychology. The final results of the study once again illustrated the objectivity of Hegel's dialectical laws and, perhaps, once again shattered illusions about the possibility of achieving high levels of road safety in cities by building rigid systems to regulate the actions of traffic participants. In the author's opinion, the results of the presented philosophical analysis will be useful to managers specializing in the management of complex systems (not only transport but also other fields) to comprehend the contradictions of the complex nature of humans and the paradoxes of their behavior when their freedom of action is restricted through external control. [ABSTRACT FROM AUTHOR]

The optimization objective function of sensor management for target identification is commonly established based on information theory indicators such as information gain, discrimination, discrimination gain, and quadratic entropy, which can control the sensors to reduce the overall uncertainty of all targets to be identified but ignores the speed of target being confirmed as identified. Therefore, inspired by the maximum posterior criterion of target identification and the target identification confirmation mechanism, we study a sensor management method that preferentially allocates resources to identifiable targets. Firstly, in the distributed target identification framework based on Bayesian theory, an improved identification probability prediction method that provides feedback the global identification results to local classifiers is proposed, which can improve the accuracy of identification probability prediction. Secondly, an effective sensor management function based on information entropy and expected confidence level is proposed to optimize the identification uncertainty itself rather than its variation, which can increase the priority of targets that satisfy the desired confidence level. In the end, the sensor management for target identification is modeled as a sensor allocation problem, and the optimization objective function based on the effective function is constructed, which can improve the target identification speed. The experimental results show that the correct identification rate of the proposed method is comparable to the methods based on information gain, discrimination, discrimination gain, and quadratic entropy in different scenarios, but the average time to confirm the identification is the shortest. [ABSTRACT FROM AUTHOR]

Modeling and inference are central tomost areas of science and especially to evolving and complex systems. Critically, the information we have is often uncertain and insufficient, resulting in an underdetermined inference problem; multiple inferences, models, and theories are consistent with available information. Information theory (in particular, the maximum information entropy formalism) provides a way to deal with such complexity. It has been applied to numerous problems, within and across many disciplines, over the last few decades. In this perspective, we review the historical development of this procedure, provide an overview of the many applications of maximum entropy and its extensions to complex systems, and discuss in more detail some recent advances in constructing comprehensive theory based on this inference procedure. We also discuss efforts at the frontier of information-theoretic inference: application to complex dynamic systems with time-varying constraints, such as highly disturbed ecosystems or rapidly changing economies. [ABSTRACT FROM AUTHOR]

One of the most powerful laws in physics is the second law of thermodynamics, which states that the entropy of any system remains constant or increases over time. In fact, the second law is applicable to the evolution of the entire universe and Clausius stated, "The entropy of the universe tends to a maximum." Here, we examine the time evolution of information systems, defined as physical systems containing information states within Shannon's information theory framework. Our observations allow the introduction of the second law of information dynamics (infodynamics). Using two different information systems, digital data storage and a biological RNA genome, we demonstrate that the second law of infodynamics requires the information entropy to remain constant or to decrease over time. This is exactly the opposite to the evolution of the physical entropy, as dictated by the second law of thermodynamics. The surprising result obtained here has massive implications for future developments in genomic research, evolutionary biology, computing, big data, physics, and cosmology. [ABSTRACT FROM AUTHOR]

Geological entropy is based on Shannon information entropy and measures order in the structure of a spatial random variable. Metrics have been defined to quantify geological entropy in multidimensional (2D and 3D) heterogeneous systems, for instance, porous and fractured geological media. This study introduces GEOENT, a toolbox that can efficiently be used to calculate geological entropy metrics for any kind of input-gridded field. Additionally, the definition of geological entropy metrics is updated to consider anisotropy in the structure of the heterogeneous system. Directional entrograms provide more accurate descriptions of spatial order over different Cartesian directions. This study presents the development of the geological entropy metrics, a description of the toolbox, and examples of its applications in different datasets, including 2D and 3D gridded fields, representing a variety of heterogeneous environments at different scales, from pore-scale microtomography (μCT) images to aquifer analogues. [ABSTRACT FROM AUTHOR]

We consider the estimation of the entropy of a discretely-supported time series through a plug-in estimator. We provide a correction of the bias and we study the asymptotic properties of the estimator. We show that the widely-used correction proposed by Roulston (1999) is incorrect as it does not remove the $O\left ({N^{-1}}\right)$ part of the bias while ours does. We provide the asymptotic distribution and we show that it differs when the values taken by the marginal distribution of the process are equiprobable (a situation that we call degeneracy) and when they are not. We introduce estimators of the bias, the variance and the distribution under degeneracy and we study the estimation error. Finally, we propose a goodness-of-fit test based on entropy and give two motivations for it. The theoretical results are supported by specific numerical examples. [ABSTRACT FROM AUTHOR]

We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find that these axioms induce sufficient structure to establish continuity in the interior of the probability simplex and meaningful upper and lower bounds, e.g., we find that every relative entropy satisfying these axioms must lie between the Rényi divergences of order 0 and $\infty $. We further show simple conditions for positive definiteness of such relative entropies and a characterisation in terms of a variant of relative trumping. Our main result is a one-to-one correspondence between entropies and relative entropies. [ABSTRACT FROM AUTHOR]

A key result of classical information theory states that if the rate of a randomly generated codebook is less than the mutual information between the channel’s input and output, then the probability that that codebook has negligible error goes to one as the blocklength goes to infinity. In an attempt to bridge the gap between the probabilistic world of classical information theory and the combinatorial world of zero-error information theory, this work derives necessary and sufficient conditions on the rate so that the probability that a randomly generated codebook operated under list decoding (for any fixed list size) has zero error probability goes to one as the blocklength goes to infinity. Furthermore, this work extends the classical birthday problem to an information-theoretic setting, which results in the definition of a “noisy” counterpart of Rényi entropy, analogous to how mutual information can be considered a noisy counterpart of Shannon entropy. [ABSTRACT FROM AUTHOR]

Entropy is a measure of uncertainty and often used in information theory to determine the precise testimonials about unclear situations. Different entropy measures available in the literature are based on the exact form of the observations and lacks in dealing with the interval-valued data. The interval-valued data often arises from the situations having ambiguity, imprecise, unclear, indefinite, or vague states of the experiment and is called neutrosophic data. In this research modified forms of different entropy measures for normal probability distribution have been proposed by considering the neutrosophic form data. The performance of the proposed neutrosophic entropies for normal distribution has been assessed via a simulation study. Moreover, the proposed measures are also applied to two real data sets for their wide applicability. The results of the study suggested the use of the proposed methods in the presence of fuzzy, interval-valued, or neutrosophic data. [ABSTRACT FROM AUTHOR]

Differential privacy (DP) provides a framework for provable privacy protection against arbitrary adversaries, while allowing the release of summary statistics and synthetic data. We address the problem of releasing a noisy real-valued statistic vector T, a function of sensitive data under DP, via the class of K-norm mechanisms with the goal of minimizing the noise added to achieve privacy. First, we introduce the sensitivity space of T, which extends the concepts of sensitivity polytope and sensitivity hull to the setting of arbitrary statistics T. We then propose a framework consisting of three methods for comparing the K-norm mechanisms: (1) a multivariate extension of stochastic dominance, (2) the entropy of the mechanism, and (3) the conditional variance given a direction, to identify the optimal K-norm mechanism. In all of these criteria, the optimal K-norm mechanism is generated by the convex hull of the sensitivity space. Using our methodology, we extend the objective perturbation and functional mechanisms and apply these tools to logistic and linear regression, allowing for private releases of statistical results. Via simulations and an application to a housing price dataset, we demonstrate that our proposed methodology offers a substantial improvement in utility for the same level of risk. [ABSTRACT FROM AUTHOR]

An accurate leakage model is critical to side-channel attacks and evaluations. Leakage certification plays an important role to address the following question: “how good is my leakage model?” Moreover, most of the current leakage model profiling only exploit the information from lower orders of moments. They still need to tolerate assumption error and estimation error from unknown leakage models. There are many probability density functions (PDFs) satisfying given moment constraints. As such, finding an unbiased, objective, and reasonable model still remains an unresolved problem. In this article, we address a more fundamental question: “which model can approach the leakage infinitely and is the optimal in theory?” In particular, we extract information from higher order moments and propose maximum entropy distribution (MED) to estimate the leakage model as MED is an unbiased, objective, and theoretically the most reasonable PDF conditioned upon the available information. MED is a moment-based statistical PDF model in side-channel attacks. It can theoretically use information on arbitrary higher order moments to infinitely approximate the leakage distribution, and well compensates the theory vacancy of model profiling and evaluation. Experimental results demonstrate the superiority of our proposed method for approximating the leakage model using MED estimation. [ABSTRACT FROM AUTHOR]

Entropy and relative or cross entropy measures are two very fundamental concepts in information theory and are also widely used for statistical inference across disciplines. The related optimization problems, in particular the maximization of the entropy and the minimization of the cross entropy or relative entropy (divergence), are essential for general logical inference in our physical world. In this paper, we discuss a two parameter generalization of the popular Rényi entropy and associated optimization problems. We derive the desired entropic characteristics of the new generalized entropy measure including its positivity, expandability, extensivity and generalized (sub-)additivity. More importantly, when considered over the class of sub-probabilities, our new family turns out to be scale-invariant; this property does not hold for most existing generalized entropy measures. We also propose the corresponding cross entropy and relative entropy measures and discuss their geometric properties including generalized Pythagorean results over $\beta $ -convex sets. The maximization of the new entropy and the minimization of the corresponding cross or relative entropy measures are carried out explicitly under the non-extensive (‘third-choice’) constraints given by the Tsallis’ normalized $q$ -expectations which also correspond to the $\beta $ -linear family of probability distributions. Important properties of the associated forward and reverse projection rules are discussed along with their existence and uniqueness. In this context, we have come up with, for the first time, a class of entropy measures – a subfamily of our two-parameter generalization – that leads to the classical (extensive) exponential family of MaxEnt distributions under the non-extensive constraints; this discovery has been illustrated through the useful concept of escort distributions and can potentially be important for future research in information theory. Other members of the new entropy family, however, lead to the power-law type generalized $q$ -exponential MaxEnt distributions which is in conformity with Tsallis’ nonextensive theory. Therefore, our new family indeed provides a wide range of entropy and associated measures combining both the extensive and nonextensive MaxEnt theories under one umbrella. [ABSTRACT FROM AUTHOR]

It is of central interest in information theory to determine whether a given vector in the entropy space is an almost entropic vector. This problem can be answered if all the information inequalities are known, but this is an extremely challenging problem. On the other hand, we can establish that a given vector is an entropy vector if we can show the existence of distribution such that the corresponding entropy vector is the same as the given vector. However, there is no known algorithm to solve this problem. Only for the simplest case of binary entropy vectors, an algorithm is known to solve this problem. In this paper, we present a recursive algorithm to determine whether a given vector is a quasi-uniform entropy vector and, if it is, to return a consistent quasi-uniform distribution. We also present two applications of the recursive procedure: (i) to generate all quasi-uniform distributions motivated by the problem of finding the smallest quasi-uniform distribution such that its entropy vector violates the well known Ingleton inequality and (ii) to obtain an entropy vector (not necessarily quasi-uniform) near to a target vector in the entropy space for random variables with given alphabet size. [ABSTRACT FROM AUTHOR]

• The process and characteristics of overburden deformation are obtained by using optical fiber sensors. • The dynamic evolution characteristics of overburden mining fissures and separations are obtained. • The overburden system state is quantitatively analyzed based on the information entropy theory. The water inrush from overburden separation is large and destructive, which is very easy to cause serious casualties and property losses. Therefore, a three-dimensional model of 150 cm × 150 cm × 60 cm (length × height × width) is established in the laboratory. And the dynamic development process of overburden separations and water flowing fractures in the process of coal mining in real time are analyzed based on distributed optical fiber and information entropy theory. The distributed optical fiber strain curve can well reflect the process and characteristics of overburden deformation and collapse. The strain curve of optical fiber shows a single peak or step shape at different mining stages. The separation position corresponds to the peak of the optical fiber strain curve or above the step. The change of strain value of optical fiber reflects the development, expansion and closure process of separation. The development height of the water flowing fractured zone corresponds to the lower part of the step of the strain curve. Strain information entropy realizes quantitative analysis of overburden system state. Strain information entropy can be divided into small amplitude rapid fluctuation period, sharp rapid adjustment period and stable slow fluctuation period. Water flowing fractures and separations are mainly produced during the period of the sharp rapid adjustment period. The height of the water flowing fractured zone and the height of the new separation increase step by step with the advance of the working face. The water flowing fractured zone finally develops to 86 cm of the model height. The highest position of the separation appears at the 112 cm height of the model. A new variable named growth rate of strain information entropy is defined, which can reflect the severity of the change of the overburden system state. When the key stratum breaks, the growth rate of strain information entropy increases significantly, which show that the key stratum plays a key role in controlling the overburden deformation process. In addition, the optical fiber strain curve can also reflect important information such as the change of advance support pressure, the distribution state of gangue in goaf and the breaking characteristics of key strata. [ABSTRACT FROM AUTHOR]

An unplanned breakdown in power generation or chemical plant due to the component failure of turbomachines often results in a huge loss of property and productivity as well as a significant increase in maintenance costs. It has become of paramount importance to predict component damage in a turbomachine using instrumented data. Most existing models, however, are obtained from multiple assumptions, resulting in a high false detection ratio due to various data uncertainties. In this article, we present a novel model-free probabilistic methodology for damage detection to resolve the drawbacks of the classical methods. The proposed method adeptly integrates Bayesian inference, wavelets signal processing, probabilistic principal components analysis, and entropy information theory. Bayesian inference is developed for denoising raw data by integrating with multiscale discrete wavelet packets transform and reducing multivariate dimension by combining with principal components analysis. The entropy information theory has been proposed to extract the feature from principal components as a precursor of the event. A multimetric hierarchical alerting strategy is proposed to predict component damage to enhance the accuracy. The feasibility of the presented novel pattern recognition methodology is demonstrated with the detection of blade damage events in a real-world steam turbine using sensor data. [ABSTRACT FROM AUTHOR]

This work presents an axiomatization for entropy based on an extension of concavity called core-concavity. We show that core-concavity characterizes the largest class of functions for which the data-processing inequality holds, under the assumption that conditional entropy is defined as a generalized average. Also, under the same assumption, we show that data-processing and “conditioning reduces entropy” properties are equivalent. We prove several properties of core-concave functions, including generalization of perfect secrecy and of Fano’s inequality. We also show that definitions of conditional entropy based on worst-case can be retrieved as limit cases of generalized averages. A connection between statistical decision making and this axiomatic approach is also presented. [ABSTRACT FROM AUTHOR]

A measure of health based on exact sciences is not considered by social sciences to evaluate human development. We propose the H measure of information of Shannon ([1948]. "A Mathematical Theory of Communication." The Bell Systems Technical Journal 27 (3): 379–423, 623–656), renamed as Hcto, as a general indicator of human health which can be subsequently aggregated at any statistical level. Several advantages of Hcto are explained: (i) connections with thermodynamics and information theory; (ii) combination of physical and statistical elements; (iii) measurements of entropy and/or information depending on the reference point, and (iv) a collection of examples based on the scientific literature that supports the application of Hcto. Simplified graphical models of the ontogeny of health condition are also shown. The empirical feasibility of using Hcto is also analysed, by concluding that it could be a tangible reality in the near future. [ABSTRACT FROM AUTHOR]

Entropy, as a complexity measure, has been widely applied for time series analysis. One preeminent example is the design of machine condition monitoring and industrial fault-diagnostic systems. The occurrence of failures in a machine will typically lead to nonlinear characteristics in the measurements, caused by instantaneous variations, which can increase the complexity in the system response. Entropy measures are suitable to quantify such dynamic changes in the underlying process, distinguishing between different system conditions. However, notions of entropy are defined differently in various contexts (e.g., information theory and dynamical systems theory), which may confound researchers in the applied sciences. In this article, we have systematically reviewed the theoretical development of some fundamental entropy measures and clarified the relations among them. Then, typical entropy-based applications of machine fault-diagnostic systems are summarized. Furthermore, insights into possible applications of the entropy measures are explained, as to where and how these measures can be useful toward future data-driven fault diagnosis methodologies. Finally, potential research trends in this area are discussed, with the intent of improving online entropy estimation and expanding its applicability to a wider range of intelligent fault-diagnostic systems. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the partition inequality, joint convexity, and Pinsker’s inequality, for a divergence that generalizes the Tsallis Relative Entropy and Kullback–Leibler divergence. The generalized divergence is defined in terms of a deformed exponential function, which replaces the Tsallis $q$ -exponential. We also constructed a family of probability distributions related to the generalized divergence. We found necessary and sufficient conditions for the partition inequality to be satisfied. A sufficient condition for the joint convexity was established. We proved that the generalized divergence satisfies the partition inequality, and is jointly convex, if, and only if, it coincides with the Tsallis relative entropy. As an application of partition inequality, a criterion for the Pinsker’s inequality was found. [ABSTRACT FROM AUTHOR]

A $(k, \varepsilon)$ -non-malleable extractor is a function $ {\sf nmExt}: \{0,1\}^{n} \times \{0,1\}^{d} \to \{0,1\}$ that takes two inputs, a weak source $X \sim \{0,1\}^{n}$ of min-entropy $k$ and an independent uniform seed $s \in \{0,1\}^{d}$ , and outputs a bit $ {\sf nmExt}(X, s)$ that is $ \varepsilon $ -close to uniform, even given the seed $s$ and the value $ {\sf nmExt}(X, s')$ for an adversarially chosen seed $s' \neq s$. Dodis and Wichs (STOC 2009) showed the existence of $(k, \varepsilon)$ -non-malleable extractors with seed length $d = \log (n-k-1) + 2\log (1/ \varepsilon) + 6$ that support sources of min-entropy $k > \log (d) + 2 \log (1/ \varepsilon) + 8$. We show that the foregoing bound is essentially tight, by proving that any $(k, \varepsilon)$ -non-malleable extractor must satisfy the min-entropy bound $k > \log (d) + 2 \log (1/ \varepsilon) - \log \log (1/ \varepsilon) - C$ for an absolute constant $C$. In particular, this implies that non-malleable extractors require min-entropy at least $\Omega (\log \log (n))$. This is in stark contrast to the existence of strong seeded extractors that support sources of min-entropy $k = O(\log (1/ \varepsilon))$. Our techniques strongly rely on coding theory. In particular, we reveal an inherent connection between non-malleable extractors and error correcting codes, by proving a new lemma which shows that any $(k, \varepsilon)$ -non-malleable extractor with seed length $d$ induces a code $ {\mathcal C} \subseteq \{0,1\}^{2^{k}}$ with relative distance $ \frac {1}{2}- 2 \varepsilon $ and rate $\frac {d-1}{2^{k}}$. [ABSTRACT FROM AUTHOR]

We present a sample path dependent measure of causal influence between time series. The proposed causal measure is a random sequence, a realization of which enables identification of specific patterns that give rise to high levels of causal influence. We show that these patterns cannot be identified by existing measures such as directed information (DI). We demonstrate how sequential prediction theory may be leveraged to estimate the proposed causal measure and introduce a notion of regret for assessing the performance of such estimators. We prove a finite sample bound on this regret that is determined by the worst case regret of the sequential predictors used in the estimator. Justification for the proposed measure is provided through a series of examples, simulations, and application to stock market data. Within the context of estimating DI, we show that, because joint Markovicity of a pair of processes does not imply the marginal Markovicity of individual processes, commonly used plug-in estimators of DI will be biased for a large subset of jointly Markov processes. We introduce a notion of DI with “stale history”, which can be combined with a plug-in estimator to upper and lower bound the DI when marginal Markovicity does not hold. [ABSTRACT FROM AUTHOR]

Point processes are often described with functionals, such as the probability generating functional, the Laplace functional, and the factorial cumulant generating functional. These are used to facilitate modelling of different processes and to determine important statistics via functional differentiation. In information theory, generating functions have also been defined for probability densities to determine information quantities such as the Shannon information and Kullback-Leibler divergence, though as yet there are no such analogues for point processes. The purpose of this article is to exploit the advantages of both types of generating function to facilitate the derivation of information statistics for point processes. In particular, a generating functional for point processes is introduced for determining statistics related to entropy and relative entropy based on Golomb’s information function and Moyal’s probability generating functional. It is shown that the information generating functional permits the derivation of a suite of statistics, including localised Shannon entropy and Kullback-Leibler divergence calculations. [ABSTRACT FROM AUTHOR]

This paper concerns the folklore statement that "entropy is a lower bound for compression." More precisely, we derive from the entropy theorem a simple proof of a pointwise inequality first stated by Ornstein and Shields and which is the almost-sure version of an average inequality first stated by Khinchin in 1953. We further give an elementary proof of the original Khinchin inequality, which can be used as an exercise for information theory students, and we conclude by giving historical and technical notes of such inequality. [ABSTRACT FROM AUTHOR]

We study random string-duplication systems, which we call Pólya string models. These are motivated by a class of mutations that are common in most organisms and lead to an abundance of repeated sequences in their genomes. Unlike previous works that study the combinatorial capacity of string-duplication systems, or in a probabilistic setting, various string statistics, this work provides the exact entropy rate or bounds on it, for several probabilistic models. The entropy rate determines the compressibility of the resulting sequences, as well as quantifying the amount of sequence diversity that these mutations can create. In particular, we study the entropy rate of noisy string-duplication systems, including the tandem-duplication, end-duplication, and interspersed-duplication systems, where in all cases we study duplication of length 1 only. Interesting connections are drawn between some systems and the signature of random permutations, as well as to the beta distribution common in population genetics. [ABSTRACT FROM AUTHOR]

The paper presents four distinct new ideas and results for communication networks: 1) We show that relay-networks (i.e. communication networks where different nodes use the same coding functions) can be used to model dynamic networks, in a way, akin to Kripke's possible worlds. Changes in the network are modelled by considering a multiverse where different possible situations arise as worlds existing in parallel. 2) We introduce the term model , which is a simple, graph-free symbolic approach to communication networks. This model yields an algorithm to calculate the capacity of a given communication network. 3) We state and prove variants of a theorem concerning the dispersion of information in single-receiver communications. The dispersion theorem resembles the max-flow min-cut theorem for commodity networks. The proof uses a very weak kind of network coding, called routing with dynamic headers. 4) We show that the solvability of an abstract multi-user communication problem is equivalent to the solvability of a single-target communication in a suitable relay network. In the paper, we develop a number of technical ramifications of these ideas and results. We prove a max-flow min-cut theorem for the Rényi entropy with order less than one, given that the sources are equiprobably distributed; conversely, we show that the max-flow min-cut theorem fails for order greater than one. We also show that linear network coding fails for relay networks, although routing with dynamic headers is asymptotically sufficient to reach capacity. [ABSTRACT FROM AUTHOR]

We develop a general method for computing logarithmic and log-gamma expectations of distributions. As a result, we derive series expansions and integral representations of the entropy for several fundamental distributions, including the Poisson, binomial, beta-binomial, negative binomial, and hypergeometric distributions. Our results also establish connections between the entropy functions and to the Riemann zeta function and its generalizations. [ABSTRACT FROM AUTHOR]

Calculating the Shannon entropy for symbolic sequences has been widely considered in many fields. For descriptive statistical problems such as estimating the N-gram entropy of English language text, a common approach is to use as much data as possible to obtain progressively more accurate estimates. However, in some instances, only short sequences may be available. This gives rise to the question of how many samples are needed to compute entropy. In this paper, we examine this problem and propose a method for estimating the number of samples required to compute Shannon entropy for a set of ranked symbolic “natural” events. The result is developed using a modified Zipf-Mandelbrot law and the Dvoretzky–Kiefer–Wolfowitz inequality, and we propose an approximation which yields an estimate for the minimum number of samples required to obtain an estimate of entropy with a given confidence level and degree of accuracy. [ABSTRACT FROM AUTHOR]

The sampling method is widely used for the quantification of reactor physics analysis uncertainty, while the traditional statistical sampling methods often require a huge number of samples to ensure the stability of uncertainty quantification results. For this reason, a new two-step sampling scheme based on the combination of information entropy theory and moment matching technique is proposed for the quantification of the reactor physics uncertainty. We construct a convex programming model to optimize initial samples by simultaneously maximizing information entropy and minimizing moment deviations. The optimized samples can not only characterize the statistics of the real distribution better, but also control the information entropy with a smaller sample size. Moreover, a quantitative criterion for sample selection induced by the maximum entropy and moment matching is proposed. Finally, a series of empirical studies are carried out to show the capability and efficiency of the introduced two-step sampling method. • A quantitative criterion for selecting samples is introduced. • A convex optimization model is constructed to optimize the initial sample set. • A new two-step sampling method is proposed for uncertainty quantification. • The proposed method can reduce the sample size significantly. • The stability of the uncertain quantitative results is enhanced by the MEMM method. [ABSTRACT FROM AUTHOR]

Abstract In many real-world problems, data samples are simultaneously associated with multiple labels, instead of a single label. Multi-label classification deals with such problems, and has extensive applications in many fields. Among the many methods proposed for multi-label classification tasks, classifier chains (CC) is an appealing one. In the classifier chains method, the label order has a strong effect on the classification performance. However, it is difficult to determine a proper order. In this paper, we propose ordering methods based on the conditional entropy of labels. We generate a single order instead of multiple orders. Unlike existing ordering methods, there is no need to train more classifiers than CC. Experimental results on nine benchmark datasets evaluated by eight measures show that the proposed methods achieve good performance. [ABSTRACT FROM AUTHOR]

Abstract The main purpose of this work is to investigate the relation between some measures in information theory and the accuracy of volatility forecasting using a model of asset returns. First we highlight the dependence between volatility forecasting and entropy and then we determine the relation between predictability and volatility. The study is conducted using a database of 65 stocks of the Dow Jones Composite Average from 1973 to 2014 and by computing the daily volatility of the market index. To this end we use the standard GARCH approach to model and forecast the daily volatility. The main result of this paper is the establishment of a relationship between the accuracy of the volatility forecast and the entropy of the time series of price returns. Since the entropy changes in time, before computing a forecast of the volatility it is recommended to compute the entropy of the time series that furnishes an important indicator on the limit of successive volatility forecast. Highlights • We apply Information Theory to financial markets. • We show the relation between entropy e predictability. • We apply our analysis on a DJ index. [ABSTRACT FROM AUTHOR]

Abstract We study the direction of information flow between Ibovespa index and its constituent companies using the Transfer entropy method. We find stronger information transfer from individual stocks towards the composite index, than in the opposite direction. Our results differ from those found for developed market, where market index was identified as driving force, indicating that the role of index increases with the maturity of the market. Highlights • This article applied the transfer entropy method to analyze the information flow of Brazilian Market Index (Ibovespa) and its constituents. • An asymmetric behavior was found about the information transfer between the Ibovespa and its constituents. • The analysis procedure employed shown that the companies have more impact on the Ibovespa than the Ibovespa impacts on the companies. • The transfer entropy analysis was done for three periods: before the 2008 crisis, in the 2008 crisis, and after 2008 crisis. [ABSTRACT FROM AUTHOR]

There is no known strategy to differentiate which multicomponent electrograms in sinus rhythm maintain reentrant ventricular tachycardia (VT). Low entropy in the voltage breakdown of a multicomponent electrogram can localize conditions suitable for reentry but has not been validated against the classic VT activation mapping. We examined whether low entropy in a late and diversely activated ventricular scar region characterizes and differentiates the diastolic path of VT and represents protected tissue channels devoid of side branches. Intraoperative bipolar electrogram (BiEGM) activation and entropy maps were obtained during sinus rhythm in 17 patients with ischemic cardiomyopathy and compared with diastolic activation paths of VT (total of 39 VTs). Mathematical modeling of a zigzag main channel with side branches was also used to further validate structural representation of low entropy in the ventricular scar. A median of one region per patient (range: 1-2 regions) was identified in sinus rhythm, in which BiEGM with the latest mean activation time and adjacent minimum entropy were assembled together in a high-activation dispersion region. These regions accurately recognized diastolic paths of 34 VTs, often to multiple inducible VTs within a single individual arrhythmogenic region. In mathematical modeling, side branching from the main channel had a strong influence on the BiEGM composition along the main channel. The BiEGM obtained from a long unbranched channel had the lowest entropy compared with those with multiple side branches. In conclusion, among a population of multicomponent sinus electrograms, those that demonstrate low entropy and are delayed colocalize to critical long-protected channels of VT. This information is pertinent for planning VT ablation in sinus rhythm. [ABSTRACT FROM AUTHOR]

Healthcare communication on Twitter is challenging because the space for a tweet is limited, but the topic is too sophisticated to be concise. Comparing medical-terminology hashtags versus lay-language hashtags, this paper explores the characteristics of healthcare hashtags using an entropy matrix which derived from information theory. In this paper, the entropy matrix comprises of six different components used for constructing a tweet and serves as a framework for the structural analysis with the granularity of tweet composition. These granular components include image(s), text with semantic meanings, hashtag(s), @ username(s), hyperlink, and unused space. The entropy matrix proposed in this paper contributes to a new approach to visualizing the complexity level of hashtag collections. In addition, the calculated entropy could be an indicator of the diversity of a user's choice across those tweet components. Furthermore, the visualizations (radar graph and scatterplot) illustrate statistical structures and the dynamics of the hashtag collections measured by entropy. The results from this study demonstrate a manifest relationship between tweet composition and the number of being retweeted. [ABSTRACT FROM AUTHOR]

Electroencephalograms reflect the electrical activity of the brain, which can be considered ruled by a chaotic nonlinear dynamics. We consider human electroencephalogram recordings during different motor type activities, and when imagining that they perform this activity. We characterize the different dynamics of the cortex according to distinct motor and imagined movement tasks using an information theory approach and a wavelet decomposition. More specifically, we use the entropy-complexity plane H × C in combination with the wavelet decomposition to precisely quantify the dynamics of the neuronal activity showing that the current theoretical framework allows us to distinguish realized and imagined tasks within the cortex. [ABSTRACT FROM AUTHOR]

Understanding information processing in the brain requires the ability to determine the functional connectivity between the different regions of the brain. We present a method using transfer entropy to extract this flow of information between brain regions from spike-train data commonly obtained in neurological experiments. Transfer entropy is a statistical measure based in information theory that attempts to quantify the information flow from one process to another, and has been applied to find connectivity in simulated spike-train data. Due to statistical error in the estimator, inferring functional connectivity requires a method for determining significance in the transfer entropy values. We discuss the issues with numerical estimation of transfer entropy and resulting challenges in determining significance before presenting the trial-shuffle method as a viable option. The trial-shuffle method, for spike-train data that is split into multiple trials, determines significant transfer entropy values independently for each individual pair of neurons by comparing to a created baseline distribution using a rigorous statistical test. This is in contrast to either globally comparing all neuron transfer entropy values or comparing pairwise values to a single baseline value. In establishing the viability of this method by comparison to several alternative approaches in the literature, we find evidence that preserving the inter-spike-interval timing is important. We then use the trial-shuffle method to investigate information flow within a model network as we vary model parameters. This includes investigating the global flow of information within a connectivity network divided into two well-connected subnetworks, going beyond local transfer of information between pairs of neurons. [ABSTRACT FROM AUTHOR]

Fundamental relations between information and estimation have been established in the literature for the continuous time Gaussian and Poisson channels. In this paper, we demonstrate that such relations hold for a much larger family of continuous-time channels. We introduce the family of semi-martingale channels where the channel output is a semi-martingale stochastic process, and the channel input modulates the characteristics of the semi-martingale. For these channels, which includes as a special case the continuous time Gaussian and Poisson models, we establish new representations relating the mutual information between the channel input and output to an optimal causal filtering loss, thereby unifying and considerably extending results from the Gaussian and Poisson settings. Extensions to the setting of mismatched estimation are also presented where the relative entropy between the laws governing the output of the channel under two different input distributions is equal to the cumulative difference between the estimation loss incurred by using the mismatched and optimal causal filters, respectively. The main tool underlying these results is the Doob–Meyer decomposition of a class of sub-martingales. The results in this paper can be viewed as the continuous-time analogues of recent generalizations for relations between information and estimation for discrete-time Lévy channels. [ABSTRACT FROM AUTHOR]

We prove a number of quantitative stability bounds for the cases of equality in Petz’s monotonicity theorem for quasi-relative entropies $S_{f}(\rho ||\sigma)$ defined in terms of an operator monotone decreasing functions $f$ and, in particular, the Rényi relative entropies. Included in our results are bounds in terms of the Petz recovery map, but we obtain more general results. The present treatment is entirely elementary and developed in the context of finite dimensional von Neumann algebras, where the results are already non-trivial and of interest in quantum information theory. [ABSTRACT FROM AUTHOR]

Estimating mutual information from independent identically distributed samples drawn from an unknown joint density function is a basic statistical problem of broad interest with multitudinous applications. The most popular estimator is the one proposed by Kraskov, Stögbauer, and Grassberger (KSG) in 2004 and is nonparametric and based on the distances of each sample to its $k^{\mathrm{ th}}$ nearest neighboring sample, where $k$ is a fixed small integer. Despite of its widespread use (part of scientific software packages), theoretical properties of this estimator have been largely unexplored. In this paper, we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the $\ell _{2}$ error as a function of a number of samples. We argue that the performance benefits of the KSG estimator stems from a curious “correlation boosting” effect and build on this intuition to modify the KSG estimator in novel ways to construct a superior estimator. As a by-product of our investigations, we obtain nearly tight rates of convergence of the $\ell _{2}$ error of the well-known fixed $k$ -nearest neighbor estimator of differential entropy by Kozachenko and Leonenko. [ABSTRACT FROM AUTHOR]

We consider denumerable stochastic processes with (or without) memory. Their evolution is encoded by a finite or infinite rooted tree. The main goal is to compare the entropy rates of a given base process and a second one, to be considered as a perturbation of the former. The processes are described by probability measures on the boundary of the given tree and by corresponding forward transition probabilities at the inner nodes. The comparison is in terms of Kullback–Leibler divergence. We elaborate and extend ideas and results of Böcherer and Amjad. Our extensions involve length functions on the edges of the tree as well as nodes with countably many successors. In particular, in , we consider trees with infinite nonbacktracking paths and random perturbations of a given process. [ABSTRACT FROM AUTHOR]

We establish the rate region of an extended Gray–Wyner system (EGW) for 2-DMS $(X,Y)$ with two additional decoders having complementary causal side information. We show that the 5-D rate region of the EGW system is equivalent to the 3-D mutual information region consisting of the set of all triples of the form $(I(X;U),\,I(Y;U),\,I(X,Y;U))$ for some $p_{U|X,Y}$. This correspondence greatly simplifies the exploration of the the extreme points of the rate region. In addition to the operationally significant extreme points of the original Gray–Wyner rate region, which include Wyner’s common information, Gács-Körner common information, and the information bottleneck, the extreme points of the rate region for the EGW system also include the Körner graph entropy, the privacy funnel and excess functional information, as well as three new quantities of potential interest. We further show that projections of the mutual information region yield the rate regions for many settings involving a two discrete memoryless source (2-DMS), including lossless source coding with causal side information, distributed channel synthesis, and lossless source coding with a helper. To further motivate the mutual information region itself, we draw analogies between set operations and its extreme points. This allows us to find random variables that can be considered as intersection, difference, and symmetric difference of two random variables. Finally, we establish the rate regions for two related setups to the EGW system, namely, the noncausal EGW system and the lossy EGW system. [ABSTRACT FROM AUTHOR]

“Bounds on information combining” are entropic inequalities that determine how the information (entropy) of a set of random variables can change when these are combined in certain prescribed ways. Such bounds play an important role in classical information theory, particularly in coding and Shannon theory; entropy power inequalities are special instances of them. The arguably most elementary kind of information combining is the addition of two binary random variables (a CNOT gate), and the resulting quantities play an important role in belief propagation and polar coding. We investigate this problem in the setting where quantum side information is available, which has been recognized as a hard setting for entropy power inequalities. Our main technical result is a non-trivial, and close to optimal, lower bound on the combined entropy, which can be seen as an almost optimal “quantum Mrs. Gerber’s Lemma”. Our proof uses three main ingredients: 1) a new bound on the concavity of von Neumann entropy, which is tight in the regime of low pairwise state fidelities; 2) the quantitative improvement of strong subadditivity due to Fawzi-Renner, in which we manage to handle the minimization over recovery maps; and 3) recent duality results on classical-quantum-channels due to Renes et al. We furthermore present conjectures on the optimal lower and upper bounds under quantum side information, supported by interesting analytical observations and strong numerical evidence. We finally apply our bounds to polar coding for binary-input classical-quantum channels, and show the following three results: 1) even non-stationary channels polarize under the polar transform; 2) the blocklength required to approach the symmetric capacity scales at most sub-exponentially in the gap to capacity; and 3) under the aforementioned lower bound conjecture, a blocklength polynomial in the gap suffices. [ABSTRACT FROM AUTHOR]

We consider a stochastic process and a given $n$ -string. We study the shortest possible return time (or shortest return path) of the string over all the realizations of process starting from this string. For a $\beta $ -mixing process having complete grammar, and for each size $n$ of the strings, we approximate the distribution of this short return (properly re-scaled) by a non-degenerated distribution. Under mild conditions on the $\beta $ coefficients, we prove the existence of the limit of this distribution to a non-degenerated distribution. We also prove that ergodicity is not enough to guaranty this convergence. Finally, we present a connection between the shortest return and the Shannon entropy, showing that maximum of the re-scaled variables grow as the matching function of Wyner and Ziv. [ABSTRACT FROM AUTHOR]