Search

‘Information transmitted’ is defined as the amount by which added evidence (or ‘message received’) diminishes ‘uncertainty’. The latter is characterized by some properties intuitively suggested by this word and possessed by conditional entropy, a parameter of the posterior probability distribution. However, conditional entropy shares these properties with some other concave symmetric functions on the probability space. Moreover, a given transmission channel (or, in the context of statistical inference, a given experiment) yields a higher maximum expected benefit than anotherto any user if and only ifall concave functions of the posterior probability vector have higher values for the former channel (or experiment). Hence one information system (channel, experiment) may be preferable to another for a given user although its transmission rate, in entropy terms, is lower. But only entropy has the economically relevant property of measuring, in the limit, the expected length of efficiently coded messages sent in long sequences. Thus, while irrelevant to the value (maximum expected benefit) of an information system and to the costs of observing, estimating, and deciding, entropy formulas are indeed relevant to the cost of communicating, i.e., of storing, coding and transmitting messages.

Although mutual information (MI) has been proposed for some time as a measure of the dependence between the class variable and pattern recognition features, it is only recently that the practical problems of designing computer programs to use MI have been raised. Within the two-class context, this paper compares two traditional approaches to the requisite entropy estimation (using the maximum likelihood and expected value estimators of class probabilities) with a new estimator: the expected value of binomial entropy (E). The latter is shown to be superior where one class has a priori dominance. E is also related to expected probability of error and, in a surprising result, it is shown that E is a better estimator of class probabilities than the maximum likelihood and expected value estimators over a wide range.

In this paper the entropy functional is derived in the context of decision making under uncertainty. It is shown that the mathematical description of a decision problem can be partitioned into two parts: a probability distribution p and a convex set LN(A) that describes the remaining structure of the problem. The question of selecting a general representative form for this convex set is explored, and a set of assumptions is proposed that specifies a form for such a representative set LN. Corresponding to this set LN, the entropy of the probability p can be interpreted as an information measure with respect to decision problems. In addition, it is shown that if this set LN is substituted for the set LN(A) in a given decision problem, the use of the maximum entropy criterion for probability selection corresponds to a minimax solution to the representative form of the problem. To the extent that the set LN resembles the set LN(A) in a given decision problem, the use of the maximum entropy criterion for probability selection corresponds to a minimax criterion for decision making.

Information-theoretic concepts are developed and employed to obtain conditions for a minimax error entropy stochastic approximation algorithm to estimate the state of a non-linear discrete time system baaed on noisy linear measurements of the state. Two recursive suboptimal error entropy estimation procedures are presented along with an upper bound formula for the resulting error entropy. A simple example is utilized to compare the optimal and suboptimal error entropy estimators and the minimum mean Square error linear estimator.

We discuss the Baron-Jauch definition of entropy and show that it is the unique answer to an entropy which has the properties of extensivity, positivity and continuity in a weak sense. As an application we also show how one easily can derive the canonical distribution from this definition of entropy using information theoretical arguments.

The conditional entropy between two states of a quantum system is shown to be nonincreasing when a complete measurement is performed on the system. The information between two quantum systems is defined and is shown to be bounded above by the logarithmic correlation. This inequality is then applied to the measurement process. The entropy changes in the observed system and the measuring apparatus are compared with the information gain in the measurement.

The quantum analogue of the classical theory of the joint microcanonical entropy of a pair of observables is investigated for a system of a large number of identical non-interacting subsystems. It is shown that the quantum joint entropy coincides with the classical joint entropy of an appropriately chosen auxiliary classical system, and known results for classical systems are applied to prove the equivalence of the quantum microcanonical and quantum canonical ensembles.

In the representation of natural catchment systems with mathematical models, a relatively wide range of choice exists among models of various degrees of completeness and sophistication. When the cost, as well as the appropriateness, of a model is considered for a particular purpose, it is desirable to have an objective criterion to make this selection. The concept of entropy as used in information theory provides one such criterion. Entropy is a measure of the degree of uncertainty of a particular outcome in a process; therefore, in dealing with the prediction of a hydrologie variable such as streamflow, one can compute the entropy of this variable from historical data and thus characterize the unexpectedness or variability inherent in the process. This represents a property of the system and is called marginal entropy. It is likewise possible to evaluate the uncertainty of the predictions made by a given mathematical model of the catchment by comparing these predictions with the measured flow. This is done through the conditional entropy. By combination of these two entropy functions a criterion called ‘transinformation,’ providing an objective evaluation of the goodness of the model, is obtained. The important point to note is that this assessment depends not only on the model but on the characteristics of the output series resulting from the natural process. The above criteria, which have been discussed in other contexts, were applied to a basin for which a number of years of historical flow record and concurrent model simulations were available. The results show the value of the entropy criterion in judging model performance as well as the difficulties and limitations of the method.

It is shown that the nonequilibrium statistical operator which was obtained previously by one of us (D.N.Z.) can be derived from the requirement that the information entropy of the nonequilibrium system must have an extremum with a set of additional conditions. The latter conditions imply that the average values of dynamical variables which determine the macroscopic state of the system are given at any moment of the past t1 in the time interval -∞ ≤ t1 ≤ t.

1. In normalizing the precision of measurements and measuring equipment it is necessary to use in addition to the entropy error also information criteria which account for the information content obtained in measurement as well as the particular features of the error distribution law. 2. As a practical criterion it is advisable to use the maximum information error which is defined as a boundary of the zones within which the main part of the error entropy power is concentrated. The loss of entropy due to the narrowing of the integration band can then be normalized on the basis of the insignificant error criterion. 3. From the information theory point of view it is possible to consider as insignificant the errors which change the error entropy by not more then 0.07 bit. 4. For characterizing various error distribution laws it is advisable to adopt the concepts of the entropy density and the entropy integral function. 5. The particular features of the error distribution can be represented by adopting the entropy density factor which characterizes the nonuniformity of the entropy density distribution within the maximum information error band. Uniform distribution possesses the maximum mean entropy concentration per unit length of the error band for a given entropy value. 6. For a normal distribution law the maximum information error determined on the basis of the 0.07 bit criterion amounts to 2.6σ.

It is proposed to use the time rate of decrease of the entropy defined by the response probabilities as a measure of the speed of learning. An empirical formula, see

The properties and algebraic structure of error correcting codes are shown to be applicable to adaptive pattern recognition. Mappings to encode pattern classes into a linear vector space are discussed. The relationship of that space to a polynominal ring with its ideal and cosets is developed, and features and pattern classes are presented in terms of their contribution to the elements of the vector space. The concept that each feature conveys information and thus has entropy is developed in its relation to candidate vectors and their joint information with ideal elements. The redundancy of feature information is utilized, and a decision algorithm is postulated upon the basis of a criterion of maximum joint entropy in excess of a calculable minimum. The correspondence of the decision criterion to a maximum likelihood decision is noted, and its relation to a learning algorithm is presented. The methodology appears to be general but requires a determination of the information value of features. A hierarchical network structure for higher order recognition sequences is an extension of the methodology presented, but work in this area is not yet sufficiently complete for presentation.

For a residual subset of the space of discrete stationary stochastic processes (under the weak topology for measures) the entropy is 0. The processes with arbitrarily large entropy are dense. For a residual subset of subshifts of the shift on a finite alphabet, the topological entropy is 0.

It has been shown by several authors that in operations involving information a quantity appears which is the negative of the quantity usually defined as entropy in similar situations. This quantity ℜ = − KI has been termed “negentropy” and it has been shown that the negentropy of information and the physical entropy S are mirrorlike representations of the same train of events. In physical terminology the energy is degraded by an increase in entropy due to an increased randomness in the positions or velocities of components, wave functions, complexions in phase space; in informational terminology some information about the same components has been lost or the negentropy has been decreased. In equilibrium the system has for a given energy content maximum randomness (or minimum information is required). One consequence of this dual aspect was the idea to apply the methods of statistical mechanics to problems of communication and Brillouin showed that Fermi-Dirac statistics or generalized Fermi statistics are applicable for example to a transmission of signals such as telegrams.

The switching variables in a switching circuit are considered as binary random variables. The input signals are given joint probabilities, from which the output signals' probabilities are known. The joint entropy, or average information content, of subsets of variables is defined, and a proof is given that a set of these entropies are characteristic invariants of the PCN equivalence classes.

An inequality for two positive operators is used to discuss entropy theorems for time smoothing, mixing, and other processes. Especially, it is proved that neglect of nondiagonal matrix elements for the density matrix causes the entropy to increase.

In this paper the entropy of a point process is defined. This entropy can be expressed as a sum of two terms: (a) the numerical entropy, associated with the number of events in a fixed interval, and (b) the locational entropy, associated with the locations of those events in the interval. For point processes describable as a generalized birth process, with fixed intensity function, the rate of change of entropy at a given time is a maximum for the inhomogeneous Poisson process. For the mixed Poisson process with fixed intensity, the locational entropy for a given interval is a maximum for the Poisson process. Other properties of the numerical and locational entropies are also discussed.

The use of the maximum entropy principle for determining prior distribution is compared with other techniques in statistical-decision theory for estimating reliability. The comparison is made in the context of estimating the parameter representing the probability of success in a binomial model and the parameter representing the mean time to failure in a simple exponential model. The existence of partial prior information of varying degrees is also assumed. When a quadratic loss function is used, the techniques based on the maximum entropy principle lose much of their appeal.

The laws of statistical thermodynamics are used for the definition of entropy, and it is shown that the definition of information can be reduced to a problem of Fermi‐Dirac statistics or to a generalized Fermi statistics. With these definitions, the entropy of a certain message can be defined, and the information contained in the message can be directly connected with the decrease of entropy in the system.This definition leads directly to the formulas proposed by C. E. Shannon for the measure of information, and shows that Shannon's ``entropy of information'' corresponds to an equal amount of negative entropy in the physical system. The physical background of the whole method is discussed and found in agreement with previous discussions.

The processes connected with the entropy of measured quantum mechanical objects are investigated. The entropic characteristics of the measurement process are determined by means of the so-called entropy balance. It is shown that the Shannon entropy of quantal objects described by non-commutative operators is always positive in the post-measurement state. From the entropy balance of quantum mechanical measurement also follows that the measurement of the characteristics of quantal objects provides some information on quantities not measured.

Entropy production, originated from an ideal gas second-order reaction taking place in a closed system, has been stochastically formulated by defining the entropy in a form similar to the conditional entropy used in the theory of information. The entropy is introduced in the natural way as a result of the partial factorization of joint probability for reaction states devised for the approximate solution of the stochastic process of the second-order reaction. The entropy production has also been discussed in comparison with the usual one.

The aim of this paper is to provide a mathematically rigorous and sufficiently general treatment of the basic information-theoretic problems concerning sources with symbols of different costs and noiseless coding in a general sense. The main new concepts defined in this paper are the entropy rate (entropy per unit cost) of a source with respect to a stochastic cost scale and the encoding (in particular decodable encoding) of a source in a general sense. On the basis of these concepts, we prove some general theorems on the relation of entropy rates with respect to different cost scales and on the effect of encoding to the entropy rate. In particular, the “principle of conservation of entropy” and the “noiseless coding theorem” are proved under very general conditions.

Systems investigated by geographers include components of inert nature, living organisms and social phenomena. The use of information theory makes it possible to overcome some of the problems involved in the modeling of systems consisting of such a diversity of elements, particularly the problem of a single system of units. The building of inductive models of natural systems is designed to identify the programs of behavior of natural complexes, which in the original systems are merged with structure. An example of such an inductive model is the model of the structure of a southern tayga landscape system. It was built by identifying information relationships between 160 elements on the basis of the equation: JAB = HA+HB—HAB, where JAB is the amount of transmitted information, and HA, HB, HAB are the entropies of the receiver, of the transmitter and their joint entropy. The model interrelated a number of properties of soils, unconsolidated sediment, groundwater, relief, plant cover and microclimate....

A study of the use of entropy as a criterion function for analyzing the performance of sampled-data estimating systems is presented, and performance bounds are obtained for a broad class of such systems. The general result is that the difference between the entropy of the signal to be estimated and the entropy of its estimate based on the output of a noisy sensor can never be larger than the mutual information between the sensor input and output. An interesting, but not totally satisfactory, sufficient condition for attainment of the bound is developed.

There have been several axiomatic characterizations [1, 2, 3, 4] of Shannon's entropy and more recently [5, 6, 7, 8] of R6nyi's entropy of order g. These axioms have been chosen to provide a convenient measure of the uncertainty of the outcome of a random experiment. However, these axiomatic characterizations have little apparent connection with the coding theorems of information theory. In this note a generalized entropy is defined in a way tha t connects entropy directly with a coding problem. I f additional hypotheses are imposed this generalized entropy becomes the l~6nyi or Shannon entropy. In fact, the final theorem of this paper is a characterization of the entropy of order g which is different from those mentioned above. Let X ~ { x l , x2 . . . . . Xm} be a finite set of events and let P : (Pl, P2, . . . , Pro) be an associated distribution of probabilities, so tha t the probabili ty of x~ is p~ and p~ : 1. Suppose tha t we wish to represent the events in X by finite sequences of elements of the set {0, 1 . . . . , D 1} where D > 1. There is a uniquely decipherable code [9] which represents x~ by a sequence of n~ elements ff and only if the integers n~ satisfy the inequality

n/a

1. An information-theory approach to the evaluation of errors in measurements or measuring instruments serves to obtain a single effective determination of the error in the form of its entropy value, which determines unambiguously the error entropy and, hence, the information content obtained as a result of measurements. 2. The determination of errors in the form of their entropy values holds for any arbitrary distribution laws, even in the cases when their evaluation by means of a variance is impossible. Thus, it is known that a distribution far removed from the normal (for instance, a Cauchy distribution) cannot be determined by its rms value, since the integral which determines variance is diverging. The entropy value of the error, however, in this case remains finite and can be calculated. 3. The determination of errors by their entropy value, which is directly related to the received information, eliminates the necessity of adopting any conditional solutions which to date had to be resorted to inevitably (for instance, by assuming the error equal to 2σ or 3σ, by evaluating the maximum value of the error from a series of 100 observations or a series of 20 observations, by assuming a fiducial probability of 0.9, 0.95, or 0.995, etc.).

Maximum-entropy processing is a method for computing the power density spectrum from the first N lags of the autocorrelation function. Unlike the discrete Fourier transform, maximum-entropy processing does not assume that the other lag values are zero. Instead, one mathematically ensures that the fewest possible assumptions about unmeasured data are made by choosing the spectrum that maximizes the entropy for the process. The use of the maximum entropy approach to spectral analysis was introduced by Burg [1]. In this correspondence, the authors derive the maximum-entropy spectrum by obtaining a spectrum that is forced to maximize the entropy of a stationary random process.

A probability space of a special type is put into correspondence with a measure space. Under this correspondence, sets in the measure space correspond to partitions of the probability space and the measure of a set equals the entropy of the corresponding partition.

From a stand-point that entropy is numerical measure of the second law of thermadynamics, a definition that "Increase of entropy in an isolated system when change occurs is defined as degree of irreversibility of the process in the system" is adopted, and from this qualitative definition, quantitative definition of entropy of materials is introduced.

In the simulation of discrete systems, such as those using the popular GPSS simulator, the following problem is often encountered: What probability distribution should we employ for the value of a system parameter in the absence of any knowledge except its range? Based on the concept of entropy, this note provides justification for the use of several well-known distributions in various cases.

Pfaffelhuber in his paper [1] deals with the approximation of the entropy H of finite information sources on the basis of independent observations. He derives an error estimation for the experimental entropy, which depends only on the number of the possible source-cutputs. Using this result he succeeded in giving an upper estimation for the number of independent observations needed for obtaining a “reliable” approximation of the exact entropy. The aim of this short correspondence is to improve his error estimation and thereby to give a new upper bound for the number of the necessary observations which is nearly the logarithm of E. Pfaffelhuber's one. We mention only that the same assertion holds for the information rate T of observation channels, too.

We prove that the relative entropy is decreasing under a trace-preserving expectation inB(K1), and we show the connection between this theorem and the strong subadditivity of the entropy. It is also proved that a linear, positive, trace-preserving map Φ ofB(K) into itself such that ‖Φ‖≦1 decreases the value of any convex trace function.

Central to the foregoing descriptions were views of information as a real connection. These views generally followed the doctrine of reflection. The common starting point of the discussions which follow is the Shannon information measure, entropy H.376 However, the doctrine of reflection also plays a role here. Relative to entropy H, Marxist-Leninist philosophers have discussed a great variety of problems.

I conceive my role to be that of a trigger for what, I hope, will turn out to be a lively discussion.

"Information transmitted" is defined as the amount by which added evidence (or "message received") diminishes "uncertainty". The latter is characterized by some properties intuitively suggested by this word and possessed by conditional entropy, a parameter of the posterior probability distribution. However, conditional entropy shares these properties with all other concave symmetric functions on the probability space.

It has been mentioned in Chapter 1 that the second law of thermodynamics (the principle of the increase of entropy) is based on the experience that all spontaneously-occurring processes have a particular direction, i.e. are irreversible. The second law appeared to be so generally applicable that eventually it (like the first law) was raised to the status of a postulate. The enormous success achieved in the second half of the nineteenth century by the atomic picture of the universe, gave rise to the need to base the second law, too, on this picture.

The limitations of a thermodynamic theory of a continuum by the atomic structure of matter are briefly discussed. They imply that any thermodynamic theory is of an approximate nature. Main emphasis is laid on the concept of a nonequilibrium entropy. It is shown that to an isothermal electrical network one can apply thermodynamic reasoning. When it is considered as a black box, that is, characterised by its impedance only, one can assign to the same state an infinite number of different entropy values. Thus one is led to doubt the existence of a unique nonequilibrium entropy in continuous matter, if one adopts here again the black box point of view, that is, if constitutive equations are used which relate the external variables only. Network models which also model the temperature by a voltage at one pair of terminals confirm the conjecture.

The claim that information theory provides a foundation for statistical thermodynamics which is independent of the use of ensembles can be sustained only if probabilities can be determined numerically without treating them as frequencies. The use of Shannon's measure of information to assign probabilities is correct only in the same conditions as justify Gibbs's use of ensembles.