Search

In hyperspectral anomaly detection, anomalies are rare targets that exhibit distinct spectral signatures from the background. Thus, anomalies are with low probabilities of occurrence in hyperspectral images. In this article, we develop a new technique for hyperspectral anomaly detection that adopts a new information theory perspective, to fully utilize the aforementioned concepts. Our goal is to transform system entropy into quantitative metrics of anomaly conspicuousness of pixels. To do so, two tasks are first completed: first, the construction of occurrence probability of pixels based on the density peak clustering algorithm, and second, the valid system definitions for pixels in specific anomaly detection problems with multiviews. Specifically, three types of systems are separately established by pixel pairs to conform to the definitions of three entropy definitions in information theory, i.e., Shannon entropy, joint entropy, and relative entropy. Then, three individual entropy-based metrics that assess the anomaly conspicuousness are defined. In addition, we design a standard deviation-based ensemble strategy for the integrated representation of the three individual metrics, which considers both logic ``or'' and ``and'' operations to simultaneously improve the detection rate and reduce the false alarm rate. Our experimental results obtained on two publicly available datasets with anomalies of different sizes and shapes demonstrate the superiority of our newly proposed anomaly detection method.

Blind image quality assessment (BIQA) aims to evaluate the quality of an image without pristine image objectively, which is highly desired in many perception-oriented image processing systems. Distortions degrade the visual contents and cause image quality degradation. Moreover, the visual contents of an image suffer from individual degradations by different types and different levels of distortions, which makes us difficult to analyze the quality degradation. From the perspective of information theory, there is a decrease in the amount of the visual contents when images are distorted. Therefore, in this paper, the image quality is assessed through its visual entropy degradation. Researches on the neuroscience indicate that the simple cells in the local receptive field can be characterized as being spatially localized and oriented, then the local intensity, gradient, and orientation features are extracted to represent the visual contents of an image. By deducing the joint entropy equation, the joint entropy is related to the statistical distributions. Next, in order to measure the visual entropy, the joint statistical distributions of those features are calculated. Finally, by measuring the degradations on these distributions of distorted images, a novel BIQA method is proposed. The experimental results on the databases of LIVE, CSIQ, and TID2013 show that the proposed method has superior performance than other state-of-the-art BIQA methods.

Multi-label feature selection that identifies important features from the original feature set of multi-labeled datasets has been attracting considerable attention owing to its generality compared to conventional single-label feature selection. The unimportant features are filtered by scoring the dependency of features to labels. In conventional multi-label feature filter studies, the score function is obtained by approximating a dependency measure such as joint entropy because direct calculation is often impractical due to the presence of multiple labels with limited training patterns. Although the efficacy of approximation can differ depending on the characteristics of the multi-label dataset, conventional methods presume a certain approximation method, leading to a degenerated feature subset if the presumed approximation is inappropriate for the given dataset. In this study, we propose a strategy for selecting an approximation among a series of approximations depending on the number of involved variables and consequently instantiate a score function based on the chosen approximation. The experimental results demonstrate that the proposed strategy and score function outperform conventional multi-label feature selection methods.

A microsleep is a brief and an involuntary sleep-related loss of consciousness of up to 15 s. We investigated the performances of seven pairwise inter-channel relationships–covariance, Pearson’s correlation coefficient, wavelet cross-spectral power, wavelet coherence, joint entropy, mutual information, and phase synchronization index–in continuous prediction of microsleep states from EEG. These relationships were used as the feature sets of a linear discriminant analysis (LDA) and a linear support vector machine classifiers. Priors for both classifiers were incorporated to address the class imbalance in the training data sets. Each feature set was extracted from a 5-s window of EEG with the step of 0.25 s and was demeaned with respect to the mean of first 2 min. The sequential forward selection (SFS) method, based on a serial combination of the correlation coefficient, Fisher score-based filter, and an LDA-based wrapper, was used to select features from each training set. The comparison was based on 16-channel EEG data from eight subjects who had performed a 1-D visuomotor task for two 1-h sessions. The prediction performances were evaluated using leave-one-subject-out cross-validation. For both classifiers, non-normalized feature sets were found to perform better than normalized feature sets. Furthermore, demeaning the non-normalized features considerably improved the prediction performance. Overall, the LDA classifier with joint entropy features resulted in the best average prediction performances (phi, AUCPR, and AUCROC) of (0.47, 0.50, and 0.95). Joint entropy between O1 and O2 from theta frequency band was the most informative feature.

Software-defined networking (SDN) is a communication paradigm that brings cost efficiency and flexibility through software-defined functions resident on centralized controllers. Although SDN applications are introduced in a limited scope with related technologies still under development, operational SDN networks already face major security threats. Therefore, comprehensive and efficient solutions are crucial. Especially, large-scale security threats such as distributed-denial-of-service (DDoS) attacks are jeopardizing safety and availability of data and services in these systems. A DDoS attack is aimed at making resources unavailable to legitimate users via overloading systems with excessive superfluous traffic from distributed sources. In this paper, we describe and evaluate a joint entropy-based security scheme (JESS) to enhance the SDN security with the aim of a reinforced SDN architecture against DDoS attacks. In particular, our proposed model devises a statistical solution to detect and mitigate these hazards. To the best of our knowledge, JESS is the first model that utilizes joint entropy for DDoS detection and mitigation in the SDN environment. Since it relies on a statistical model, it mitigates not only known attacks but also unfamiliar types in an efficient manner.

Characterising the capacity region for a network can be extremely difficult. Even with independent sources, determining the capacity region can be as hard as the open problem of characterising all information inequalities. The majority of computable outer bounds in the literature are relaxations of the Linear Programming bound which involves entropy functions of random variables related to the sources and link messages. When sources are not independent, the problem is even more complicated. Extension of Linear Programming bounds to networks with correlated sources is largely open. Source dependence is usually specified via a joint probability distribution, and one of the main challenges in extending linear program bounds is the difficulty (or impossibility) of characterising arbitrary dependencies via entropy functions. This paper tackles the problem by answering the question of how well entropy functions can characterise correlation among sources. We show that by using carefully chosen auxiliary random variables, the characterisation can be fairly "accurate" Using such auxiliary random variables we also give implicit and explicit outer bounds on the capacity of networks with correlated sources. The characterisation of correlation or joint distribution via Shannon entropy functions is also applicable to other information measures such as Renyi entropy and Tsallis entropy., Comment: 24 pager, 1 figure, submitted to IEEE Transactions on Information Theory. arXiv admin note: text overlap with arXiv:1309.1517

The key issue in image fusion is the process of defining evaluation indices for the output image and for multi-scale image data set. This paper attempted to develop a fusion model for plantar pressure distribution images, which is expected to contribute to feature points construction based on shoe-last surface generation and modification. First, the time series plantar pressure distribution image was preprocessed, including back removing and Laplacian of Gaussian (LoG) filter. Then, discrete wavelet transform and a multi-scale pixel conversion fusion operating using a parameter estimation optimized Gaussian mixture model (PEO-GMM) were performed. The output image was used in a fuzzy weighted evaluation system, that included the following evaluation indices: mean, standard deviation, entropy, average gradient, and spatial frequency; the difference with the reference image, including the root mean square error, signal to noise ratio (SNR), and the peak SNR; and the difference with source image including the cross entropy, joint entropy, mutual information, deviation index, correlation coefficient, and the degree of distortion. These parameters were used to evaluate the results of the comprehensive evaluation value for the synthesized image. The image reflected the fusion of plantar pressure distribution using the proposed method compared with other fusion methods, such as up-down, mean-mean, and max-min fusion. The experimental results showed that the proposed LoG filtering with PEO-GMM fusion operator outperformed other methods.

In 1953, Shannon proposed the question of quantification of structural information to analyze communication systems. The question has become one of the longest great challenges in information science and computer science. Here, we propose the first metric for structural information. Given a graph $G$ , we define the $K$ -dimensional structural information of $G$ (or structure entropy of $G$ ), denoted by $\mathcal {H}^{K}(G)$ , to be the minimum overall number of bits required to determine the $K$ -dimensional code of the node that is accessible from random walk in $G$ . The $K$ -dimensional structural information provides the principle for completely detecting the natural or true structure, which consists of the rules, regulations, and orders of the graphs, for fully distinguishing the order from disorder in structured noisy data, and for analyzing communication systems, solving the Shannon’s problem and opening up new directions. The $K$ -dimensional structural information is also the first metric of dynamical complexity of networks, measuring the complexity of interactions, communications, operations, and even evolution of networks. The metric satisfies a number of fundamental properties, including additivity, locality, robustness, local and incremental computability, and so on. We establish the fundamental theorems of the one- and two-dimensional structural information of networks, including both lower and upper bounds of the metrics of classic data structures, general graphs, the networks of models, and the networks of natural evolution. We propose algorithms to approximate the $K$ -dimensional structural information of graphs by finding the $K$ -dimensional structure of the graphs that minimizes the $K$ -dimensional structure entropy. We find that the $K$ -dimensional structure entropy minimization is the principle for detecting the natural or true structures in real-world networks. Consequently, our structural information provides the foundation for knowledge discovering from noisy data. We establish a black hole principle by using the two-dimensional structure information of graphs. We propose the natural rank of locally listing algorithms by the structure entropy minimization principle, providing the basis for a next-generation search engine.

We develop new methods for the representation of uncertainty in the granularized information source values by making use of the entropy framework in the possibilistic domain. An information-theoretic entropy function is used to map the information source values to information (entropy) values. We term a collection of such information values as an information set. The information values are then used in an adaptive form of this entropy function to formulate Shannon transforms. A few uncertainty measures are derived from these transforms for the quantification of uncertainty. Information set is also extended to other domains, such as probabilistic, intuitionistic, and probabilistic-intuitionistic domains. A biometric application is included to demonstrate the usefulness of the study.

The transformation of belief function into probability is one of the most important and common ways for decision making under the framework of evidence theory. In this paper, we focus on the evaluation of such probability transformations (PTs), which are crucial for their proper applications and the design of new ones. Shannon entropy or probabilistic information content (PIC) measure is traditionally used in evaluating PTs. The transformation having the lowest entropy or highest PIC is considered as the best one. This standpoint is questioned in this paper by comparing a PT based on uncertainty minimization with other available PTs. It shows experimentally that entropy or PIC is not comprehensive to evaluate a PT. To make a comprehensive evaluation, some new approaches are proposed by the joint use of PIC and the distance of evidence according to the value- and rank-based fusion. A pattern classification application oriented evaluation approach for PTs is also proposed. Some desired properties for PTs are also discussed. Experimental results and related analysis are provided to show the rationality of the new evaluation approaches.

Many methods have been proposed to select sites for grid-scale soil moisture monitoring networks; however, calibration/validation activities also require information about where to place grid representative monitoring sites. In order to design a soil moisture network for this task in the Great Lakes Basin (522 000 km2), the dual-entropy multiobjective optimization algorithm was used to maximize the information content and minimize the redundancy of information in a potential soil moisture monitoring network. Soil moisture retrieved from the Soil Moisture and Ocean Salinity (SMOS) mission during the frost-free periods of 2010–2013 were filtered for data quality and then used in a multiobjective search to find Pareto optimum network designs based on the joint entropy and total correlation measures of information content and information redundancy, respectively. Differences in the information content of SMOS ascending and descending overpasses resulted in distinctly different network designs. Entropy from the SMOS ascending overpass was found to be spatially consistent, whereas descending overpass entropy had many peaks that coincided with areas of high subgrid heterogeneity. A combination of both ascending and descending overpasses produced network designs that incorporated aspects of information from each overpass. Initial networks were designed to include 15 monitoring sites, but the addition of network cost as an objective demonstrated that a network with similar information content could be achieved with fewer monitoring stations.

Detection of causality is an important and challenging problem in root cause and hazard propagation analysis. It has been shown that the transfer entropy approach is a very useful tool in quantifying directional causal influence for both linear and nonlinear relationships. A key assumption for this method is that the sampled data should follow a well-defined probability distribution; yet this assumption may not hold for some industrial process data. In this paper, a new information theory-based measure, transfer 0-entropy (T0E), is proposed for causality analysis on the basis of the definitions of 0-entropy and 0-information without assuming a probability space. For the cases of more than two variables, a direct T0E (DT0E) concept is presented to detect whether there is a direct information and/or material flow pathway from one variable to another. Estimation methods for the T0E and the DT0E are addressed. The effectiveness of the proposed method is illustrated by two data sets, one based on data from a pilot scale process and a second evaluation based on data from a benchmark industrial case study.

By assuming latent sources are statistically independent, independent component analysis (ICA) separates underlying sources from a given linear mixture. Since in many applications, latent sources are non-Gaussian, noncircular, and have sample dependence, it is desirable to exploit all these properties jointly. Mutual information rate, which leads to the minimization of entropy rate, provides a natural cost for the task. In this paper, we establish the theory for complex-valued ICA giving Cramer-Rao lower bound and identification conditions, and present a new algorithm that takes all these properties into account. We propose an effective estimator of entropy rate and a complex-valued entropy rate bound minimization algorithm based on it. We show that the new method exploits all these properties effectively by comparing the estimation performance with the Cramer-Rao lower bound and by a number of examples.

The classical entropy power inequality is extended to the Renyi entropy. We also discuss the question of the existence of the entropy for sums of independent random variables.

A theoretical framework based on the maximum Shannon entropy principle with the specification of geometric mean or shifted geometric mean is proposed to generate the power law behavior of the system size in broadband communication networks. It is shown that the equilibrium distribution of an M/M/1 queue is obtained as a limiting case when the shifted geometric mean is specified. The various quality-of-service parameters, such as the probability of exceeding buffer size, exhibit the power law behavior. It is noted that the results based on the Shannon entropy with shifted geometric mean are found to be similar to the results when the Tsallis entropy subject to expected number of jobs in the system is prescribed. This brings out a deeper issue of the relevance of shifted geometric mean within Shannon entropy framework when the input traffic has broadband characteristics. The proposed approach gives closed-form expressions for a queuing distribution in a much simpler way and thus establishes the efficacy of Shannon entropy in communication networks exhibiting power law.

Joint entropy is used to quantify the complex nonlinear states of dc-dc switching converters in this brief. The switching instant symbolic sequence (SISS) and the topological structural symbolic sequence (TSSS) are presented based on the characteristics of dc-dc converters. Transforming SISS and TSSS into the decimal symbolic sequences, joint entropy is obtained as the index to quantify the nonlinearity of the dc-dc converter. Finally, a Cuk converter is studied as an example to explain how the joint entropy can quantify the complex state of the dc-dc switching converter.

A fuzzy relation information system may be viewed as an information system with fuzzy relations. Uncertainty measurement is a critical evaluating tool. This paper investigates uncertainty measurement for a fuzzy relation information system. The concept of information structures in a fuzzy relation information system is first described by using set vectors. Then, dependence between information structures in a fuzzy relation information system is given. Next, the axiom definition of the granularity measurement of the uncertainty for fuzzy relation information systems is proposed by means of its information structures. Based upon this axiom definition, information granulation and rough entropy in a fuzzy relation information system are proposed. Moreover, information entropy, information amount, joint entropy, and condition entropy in a fuzzy relation information system are also considered. To show the feasibility of the proposed measures for uncertainty of a fuzzy relation information system, effectiveness analysis is conducted from the angle of statistics. Finally, characterizations of fuzzy relation information systems under a compatible homomorphism are obtained. These results will be helpful for understanding the essence of uncertainty in a fuzzy relation information system.

This paper proposes a unifying variational approach for proving and extending some fundamental information theoretic inequalities. Fundamental information theory results such as maximization of differential entropy, minimization of Fisher information (Cram\'er-Rao inequality), worst additive noise lemma, entropy power inequality (EPI), and extremal entropy inequality (EEI) are interpreted as functional problems and proved within the framework of calculus of variations. Several applications and possible extensions of the proposed results are briefly mentioned.

We propose an approach for computing mutual information in rigid multimodality image registration. Images to be registered are modeled as functions defined on a continuous image domain. Analytic forms of the probability density functions for the images and the joint probability density function are first defined in 1D. We describe how the entropies of the images, the joint entropy, and mutual information can be computed accurately by a numerical method. We then extend the method to 2D and 3D. The mutual information function generated is smooth and does not seem to have the typical interpolation artifacts that are commonly observed in other standard models. The relationship between the proposed method and the partial volume (PV) model is described. In addition, we give a theoretical analysis to explain the nonsmoothness of the mutual information function computed by the PV model. Numerical experiments in 2D and 3D are presented to illustrate the smoothness of the mutual information function, which leads to robust and accurate numerical convergence results for solving the image registration problem.

Two existing methods of probabilistic inference are based on variational principles: maximum entropy and minimum Fisher information. In each case, a probability density function is inferred by setting the first variation of a functional to zero, subject to information constraints. This study considers whether other functionals could be used for this purpose, and by starting with requirements for self-consistency and invariance, it is shown that the most general admissible functional is just a linear combination of entropy and Fisher information, with the proviso that the normal definition of Fisher information is modified by the inclusion of a prior. This amounts to an axiomatic derivation of entropy and Fisher information. The concern is with continuous random variables and both the single- and multivariable cases are considered. A number of examples are considered to compare inference based on entropy with that based on Fisher information, and to highlight the role of boundary conditions for inference based on Fisher information.

In this paper, we consider the training of complex-valued filter based on the information theoretic method. We first generalize the error entropy criterion to complex domain to present the complex error entropy criterion (CEEC). Due to the difficulty in estimating the entropy of complex-valued error directly, the entropy bound minimization (EBM) method is used to compute the upper bounds of the entropy of the complex-valued error, and the tightest bound selected by the EBM algorithm is used as the estimator of the complex-error entropy. Then, based on the minimization of complex-error entropy (MCEE) and the complex gradient descent approach, complex-valued learning algorithms for both the (linear) transverse filter and the (nonlinear) neural network are derived. The algorithms are applied to complex-valued linear filtering and complex-valued nonlinear channel equalization to demonstrate their effectiveness and advantages.

The principle of maximum entropy provides a powerful framework for estimating joint, conditional, and marginal probability distributions. However, there are many important distributions with elements of interaction and feedback where its applicability has not been established. This paper presents the principle of maximum causal entropy-an approach based on directed information theory for estimating an unknown process based on its interactions with a known process. We demonstrate the breadth of the approach using two applications: a predictive solution for inverse optimal control in decision processes and computing equilibrium strategies in sequential games.

There is no generally accepted definition of conditional Renyi entropy. The (unconditional) Renyi entropy depends on a parameter α, which for the case of min-entropy takes the value ∞. Even for this particular case, there are several proposals for the definition of conditional entropy. This paper describes three general definitions of conditional Renyi entropy that were found or suggested in the literature. Their properties are studied and their values, as a function of α, are compared. The particular case of min-entropy is widely used in cryptography as a security parameter; this case is studied in some detail.

This paper presents a novel iterative active learning (AL) technique aimed at defining effective multitemporal training sets to be used for the supervised detection of land-cover transitions in a pair of remote sensing images acquired on the same area at different times. The proposed AL technique is developed in the framework of the Bayes' rule for compound classification. At each iteration, it selects the pair of spatially aligned unlabeled pixels in the two images that are classified with the maximum uncertainty. These pixels are then labeled by an external supervisor and included in the training set. The uncertainty of a pair of pixels is assessed by the joint entropy defined by considering two possible different simplifying assumptions: 1) class-conditional independence and 2) temporal independence between multitemporal images. Accordingly, different algorithms are introduced. The proposed joint-entropy-based AL algorithms for compound classification are compared with each other and with a marginal-entropy-based AL technique (in which the entropy is computed separately on single-date images) applied to the postclassification comparison method. The experimental results obtained on two multispectral and multitemporal data sets show the effectiveness of the proposed technique.

We express the joint entropy of progressively Type-II censored order statistics in terms of an incomplete integral of the hazard function, and use it to develop a simple estimate of the joint entropy of progressively Type-II censored data, considered earlier by Balakrishnan , IEEE Trans. Reliability, vol. 56, pp. 349-356. We then construct a goodness-of-fit test statistic based on the Kullback-Leibler information for Pareto, log-normal, and Weibull distributions by using maximum likelihood estimates and approximate maximum likelihood estimates of the model parameters. Finally, we use Monte Carlo simulations to evaluate the power of the proposed test for several alternatives under different sample sizes and progressive censoring schemes.

We describe a nonparametric framework for incorporating information from co-registered anatomical images into positron emission tomographic (PET) image reconstruction through priors based on information theoretic similarity measures. We compare and evaluate the use of mutual information (MI) and joint entropy (JE) between feature vectors extracted from the anatomical and PET images as priors in PET reconstruction. Scale-space theory provides a framework for the analysis of images at different levels of detail, and we use this approach to define feature vectors that emphasize prominent boundaries in the anatomical and functional images, and attach less importance to detail and noise that is less likely to be correlated in the two images. Through simulations that model the best case scenario of perfect agreement between the anatomical and functional images, and a more realistic situation with a real magnetic resonance image and a PET phantom that has partial volumes and a smooth variation of intensities, we evaluate the performance of MI and JE based priors in comparison to a Gaussian quadratic prior, which does not use any anatomical information. We also apply this method to clinical brain scan data using F(18) Fallypride, a tracer that binds to dopamine receptors and therefore localizes mainly in the striatum. We present an efficient method of computing these priors and their derivatives based on fast Fourier transforms that reduce the complexity of their convolution-like expressions. Our results indicate that while sensitive to initialization and choice of hyperparameters, information theoretic priors can reconstruct images with higher contrast and superior quantitation than quadratic priors.

For discrete random variables X1, ..., Xn we construct an n by n matrix. In the (i, j)-entry we put the mutual information I(Xi ; Xj) between Xi and Xj. In particular, in the (i, i)-entry we put the entropy H(Xi) = I(Xi; Xi) of Xi. This matrix, called the mutual information matrix of (X1, ..., Xn), has been conjectured to be positive semidefinite. In this paper, we give counterexamples to the conjecture, and show that the conjecture holds for up to three random variables.

Human faces captured inside videos are often presented with variable poses, making it difficult to recognize and thus pose detection becomes crucial for such face recognition under non-controlled environment. While existing mutual in formation (MI) primarily considers the relationship between corresponding individual pixels, we propose a normalized co occurrence mutual information in this letter to capture the information embedded not only in corresponding pixel values but also in their geographical locations. In comparison with the existing Mis, the proposed presents an essential advantage that both marginal entropy and joint entropy can be optimally exploited in measuring the similarity between two given images. When developed into a facial pose detection algorithm inside video sequences, we show, through extensive experiments, that such design is capable of achieving the best performances among all the representative existing techniques compared.

A novel (differential) entropy estimator is introduced where the maximum entropy bound is used to approximate the entropy given the observations, and is computed using a numerical procedure thus resulting in accurate estimates for the entropy. We show that such an estimator exists for a wide class of measuring functions, and provide a number of design examples to demonstrate its flexible nature. We then derive a novel independent component analysis (ICA) algorithm that uses the entropy estimate thus obtained, ICA by entropy bound minimization (ICA-EBM). The algorithm adopts a line search procedure, and initially uses updates that constrain the demixing matrix to be orthogonal for robust performance. We demonstrate the superior performance of ICA-EBM and its ability to match sources that come from a wide range of distributions using simulated and real-world data.

We first present a new (differential) entropy estimator for complex random variables by approximating the entropy estimate using a numerically computed maximum entropy bound. The associated maximum entropy distributions belong to the class of weighted linear combinations and elliptical distributions, and together, they provide a rich array of bivariate distributions for density matching. Next, we introduce a new complex independent component analysis (ICA) algorithm, complex ICA by entropy-bound minimization (complex ICA-EBM), using this new entropy estimator and a line search optimization procedure. We present simulation results to demonstrate the superior separation performance and computational efficiency of complex ICA-EBM in separation of complex sources that come from a wide range of bivariate distributions.

Maximization of mutual information (MI) (MMI) has a wide application in multimodal image registration. The conventional MMI algorithm is based on the statistics derived from image sampling and does not use spatial features. In this paper, we proposed an improved MMI registration algorithm that combines the spatial information through a feature-based selection mechanism. A Harris corner filter is chosen to classify the pixels. The classification results are used to focus sample selection, joint entropy estimation, and weighted MI information calculation. Although features play an important role, conventional feature matching is not used, which removes an important challenge to the use of feature-based information. The advantages of the proposed algorithm are improved robustness by incorporation of spatial features and significant reduction in computing time for registration. Experiments with challenging synthetic images and multimodal airborne infrared images are provided to demonstrate the improvement.

This paper studies the entropy rate of hidden Markov processes (HMPs) which are generated by observing a discrete-time binary homogeneous Markov chain through an arbitrary memoryless channel. A fixed-point functional equation is derived for the stationary distribution of an input symbol conditioned on all past observations. While the existence of a solution to the fixed-point functional equation is guaranteed by martingale theory, its uniqueness follows from the fact that the solution is the fixed point of a contraction mapping. The entropy or differential entropy rate of the HMP can then be obtained through computing the average entropy of each input symbol conditioned on past observations. In absence of an analytical solution to the fixed-point functional equation, a numerical method is proposed in which the fixed-point functional equation is first converted to a discrete linear system using uniform quantization and then solved efficiently. The accuracy of the computed entropy rate is shown to be proportional to the quantization interval. Unlike many other numerical methods, this numerical solution is not based on averaging over a sample path of the HMP.

Erasure entropy rate differs from Shannon's entropy rate in that the conditioning occurs with respect to both the past and the future, as opposed to only the past (or the future). In this paper, consistent universal algorithms for estimating erasure entropy rate are proposed based on the basic and extended context-tree weighting (CTW) algorithms. Simulation results for those algorithms applied to Markov sources, tree sources, and English texts are compared to those obtained by fixed-order plug-in estimators with different orders.

Image analysis usually refers to processing of images with the goal of finding objects presented in the image. Image segmentation is one of the most critical tasks in automatic image analysis. The non-extensive entropy, also known as Tsallis entropy, is a recent development in statistical mechanics and has been considered as a useful measure in describing termostatistical properties of physical systems. In this new formalism a real quantity q was introduced as parameter for physical systems that presents long range interactions, long time memories and fractal-type structures. In image processing, one of the most efficient techniques for image segmentation is entropy-based thresholding. This approach uses the Shannon entropy from the information theory considering the gray level image histogram as a probability distribution. In this work, it was applied the Tsallis entropy as a generalized entropy formalism for information theory. For the first time it was proposed an image thresholding method using a non-extensive relative entropy.

Group testing has been used in many applications to efficiently identify rare events in a large population. In this paper, the concept of group testing is generalized to applications with correlated source models to derive scheduling policies for sensors' adopting cooperative transmissions. The tenet of our work is that in a wireless sensor network it is advantageous to allocate the same channel dimensions to all sensor sources that have the same response to a sequence of queries or tests. That is, nodes that have the same data attributes should transmit as a cooperative super-source. Specifically, we consider the case where sensors' data are modeled spatially as a one-dimensional Markov chain. Two strategies are considered: the recursive algorithm and the tree-based algorithm. The recursive scheme allows us to illustrate the performance of group testing for finite populations while the tree-based algorithm is used to derive the achievable scaling performances of the class of group testing strategies as the number of sensors increases. We show that the total number of queries required to gather all sensors' data scales in the order of the joint entropy. A further generalization of this concept provides the basis of deriving efficient data-gathering algorithms for correlated sources.

Tight bounds on the block entropy of patterns of sequences generated by independent and identically distributed (i.i.d.) sources are derived. A pattern of a sequence is a sequence of integer indices with each index representing the order of first occurrence of the respective symbol in the original sequence. Since a pattern is the result of data processing on the original sequence, its entropy cannot be larger. Bounds derived here describe the pattern entropy as function of the original i.i.d. source entropy, the alphabet size, the symbol probabilities, and their arrangement in the probability space. Matching upper and lower bounds derived provide a useful tool for very accurate approximations of pattern block entropies for various distributions, and for assessing the decrease of the pattern entropy from that of the original i.i.d. sequence.

This paper deals with the degree of uncertainty associated with fuzzy variables. Based on the notion of credibility measure, a definition of entropy is formulated from an information theoretical view and its properties are investigated. Finally, some comments are given on the construction of alternative definitions of entropy and axiomatic characterization.

Conceptual and practical encoding/decoding, aimed at accurately reproducing remotely collected observations, has been heavily investigated since the pioneering works by Shannon about source coding. However, when the goal is not to reproduce the observables, but making inference about an embedded parameter and the scenario consists of many unconnected remote nodes, the landscape is less certain. We consider a multiterminal system designed for efficiently estimating a random parameter according to the minimum mean square error (MMSE) criterion. The analysis is limited to scalar quantizers followed by a joint entropy encoder, and it is performed in the high-resolution regime where the problem can be more easily mathematically tackled. Focus is made on the peculiarities deriving from the estimation task, as opposed to that of reconstruction, as well as on the multiterminal, as opposite to centralized, character of the inference. The general form of the optimal nonuniform quantizer is derived and examples are given.

Partition entropy is the numerical metric of uncertainty within a partition of a finite set, while conditional entropy measures the degree of difficulty in predicting a decision partition when a condition partition is provided. Since two direct methods exist for defining conditional entropy based on its partition entropy, the inequality postulates of monotonicity, which conditional entropy satisfies, are actually additional constraints on its entropy. Thus, in this paper partition entropy is defined as a function of probability distribution, satisfying all the inequalities of not only partition entropy itself but also its conditional counterpart. These inequality postulates formalize the intuitive understandings of uncertainty contained in partitions of finite sets. We study the relationships between these inequalities, and reduce the redundancies among them. According to two different definitions of conditional entropy from its partition entropy, the convenient and unified checking conditions for any partition entropy are presented, respectively. These properties generalize and illuminate the common nature of all partition entropies.

Estimation of the differential entropy from observations of a random variable is of great importance for a wide range of signal processing applications such as source coding, pattern recognition, hypothesis testing, and blind source separation. In this paper, we present a method for estimation of the Shannon differential entropy that accounts for embedded manifolds. The method is based on high-rate quantization theory and forms an extension of the classical nearest-neighbor entropy estimator. The estimator is consistent in the mean square sense and an upper bound on the rate of convergence of the estimator is given. Because of the close connection between compression and Shannon entropy, the proposed method has an advantage over methods estimating the Renyi entropy. Through experiments on uniformly distributed data on known manifolds and real-world speech data we show the accuracy and usefulness of our proposed method.

We express the joint entropy of progressively censored order statistics in terms of an incomplete integral of the hazard function, and provide a simple estimate of the joint entropy of progressively Type-II censored data. We then construct a goodness-of-fit test statistic based on Kullback-Leibler information with progressively Type-II censored data. Finally, by using Monte Carlo simulations, the power of the test is estimated, and compared against several alternatives under different progressive censoring schemes

Time delay estimation (TDE) is a basic technique for numerous applications where there is a need to localize and track a radiating source. The most important TDE algorithms for two sensors are based on the generalized cross-correlation (GCC) method. These algorithms perform reasonably well when reverberation or noise is not too high. In an earlier study by the authors, a more sophisticated approach was proposed. It employs more sensors and takes advantage of their delay redundancy to improve the precision of the time difference of arrival (TDOA) estimate between the first two sensors. The approach is based on the multichannel cross-correlation coefficient (MCCC) and was found more robust to noise and reverberation. In this letter, we show that this approach can also be developed on a basis of joint entropy. For Gaussian signals, we show that, in the search of the TDOA estimate, maximizing MCCC is equivalent to minimizing joint entropy. However, with the generalization of the idea to non-Gaussian signals (e.g., speech), the joint entropy-based new TDE algorithm manifests a potential to outperform the MCCC-based method

The entropy of a set of data is a measure of the amount of information contained in it. Entropy calculations for fully specified data have been used to get a theoretical bound on how much that data can be compressed. This paper extends the concept of entropy for incompletely specified test data (i.e., that has unspecified or don't care bits) and explores the use of entropy to show how bounds on the maximum amount of compression for a particular symbol partitioning can be calculated. The impact of different ways of partitioning the test data into symbols on entropy is studied. For a class of partitions that use fixed-length symbols, a greedy algorithm for specifying the don't cares to reduce entropy is described. It is shown to be equivalent to the minimum entropy set cover problem and thus is within an additive constant error with respect to the minimum entropy possible among all ways of specifying the don't cares. A polynomial time algorithm that can be used to approximate the calculation of entropy is described. Different test data compression techniques proposed in the literature are analyzed with respect to the entropy bounds. The limitations and advantages of certain types of test data encoding strategies are studied using entropy theory

This paper discusses a possible program for improving the outer (converse) bounds on the finite-blocklength performance of multiple-access codes. The program is based on a certain conjecture involving Rényi entropy of a sum of two independent binary vectors. Some partial results towards showing the conjecture are presented. The problem of bounding the joint Rényi entropy in terms of the marginal entropies is addressed., National Science Foundation (U.S.) (Grant CCF- 12-53205)

Recent work by Orlitsky et al. (2004) has motivated the study of pattern sequences and their compressibility properties. Emphasis in this recent line of work has been on compressing pattern sequences under uncertainty in the source that has generated them, thus focusing on universal schemes and their redundancy. Our interest in this work is in the entropy rate of pattern sequences of stochastic processes, and its relationship to the entropy rate of the original process. We give a complete characterization of this relationship for i.i.d. processes over arbitrary alphabets, stationary and ergodic: processes over discrete alphabets, as well as more general processes that can be represented as the output of an additive white-noise channel. For cases where the entropy rate of the pattern process is infinite, we characterize the possible growth rate of the block entropy.

Rough set theory has proven to be an efficient tool for modeling and reasoning with uncertainty information. By introducing probability into fuzzy approximation space, a theory about fuzzy probabilistic approximation spaces is proposed in this paper, which combines three types of uncertainty: probability, fuzziness, and roughness into a rough set model. We introduce Shannon's entropy to measure information quantity implied in a Pawlak's approximation space, and then present a novel representation of Shannon's entropy with a relation matrix. Based on the modified formulas, some generalizations of the entropy are proposed to calculate the information in a fuzzy approximation space and a fuzzy probabilistic approximation space, respectively. As a result, uniform representations of approximation spaces and their information measures are formed with this work

Image restoration is an important and widely studied problem in computer vision and image processing. Various image filtering strategies have been effective, but invariably make strong assumptions about the properties of the signal and/or degradation. Hence, these methods lack the generality to be easily applied to new applications or diverse image collections. This paper describes a novel unsupervised, information-theoretic, adaptive filter (UINTA) that improves the predictability of pixel intensities from their neighborhoods by decreasing their joint entropy. In this way, UINTA automatically discovers the statistical properties of the signal and can thereby restore a wide spectrum of images. The paper describes the formulation to minimize the joint entropy measure and presents several important practical considerations in estimating neighborhood statistics. It presents a series of results on both real and synthetic data along with comparisons with current state-of-the-art techniques, including novel applications to medical image processing.

A fundamental step in decision analysis is the elicitation of the decision maker's information about the uncertainties of the decision situation in the form of a joint probability distribution. This paper presents a method based on the maximum entropy principle to obtain a joint probability distribution using lower order joint probability assessments. The approach reduces the number of assessments significantly and also reduces the number of conditioning variables in these assessments. We discuss the order of the approximation provided by the maximum entropy distribution with each lower order assessment using a Monte Carlo simulation and discuss the implications of using the maximum entropy distribution in Bayesian inference. We present an application to a practical decision situation faced by a semiconductor testing company in the Silicon Valley.

Sample entropy and approximate entropy are measures that have been successfully utilized to study the deterministic dynamics of heart rate (HR). A complementary stochastic point of view and a heuristic argument using the Central Limit Theorem suggests that the Gaussianity of HR is a complementary measure of the physiological complexity of the underlying signal transduction processes. Renyi entropy (or q-entropy) is a widely used measure of Gaussianity in many applications. Particularly important members of this family are differential (or Shannon) entropy (q = 1) and quadratic entropy (q = 2). We introduce the concepts of differential and conditional Renyi entropy rate and, in conjunction with Burg's theorem, develop a measure of the Gaussianity of a linear random process. Robust algorithms for estimating these quantities are presented along with estimates of their standard errors.

A method to perform convolutive blind source separation of super-Gaussian sources by minimizing the mutual information between segments of output signals is presented. The proposed approach is essentially an implementation of an idea previously proposed by Pham. The formulation of mutual information in the proposed criterion makes use of a nonparametric estimator of Renyi's /spl alpha/-entropy, which becomes Shannon's entropy in the limit as /spl alpha/ approaches 1. Since /spl alpha/ can be any number greater than 0, this produces a family of criteria having an infinite number of members. Interestingly, it appears that Shannon's entropy cannot be used for convolutive source separation with this type of estimator. In fact, only one value of /spl alpha/ appears to be appropriate, namely /spl alpha/=2, which corresponds to Renyi's quadratic entropy. Four experiments are included to show the efficacy of the proposed criterion.