Search

Smooth entropy of X is defined as possibly biggest entropy of a distribution Y close to X. It has found many applications including privacy amplification, information reconciliation, quantum information theory and even constructing random number generators. However the basic question about the optimal shape for the distribution Y has not been answered yet. In this paper we solve this problem for Renyi entropies in non-quantum settings, giving a formal treatment to an approach suggested at TCC'05 and ASIACRYPT'05. The main difference is that we use a threshold cut instead of a quantile cut to rearrange probability masses of X. As an example of application, we derive tight lower bounds on the number of bits extractable from Shannon memoryless sources.

The classical expression for entropy production is a bilinear product of generalized forces and fluxes. The combination of the Cattaneo-Vernotte model with the classical entropy expression gives a non-quadratic form, which needs to be mended to avoid the paradox of negative entropy production. Based on the thermomass theory, it is shown that the entropy production corresponds to the dissipation of the mechanical energy of thermomass flow. Therefore, the generalized forces in the entropy production should be the friction force rather than the driving force. The friction force is proportional to the heat flux. The general entropy production is thus derived as a quadratic form of heat flux, avoiding the paradox of the negative entropy production. The generalized forces and fluxes in other irreversible transport processes are investigated following the similar framework. The friction forces, driving forces and drift velocities are clarified for these transports and the general entropy production for various transport processes are derived.

Random numbers are essential for cryptography. In most real-world systems, these values come from a cryptographic pseudorandom number generator (PRNG), which in turn is seeded by an entropy source. The security of the entire cryptographic system then relies on the accuracy of the claimed amount of entropy provided by the source. If the entropy source provides less unpredictability than is expected, the security of the cryptographic mechanisms is undermined, as in [5, 7, 10]. For this reason, correctly estimating the amount of entropy available from a source is critical.

Research knowledge acquisition in the real valued decision system is the main directions of the granular computing. Based on the neighborhood of rough set theory and according to the characteristics of the real value, an efficient knowledge acquisition method is discussed in this paper. Firstly, the definition of the conditional entropy and reduction theorem in a real value decision system is given and its properties are studied. Then, the algorithm of attribute reduction based on conditional entropy in value decision system is proposed. Lastly, the efficiency of the algorithm is showed by an example.

Compressed domain based steganography (CDBS) is a kind of relatively new and secure audio steganography. Up to date, there is little research on the steganalysis against this kind of audio steganography. In this paper, we introduce two methods to detect various CDBS on ACELP speech. One is the Markov method and the other is the entropy method. Both methods are based on the observation that the steganography behavior has certain effects on the relationship among the pulses in the same track. The Markov transition probabilities are utilized to evaluate the interrelationships between adjacent pulses and entropy is employed to measure the “disorder” of combined pulse distributions. First, Markov transition probabilities, joint entropy and conditional entropy features of the track pulses are extracted; a support vector machine (SVM) is then applied to the features for discovering the existence of hidden data in compressed speech signals, respectively. Some famous CDBS methods on ACELP encoded speech are considered. Experimental results have proven the effectiveness of the two methods.

A fundamental issue in active learning of Gaussian processes is that of the exploration-exploitation trade-off. This paper presents a novel nonmyopic e-Bayes-optimal active learning (e-BAL) approach [4] that jointly optimizes the trade-off. In contrast, existing works have primarily developed greedy algorithms or performed exploration and exploitation separately. To perform active learning in real time, we then propose an anytime algorithm [4] based on e-BAL with performance guarantee and empirically demonstrate using a real-world dataset that, with limited budget, it outperforms the state-of-the-art algorithms.

In this paper we explore the notion of entropy for non–autonomous discrete systems and solve an open question stated in Zhu et al. (J Korean Math Soc 49:165–185, 2012). Some other open questions are also proposed.

Co-clustering is a powerful technique with varied applications in text clustering and recommender systems. For large scale high dimensional and sparse real world data, there is a strong need to provide an overlapped co-clustering algorithm that mitigates the effect of noise and non-discriminative information, generalizes well to the unseen data, and performs well with respect to several quality measures. In this paper, we introduce a novel fuzzy co-clustering algorithm that incorporates multiple regularizers to address these important issues. Specifically, we propose MRegFC that considers terms corresponding to Entropy, Gini Index, and Joint Entropy simultaneously. We demonstrate that MRegFC generates significantly higher quality results compared to many existing approaches on several real world benchmark datasets.

Fuzzy entropy is used to characterize the uncertainty on the possible values of fuzzy variables, which has been studied by many researchers. Within the framework of credibility theory, Li and Liu presented a Shannon-like entropy for both discrete fuzzy variable and continuous fuzzy variable. Furthermore, Li and Liu proposed the maximum entropy principle, and proved that out of all the credibility functions with fixed expected value and variance, the normal credibility function has the maximum entropy. Based on the concept of fuzzy entropy, Li et al. proposed an entropy optimization model by minimizing the uncertainty of the fuzzy objective under certain expected constraints. This chapter mainly includes the definition of fuzzy entropy, maximum entropy theorems, entropy optimization model and its crisp equivalents, fuzzy simulation, and applications in portfolio selection problem.

It is known that the security evaluation can be done by smoothing of Renyi entropy of order 2 in the quantum setting when we apply universal2 hash functions. This fact can be extended to the case when we apply e-almost dual universal2 hash functions, which is a generalized concept of universal2 hash functions. Demonstrating the smoothing of Renyi entropy of order 2, we derived security bounds for universal composability and mutual information criterion under the condition in the quantum setting.

In order to find out the limiting speed of solving a specific problem using computer, this essay provides a method based on the entropy of information. The relationship between the minimum time complexity and the information entropy change is illustrated. Several examples are served as evidence of such connection. Meanwhile some notices of modeling these problems are proposed. Finally, the nature of solving problems with computer programs is disclosed to support the theory and a redefinition of the information entropy in this field is proposed. This will develop a new field of science.

This chapter is dedicated to the study of metric entropy, including its relation to topological entropy. After establishing some basic properties of metric entropy, we consider the notion of conditional entropy, and we show how generators can be used to compute metric entropy. We then establish the Shannon–McMillan–Breiman theorem, which can be seen as the fundamental theorem of entropy theory. In particular, it shows that metric entropy can be computed in terms of an invariant local quantity. We also introduce the notion of topological entropy for a continuous transformation of a compact metric space, and we establish the variational principle showing that the topological entropy is the supremum of the metric entropies over all invariant probability measures.

In order to discuss the information entropy on quasi-probability space, combining Shannon entropy on probability space with quasi-probability on quasi-probability space, the definitions of self-information and information entropy on quasi-probability space are proposed, and basic properties of information entropy of discrete quasi-random variables on quasi-probability space are also given.

We propose a formal expansion of the transfer entropy to address the problem or partial conditioning evaluating information flow in multivariate datasets. This approach will then be adapted to put in evidence irreducible sets of variables which provide information for the future state of each assigned target. Multiplets characterized by an high value will be associated to informational circuits present in the system, with an informational character (synergetic or redundant) which can be associated to the sign of the contribution. These methods are then applied to the analysis of fMRI data.

Numerous entropy-type characteristics (functionals) generalizing Renyi entropy are widely used in mathematical statistics, physics, information theory, and signal processing for characterizing uncertainty in probability distributions and distribution identification problems. We consider estimators of some entropy (integral) functionals for discrete and continuous distributions based on the number of epsilon-close vector records in the corresponding independent and identically distributed samples from two distributions. The proposed estimators are generalized U-statistics. We show the asymptotic properties of these estimators (e.g., consistency and asymptotic normality). The results can be applied in various problems in computer science and mathematical statistics (e.g., approximate matching for random databases, record linkage, image matching). AMS 2000 subject classification: 94A15, 62G20

Entropy is a measurement of the degree of uncertainty. Mean-entropy method can be used for modeling the choice among uncertain outcomes. In this paper, we consider the portfolio selection problem under the assumption that security returns are characterized by type-2 fuzzy variables. Since the expectation and entropy of type-2 fuzzy variables haven't been well defined, type-2 fuzzy variables need to be reduced firstly. Then we propose a mean-entropy model with reduced variables. To solve the proposed model, we use the entropy formula of reduced fuzzy variable and transform the mean-entropy model to its equivalent parametric form, which can be solved by standard optimization solver.

This paper considers an entropy model of portfolio selection problem with fuzzy random variables to future returns. Since standard mean-variance portfolio models suffer from some shortcomings, the entropy is introduced as a risk measure instead of variances to overcome the shortcomings. Furthermore, introducing the sum of entropy to each portfolio as well as the entropy of fuzzy random variables, the previous entropy-based fuzzy random portfolio selection problem is extended, the exact optimal portfolio is explicitly obtained using nonlinear programming such as Karush-Kuhn-Tucker condition.

In this paper, data analysis for multiple facts was carried out via fuzzy entropy. The entropy for the fuzzy data with respect to multiple facts was designed through distance measure. The obtained fuzzy entropy was used to analyze the uncertainties with respect to each fact. By summarizing fuzzy entropy, data uncertainty information was limited by the total fact (n) minus one, that is, n -1. The bounded calculation of data uncertainty to each fact was also proven for multiple facts and the decision of fuzzy data to the certain fact among multiple facts has been considered with the help of fuzzy entropy calculation.

Although information theory is primarily concerned with the transmission of information it can be also applied to the quantification of the intrinsic information that emerges from a given physical system. Over the past years, principles of information theory have been applied to many environmental and ecological studies. However, it still appears that the initial concept of entropy as identified by Shannon's initial contribution cannot be directly applied to evolving geographical systems. The research introduced in this paper suggests an extension of the concept of entropy to the spatial and temporal dimensions, by taking into account the distribution of entities in space and time. We propose a series of entropy measures that together form a set of complementary indices to evaluate the distribution of entities, events and categories in space and time. The whole approach is exemplified by several illustrative configurations.

Formulation of the problem. In most practical situations, our knowledge is incomplete: there are several (n) different states which are consistent with our knowledge. How can we gauge this uncertainty? A natural measure of uncertainty is the average number of binary (“yes”-“no”) questions that we need to ask to find the exact state. This idea is behind Shannon’s information theory: according to this theory, when we know the probabilities p1,..., p n of different states (for which ∑ p i = 1), then this average number of questions is equal to S = – \(\sum\limits^{n}_{i=1}\) p i ·log2(p i ). In information theory, this average number of question is called the amount of information.

What is entropy? This term is used with several meanings in science and technology. For a chemist, “it is a function of the state of a thermodynamic system”, whereas a physicist would say that “it is a measure of the disorder of a given system”, and a communications engineer would come with a very different idea, since for him entropy would be “the average information transmitted or received in a series of messages”.

The Shannon information content is a fundamental quantity and it is of great importance to estimate it from observed dataset in the field of statistics, information theory, and machine learning. In this study, an estimator for the information content using a given set of weighted data is proposed. The empirical data distribution varies depending on the weight. The notable features of the proposed estimator are its computational efficiency and its ability to deal with weighted data. The proposed estimator is extended in order to estimate cross entropy, entropy and KL divergence with weighted data. Then, the estimators are applied to classification with one-class samples, and distribution preserving data compression problems.

In this paper we show how to approximate the von Neumann entropy associated with the Laplacian eigenspectrum of graphs and exploit it as a characteristic for the clustering and classification of graphs. We commence from the von Neumann entropy and approximate it by replacing the Shannon entropy by its quadratic counterpart. We then show how the quadratic entropy can be expressed in terms of a series of permutation invariant traces. This leads to a simple approximate form for the entropy in terms of the elements of the adjacency matrix which can be evaluated in quadratic time. We use this approximate expression for the entropy as a unary characteristic for graph clustering. Experiments on real world data illustrate the effectiveness of the method.

The information entropy has general applications in different subjects, such as information theory, linear algebra, signal processing, dynamical systems, ergodic theory, probability and statistical. Then the study of inequality on the information entropy has important signification in theory. Schur-convexity and Schur-geometric convexity and Schur-harmonic convexity entropy are studied for the generalized information based on the well-known Schur's condition. As applications, some inequalities of the entropy are established by use of majorization.

Although the concept of entropy was originally a thermodynamic construct, it has been adapted to other fields, including information theory. In this theory, entropy is a measure of the uncertainty associated with a random variable. It measures the average information content one is missing when one does not know the value of the random variable. The concept was introduced by Claude E. Shannon in his 1948 paper “A Mathematical Theory of Communication.”

In this paper we explore and compare two contrasting graph characterizations. The first of these is Estrada's heterogeneity index, which measures the heterogeneity of the node degree across a graph. Our second measure is the the von Neumann entropy associated with the Laplacian eigenspectrum of graphs. Here we show how to approximate the von Neumann entropy by replacing the Shannon entropy by its quadratic counterpart. This quadratic entropy can be expressed in terms of a series of permutation invariant traces, which can be computed from the node degrees in quadratic time. We compare experimentally the effectiveness of the approximate expression for the entropy with the heterogeneity index.

The aggregate uncertainty is the only known functional for Dempster-Shafer theory that generalizes the Shannon and Hartley measures and satisfies all classical requirements for uncertainty measures, including subadditivity. Although being posed several times in the literature, it is still an open problem whether the aggregate uncertainty is unique under these properties. This paper derives an uncertainty measure based on the theory of hints and shows its equivalence to the pignistic entropy. It does not satisfy subadditivity, but the viewpoint of hints uncovers a weaker version of subadditivity. On the other hand, the pignistic entropy has some crucial advantages over the aggregate uncertainty. i.e. explicitness of the formula and sensitivity to changes in evidence. We observe that neither of the two measures captures the full uncertainty of hints and propose an extension of the pignistic entropy called hints entropy that satisfies all axiomatic requirements, including subadditivity, while preserving the above advantages over the aggregate uncertainty.

A definition for the entropy of fuzzy measures defined on set systems is proposed. The underlying set is not necessarily the whole power set, but satisfy a condition of regularity. This definition encompasses the classical definition of Shannon for probability measures, as well as the definition of Marichal et al. for classical fuzzy measures, and may have applicability to most fuzzy measures which appear in applications. We give several characterizations of this entropy which are natural and understandable as the entropy.

Mutual information (MI) was introduced for use in multimodal image registration over a decade ago [1,2,3,4]. The MI between two images is based on their marginal and joint/conditional entropies. The most common versions of entropy used to compute MI are the Shannon and differential entropies; however, many other definitions of entropy have been proposed as competitors. In this article, we show how to construct normalized versions of MI using any of these definitions of entropy. The resulting similarity measures are analogous to normalized mutual information (NMI), entropy correlation coefficient (ECC), and symmetric uncertainty (SU), which have all been shown to be superior to MI in a variety of situations. We use publicly available CT, PET, and MR brain images1 with known ground truth transformations to evaluate the performance of the normalized measures for rigid multimodal registration. Results show that for a number of different definitions of entropy, the proposed normalized versions of mutual information provide a statistically significant improvement in target registration error (TRE) over the non-normalized versions.

The word “entropy” was coined by the German physicist R. Clausius (1822–1888), who introduced it in thermodynamics in 1865 to measure the amount of energy in a system that cannot produce work. The fact that the entropy of an isolated system never decreases constitutes the second law of thermodynamics and clearly shows the central role of entropy in many-particle physics. The direction of time is then explained as a consequence of the increase of entropy in all irreversible processes. Later on the concept of entropy was given a microscopic interpretation in the foundational works of L. Boltzmann (1844–1906) on gas kinetics and statistical mechanics [184]. The celebrated Boltzmann’s equation reads in the usual physical notation

Multiple microphone devices aimed at hands-free capabilities and speech recognition via acoustic beamforming require reliable estimates of the position of the acoustic source. Two-stage approaches based on the timedifference- of-arrival (TDOA) are computationally simple but lack robustness in practical environments. This chapter presents a family of broadband source localization algorithms based on parameterized spatiotemporal correlation, including the popular and robust steered response power (SRP) algorithm. Before forming the conventional spatial correlation matrix, the vector of microphone signals is time-aligned with respect to a hypothesized source location. It is shown that this parametrization easily generalizes classical narrowband techniques to the broadband setting. Methods based on minimum information entropy and temporally constrained minimum variance are developed. In order to ease the high computational demands imposed by a location parameterized scheme, a sparse representation of the parameterized spatial correlation matrix is proposed.

As sensor networks increase in size and number, efficient techniques are required to process the very large data sets that they generate. Frequently, sensor networks monitor objects in motion within their vicinity; the data associated with the movement of these objects are known as kinetic data. In an earlier paper we introduced an algorithm which, given a set of sensor observations, losslessly compresses this data to a size that is within a constant factor of the asymptotically optimal joint entropy bound. In this paper we present an efficient algorithm for answering spatio-temporal range queries. Our algorithm operates on a compressed representation of the data, without the need to decompress it. We analyze the efficiency of our algorithm in terms of a natural measure of information content, the joint entropy of the sensor outputs. We show that with space roughly equal to entropy, queries can be answered in time that is roughly logarithmic in entropy. In addition, we show experimentally that on real-world data our range searching structures use less space and have faster query times than the naive versions. These results represent the first solutions to range searching problems over compressed kinetic sensor data.

A metric has been developed for measuring privacy degradation when various types of media reveal private information about a person. The metric, which is based on joint entropy, quantifies accumulated revelations of information about various personal attributes. Application of this metric to entries posted on a social networking service and to leaks from a company database containing personal information showed that it is effective; that is, it can quantify accumulated revelations of information about multiple attributes and can cope with cases in which the attributes affect each other.

In recent years, there has been a growing interest in considering the quantitative aspects of Information Flow, partly because often the a priori knowledge of the secret information can be represented by a probability distribution, and partly because the mechanisms to protect the information may use randomization to obfuscate the relation between the secrets and the observables.

A non-probabilistic-type entropy measure for interval-valued fuzzy set (IVFS) is proposed. It is a result of a geometric interpretation of IVFS and uses a ratio of distances between them. It is also shown that the proposed measure can be defined in terms of the ratio of interval-valued fuzzy cardinalities: of F ∩ F c and F ∪ F c .

This paper proposes the use of non-extensive entropy for text classification. Non-extensive entropy technique is used for text classification by estimating the conditional distribution of the class variable given the document. The underlying principle of non-extensive entropy is that without external knowledge, one should prefer distributions that are uniform. This paper proposes two models for text classification based on maximum entropy principle. The first model extends Shannon entropy into non-extensive entropy to simplify the form of classifier, the other one introduces high-level constraints into non-extensive model to impose constraints on the pairs of entities. Model with high-level constraints constructs relations between word pairs which builds semantic constraints, for the sake of advancing accuracy of text classification. Experiments on the 20 newsgroup set demonstrate the advantage of non-extensive model and non-extensive model with high-level constraints.

We introduce a framework for storing and processing kinetic data observed by sensor networks. These sensor networks generate vast quantities of data, which motivates a significant need for data compression. We are given a set of sensors, each of which continuously monitors some region of space. We are interested in the kinetic data generated by a finite set of objects moving through space, as observed by these sensors. Our model relies purely on sensor observations; it allows points to move freely and requires no advance notification of motion plans. Sensor outputs are represented as random processes, where nearby sensors may be statistically dependent. We model the local nature of sensor networks by assuming that two sensor outputs are statistically dependent only if the two sensors are among the k nearest neighbors of each other. We present an algorithm for the lossless compression of the data produced by the network. We show that, under the statistical dependence and locality assumptions of our framework, asymptotically this compression algorithm encodes the data to within a constant factor of the information-theoretic lower bound optimum dictated by the joint entropy of the system.

The cumulative entropy is a new measure of information, alternative to the classical differential entropy. It has been recently proposed in analogy with the cumulative residual entropy studied by Wang et al. (2003a) and (2003b). After recalling its main properties, including a connection to reliability theory, we discuss estimates of random lifetimes based on the empirical cumulative entropy, which is suitably expressed in terms of the dual normalized sample spacings.

We introduce a new hybrid approach for spline-based elastic registration of multimodal medical images. The approach uses point landmarks as well as intensity information based on local analytic measures for joint entropy and mutual information. The information-theoretic similarity measures are computationally efficient and can be optimized independently for each voxel. We have applied our approach to synthetic images, brain phantom images, as well as clinically relevant multimodal medical images. We also compared our measures with previous measures.

Some approaches to the covering information entropy and some definitions of orderings and quasi---orderings of coverings will be described, generalizing the case of the partition entropy and ordering. The aim is to extend to covering the general result of anti---tonicity (strictly decreasing monotonicity) of partition entropy. In particular an entropy in the case of incomplete information systems is discussed, with the expected anti-tonicity result, making use of a partial partition strategy in which the missing information is treated as a peculiar value of the system. On the other side, an approach to generate a partition from a covering is illustrated. In particular, if we have a covering ? coarser than another covering ? with respect to a certain quasi order relation on coverings, the induced partition ?(?) results to be coarser than ?(?) with respect to the standard partial ordering on partitions. Thus, one can compare the two coverings through the entropies of the induced partitions.

In this paper, we propose a Robust Discriminant Analysis based on maximum entropy (MaxEnt) criterion (MaxEnt-RDA), which is derived from a nonparametric estimate of Renyi's quadratic entropy. MaxEnt-RDA uses entropy as both objective and constraints; thus the structural information of classes is preserved while information loss is minimized. It is a natural extension of LDA from Gaussian assumption to any distribution assumption. Like LDA, the optimal solution of MaxEnt-RDA can also be solved by an eigen-decomposition method, where feature extraction is achieved by designing two Parzen probability matrices that characterize the within-class variation and the between-class variation respectively. Furthermore, MaxEnt-RDA makes use of high order statistics (entropy) to estimate the probability matrix so that it is robust to outliers. Experiments on toy problem , UCI datasets and face datasets demonstrate the effectiveness of the proposed method with comparison to other state-of-the-art methods.

The discovery of diversity patterns from binary data is an important data mining task. This paper proposes entropy l -diversity patterns based on information theory, and develops techniques for discovering such diversity patterns. We study the properties of the entropy l -diversity patterns, and propose some pruning strategies to speed our mining algorithm. Experiments show that our mining algorithm is fast in practice. For real datesets the running time are improved by serval orders of magnitude over brute force method.

Information theoretic measures to incorporate anatomical priors have been explored in the field of emission tomography, but not in transmission tomography. In this work, we apply the joint entropy prior to the case of limited angle transmission tomography. Due to the data insufficiency problem, the joint entropy prior is found to be very sensitive to local optima. Two methods for robust joint entropy minimization are proposed. The first approximates the joint probability density function by a single 2D Gaussian, and is found to be appropriate for reconstructions where the ground truth joint histogram is dominated by two clusters, or multiple clusters that are roughly aligned. The second method is an extension to the case of multiple Gaussians. The intended application for the single Gaussian approximation is digital breast tomosynthesis, where reconstructed volumes are approximately bimodal, consisting mainly of fatty and fibroglandular tissues.

Exploded views are often used in illustration to overcome the problem of occlusion when depicting complex structures. In this paper, we propose a volume visualization technique inspired by exploded views that partitions the volume into a number of parallel slabs and shows them apart from each other. The thickness of slabs is driven by the similarity between partitions. We use an information-theoretic technique for the generation of exploded views. First, the algorithm identifies the viewpoint from which the structure is the highest. Then, the partition of the volume into the most informative slabs for exploding is obtained using two complementary similarity-based strategies. The number of slabs and the similarity parameter are freely adjustable by the user. acceptedVersion

Tomosynthesis as a technique is being developed and studied with the goal of overcoming mammography's limitations due to overlying tissue. Various algorithms exist for tomosynthesis datasets including a novel Computer Aided Detection (CADe) algorithm using a featureless False Positive (FP) reduction stage. The goal of this project is to study the previously unexplored effects of variation of Region of Interest (ROI) sizes as well as the crucial similarity metrics for such a CADe algorithm's performance. Four datasets consisting of 1479 tomosynthesis ROIs were generated by a CADe algorithm from reconstructed volumes of one hundred subjects consisting of 4 different sizes --- 128 x 128, 256 x 256, 512 x 512, and 1024 x 1024 pixels. Five different similarity metrics --- (1) mutual information, (2) average conditional entropy, (3) joint entropy, (3) Jensen divergence and (4) average Kullback-Leibler divergence were used for the task of FP reduction using a leave-one-case-out sampling scheme. Mutual information and average conditional entropy were the best performing metrics with an Area Under Curve (AUC) of 0.88. Cross-bin measures performed consistently higher than those that rely on only marginal distributions. Also, for all metrics, the datatset consisting of 256 x 256 pixel ROIs gave the best performance. In conclusion, for the tomosynthesis dataset, cross-bin measures such as MI and average conditional entropy should be used over other metrics using a ROI size of 256 x 256 pixels.

The signal processing community has shown to be increasingly reliant upon information theoretic concepts to develop algorithms for a wide variety of important problems ranging e. g. from audio-visual signal processing to medical imaging. When comparing the proposed algorithms, two facts are particularly surprising. First, the range of practical problems which, are solved with the fundamentally very compact mathematical concepts of information theory seem to be very broad and unrelated. The second striking fact is that the mathematical expressions governing the final algorithms seem not to be much related, even though the employed fundamental concepts were identical. The main aim of this chapter consists of developing an information theoretic framework for multimodal signal processing, closely related to information theoretic feature extraction/selection. This important relationship will indicate how we can unify to a large extent multimodal medical image processing, e. g. multi-channel segmentation and image registration, and extend information theoretic registration to other features than image intensities. The framework is not at all restricted to medical images though and we will illustrate this by applying it to multimedia sequences as well.

The abstract notion of rough approximation space is applied to the concrete cases of topological spaces with the particular situation of clopen–topologies generated by partitions, according to the Pawlak approach to rough set theory. In this partition context of a finite universe, typical of complete information systems, the probability space generated by the counting measure is analyzed, with particular regard to a local notion of rough entropy linked to the Shannon approach to these arguments. In the context of partition the notion of entropy as measure of uncertainty is distinguished from the notion of co–entropy as measure of granularity. The above considerations are extended to the case of covering, typical situation of incomplete information systems with the associated similarity relation.

This paper deals with video segmentation based on motion and spatial information. Classically, the nucleus of the motion term is the motion compensation error (MCE) between two consecutive frames. Defining a motion-based energy as the integral of a function of the MCE over the object domain implicitly results in making an assumption on the MCE distribution: Gaussian for the square function, Laplacian for the absolute value, or other parametric distributions for functions used in robust estimation. However, these assumptions are generally false. Instead, it is proposed to integrate a function of (an estimation of) the MCE distribution. The function is taken such that the integral is the Ahmad-Lin entropy of the MCE, the purpose being to be more robust to outliers. Since a motion-only approach can fail in homogeneous areas, the proposed energy is the joint entropy of the MCE and the object color. It is minimized using active contours.

Beaubouef, Petry and Buckles proposed the generalized rough set database analysis (GRSDA) to discuss rough relational databases. Given any rough relational database (U, A) and an attribute a ∈ A, as in rough set theory, a definition of the lower and upper approximations based on φa is given. The entropy and conditional entropy of similarity relations in a rough relational database are defined. The examples show that the entropy of a similarity relation does not decrease as the similarity relation is refined. It will be proved that given any two similarity relations φ and ψ, defined by a set C of conditional attributes and a decision attribute d, respectively, if d similarly depends on C in a rough relational database then the conditional entropy of φ with respect to ψ is equal to the entropy of φ.