Search

Real-world adversarial patches were shown to be successful in compromising state-of-the-art models in various computer vision applications. Most existing defenses rely on analyzing input or feature level gradients to detect the patch. However, these methods have been compromised by recent GAN-based attacks that generate naturalistic patches. In this paper, we propose a new perspective to defend against adversarial patches based on the entropy carried by the input, rather than on its saliency. We present Jedi, a new defense against adversarial patches that tackles the patch localization problem from an information theory perspective; leveraging the high entropy of adversarial patches to identify potential patch zones, and using an autoencoder to complete patch regions from high entropy kernels. Jedi achieves high-precision adversarial patch localization and removal, detecting on average 90% of adversarial patches across different benchmarks, and recovering up to 94% of successful patch attacks. Since Jedi relies on an input entropy analysis, it is model-agnostic, and can be applied to off-the-shelf models without changes to the training or inference of the models. Moreover, we propose a comprehensive qualitative analysis that investigates the cases where Jedi fails, comparatively with related methods. Interestingly, we find a significant core failure cases among the different defenses share one common property: high entropy. We think that this work offers a new perspective to understand the adversarial effect under physical-world settings. We also leverage these findings to enhance Jedi's handling of entropy outliers by introducing Adaptive Jedi, which boosts performance by up to 9% in challenging images., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier Ltd. All rights reserved.)

Entropy is not just a property of a system - it is a property of a system and an observer. Specifically, entropy is a measure of the amount of hidden information in a system that arises due to an observer's limitations. Here we provide an account of entropy from first principles in statistical mechanics with the aid of toy models of neural systems. Specifically, we describe the distinction between micro and macrostates in the context of simplified binary-state neurons and the characteristics of entropy required to capture an associated measure of hidden information. We discuss the origin of the mathematical form of entropy via the indistinguishable re-arrangements of discrete-state neurons and show the way in which the arguments are extended into a phase space description for continuous large-scale neural systems. Finally, we show the ways in which limitations in neuroimaging resolution, as represented by coarse graining operations in phase space, lead to an increase in entropy in time as per the second law of thermodynamics. It is our hope that this primer will support the increasing number of studies that use entropy as a way of characterising neuroimaging timeseries and of making inferences about brain states., (Copyright © 2023 Elsevier Ltd. All rights reserved.)

In this paper, we present several new theoretical measures based on information entropy that can be used to analyze the information content of a chiral molecule. Starting from a differentiation between "chiral" and "achiral" portions in a chiral molecule, we define a new concept that allows us to quantify the complexity of chiral constitutional 2D-isomers of C 10 to C 20 alkanes. Various new chiral and achiral information measures founded on joint entropy, mutual information, and conditional entropy are presented providing an access to a set of regression equations. Then, introducing a case-based measure of entropy, we demonstrate that the distribution of the chiral complexity in these molecules is mostly skewed-right: 60% of the chiral isomers follow a 60/40 distribution rule, which indicates a concentration of chiral complexity in a small number of topological features. Furthermore, by replacing 2D topological distances by 3D distances, the application of these new information measures goes from conformational to racemization and deracemization studies. Interestingly, when the geometrical distances between atoms and the chiral center(s) are taken into account when determining the chiral information entropy, one can observe a significative Pearson correlation coefficient (R = 0.70) between the chiral entropy of 3D molecules and the continuous chirality measure. Finally, we show that our approach is applicable to almost any type of chiral organic chemical structures if in the entropy equation, atoms are represented by their electrotopological state (E-state) index instead of connectivity., (© 2022 Wiley Periodicals LLC.)

The accurate identification of the content and number of latent factors underlying multivariate data is an important endeavor in many areas of Psychology and related fields. Recently, a new dimensionality assessment technique based on network psychometrics was proposed (Exploratory Graph Analysis, EGA), but a measure to check the fit of the dimensionality structure to the data estimated via EGA is still lacking. Although traditional factor-analytic fit measures are widespread, recent research has identified limitations for their effectiveness in categorical variables. Here, we propose three new fit measures (termed entropy fit indices) that combines information theory, quantum information theory and structural analysis: Entropy Fit Index (EFI), EFI with Von Neumman Entropy (EFI.vn) and Total EFI.vn (TEFI.vn). The first can be estimated in complete datasets using Shannon entropy, while EFI.vn and TEFI.vn can be estimated in correlation matrices using quantum information metrics. We show, through several simulations, that TEFI.vn, EFI.vn and EFI are as accurate or more accurate than traditional fit measures when identifying the number of simulated latent factors. However, in conditions where more factors are extracted than the number of factors simulated, only TEFI.vn presents a very high accuracy. In addition, we provide an applied example that demonstrates how the new fit measures can be used with a real-world dataset, using exploratory graph analysis.

Background: The early diagnosis of lung cancer has been a critical problem in clinical practice for a long time and identifying differentially expressed gene as disease marker is a promising solution. However, the most existing gene differential expression analysis (DEA) methods have two main drawbacks: First, these methods are based on fixed statistical hypotheses and not always effective; Second, these methods can not identify a certain expression level boundary when there is no obvious expression level gap between control and experiment groups., Methods: This paper proposed a novel approach to identify marker genes and gene expression level boundary for lung cancer. By calculating a kernel maximum mean discrepancy, our method can evaluate the expression differences between normal, normal adjacent to tumor (NAT) and tumor samples. For the potential marker genes, the expression level boundaries among different groups are defined with the information entropy method., Results: Compared with two conventional methods t-test and fold change, the top average ranked genes selected by our method can achieve better performance under all metrics in the 10-fold cross-validation. Then GO and KEGG enrichment analysis are conducted to explore the biological function of the top 100 ranked genes. At last, we choose the top 10 average ranked genes as lung cancer markers and their expression boundaries are calculated and reported., Conclusion: The proposed approach is effective to identify gene markers for lung cancer diagnosis. It is not only more accurate than conventional DEA methods but also provides a reliable method to identify the gene expression level boundaries.

Younger adults with normal hearing can typically understand speech in the presence of a competing speaker without much effort, but this ability to understand speech in challenging conditions deteriorates with age. Older adults, even with clinically normal hearing, often have problems understanding speech in noise. Earlier auditory studies using the frequency-following response (FFR), primarily believed to be generated by the midbrain, demonstrated age-related neural deficits when analyzed with traditional measures. Here we use a mutual information paradigm to analyze the FFR to speech (masked by a competing speech signal) by estimating the amount of stimulus information contained in the FFR. Our results show, first, a broadband informational loss associated with aging for both FFR amplitude and phase. Second, this age-related loss of information is more severe in higher-frequency FFR bands (several hundred hertz). Third, the mutual information between the FFR and the stimulus decreases as noise level increases for both age groups. Fourth, older adults benefit neurally, i.e., show a reduction in loss of information, when the speech masker is changed from meaningful (talker speaking a language that they can comprehend, such as English) to meaningless (talker speaking a language that they cannot comprehend, such as Dutch). This benefit is not seen in younger listeners, which suggests that age-related informational loss may be more severe when the speech masker is meaningful than when it is meaningless. In summary, as a method, mutual information analysis can unveil new results that traditional measures may not have enough statistical power to assess. NEW & NOTEWORTHY Older adults, even with clinically normal hearing, often have problems understanding speech in noise. Auditory studies using the frequency-following response (FFR) have demonstrated age-related neural deficits with traditional methods. Here we use a mutual information paradigm to analyze the FFR to speech masked by competing speech. Results confirm those from traditional analysis but additionally show that older adults benefit neurally when the masker changes from a language that they comprehend to a language they cannot.

The maximum entropy principle offers a constructive criterion for setting up probability distributions on the basis of partial knowledge. In the present work, the principle is applied to tackle an important problem in the surrogate marker field, namely, the evaluation of a binary outcome as a putative surrogate for a binary true endpoint within a causal inference framework. In the first step, the maximum entropy principle is used to determine the relative frequencies associated with the values of the vector of potential outcomes. Subsequently, in the second step, these relative frequencies are used in combination with two newly proposed metrics of surrogacy, the so-called individual causal association and the surrogate predictive function, to assess the validity of the surrogate. The procedure is conceptually similar to the use of noninformative or reference priors in Bayesian statistics. Additionally, approximate, identifiable bounds are proposed for the estimands of interest, and their performance is studied via simulations. The methods are illustrated using data from a clinical trial involving schizophrenic patients, and a newly developed and user-friendly R package Surrogate is provided to carry out the validation exercise., (© 2018 John Wiley & Sons, Ltd.)

A new class of functions, called the 'information sensitivity functions' (ISFs), which quantify the information gain about the parameters through the measurements/observables of a dynamical system are presented. These functions can be easily computed through classical sensitivity functions alone and are based on Bayesian and information-theoretic approaches. While marginal information gain is quantified by decrease in differential entropy, correlations between arbitrary sets of parameters are assessed through mutual information. For individual parameters, these information gains are also presented as marginal posterior variances, and, to assess the effect of correlations, as conditional variances when other parameters are given. The easy to interpret ISFs can be used to (a) identify time intervals or regions in dynamical system behaviour where information about the parameters is concentrated; (b) assess the effect of measurement noise on the information gain for the parameters; (c) assess whether sufficient information in an experimental protocol (input, measurements and their frequency) is available to identify the parameters; (d) assess correlation in the posterior distribution of the parameters to identify the sets of parameters that are likely to be indistinguishable; and (e) assess identifiability problems for particular sets of parameters., (© 2018 The Authors.)

Currently, the comparison between GC usage pattern at the 3rd codon position and codon usage index is commonly used to estimate the roles of evolutionary forces in shaping synonymous codon usages, however, this kind of analysis often losses the information about the role of A/T usage bias in shaping synonymous codon usage bias. To overcome this limitation and better understand the interplay between nucleotide and codon usages for the evolution of bacteria at gene levels, in this study, we employed the information entropy method with some modification to estimate roles of nucleotide compositions in the overall codon usage bias for 18 mycoplasma species in combination with Davies-Bouldin index. At gene levels, the overall nucleotide usage bias represents A content as the highest, followed by T, G and C for mycoplasmas, resulting in a low GC content. This feature is universal across these species derived from different hosts, suggesting that the hosts have the limited impact on nucleotide usage bias of mycoplasmas. Information entropy and Davies-Bouldin index can better reveal that the nucleotide usage bias at the 3rd codon position is essential in shaping the overall nucleotide bias for all given mycoplasmas except M. pneumoniae M129. Davies-Bouldin index revealed that the 1st and 2nd codon position play more important role in synonymous codon usage bias than that of the 3rd position at gene levels. To our knowledge, this is the first comprehensive investigation into cooperation between nucleotide and codon usages for mycoplasma and extends our knowledge of the mechanisms that contribute to codon usage and evolution of this microorganism., (Copyright © 2017 Elsevier B.V. All rights reserved.)

The principle of entropy rate constancy (ERC) states that language users distribute information such that words tend to be equally predictable given previous contexts. We examine the applicability of this principle to spoken dialogue, as previous findings primarily rest on written text. The study takes into account the joint-activity nature of dialogue and the topic shift mechanisms that are different from monologue. It examines how the information contributions from the two dialogue partners interactively evolve as the discourse develops. The increase of local sentence-level information density (predicted by ERC) is shown to apply to dialogue overall. However, when the different roles of interlocutors in introducing new topics are identified, their contribution in information content displays a new converging pattern. We draw explanations to this pattern from multiple perspectives: Casting dialogue as an information exchange system would mean that the pattern is the result of two interlocutors maintaining their own context rather than sharing one. Second, we present some empirical evidence that a model of Interactive Alignment may include information density to explain the effect. Third, we argue that building common ground is a process analogous to information convergence. Thus, we put forward an information-theoretic view of dialogue, under which some existing theories of human dialogue may eventually be unified., (Copyright © 2017 Elsevier B.V. All rights reserved.)

Directed information transfer measures are increasingly being employed in modeling neural system behavior due to their model-free approach, applicability to nonlinear and stochastic signals, and the potential to integrate repetitions of an experiment. Intracellular physiological recordings of graded synaptic potentials provide a number of additional challenges compared to spike signals due to non-stationary behaviour generated through extrinsic processes. We therefore propose a method to overcome this difficulty by using a preprocessing step based on Singular Spectrum Analysis (SSA) to remove nonlinear trends and discontinuities. We apply the method to intracellular recordings of synaptic responses of identified motor neurons evoked by stimulation of a proprioceptor that monitors limb position in leg of the desert locust. We then apply normalized delayed transfer entropy measures to neural responses evoked by displacements of the proprioceptor, the femoral chordotonal organ, that contains sensory neurones that monitor movements about the femoral-tibial joint. We then determine the consistency of responses within an individual recording of an identified motor neuron in a single animal, between repetitions of the same experiment in an identified motor neurons in the same animal and in repetitions of the same experiment from the same identified motor neuron in different animals. We found that delayed transfer entropy measures were consistent for a given identified neuron within and between animals and that they predict neural connectivity for the fast extensor tibiae motor neuron.

Based on the Shannon's information communication theory, information amount of the entire length of a polymeric macromolecule can be calculated in bits through adding the entropies of each building block. Proteins, DNA and RNA are such macromolecules. When only the building blocks' variation is considered as the source of entropy, there is seemingly lower information in case of the protein if this approach is applied directly on a protein of specific size and the coding sequence size of the mRNA corresponding to the particular length of the protein. This decrease in the information amount seems contradictory but this apparent conflict is resolved by considering the conformational variations in proteins as a new variable in the calculation and balancing the approximated entropy of the coding part of the mRNA and the protein. Probabilities can change therefore we also assigned hypothetical probabilities to the conformational states, which represent the uneven distribution as the time spent in one conformation, providing the probability of the presence in either or one of the possible conformations. Results that are obtained by using hypothetical probabilities are in line with the experimental values of variations in the conformational-state of protein populations. This equalization approach has further biological relevance that it compensates for the degeneracy in the codon usage during protein translation and it leads to the conclusion that the alphabet size for the protein is rather optimal for the proper protein functioning within the thermodynamic milieu of the cell. The findings were also discussed in relation to the codon bias and have implications in relation to the codon evolution concept. Eventually, this work brings the fields of protein structural studies and molecular protein translation processes together with a novel approach., (Copyright © 2017 Elsevier B.V. All rights reserved.)

The Shannon entropy measures the expected information value of messages. As with thermodynamic entropy, the Shannon entropy is only defined within a system that identifies at the outset the collections of possible messages, analogous to microstates, that will be considered indistinguishable macrostates. This fundamental insight is applied here for the first time to amino acid alphabets, which group the twenty common amino acids into families based on chemical and physical similarities. To evaluate these schemas objectively, a novel quantitative method is introduced based the inherent redundancy in the canonical genetic code. Each alphabet is taken as a separate system that partitions the 64 possible RNA codons, the microstates, into families, the macrostates. By calculating the normalized mutual information, which measures the reduction in Shannon entropy, conveyed by single nucleotide messages, groupings that best leverage this aspect of fault tolerance in the code are identified. The relative importance of properties related to protein folding - like hydropathy and size - and function, including side-chain acidity, can also be estimated. This approach allows the quantification of the average information value of nucleotide positions, which can shed light on the coevolution of the canonical genetic code with the tRNA-protein translation mechanism., (Copyright © 2016 Elsevier Ltd. All rights reserved.)

Epistasis is a key concept in the theory of adaptation. Indicators of epistasis are of interest for large systems where systematic fitness measurements may not be possible. Some recent approaches depend on information theory. We show that considering shared entropy for pairs of loci can be misleading. The reason is that shared entropy does not imply epistasis for the pair. This observation holds true also in the absence of higher order epistasis. We discuss a method for reducing the number of false positives. However, our main conclusion is that entropy-based approaches have serious limitations in this context., Competing Interests: The author has declared that no competing interests exist.

Objective: Clinical assessments that rely on behavioral responses to differentiate Disorders of Consciousness are at times inapt because of some patients' motor disabilities. To objectify patients' conditions of reduced consciousness the present study evaluated the use of electroencephalography to measure residual brain activity., Methods: We analyzed entropy values of 18 scalp EEG channels of 15 severely brain-damaged patients with clinically diagnosed Minimally-Conscious-State (MCS) or Unresponsive-Wakefulness-Syndrome (UWS) and compared the results to a sample of 24 control subjects. Permutation entropy (PeEn) and symbolic transfer entropy (STEn), reflecting information processes in the EEG, were calculated for all subjects. Participants were tested on a modified active own-name paradigm to identify correlates of active instruction following., Results: PeEn showed reduced local information content in the EEG in patients, that was most pronounced in UWS. STEn analysis revealed altered directed information flow in the EEG of patients, indicating impaired feed-backward connectivity. Responses to auditory stimulation yielded differences in entropy measures, indicating reduced information processing in MCS and UWS., Conclusions: Local EEG information content and information flow are affected in Disorders of Consciousness. This suggests local cortical information capacity and feedback information transfer as neural correlates of consciousness., Significance: The utilized EEG entropy analyses were able to relate to patient groups with different Disorders of Consciousness., (Copyright © 2015 International Federation of Clinical Neurophysiology. Published by Elsevier Ireland Ltd. All rights reserved.)

A new approach for assessing parameter identifiability of dynamical systems in a Bayesian setting is presented. The concept of Shannon entropy is employed to measure the inherent uncertainty in the parameters. The expected reduction in this uncertainty is seen as the amount of information one expects to gain about the parameters due to the availability of noisy measurements of the dynamical system. Such expected information gain is interpreted in terms of the variance of a hypothetical measurement device that can measure the parameters directly, and is related to practical identifiability of the parameters. If the individual parameters are unidentifiable, correlation between parameter combinations is assessed through conditional mutual information to determine which sets of parameters can be identified together. The information theoretic quantities of entropy and information are evaluated numerically through a combination of Monte Carlo and k-nearest neighbour methods in a non-parametric fashion. Unlike many methods to evaluate identifiability proposed in the literature, the proposed approach takes the measurement-noise into account and is not restricted to any particular noise-structure. Whilst computationally intensive for large dynamical systems, it is easily parallelisable and is non-intrusive as it does not necessitate re-writing of the numerical solvers of the dynamical system. The application of such an approach is presented for a variety of dynamical systems--ranging from systems governed by ordinary differential equations to partial differential equations--and, where possible, validated against results previously published in the literature., (Copyright © 2015 Elsevier Inc. All rights reserved.)

We propose a general information-theoretic approach to semi-supervised metric learning called SERAPH (SEmi-supervised metRic leArning Paradigm with Hypersparsity) that does not rely on the manifold assumption. Given the probability parameterized by a Mahalanobis distance, we maximize its entropy on labeled data and minimize its entropy on unlabeled data following entropy regularization. For metric learning, entropy regularization improves manifold regularization by considering the dissimilarity information of unlabeled data in the unsupervised part, and hence it allows the supervised and unsupervised parts to be integrated in a natural and meaningful way. Moreover, we regularize SERAPH by trace-norm regularization to encourage low-dimensional projections associated with the distance metric. The nonconvex optimization problem of SERAPH could be solved efficiently and stably by either a gradient projection algorithm or an EM-like iterative algorithm whose M-step is convex. Experiments demonstrate that SERAPH compares favorably with many well-known metric learning methods, and the learned Mahalanobis distance possesses high discriminability even under noisy environments.

The maximum entropy principle (MEP) is a method for obtaining the most likely distribution functions of observables from statistical systems by maximizing entropy under constraints. The MEP has found hundreds of applications in ergodic and Markovian systems in statistical mechanics, information theory, and statistics. For several decades there has been an ongoing controversy over whether the notion of the maximum entropy principle can be extended in a meaningful way to nonextensive, nonergodic, and complex statistical systems and processes. In this paper we start by reviewing how Boltzmann-Gibbs-Shannon entropy is related to multiplicities of independent random processes. We then show how the relaxation of independence naturally leads to the most general entropies that are compatible with the first three Shannon-Khinchin axioms, the (c,d)-entropies. We demonstrate that the MEP is a perfectly consistent concept for nonergodic and complex statistical systems if their relative entropy can be factored into a generalized multiplicity and a constraint term. The problem of finding such a factorization reduces to finding an appropriate representation of relative entropy in a linear basis. In a particular example we show that path-dependent random processes with memory naturally require specific generalized entropies. The example is to our knowledge the first exact derivation of a generalized entropy from the microscopic properties of a path-dependent random process.

We showed recently that exposure to whisker vibrations enhances coding efficiency in rat barrel cortex despite increasing correlations in variability (Adibi et al., 2013). Here, to understand how adaptation achieves this improvement in sensory representation, we decomposed the stimulus information carried in neuronal population activity into its fundamental components in the framework of information theory. In the context of sensory coding, these components are the entropy of the responses across the entire stimulus set (response entropy) and the entropy of the responses conditional on the stimulus (conditional response entropy). We found that adaptation decreased response entropy and conditional response entropy at both the level of single neurons and the pooled activity of neuronal populations. However, the net effect of adaptation was to increase the mutual information because the drop in the conditional entropy outweighed the drop in the response entropy. The information transmitted by a single spike also increased under adaptation. As population size increased, the information content of individual spikes declined but the relative improvement attributable to adaptation was maintained.

A total of 74 flavonoid compounds are classified into a periodic table by using an algorithm based on the entropy of information theory. Seven features in hierarchical order are used to classify structurally the flavonoids. From these features, the first three mark the group or column, while the last four are used to indicate the row or period in a table of periodic classification. Those flavonoids in the same group and period are suggested to show maximum similarity in properties. Furthermore, those with only the same group will present moderate similarity. In this report, the flavonoid compounds in the table, whose experimental data in bioactivity and antioxidant properties have been previously published, are related., (Copyright © 2013 Elsevier Ltd. All rights reserved.)

Gene-gene interactions may play an important role in the genetics of a complex disease. Detection and characterization of gene-gene interactions is a challenging issue that has stimulated the development of various statistical methods to address it. In this study, we introduce a method to measure gene interactions using entropy-based statistics from a contingency table of trait and genotype combinations. We also developed an exploration procedure by using graphs. We propose a standardized relative information gain (RIG) measure to evaluate the interactions between single nucleotide polymorphism (SNP) combinations. To identify the k (th) order interactions, contingency tables of trait and genotype combinations of k SNPs are constructed, with which RIGs are calculated. The RIGs are standardized using the mean and standard deviation from the permuted datasets. SNP combinations yielding high standardized RIG are chosen for gene-gene interactions. Detection of high-order interactions and comparison of interaction strengths between different orders are made possible by using standardized RIG. We have applied the proposed standardized entropy-based method to two types of data sets from a simulation study and a real genetic association study. We have compared our method and the multifactor dimensionality reduction (MDR) method through power analysis of eight different genetic models with varying penetrance rates, number of SNPs, and sample sizes. Our method shows successful identification of genetic associations and gene-gene interactions both in simulation and real genetic data. Simulation results suggest that the proposed entropy-based method is better able to detect high-order interactions and is superior to the MDR method in most cases. The proposed method is well suited for detecting interactions without main effects as well as for models including main effects.

In this paper we discuss the entropy and information aspects of a living cell. Particular attention is paid to the information gain on assembling and maintaining a living state. Numerical estimates of the information and entropy reduction are given and discussed in the context of the cell's metabolic activity. We discuss a solution to an apparent paradox that there is less information content in DNA than in the proteins that are assembled based on the genetic code encrypted in DNA. When energy input required for protein synthesis is accounted for, the paradox is clearly resolved. Finally, differences between biological information and instruction are discussed., (Copyright © 2012 Elsevier Ireland Ltd. All rights reserved.)

To understand the function of networks we have to identify the structure of their interactions, but also interaction timing, as compromised timing of interactions may disrupt network function. We demonstrate how both questions can be addressed using a modified estimator of transfer entropy. Transfer entropy is an implementation of Wiener's principle of observational causality based on information theory, and detects arbitrary linear and non-linear interactions. Using a modified estimator that uses delayed states of the driving system and independently optimized delayed states of the receiving system, we show that transfer entropy values peak if the delay of the state of the driving system equals the true interaction delay. In addition, we show how reconstructed delays from a bivariate transfer entropy analysis of a network can be used to label spurious interactions arising from cascade effects and apply this approach to local field potential (LFP) and magnetoencephalography (MEG) data.

The identification of molecular descriptors that are able to distinguish between different compound classes is of paramount importance in chemoinformatics. To aid in the identification of such discriminatory descriptors, concepts from information theory have been adapted. In an earlier study, an approach termed Differential Shannon Entropy (DSE) has been introduced for descriptor profiling to detect and quantify compound database-dependent differences in the information content and value range distribution of descriptors. Because the DSE approach was intrinsically limited in its ability to select compound class-specific descriptors by comparing data sets of very different size, this approach has recently been extended to Mutual Information-DSE (MI-DSE). Herein, DSE, MI-DSE, and the Shannon entropy concept underlying both information theoretic approaches are introduced and compared, and differences between their application areas are discussed.

We propose a formal expansion of the transfer entropy to put in evidence irreducible sets of variables which provide information for the future state of each assigned target. Multiplets characterized by an high value will be associated to informational circuits present in the system, with an informational character (synergetic or redundant) which can be associated to the sign of the contribution. We also present preliminary results on fMRI and EEG data sets.

Spectral measures of causality are used to explore the role of different rhythms in the causal connectivity between brain regions. We study several spectral measures related to Granger causality, comprising the bivariate and conditional Geweke measures, the directed transfer function, and the partial directed coherence. We derive the formulation of dependence and causality in the spectral domain from the more general formulation in the information-theory framework. We argue that the transfer entropy, the most general measure derived from the concept of Granger causality, lacks a spectral representation in terms of only the processes associated with the recorded signals. For all the spectral measures we show how they are related to mutual information rates when explicitly considering the parametric autoregressive representation of the processes. In this way we express the conditional Geweke spectral measure in terms of a multiple coherence involving innovation variables inherent to the autoregressive representation. We also link partial directed coherence with Sims' criterion of causality. Given our results, we discuss the causal interpretation of the spectral measures related to Granger causality and stress the necessity to explicitly consider their specific formulation based on modeling the signals as linear Gaussian stationary autoregressive processes.

Understanding causal relationships, or effective connectivity, between parts of the brain is of utmost importance because a large part of the brain's activity is thought to be internally generated and, hence, quantifying stimulus response relationships alone does not fully describe brain dynamics. Past efforts to determine effective connectivity mostly relied on model based approaches such as Granger causality or dynamic causal modeling. Transfer entropy (TE) is an alternative measure of effective connectivity based on information theory. TE does not require a model of the interaction and is inherently non-linear. We investigated the applicability of TE as a metric in a test for effective connectivity to electrophysiological data based on simulations and magnetoencephalography (MEG) recordings in a simple motor task. In particular, we demonstrate that TE improved the detectability of effective connectivity for non-linear interactions, and for sensor level MEG signals where linear methods are hampered by signal-cross-talk due to volume conduction.

Recently, the minimum error entropy (MEE) criterion has been used as an information theoretic alternative to traditional mean-square error criterion in supervised learning systems. MEE yields nonquadratic, nonconvex performance surface even for adaptive linear neuron (ADALINE) training, which complicates the theoretical analysis of the method. In this paper, we develop a unified approach for mean-square convergence analysis for ADALINE training under MEE criterion. The weight update equation is formulated in the form of block-data. Based on a block version of energy conservation relation, and under several assumptions, we carry out the mean-square convergence analysis of this class of adaptation algorithm, including mean-square stability, mean-square evolution (transient behavior) and the mean-square steady-state performance. Simulation experimental results agree with the theoretical predictions very well.

Entropy rate quantifies the change of information of a stochastic process (Cover & Thomas, 2006). For decades, the temporal dynamics of spike trains generated by neurons has been studied as a stochastic process (Barbieri, Quirk, Frank, Wilson, & Brown, 2001; Brown, Frank, Tang, Quirk, & Wilson, 1998; Kass & Ventura, 2001; Metzner, Koch, Wessel, & Gabbiani, 1998; Zhang, Ginzburg, McNaughton, & Sejnowski, 1998). We propose here to estimate the entropy rate of a spike train from an inhomogeneous hidden Markov model of the spike intervals. The model is constructed by building a context tree structure to lay out the conditional probabilities of various subsequences of the spike train. For each state in the Markov chain, we assume a gamma distribution over the spike intervals, although any appropriate distribution may be employed as circumstances dictate. The entropy and confidence intervals for the entropy are calculated from bootstrapping samples taken from a large raw data sequence. The estimator was first tested on synthetic data generated by multiple-order Markov chains, and it always converged to the theoretical Shannon entropy rate (except in the case of a sixth-order model, where the calculations were terminated before convergence was reached). We also applied the method to experimental data and compare its performance with that of several other methods of entropy estimation.

The maximal information that the spike train of any neuron can pass on to subsequent neurons can be quantified as the neuronal firing pattern entropy. Difficulties associated with estimating entropy from small datasets have proven an obstacle to the widespread reporting of firing pattern entropies and more generally, the use of information theory within the neuroscience community. In the most accessible class of entropy estimation techniques, spike trains are partitioned linearly in time and entropy is estimated from the probability distribution of firing patterns within a partition. Ample previous work has focused on various techniques to minimize the finite dataset bias and standard deviation of entropy estimates from under-sampled probability distributions on spike timing events partitioned linearly in time. In this manuscript we present evidence that all distribution-based techniques would benefit from inter-spike intervals being partitioned in logarithmic time. We show that with logarithmic partitioning, firing rate changes become independent of firing pattern entropy. We delineate the entire entropy estimation process with two example neuronal models, demonstrating the robust improvements in bias and standard deviation that the logarithmic time method yields over two widely used linearly partitioned time approaches.

We give a short unified proof of the following theorem, valid in the context of both classical probability theory and Voiculescu's free probability theory: let (X(j)((1)), ..., X(j)((n))) be independent (resp., freely independent) n-tuples of random variables. Let Z(N)((p)) = N(-1/2)(X(1)((p)) + ... + X(N)((p))) be their central limit sums. Then the entropy (resp., free entropy) of the n-tuple (Z(N)((1)), ..., Z(N)((n))) is a monotone function of N. The classical case (for n = 1) is a celebrated result of Artstein, Ball, Barthe, and Naor, and our proof is an adaptation and simplification of their argument.

Competitive learning neural networks are regarded as a powerful tool for online data clustering to represent a non-stationary probability distribution with a fixed number of weight vectors. One difficulty in practical applications of competitive learning neural networks to online data clustering is that most of them require heuristically-predetermined threshold parameters for balancing a trade-off between convergence accuracy, i.e. error minimization performance, and speed of adaptation to the changes in source statistics. Although adaptation acceleration is achievable by relocating a "useless" node so that it becomes useful, excessive relocation often disturbs error minimization. Hence, both of the adaptation speed and the error minimization performance sensitively depend on threshold parameters to determine whether a node should be relocated or not. In general, it is difficult to know adequate threshold parameters a priori. This paper proposes a novel criterion for decision making of node relocation without heuristically predetermined thresholds. According to the proposed criterion, a node is relocated only if the relocation task improves partial distortion entropy, which is an online optimality metric reliable from the viewpoint of error minimization. Hence, node relocation is carried out without disturbing error minimization. As a result, both quick adaptation and error minimization are simultaneously accomplished without any carefully predefined parameters. Experimental results clarify the validity of the proposed criterion. Competitive learning with the criterion is clearly superior to other representative algorithms in terms of both quick adaptation and error minimization performance.

The exchange of energy between organism and environment perhaps may be to describe with help "entropia"--notion of thermodynamics. Point of view information technologies authors suggest to use "factor of controls", which work with help principle of reverse biological connection. This principle may be to use for control aging, when complicated modulated electromagnetic (information) field has regulationing negative entropic component. There are three principles of information control. All principles have on the basis of utilization by quantitative exponents of functional asymmetry.

The paper is devoted to the analysis of digitized sequences of real numbers and discrete strings, by means of the concepts of entropy and complexity. Special attention is paid to the random character of these quantities and their fluctuation spectrum. As applications, we discuss neural spike-trains and DNA sequences. We consider a given sequence as one realization of finite length of certain random process. The other members of the ensemble are defined by appropriate surrogate sequences and surrogate processes. We show that n-gram entropies and the context-free grammatical complexity have to be considered as fluctuating quantities and study the corresponding distributions. Different complexity measures reveal different aspects of a sequence. Finally, we show that the diversity of the entropy (that takes small values for pseudorandom strings) and the context-free grammatical complexity (which takes large values for pseudorandom strings) give, nonetheless, consistent results by comparison of the ranking of sample sequences taken from molecular biology, neuroscience, and artificial control sequences.

We applied the information theory concepts to notes repertoire characteristics combined with temporal parameters of the Rufous-bellied thrush Turdus rufiventris song, using this particular case to test a new method of analysing quantitatively complex animal communication systems. Like most Turdus thrushes, Rufous-bellied thrushes are remarkable for their long, varied and melodious songs. For the analysis of the species repertoire, we used recordings of 44 individuals from 24 localities covering its full geographical range. We measured the repertoire size, note duration and rhythm (frequency of note utterance), and combined these parameters with the Shannon entropy values calculated for each individual. Although individuals maintain species-specific recognition capacity, we find a large variation between their song parameters and show that the information theory can be useful to analyse large and varied animal vocal repertoires. We are introducing two new parameters, temporal average entropy (E(t)) and utterance frequency average entropy (E(f)), for measuring such communication systems., (Copyright 2000 Academic Press.)

The aim of this paper is to present a study of polynomial functional-link neural units that learn through an information-theoretic-based criterion. First the structure of the neuron is presented and the unsupervised learning theory is explained and discussed, with particular attention being paid to its probability density function and cumulative distribution function approximation capability. Then a neural network formed by such neurons (the polynomial functional-link artificial neural network, or PFANN) is shown to be able to separate out linearly mixed eterokurtic source signals, i.e. signals endowed with either positive or negative kurtoses. In order to compare the performance of the proposed blind separation technique with those exhibited by existing methods, the mixture of densities (MOD) approach of Xu et al, which is closely related to PFANN, is briefly recalled; then comparative numerical simulations performed on both synthetic and real-world signals and a complexity evaluation are illustrated. These results show that the PFANN approach gives similar performance with a noticeable reduction in computational effort.

The structure of linear strings carrying information is investigated by means of entropy concepts. First conditional entropy and transinformation are introduced and several generalizations are discussed. The capability to describe the structure of information carriers as DNA, proteins, texts and musical strings is investigated. The relation between order and the predictability of informational strings is discussed. As examples we study the mutual information function of virus DNA and several long proteins. Further we show some (rather formal) analogies to the structure of texts, and strings generated by musical melodies. It is shown that several information carriers show long-range correlations.

Discusses relationships between the frequency-based characteristics of neighboring terms in natural language and the rank or frequency of the terms. Topics include information theory measures, including expected mutual information measure (EMIM); entropy and rank; Luhn's model of term aboutness; Zipf's law; and implications for indexing and information retrieval. (LRW)

Proposes an alternative real valued representation of color based on the information theoretic concept of entropy. A theoretical presentation of image entropy is accompanied by a practical description of the merits and limitations of image entropy compared to color histograms. Results suggest that image entropy is a promising approach to image description and representation. (AEF)

The general-information-theoretic approach was used to identify informational overlap and asymmetry between variables, using affective, cognitive, and behavioral measures. Using the chi-squared test, no significant differences were found in response rates, demographics, or patronage frequency of three stores between numerical (n=453) and graphic (n=441) scales. (TJH)

Compares the notion of the conservation or inconservation of information with the laws of conservation of matter and energy. The theory of entropy in physics and its significance in information theory is discussed, and the essential nature of information is considered. (LRW)

Background: A variant can be pathogenic or benign with relation to a human disease. Current classification categories from benign to pathogenic reflect a probabilistic summary of the current understanding. A primary metric of clinical utility for multiplexed assays of variant effect (MAVE) is the number of variants that can be reclassified from uncertain significance (VUS). However, a gap in this measure of utility is that it underrepresents the information gained from MAVEs. The aim of this study was to develop an improved quantification metric for MAVE utility. We propose adopting an information content approach that includes data that does not reclassify variants will better reflect true information gain. We adopted an information content approach to evaluate the information gain, in bits, for MAVEs of BRCA1, PTEN, and TP53. Here, one bit represents the amount of information required to completely classify a single variant starting from no information. Results: BRCA1 MAVEs produced a total of 831.2 bits of information, 6.58% of the total missense information in BRCA1 and a 22-fold increase over the information that only contributed to VUS reclassification. PTEN MAVEs produced 2059.6 bits of information which represents 32.8% of the total missense information in PTEN and an 85-fold increase over the information that contributed to VUS reclassification. TP53 MAVEs produced 277.8 bits of information which represents 6.22% of the total missense information in TP53 and a 3.5-fold increase over the information that contributed to VUS reclassification. Conclusions: An information content approach will more accurately portray information gained through MAVE mapping efforts than by counting the number of variants reclassified. This information content approach may also help define the impact of guideline changes that modify the information definitions used to classify groups of variants. [ABSTRACT FROM AUTHOR]

The problem of quantifying the distance between distributions arises in various fields, including cryptography, information theory, communication networks, machine learning, and data mining. In this article, the analogy with the cumulative Jensen–Shannon divergence, defined in Nguyen and Vreeken (2015), we propose a new divergence measure based on the cumulative distribution function and call it the cumulative α-Jensen–Shannon divergence, denoted by CJS (α) . Properties of CJS (α) are studied in detail, and also two upper bounds for CJS (α) are obtained. The simplified results under the proportional reversed hazard rate model are given. Various illustrative examples are analyzed. [ABSTRACT FROM AUTHOR]

Anurans have several characteristics in their acoustic communication that distinguish them. These characteristics are influenced by body-related, environmental and social variables. However, the complexity of anuran repertoire and the effect of such variables on it are still understudied. Dendropsophus minutus is a Neotropical treefrog that has a wide repertoire composed of three notes, grouped in at least 71 unique combinations. To quantify complexity, we used the entropy measurement, since it measures the randomness of a sequence. Therefore, we could test if repertoire complexity in D. minutus is related with body condition, social context, and environmental variables. We recorded vocalisations of 80 males in three sites and found 44 unique note combinations not described before, increasing the species' repertoire to 115 combinations. We observed that the species has near maximum entropy for note frequency, but a low entropy for note sequencies, showing a clear a bias for some note combinations. We found a positive relationship between repertoire complexity and air temperature. Thus, based on our results, the complexity of this species' repertoire is an additional trait influenced by climate, and should be considered in future studies. [ABSTRACT FROM AUTHOR]

We present contrast information, a novel application of some specific cases of relative entropy, designed to be useful for the cognitive modelling of the sequential perception of continuous signals. We explain the relevance of entropy in the cognitive modelling of sequential phenomena such as music and language. Then, as a first step to demonstrating the utility of constrast information for this purpose, we empirically show that its discrete case correlates well with existing successful cognitive models in the literature. We explain some interesting properties of constrast information. Finally, we propose future work toward a cognitive architecture that uses it. [ABSTRACT FROM AUTHOR]

The concept of attention has proven to be very relevant in artificial intelligence. Relative entropy (RE, aka Kullback‐Leibler divergence) plays a central role in communication theory. Here, these concepts, attention, and RE are combined. RE guides optimal encoding of messages in bandwidth‐limited communication as well as optimal message decoding via the maximum entropy principle. In the coding scenario, RE can be derived from four requirements, namely being analytical, local, proper, and calibrated. Weighted RE, used for attention steering in communications, turns out to be improper. To see how proper attention communication can emerge, a scenario of a message sender who wants to ensure that the receiver of the message can perform well‐informed actions is analyzed. In case only the curvature of the utility function maxima are known, it becomes desirable to accurately communicate an attention function, in this case a by this curvature weighted and re‐normalized probability function. Entropic attention communication is here proposed as the desired generalization of entropic communication that permits weighting while being proper, thereby aiding the design of optimal communication protocols in technical applications and helping to understand human communication. It provides the level of cooperation expected under misaligned interests of otherwise honest communication partners. [ABSTRACT FROM AUTHOR]

Fuzzy extractors convert noisy signals from the physical world into reliable cryptographic keys. Fuzzy min-entropy measures the limit of the length of key that a fuzzy extractor can derive from a distribution (Fuller et al. in IEEE Trans Inf Theory 66(8):5282–5298, 2020). In general, fuzzy min-entropy that is superlogarithmic in the security parameter is required for a noisy distribution to be suitable for key derivation. There is a wide gap between what is possible with respect to computational and information-theoretic adversaries. Under the assumption of general-purpose obfuscation, keys can be securely derived from all distributions with superlogarithmic entropy. Against information-theoretic adversaries, however, it is impossible to build a single fuzzy extractor that works for all distributions (Fuller et al. 2020). A weaker information-theoretic goal is building a fuzzy extractor for each probability distribution. This is the approach taken by Woodage et al. (in: Advances in Cryptology—CRYPTO, Springer, pp 682–710, 2017). Prior approaches use the full description of the probability mass function and are inefficient. We show this is inherent: for a quarter of distributions with fuzzy min-entropy and 2 k points there is no secure fuzzy extractor that uses less 2 Θ (k) bits of information about the distribution. We show an analogous result with stronger parameters for information-theoretic secure sketches. Secure sketches are frequently used to construct fuzzy extractors. [ABSTRACT FROM AUTHOR]

Abstract Background A variant can be pathogenic or benign with relation to a human disease. Current classification categories from benign to pathogenic reflect a probabilistic summary of the current understanding. A primary metric of clinical utility for multiplexed assays of variant effect (MAVE) is the number of variants that can be reclassified from uncertain significance (VUS). However, a gap in this measure of utility is that it underrepresents the information gained from MAVEs. The aim of this study was to develop an improved quantification metric for MAVE utility. We propose adopting an information content approach that includes data that does not reclassify variants will better reflect true information gain. We adopted an information content approach to evaluate the information gain, in bits, for MAVEs of BRCA1, PTEN, and TP53. Here, one bit represents the amount of information required to completely classify a single variant starting from no information. Results BRCA1 MAVEs produced a total of 831.2 bits of information, 6.58% of the total missense information in BRCA1 and a 22-fold increase over the information that only contributed to VUS reclassification. PTEN MAVEs produced 2059.6 bits of information which represents 32.8% of the total missense information in PTEN and an 85-fold increase over the information that contributed to VUS reclassification. TP53 MAVEs produced 277.8 bits of information which represents 6.22% of the total missense information in TP53 and a 3.5-fold increase over the information that contributed to VUS reclassification. Conclusions An information content approach will more accurately portray information gained through MAVE mapping efforts than by counting the number of variants reclassified. This information content approach may also help define the impact of guideline changes that modify the information definitions used to classify groups of variants.