Search

We introduce a framework for the evaluation of multiclass classifiers by exploring their confusion matrices. Instead of using error-counting measures of performance, we concentrate in quantifying the information transfer from true to estimated labels using information-theoretic measures. First, the Entropy Triangle allows us to visualize the balance of mutual information, variation of information, and the deviation from uniformity in the true and estimated label distributions. Next, the Entropy-Modified Accuracy allows us to rank classifiers by performance while the Normalized Information Transfer rate allows us to evaluate classifiers by the amount of information accrued during learning. Finally, if the question rises to elucidate which errors are systematically committed by the classifier, we use a generalization of Formal Concept Analysis to elicit such knowledge. All such techniques can be applied either to artificially or biologically embodied classifiers—e.g., human performance on perceptual tasks. We instantiate the framework in a number of examples to provide guidelines for the use of these tools in the case of assessing single classifiers or populations of them—whether induced with the same technique or not—either on single tasks or in a set of them. These include well-known UCI tasks and the more complex KDD cup 99 competition on Intrusion Detection.

As birds migrate in a linear formation, the frequent exchange of leader-follower in the Ibises can extend the formation flight time. In this paper, we utilize the information theory to analyze the underlying mechanism of the coordination and cooperation within the Ibises. To this end, we examine and quantify the dynamic characteristics of the information transfer in a group of Ibises at the individual level. Firstly, the local transfer entropy is employed to quantify the local dynamic information transfer during the migration of the Ibises. Therefore, we provide lucid explanations of how Ibis individually influences the group decision through information transfer to generate consistent collective behavior. The local active information storage is used to represent the memory of the Ibis. The relationship between the consistency of the collective behavior and the memory of the Ibis is obtained by calculating the mutual information between the historical state sequences and the future state of the Ibis. Finally, based on the dynamic information transfer, the relationship between the group memory and the consistent collective behavior, we effectively captured the relationship between the excitation and convergence of the Ibis’ collective behavior, and the variance of entropy is obtained.

The advantages of digital methods of processing, displaying, storing and transmitting information. Currently, various conversion methods and efficient coding methods are used to increase the speed of information transfer, maintain high accuracy and provide the required latent accuracy. In particular, timer (temporary) signal constructions, which, in comparison with other coding methods, for example, positional (bitwise) coding, can reduce costs by more than two times. In the work, the information parameters of the code ensembles synthesized at a constant duration "m" are evaluated. Determined the conditions for the formation of a code ensemble and calculate the number of code dictionary implementations on a segment of a nyquist elements.In order to use the communication channel efficiently, the proposed increase in the entropy of the transmitted ensemble is due to the use of code sets with different number of information segments and at a constant length of the code word. A significant increase in the weight of the synthesized ensemble ensured an increase in the value of the module А0 =19 integer times K є 8:18 .The maximum values of the module are calculated in which the greatest number of code words is synthesized: At: КА0 =13, Np=8; КА0 =14, Np=10 КА0 =15, Np=15; КА0 =17, Np=16 КА0 =18, Np=10; КА0 =19, Np=6 КА0 =20, Np=1 For these КА0 values, the entropy value is H=3.269, which is less than the entropy of the Russian text H=4.35. In accordance with code words that satisfy the conditions of the quality equation. The methods and algorithms of reliable reception of code words under the influence of interference in the channel used were analyzed.Studies and calculations have shown that the use of temporary signal structures synthesized on the basis of a onemodule can significantly reduce the value of entropy for the transmission of Russian text.

In the fourth paper of this Special Issue, we bridge the theoretical debate on the role of memory and criticality discussed in the three earlier manuscripts, with a review of key concepts in biology and focus on cell-to-cell communication in organismal development. While all living organisms are dynamic complex networks of organization and disorder, most studies in biology have used energy and biochemical exchange to explain cell differentiation without considering the importance of information (entropy) transfer. While all complex networks are mixtures of patterns of complexity (non-crucial and crucial events), it is the crucial events that determine the efficiency of information transfer, especially during key transitions, such as in embryogenesis. With increasing multicellularity, emergent relationships from cell-to-cell communication create reaction–diffusion exchanges of different concentrations of biochemicals or morphogenetic gradients resulting in differential gene expression. We suggest that in conjunction with morphogenetic gradients, there exist gradients of information transfer creating cybernetic loops of stability and disorder, setting the stage for adaptive capability. We specifically reference results from the second paper in this Special Issue, which correlated biophotons with lentil seed germination to show that phase transitions accompany changes in complexity patterns during development. Criticality, therefore, appears to be an important factor in the transmission, transfer and coding of information for complex adaptive system development.

The measures of information transfer which correspond to non-additive entropies have intensively been studied in previous decades. The majority of the work includes the ones belonging to the Sharma–Mittal entropy class, such as the Rényi, the Tsallis, the Landsberg–Vedral and the Gaussian entropies. All of the considerations follow the same approach, mimicking some of the various and mutually equivalent definitions of Shannon information measures, and the information transfer is quantified by an appropriately defined measure of mutual information, while the maximal information transfer is considered as a generalized channel capacity. However, all of the previous approaches fail to satisfy at least one of the ineluctable properties which a measure of (maximal) information transfer should satisfy, leading to counterintuitive conclusions and predicting nonphysical behavior even in the case of very simple communication channels. This paper fills the gap by proposing two parameter measures named the α-q-mutual information and the α-q-capacity. In addition to standard Shannon approaches, special cases of these measures include the α-mutual information and the α-capacity, which are well established in the information theory literature as measures of additive Rényi information transfer, while the cases of the Tsallis, the Landsberg–Vedral and the Gaussian entropies can also be accessed by special choices of the parameters α and q. It is shown that, unlike the previous definition, the α-q-mutual information and the α-q-capacity satisfy the set of properties, which are stated as axioms, by which they reduce to zero in the case of totally destructive channels and to the (maximal) input Sharma–Mittal entropy in the case of perfect transmission, which is consistent with the maximum likelihood detection error. In addition, they are non-negative and less than or equal to the input and the output Sharma–Mittal entropies, in general. Thus, unlike the previous approaches, the proposed (maximal) information transfer measures do not manifest nonphysical behaviors such as sub-capacitance or super-capacitance, which could qualify them as appropriate measures of the Sharma–Mittal information transfer.

The purpose of this study is to introduce a method in quantifying cross-frequency information transfer to characterize directional couplers between irregular oscillations in complex systems. Importantly, the method should be able to reflect the intrinsic mechanism of interacting oscillations faithfully. Six types of interacting oscillators, including phase-amplitude, amplitude-amplitude, and component-amplitude cross-frequency transfer entropy as well as their inverse transfer entropies, are within our scope in untangling the brain connectivity. Challenges with nonlinear and nonstationary patterns are designed to validate the robustness of the proposed method. We suggest this approach could be effective in identifying driving and responding elements of interacting oscillators across different time scales. Meanwhile, an atlas of interacting oscillators in sleep is constructed. High-frequency amplitude can inversely drive low-frequency phase stronger than the standard phase-amplitude coupling, and the low-frequency amplitude can be the driving force to the high-frequency amplitude in addition to the low-frequency phase. Unlike the standard phase-amplitude coupling, the proposed cross-frequency transfer entropy is applicable to quantify the interactions across phases, amplitudes, or even the components without methodological adjustments. Meanwhile, the exploration of causal relationship enables the identification of the driving force of information flow.

Artificial neural networks (ANNs) are one of the most promising tools in the quest to develop general artificial intelligence. Their design was inspired by how neurons in natural brains connect and process, the only other substrate to harbor intelligence. Compared to natural brains that are sparsely connected and that form sparsely distributed representations, ANNs instead process information by connecting all nodes of one layer to all nodes of the next. In addition, modern ANNs are trained with backpropagation, while their natural counterparts have been optimized by natural evolution over eons. Here we measure the transfer entropy, that is the information that is transferred from one node to another, to determine how information is propagating through recurrent neural networks optimized either by backpropagation or a genetic algorithm. Surprisingly, we find no difference in how they relay information, suggesting that it is not the optimization method, but instead their topological structure, that causes these ANNs to process information differently compared to the natural brains they seek to emulate.

In this paper, we present a novel approach to identify the generators and states responsible for the small-signal stability of power networks. To this end, the newly developed notion of information transfer between the states of a dynamical system is used. In particular, using the concept of information transfer, which characterizes influence between the various states of a dynamical system, we identify the generators and states which are responsible for causing instability of the power network. While characterizing influence from state to state, information transfer can also describe influence from state to modes thereby generalizing the well known notion of participation factor while at the same time overcoming some of the limitations of the participation factor. The developed framework is applied to reproduce known results for the three bus system, identifying the various causes of instabilities, and is extended to IEEE 39 bus system.

Information transfer, measured by transfer entropy, is a key component of distributed computation. It is therefore important to understand the pattern of information transfer in order to unravel the distributed computational algorithms of a system. Since in many natural systems distributed computation is thought to rely on rhythmic processes a frequency resolved measure of information transfer is highly desirable. Here, we present a novel algorithm, and its efficient implementation, to identify separately frequencies sending and receiving information in a network. Our approach relies on the invertible maximum overlap discrete wavelet transform (MODWT) for the creation of surrogate data in the computation of transfer entropy and entirely avoids filtering of the original signals. The approach thereby avoids well-known problems due to phase shifts or the ineffectiveness of filtering in the information theoretic setting. We also show that measuring frequency-resolved information transfer is a partial information decomposition problem that cannot be fully resolved to date and discuss the implications of this issue. Last, we evaluate the performance of our algorithm on simulated data and apply it to human magnetoencephalography (MEG) recordings and to local field potential recordings in the ferret. In human MEG we demonstrate top-down information flow in temporal cortex from very high frequencies (above 100Hz) to both similarly high frequencies and to frequencies around 20Hz, i.e. a complex spectral configuration of cortical information transmission that has not been described before. In the ferret we show that the prefrontal cortex sends information at low frequencies (4-8 Hz) to early visual cortex (V1), while V1 receives the information at high frequencies (> 125 Hz)., Author summary Systems in nature that perform computations typically consist of a large number of relatively simple but interacting parts. In human brains, for example, billions of neurons work together to enable our cognitive abilities. This well-orchestrated teamwork requires information to be exchanged very frequently. In many cases this exchange happens rhythmically and, therefore, it seems beneficial for our understanding of physical systems if we could link the information exchange to specific rhythms. We here present a method to determine which rhythms send, and which rhythms receive information. Since many rhythms can interact at both sender and receiver side, we show that the above problem is tightly linked to partial information decomposition—an intriguing problem from information theory only solved recently, and only partly. We applied our novel method to information transfer in the human inferior temporal cortex, a brain region relevant for object perception, and unexpectedly found information transfer originating at very high frequencies at 100Hz and then forking to be received at both similarly high but also much lower frequencies around 20Hz. These results overturn the current standard assumption that low frequencies send information to high frequencies.

Any edge in random climate networks appears with some probability, which is determined by the entropy of climate system. Since a large part of climatic variabilities are known to have a random nature, random climate network is becoming a powerful, but only marginally explored at present, tool to quantify uncertainties of climate parameters, analyze strength and range of teleconnections, and examine structure sensitivity of climate system. The climate system has been found to possess key features of small-world networks due to relatively few edges connecting geographically very distant vertices that can stabilize the climate system, and enhance the information transfer significantly within it.

The framework of information dynamics allows the dissection of the information processed in a network of multiple interacting dynamical systems into meaningful elements of computation that quantify the information generated in a target system, stored in it, transferred to it from one or more source systems, and modified in a synergistic or redundant way. The concepts of information transfer and modification have been recently formulated in the context of linear parametric modeling of vector stochastic processes, linking them to the notion of Granger causality and providing efficient tools for their computation based on the state&ndash, space (SS) representation of vector autoregressive (VAR) models. Despite their high computational reliability these tools still suffer from estimation problems which emerge, in the case of low ratio between data points available and the number of time series, when VAR identification is performed via the standard ordinary least squares (OLS). In this work we propose to replace the OLS with penalized regression performed through the Least Absolute Shrinkage and Selection Operator (LASSO), prior to computation of the measures of information transfer and information modification. First, simulating networks of several coupled Gaussian systems with complex interactions, we show that the LASSO regression allows, also in conditions of data paucity, to accurately reconstruct both the underlying network topology and the expected patterns of information transfer. Then we apply the proposed VAR-SS-LASSO approach to a challenging application context, i.e., the study of the physiological network of brain and peripheral interactions probed in humans under different conditions of rest and mental stress. Our results, which document the possibility to extract physiologically plausible patterns of interaction between the cardiovascular, respiratory and brain wave amplitudes, open the way to the use of our new analysis tools to explore the emerging field of Network Physiology in several practical applications.

Information theory, introduced by Shannon in the context of information transfer in communication channels, is used as a foundation for research in many diverse fields. The concept Entropy in terms of information theory is seen as the average amount of information or the rate of information produced when forming a message, element by element. Entropy has found broad application in many research fields and can also be applied in software engineering for quantifying the uncertainty associated with a software code. In this paper, information entropy and its application towards measuring software complexity are explored, along with the formulation of an information entropy based complexity measure that considers logical decision-making, processes, and software statement interaction patterns in control flow graphs mapped from actual software code. To broaden the application of the proposed metric, the execution times of nodes in the control flow graphs are also incorporated. Further, the metric is evaluated against eight different axioms that a software complexity measure should satisfy.

Field monitoring strategy is critical for disaster preparedness and watershed emergency environmental management. However, development of such is also highly challenging. Despite the efforts and progress thus far, no definitive guidelines or solutions are available worldwide for quantitatively designing a monitoring network in response to river chemical spill incidents, except general rules based on administrative divisions or arbitrary interpolation on routine monitoring sections. To address this gap, a novel framework for spatial-temporal network design was proposed in this study. The framework combines contaminant transport modelling with discrete entropy theory and spectral analysis. The water quality model was applied to forecast the spatio-temporal distribution of contaminant after spills and then corresponding information transfer indexes (ITIs) and Fourier approximation periodic functions were estimated as critical measures for setting sampling locations and times. The results indicate that the framework can produce scientific preparedness plans of emergency monitoring based on scenario analysis of spill risks as well as rapid design as soon as the incident happened but not prepared. The framework was applied to a hypothetical spill case based on tracer experiment and a real nitrobenzene spill incident case to demonstrate its suitability and effectiveness. The newly-designed temporal-spatial monitoring network captured major pollution information at relatively low costs. It showed obvious benefits for follow-up early-warning and treatment as well as for aftermath recovery and assessment. The underlying drivers of ITIs as well as the limitations and uncertainty of the approach were analyzed based on the case studies. Comparison with existing monitoring network design approaches, management implications, and generalized applicability were also discussed.

This paper measures the information transmission and relationships between Chinese stock market and the commodity futures market on different time scales novelly by combining the transfer entropy from information theory and the multi-scale analysis technique, the complete empirical mode decomposition based on adaptive noise (CEEMDAN). Pairwise information transfer and average information inflow and outflow of assets and markets are investigated. Information spillover network is built based on the net transfer entropy and the direction of information spillover is identified. The results show that the direction and magnitude of information spillover have changed as types of assets and the time scale have changed. There is heterogeneity in the relationship between stock market and the metals, energy and agricultural product futures markets. Lastly by cluster analysis using minimum spanning tree based on transfer entropy, it shows that the methanol futures and information industry form the core of the network. The study finds that with the increase of time scale, the relative information transmission between the commodity futures market and the stock market gradually decreases, and the stock market is in a more influential position.

A fast and approximate method of generating allosteric communication landscapes in proteins is presented by using Schreiber's entropy transfer concept in combination with the Gaussian Network Model of proteins. Predictions of the model and the allosteric communication landscapes generated show that information transfer in proteins does not necessarily take place along a single path, but an ensemble of pathways is possible. The model emphasizes that knowledge of entropy only is not sufficient for determining allosteric communication and additional information based on time delayed correlations should be introduced, which leads to the presence of causality in proteins. The model provides a simple tool for mapping entropy sink-source relations into pairs of residues. By this approach, residues that should be manipulated to control protein activity may be determined. This should be of great importance for allosteric drug design and for understanding the effects of mutations on function. The model is applied to determine allosteric communication in three proteins, Ubiquitin, Pyruvate Kinase, and the PDZ domain. Predictions are in agreement with molecular dynamics simulations and experimental evidence. Proteins 2017; 85:1056-1064. © 2017 Wiley Periodicals, Inc.

In social network big data scheduling, it is easy for target data to conflict in the same data node. Of the different kinds of entropy measures, this paper focuses on the optimization of target entropy. Therefore, this paper presents an optimized method for the scheduling of big data in social networks and also takes into account each task&rsquo, s amount of data communication during target data transmission to construct a big data scheduling model. Firstly, the task scheduling model is constructed to solve the problem of conflicting target data in the same data node. Next, the necessary conditions for the scheduling of tasks are analyzed. Then, the a periodic task distribution function is calculated. Finally, tasks are scheduled based on the minimum product of the corresponding resource level and the minimum execution time of each task is calculated. Experimental results show that our optimized scheduling model quickly optimizes the scheduling of social network data and solves the problem of strong data collision.

This paper is concerned with decoding waves for spatially-coupled low-density parity-check (SC-LDPC) ensembles on binary memoryless symmetric (BMS) channels. We propose an approach termed as interpolated density evolution (IDE) to predict the velocities of decoding waves. In this approach, we approximate the densities of decoding waves via interpolation and establish two transfer functions for the update of interpolation functions. Comparison results indicate that the proposed IDE approach achieves a good trade-off between computational cost and accuracy as compared with the standard coupled density evolution (CDE) approach and the extrinsic information transfer (EXIT) approach based on Gaussian approximation (GA). This highlights the attractiveness of the proposed approach in the analysis of SC-LDPC ensembles on general BMS channels.

Because of the virtuality of the e-commerce transaction, its transaction behavior faces the serious security threats. However, the existing recognition algorithms of malicious transaction behaviors only rely on traditional malicious features, resulting in low recognition efficiency. This paper proposes a novel technique for identifying collaborative malicious network behavior in e-commerce: the quantum behavior-based particle swarm optimization method with global search capability, which is combined with the rapidly converging k-means method to improve efficiency and effectiveness of the algorithm. The paper first illustrates that the current malicious network behavior is diversified and collaborative, it is necessary to identify malicious network behavior from a higher level-cooperation level, and then proposes original cooperative malicious behavior features of e-commerce. Second, the use of information transfer entropy (ITE) based on quantum potential energy is introduced to designate the node with the largest ITE in the cluster as the cluster center, avoiding randomly initializing cluster center nodes and calculating the distance repeatedly. Third, the existence of collaborative malicious network behavior and the feasibility of the algorithm are proved using the fixed point theorem in the probability metric space. Meanwhile, quantum behavior-based particle swarm optimization is used to extract cluster centers as a means of improving the spatial distribution and collaborative state of particle nodes, and enables the clustering algorithm to have global search ability. The simulation results demonstrate that the our schemes improve the efficiency and accuracy of identifying cooperative malicious network behavior in e-commerce recognition.

Rainfall is one of the most fundamental components of the water cycle and is one of the fundamental inputs of hydrological models. A well-designed network can not only depict the regional precipitation characteristics, but also economically yield maximum needed rainfall information. In regions where either there is limited data or data is not available, it is a common challenge to add stations. The entropy theory-based information transfer model and geostatistical interpolation techniques are two solutions to meet the challenge. In this study, we used a representative rain gauge network to do the network design. Two models, based on information transfer and data transfer, were compared for network design. Other rain gauges in the study area were used as reference (“true values”) for assessing the model. Results showed that the information transfer model estimated transinformation between station pairs better than did the data transfer model. Different representative gauges were evaluated separately by the directional information transfer index (DIT). The candidate gauges selected with least information redundancy were similar for both information transfer and data transfer models. Though both models captured some least information-redundant areas, other areas may be bypassed because of model errors or estimation errors.

Data transformation, e.g., feature transformation and selection, is an integral part of any machine learning procedure. In this paper, we introduce an information-theoretic model and tools to assess the quality of data transformations in machine learning tasks. In an unsupervised fashion, we analyze the transformation of a discrete, multivariate source of information X&macr, into a discrete, multivariate sink of information Y&macr, related by a distribution PX&macr, Y&macr, The first contribution is a decomposition of the maximal potential entropy of (X&macr, ), which we call a balance equation, into its (a) non-transferable, (b) transferable, but not transferred, and (c) transferred parts. Such balance equations can be represented in (de Finetti) entropy diagrams, our second set of contributions. The most important of these, the aggregate channel multivariate entropy triangle, is a visual exploratory tool to assess the effectiveness of multivariate data transformations in transferring information from input to output variables. We also show how these decomposition and balance equations also apply to the entropies of X&macr, and Y&macr, respectively, and generate entropy triangles for them. As an example, we present the application of these tools to the assessment of information transfer efficiency for Principal Component Analysis and Independent Component Analysis as unsupervised feature transformation and selection procedures in supervised classification tasks.

Sensitivity analysis plays an important role in analyzing most influential factors of models. There are few studies on sensitivity analysis (SA) of continuous time models. By using the concept of information flow, this paper presents a new sensitivity analysis method for dynamical models. By combining the direction and magnitude of information transfer, the method builds a novel framework for SA, with the rigorous mathematical and statistic theories on information flow. This method is effective and feasible for both static systems and dynamical systems including continuous time models and discrete time series. The new method is applied to an example of linear dynamical models and the Lorenz system, and the results indicate that the method could find out the most influential variables, which conforms with entropy method, and it also give more detailed information for system analysis.

In this paper, we provide a novel approach to capture causal interaction in a linear dynamical system from time-series data. In [1], we have shown that the existing measures of information transfer, namely directed information, Granger causality and transfer entropy fail to capture true causal interaction in a dynamical system and proposed a new definition of information transfer that captures true causal interaction. The main contribution of this paper is to show that the proposed definition of information transfer in [1] [2] can be computed from time-series data and the computed information measure allows the identification of causal interaction and network topology in a dynamical system. The data-driven algorithm for computation of information transfer for linear systems relies on a robust optimization formulation of transfer operator theoretic framework. The proposed technique is applied to a number of different examples to establish its efficiency.

Entropy measures have been a major interest of researchers to measure the information content of a dynamical system. One of the well-known methodologies is sample entropy, which is a model-free approach and can be deployed to measure the information transfer in time series. Sample entropy is based on the conditional entropy where a major concern is the number of past delays in the conditional term. In this study, we deploy a lag-specific conditional entropy to identify the informative past values. Moreover, considering the seasonality structure of data, we propose a clustering-based sample entropy to exploit the temporal information. Clustering-based sample entropy is based on the sample entropy definition while considering the clustering information of the training data and the membership of the test point to the clusters. In this study, we utilize the proposed method for transductive feature selection in black-box weather forecasting and conduct the experiments on minimum and maximum temperature prediction in Brussels for 1-6 days ahead. The results reveal that considering the local structure of the data can improve the feature selection performance. In addition, despite the large reduction in the number of features, the performance is competitive with the case of using all features. ispartof: ENTROPY vol:20 issue:4 ispartof: location:Switzerland status: published

Objective: Measures of transfer entropy (TE) quantify the direction and strength of coupling between two complex systems. Standard approaches assume stationarity of the observations, and therefore are unable to track time-varying changes in nonlinear information transfer with high temporal resolution. In this study, we aim to define and validate novel instantaneous measures of TE to provide an improved assessment of complex nonstationary cardiorespiratory interactions. Methods: We here propose a novel instantaneous point-process TE (ipTE) and validate its assessment as applied to cardiovascular and cardiorespiratory dynamics. In particular, heartbeat and respiratory dynamics are characterized through discrete time series, and modeled with probability density functions predicting the time of the next physiological event as a function of the past history. Likewise, nonstationary interactions between heartbeat and blood pressure dynamics are characterized as well. Furthermore, we propose a new measure of information transfer, the instantaneous point-process information transfer (ipInfTr), which is directly derived from point-process-based definitions of the Kolmogorov–Smirnov distance. Results and Conclusion: Analysis on synthetic data, as well as on experimental data gathered from healthy subjects undergoing postural changes confirms that ipTE, as well as ipInfTr measures are able to dynamically track changes in physiological systems coupling. Significance: This novel approach opens new avenues in the study of hidden, transient, nonstationary physiological states involving multivariate autonomic dynamics in cardiovascular health and disease. The proposed method can also be tailored for the study of complex multisystem physiology (e.g., brain–heart or, more in general, brain–body interactions).

In this work, we leverage the information-theoretic notion of transfer entropy theory to study causal information flow in epidemic spreading over temporal networks. An improved understanding of causal information flow may lead to the early detection of population segments that should be monitored or immunized to enhance epidemic containment. We focus on activity driven networks, which constitute a powerful and elegant paradigm to capture the inherent time-varying nature of contacts and population heterogeneity. Our preliminary results confirm the intuition that individuals who have a higher propensity in contacting others are responsible for the largest information transfer. Moreover, we find that epidemic parameters such as the probability of infection and recovery may dominate the spreading phenomenon over heterogeneities in the contact formation.

Information theory is increasingly being employed in the study of complex systems, particularly in the fields of neuroscience and artificial life. While domain-specific tools for information analysis are certainly valuable, high-performance and general-purpose toolkits can ensure better reproducibility and faster research turnover. We introduce Inform, an open-source and cross-platform C library for information-theoretic analysis of complex systems. Inform provides a host of functions to estimate information-theoretic measures from time series data. This includes classical information-theoretic measures (e.g. entropy, mutual information) and measures of information dynamics (e.g. active information storage, transfer entropy), but also several less common, yet powerful information-based concepts such as effective information, information flow and integration measures. However, what makes Inform unique is that it exposes a lower-level API allowing users to construct measures of their own, and includes a suite of utility functions that can be used to augment and extend the built-in functionality. Significant effort went into designing Inform's API to make its use from other languages as simple as possible. We describe Inform's overall design and implementation including details of validation techniques and plans for future development. We present evidence that suggests that Inform's computational performance is at least comparable to the Java Information Dynamics Toolkit (JIDT), which is taken to be the gold-standard for the field. We provide several examples to guide users and provide information about higher-level language wrappers for Python, R, Julia and Mathematica.

In this study, a method combining radar and entropy was proposed to design a rainfall network. Owing to the shortage of rain gauges in mountain areas, weather radars are used to measure rainfall over catchments. The major advantage of radar is that it is possible to observe rainfall widely in a short time. However, the rainfall data obtained by radar do not necessarily correspond to that observed by ground-based rain gauges. The in-situ rainfall data from telemetering rain gauges were used to calibrate a radar system. Therefore, the rainfall intensity; as well as its distribution over the catchment can be obtained using radar. Once the rainfall data of past years at the desired locations over the catchment were generated, the entropy based on probability was applied to optimize the rainfall network. This method is applicable in remote and mountain areas. Its most important utility is to construct an optimal rainfall network in an ungauged catchment. The design of a rainfall network in the catchment of the Feitsui Reservoir was used to illustrate the various steps as well as the reliability of the method.

Joint source–channel coding (JSCC) based on double protograph low-density parity-check (DP-LDPC) codes has been shown to be a good solution for wireless communications. However, the premature error floor region of the DP-LDPC codes usually does not meet the bit-error-rate requirements. In this letter, we provide a matching criterion between the source statistics and the source coding rate using a novel protograph extrinsic information transfer (PEXIT) algorithm with a new decoding threshold. The resulting matching criterion helps predict the error floor region performance and give directions for source coding rate designs.

Summary This study proposed a spatial information estimation model for the analysis of precipitation gauge networks, to improve on previous methods based on information theory. The proposed model employs a two-dimensional transinformation–distance (T–D) relationship in conjunction with multivariate information approximation to estimate transinformation to ungauged locations from existing stations, while taking into consideration the influence of multiple stations and anisotropy. The proposed model is used to evaluate the spatial distribution of precipitation data and the characteristics of information transfer, which are then applied in a spatial optimization algorithm for the selection of additional station locations. This framework was implemented to investigate temporal and spatial patterns in information content in the Shihmen Reservoir watershed. The results demonstrate obvious anisotropy associated with the delivery of information. By comparing different cases, it was determined that the efficiency of information delivery dominates the spatial distribution of the information content, such that eccentricity is merely supplemental. Efficiency in information delivery is also heavily influenced by temporal scale. For data covering long intervals (monthly and annual), efficiency in the delivery of information is relatively high, while the uncertainty or heterogeneity of hourly or daily time series produces low spatial correlations due to the inefficient delivery of information. The proposed spatial optimization algorithm confirmed that the optimal location for new stations lies close to the center of low information zones. Additional stations could improve information content considerably; however, the margin of improvement decreases with the number of stations.

In the study of complex physical and physiological systems represented by multivariate time series, an issue of great interest is the description of the system dynamics over a range of different temporal scales. While information-theoretic approaches to the multiscale analysis of complex dynamics are being increasingly used, the theoretical properties of the applied measures are poorly understood. This study introduces for the first time a framework for the analytical computation of information dynamics for linear multivariate stochastic processes explored at different time scales. After showing that the multiscale processing of a vector autoregressive (VAR) process introduces a moving average (MA) component, we describe how to represent the resulting VARMA process using state-space (SS) models and how to exploit the SS model parameters to compute analytical measures of information storage and information transfer for the original and rescaled processes. The framework is then used to quantify multiscale information dynamics for simulated unidirectionally and bidirectionally coupled VAR processes, showing that rescaling may lead to insightful patterns of information storage and transfer but also to potentially misleading behaviors.

In this paper, we propose a new definition of information transfer in a discrete dynamical system. The information transfer is based on how much entropy (uncertainty) is transferred from state x to state y, as the dynamical system evolves in time. In our previous work, we had provided an axiomatic definition of information transfer where we had considered absolute entropy to define the transfer. Our new definition can be viewed as a natural generalization of directed information from information theory to dynamical systems and is based on transfer of conditional entropy. This new definition also satisfies the properties of a) zero transfer, b) transfer asymmetry and c) information conservation. We start with a definition of one-step information transfer and then generalize this definition to n-step information transfer and average information transfer over infinite time step. We also provide analytical expressions for information transfer for linear systems. Some basic examples are provided to understand the physical meaning of information transfer and how this can be used to infer causality and influence in dynamical system setting.

The continuously growing framework of information dynamics encompasses a set of tools, rooted in information theory and statistical physics, which allow to quantify different aspects of the statistical structure of multivariate processes reflecting the temporal dynamics of complex networks. Building on the most recent developments in this field, this work designs a complete approach to dissect the information carried by the target of a network of multiple interacting systems into the new information produced by the system, the information stored in the system, and the information transferred to it from the other systems; information storage and transfer are then further decomposed into amounts eliciting the specific contribution of assigned source systems to the target dynamics, and amounts reflecting information modification through the balance between redundant and synergetic interaction between systems. These decompositions are formulated quantifying information either as the variance or as the entropy of the investigated processes, and their exact computation for the case of linear Gaussian processes is presented. The theoretical properties of the resulting measures are first investigated in simulations of vector autoregressive processes. Then, the measures are applied to assess information dynamics in cardiovascular networks from the variability series of heart period, systolic arterial pressure and respiratory activity measured in healthy subjects during supine rest, orthostatic stress, and mental stress. Our results document the importance of combining the assessment of information storage, transfer and modification to investigate common and complementary aspects of network dynamics; suggest the higher specificity to alterations in the network properties of the measures derived from the decompositions; and indicate that measures of information transfer and information modification are better assessed, respectively, through entropy-based and variance-based implementations of the framework.

In this paper, we show through examples, how the existing definitions of information transfer, namely directed information and transfer entropy fail to capture true causal interaction between states in control dynamical system. Furthermore, existing definitions are shown to be too weak to have any implication on two of the most fundamental concepts in system theory, namely controllability and observability. We propose a new definition of information transfer, based on the ideas from dynamical system theory, and show that this new definition can not only capture true causal interaction between states, but also have implication on system controllability and observability properties. In particular, we show that non-zero transfer of information from input-to-state and state-to-output implies structural controllability and observability properties of the control dynamical system respectively. Analytical expression for information transfer between state-to-state, input-to-state, state-to-output, and input-to-output are provided for linear system. There is a natural extension of our proposed definition to define information transfer over n time steps and average information transfer over infinite time step. We show that the average information transfer in feedback control system between plant output and input is equal to the entropy of the open loop dynamics thereby re-deriving the Bode fundamental limitation results using the proposed definition of transfer.

With the technical progress in the fields of imaging and printing device, multilevel imaging and screening technologies are applied increasingly. The application of these technologies in printing field, which is as an information channel, can expand capability of information transfer. In order to obtain the characteristics of information transfer in image reproduction, in which multilevel screening technology is applied, the relationship among optical density, dot area and dot density is constructed. According to the basic definitions of information content and of information entropy, the relationship between information content/entropy and gray levels of screening dot is analyzed. Furthermore, multilevel FM screening images, which are with 4 levels, 8 levels and 16 levels and generated by a program, are analyzed and compared with bi-level screened images. The results show, that under lower imaging resolution, multilevel screening can achieve higher count of tone levels and information entropy. It is significant for the improvement of the capability of information transfer and reproduction quality of image.

As the big data science develops, efficient methods are demanded for various data analysis. Granger causality provides the prime model for quantifying causal interactions. However, this theoretic model does not meet the requirement for real-world data analysis, because real-world time series are diverse whose models are usually unknown. Therefore, model-free measures such as information transfer measures are strongly desired. Here, we propose the multi-scale extension of conditional mutual information measures using MORLET wavelet, which are named the WM and WPM. The proposed measures are computational efficient and interpret information transfer by multi-scales. We use both synthetic data and real-world examples to demonstrate the efficiency of the new methods. The results of the new methods are robust and reliable. Via the simulation studies, we found the new methods outperform the wavelet extension of transfer entropy (WTE) in both computational efficiency and accuracy. The features and properties of the proposed measures are also discussed.

Predictability is by observation a local notion in complex dynamical systems. Its spatiotemporal structure implies a flow, or transfer in discrete cases, of information that redistributes the local predictability within the state space of concern. Information flow is a fundamental concept in general physics which has applications in a wide variety of disciplines such as neuroscience, material science, atmosphere–ocean science, and turbulence research, to name but a few. In this study, it is rigorously formulated with respect to relative entropy within the framework of a system with many components, each signifying a location or a structure. Given a component, the mechanism governing the evolution of its predictability can be classified into two groups, one due to the component itself, another due to a transfer of information from its peers. A measure of the transfer is rigorously derived, and an explicit expression obtained. This measure possesses a form reminiscent of that we have obtained before with respect to absolute entropy in [X.S. Liang and R. Kleeman, A rigorous formalism of information transfer between dynamical system components, Physica D 227 (2007) 173–182]; in particular, when the system is of dimensionality 2, there is no difference between the formalisms with respect to absolute entropy and relative entropy, except for a minus sign. Properties have been explored and discussed; particularly discussed is the property of asymmetry or causality, which states that information transfer from one component to another carries no hint about the transfer in the other direction, in contrast to the transfer of other quantities such as energy. This formalism has been applied to the study of the scale–scale interaction and information transfer between the first two modes of the truncated Burgers equation. It is found that all 12 transfers are essentially zero or negligible, save for a strong transfer between the sine components from the low-frequency mode to the high-frequency mode. That is to say, the predictability of the high-frequency mode is controlled by the knowledge of the low-frequency mode. This result, though from a highly idealized system, has interesting implications about the dynamical closure problem in turbulence research and atmosphere–ocean science, i.e., the subgrid processes may to some extent be parameterized by the large-scale dynamics. This study can be adopted to investigate the propagation of uncertainties in fluid flows, which has important applications in problems such as atmospheric observing platform design, and may be utilized to identify the route of information flowing within a complex network.

Information storage is a key component of intrinsic distributed computation. Despite the existence of appropriate measures for it (e.g. excess entropy), its role in interacting with information transfer and modification to give rise to distributed computation is not yet well-established. We explore how to quantify information storage on a local scale in space and time, so as to understand its role in the dynamics of distributed computation. To assist these explorations, we introduce the active information storage, which quantifies the information storage component that is directly in use in the computation of the next state of a process. We present the first profiles of local excess entropy and local active information storage in cellular automata, providing evidence that blinkers and background domains are dominant information storage processes in these systems. This application also demonstrates the manner in which these two measures of information storage are distinct but complementary. It also reveals other information storage phenomena, including the misinformative nature of local storage when information transfer dominates the computation, and demonstrates that the local entropy rate is a useful spatiotemporal filter for information transfer structure.