Search

We study a generic ensemble of deep belief networks which is parametrized by the distribution of energy levels of the hidden states of each layer. We show that, within a random energy approach, statistical dependence can propagate from the visible to deep layers only if each layer is tuned close to the critical point during learning. As a consequence, efficiently trained learning machines are characterised by a broad distribution of energy levels. The analysis of Deep Belief Networks and Restricted Boltzmann Machines on different datasets confirms these conclusions., 16 pages, 4 figures

The Glosten-Milgrom model describes a single asset market, where informed traders interact with a market maker, in the presence of noise traders. We derive an analogy between this financial model and a Szil\'ard information engine by {\em i)} showing that the optimal work extraction protocol in the latter coincides with the pricing strategy of the market maker in the former and {\em ii)} defining a market analogue of the physical temperature from the analysis of the distribution of market orders. Then we show that the expected gain of informed traders is bounded above by the product of this market temperature with the amount of information that informed traders have, in exact analogy with the corresponding formula for the maximal expected amount of work that can be extracted from a cycle of the information engine. This suggests that recent ideas from information thermodynamics may shed light on financial markets, and lead to generalised inequalities, in the spirit of the extended second law of thermodynamics., Comment: 20 pages, 3 figures

Optimal Learning Machines (OLM) are systems that extract maximally informative representation of the environment they are in contact with, or of the data they are presented. It has recently been suggested that these systems are characterised by an exponential distribution of energy levels. In order to understand the peculiar properties of OLM within a broader framework, I consider an ensemble of optimisation problems over functions of many variables, part of which describe a sub-system and the rest account for its interaction with a random environment. The number of states of the sub-system with a given value of the objective function obeys a stretched exponential distribution, with exponent $\gamma$, and the interaction part is drawn at random from the same distribution, independently for each configuration of the whole system. Systems with $\gamma=1$ then correspond to OLM, and we find that they sit at the boundary between two regions with markedly different properties. For all $\gamma>0$ the system exhibits a freezing phase transition. The transition is discontinuous for $\gamma1$. The region $\gamma>1$ corresponds to learnable energy landscapes and the behaviour of the sub-system becomes predictable as the size of the environment exceeds a critical threshold. For $\gamma, Comment: 17 pages, 4 figures

We propose a method for recovering the structure of a sparse undirected graphical model when very few samples are available. The method decides about the presence or absence of bonds between pairs of variable by considering one pair at a time and using a closed form formula, analytically derived by calculating the posterior probability for every possible model explaining a two body system using Jeffreys prior. The approach does not rely on the optimisation of any cost functions and consequently is much faster than existing algorithms. Despite this time and computational advantage, numerical results show that for several sparse topologies the algorithm is comparable to the best existing algorithms, and is more accurate in the presence of hidden variables. We apply this approach to the analysis of US stock market data and to neural data, in order to show its efficiency in recovering robust statistical dependencies in real data with non stationary correlations in time and space., 54 pages, 26 figures

Deep learning has been successfully applied to various tasks, but its underlying mechanism remains unclear. Neural networks associate similar inputs in the visible layer to the same state of hidden variables in deep layers. The fraction of inputs that are associated to the same state is a natural measure of similarity and is simply related to the cost in bits required to represent these inputs. The degeneracy of states with the same information cost provides instead a natural measure of noise and is simply related the entropy of the frequency of states, that we call relevance. Representations with minimal noise, at a given level of resolution, are those that maximise the relevance. A signature of such efficient representations is that frequency distributions follow power laws. We show, in extensive numerical experiments, that deep neural networks extract a hierarchy of efficient representations from data, because they (i) achieve low levels of noise (i.e. high relevance) and (ii) exhibit power law distributions. We also find that the layer that is most efficient to reliably generate patterns of training data is the one for which relevance and resolution are traded at the same price, which implies that frequency distribution follows Zipf's law.

In the pursuit of ever increasing efficiency and growth, our economies have evolved to remarkable degrees of complexity, with nested production processes feeding each other in order to create products of greater sophistication from less sophisticated ones, down to raw materials. The engine of such an expansion have been competitive markets that, according to General Equilibrium Theory (GET), achieve efficient allocations under specific conditions. We study large random economies within the GET framework, as templates of complex economies, and we find that a non-trivial phase transition occurs: the economy freezes in a state where all production processes collapse when either the number of primary goods or the number of available technologies fall below a critical threshold. As in other examples of phase transitions in large random systems, this is an unintended consequence of the growth in complexity. Our findings suggest that the Industrial Revolution can be regarded as a sharp transition between different phases, but also imply that well developed economies can collapse if too many intermediate goods are introduced., 30 pages, 10 figures

A new model for cluster growth is introduced. The model is built with a long time memory which is similar to the one found in cluster growth in disordered media. The model, as a function of its parameter, interpolates between Eden growth and invasion percolation-like growth and has a behavior similar to self-avoiding walks in another limit.

The problem of directed polymers in a random medium is generalized to include a killing probability. The model is then studied using results of the theory of extreme statistics. These reveal that, in a region of parameters, the thermodynamics of polymers with killing in finite dimensions is similar to that of the mean field solution of directed polymers on a Cayley tree. The implication of these results for the behaviour of directed polymers in high dimensions is also discussed.

We introduce a new type of renormalization group for the critical dynamics of a recently proposed model for biological evolution. The method is suited to deal with the combined effects of quenched disorder and irreversible dynamics. This allows to understand the self-organized nature of the process, analyze its universality and compute the critical exponents which are in good agreement with the values of the simulations.

The properties of the shape of patterns generated by random growth in a Laplacian field are investigated using a simple model, the smoothed Laplacian model. The relation between this model and more complex systems such as Saffman-Taylor fingers or diffusion-limited aggregates, stands in the recently recognized universality of patterns whose evolution is controlled by the Laplace equation, and in an analytical approximation for the Laplacian field itself. The smoothed Laplacian model allows one to treat analytically the equilibrium of the structure in the 'smoothed' Laplacian field in a mean-field approximation. The results on this model are extended to radial geometry thus providing theoretical insight in the dependence of the shape on the geometry of boundary conditions in which the process takes place and on the parameter that tunes the strength of the Laplacian field in the same way as the parameter eta does for the dielectric-breakdown model. This new picture is compared to numerical simulation results of DLA and DBM and proves to be consistent with them.

The genetic code is redundant, as there are about three times more codons than amino acids. Because of this redundancy, a given amino acid can be specified by different codons, which are therefore considered synonymous. Despite being synonymous, however, such codons are used with different frequencies, a phenomenon known as codon bias. The origin and roles of the codon bias have not yet been fully clarified, although it is clear that it can affect the efficiency, accuracy and regulation of the translation process. In order to provide a tool to address these issues, we introduce here the codon information index (CII), which represents a measure of the amount of information stored in mRNA sequences through the codon bias. The calculation of the CII requires solely the knowledge of the mRNA sequences, without any other additional information. We found that the CII is highly correlated with the tRNA adaptation index (tAI), even if the latter requires the knowledge of the tRNA pool of an organism. We anticipate that the CII will represent a useful tool to study quantitatively the relationship between the information provided by the codon bias and various aspects of the translation process, thus identifying those aspects that are most influenced by it.

Neurons subject to a common non-stationary input may exhibit a correlated firing behavior. Correlations in the statistics of neural spike trains also arise as the effect of interaction between neurons. Here we show that these two situations can be distinguished, with machine learning techniques, provided the data are rich enough. In order to do this, we study the problem of inferring a kinetic Ising model, stationary or nonstationary, from the available data. We apply the inference procedure to two data sets: one from salamander retinal ganglion cells and the other from a realistic computational cortical network model. We show that many aspects of the concerted activity of the salamander retinal neurons can be traced simply to the external input. A model of non-interacting neurons subject to a non-stationary external field outperforms a model with stationary input with couplings between neurons, even accounting for the differences in the number of model parameters. When couplings are added to the non-stationary model, for the retinal data, little is gained: the inferred couplings are generally not significant. Likewise, the distribution of the sizes of sets of neurons that spike simultaneously and the frequency of spike patterns as function of their rank (Zipf plots) are well-explained by an independent-neuron model with time-dependent external input, and adding connections to such a model does not offer significant improvement. For the cortical model data, robust couplings, well correlated with the real connections, can be inferred using the non-stationary model. Adding connections to this model slightly improves the agreement with the data for the probability of synchronous spikes but hardly affects the Zipf plot., Comment: version in press in J Stat Mech

Using the cross-correlation of the wavelet transformation, we propose a general method of studying the scale dependence of the direction of coupling for coupled time series. The method is first demonstrated by applying it to coupled van der Pol forced oscillators and coupled nonlinear stochastic equations. We then apply the method to the analysis of the log-return time series of the stock values of the IBM and General Electric (GE) companies. Our analysis indicates that, on average, IBM stocks react earlier to possible common sector price movements than those of GE.

Inverse phase transitions are striking phenomena in which an apparently more ordered state disorders under cooling. This behavior can naturally emerge in tricritical systems on heterogeneous networks and it is strongly enhanced by the presence of disassortative degree correlations. We show it both analytically and numerically, providing also a microscopic interpretation of inverse transitions in terms of freezing of sparse subgraphs and coupling renormalization., 4 pages, 4 figures

We consider the coarsening properties of a kinetic Ising model with a memory field. The probability of a spin-flip depends on the persistence time of the spin in a state. The more a spin has been in a given state, the less the spin-flip probability is. We numerically studied the growth and persistence properties of such a system on a two dimensional square lattice. The memory introduces energy barriers which freeze the system at zero temperature. At finite temperature we can observe an apparent arrest of coarsening for low temperature and long memory length. However, since the energy barriers introduced by memory are due to local effects, there exists a timescale on which coarsening takes place as for the Ising model. Moreover the two point correlation functions of the Ising model with and without memory are the same, indicating that they belong to the same universality class., Comment: 10 pages, 7 figures; some figures and some comments added