Search

To make efficient use of limited physical resources, the brain must match its coding and computational strategies to the statistical structure of input signals. An attractive testing ground for these principles is the problem of motion estimation in the fly visual system: we understand the optics of the compound eye, have a quantitative description of input signals and noise from the retina, and can record from output neurons that encode estimates of different velocity components. Furthermore, recent work provides a nearly complete wiring diagram of the intervening circuitry. What is missing is a characterization of the visual signals and motions that flies encounter in a natural context. We attack this directly with the development of a specialized camera that matches the high temporal resolution, optical properties, and spectral sensitivity of the fly's eye; inertial motion sensors provide ground truth about rotations and translations through the world. We describe the design, construction, and performance characteristics of this FlEye camera. To illustrate the opportunities created by this instrument we use data on movies and motion to construct optimal local motion estimators that can be compared with the responses of the fly's motion sensitive neurons.

The 2024 Nobel Prize in Physics was awarded to John Hopfield and Geoffrey Hinton, "for foundational discoveries and inventions that enable machine learning with artificial neural networks." As noted by the Nobel committee, their work moved the boundaries of physics. This is a brief reflection on Hopfield's work, its implications for the emergence of biological physics as a part of physics, the path from his early papers to the modern revolution in artificial intelligence, and prospects for the future.

Perceptions and actions, thoughts and memories result from coordinated activity in hundreds or even thousands of neurons in the brain. It is an old dream of the physics community to provide a statistical mechanics description for these and other emergent phenomena of life. These aspirations appear in a new light because of developments in our ability to measure the electrical activity of the brain, sampling thousands of individual neurons simultaneously over hours or days. We review the progress that has been made in bringing theory and experiment together, focusing on maximum entropy methods and a phenomenological renormalization group. These approaches have uncovered new, quantitatively reproducible collective behaviors in networks of real neurons, and provide examples of rich parameter--free predictions that agree in detail with experiment.

In the brain, fine-scale correlations combine to produce macroscopic patterns of activity. However, as experiments record from larger and larger populations, we approach a fundamental bottleneck: the number of correlations one would like to include in a model grows larger than the available data. In this undersampled regime, one must focus on a sparse subset of correlations; the optimal choice contains the maximum information about patterns of activity or, equivalently, minimizes the entropy of the inferred maximum entropy model. Applying this ``minimax entropy" principle is generally intractable, but here we present an exact and scalable solution for pairwise correlations that combine to form a tree (a network without loops). Applying our method to over one thousand neurons in the mouse hippocampus, we find that the optimal tree of correlations reduces our uncertainty about the population activity by 14% (over 50 times more than a random tree). Despite containing only 0.1% of all pairwise correlations, this minimax entropy model accurately predicts the observed large-scale synchrony in neural activity and becomes even more accurate as the population grows. The inferred Ising model is almost entirely ferromagnetic (with positive interactions) and exhibits signatures of thermodynamic criticality. These results suggest that a sparse backbone of excitatory interactions may play an important role in driving collective neuronal activity., Comment: 15 pages, 11 figures

Maximum entropy methods provide a principled path connecting measurements of neural activity directly to statistical physics models, and this approach has been successful for populations of $N\sim 100$ neurons. As $N$ increases in new experiments, we enter an undersampled regime where we have to choose which observables should be constrained in the maximum entropy construction. The best choice is the one that provides the greatest reduction in entropy, defining a "minimax entropy" principle. This principle becomes tractable if we restrict attention to correlations among pairs of neurons that link together into a tree; we can find the best tree efficiently, and the underlying statistical physics models are exactly solved. We use this approach to analyze experiments on $N\sim 1500$ neurons in the mouse hippocampus, and show that the resulting model captures the distribution of synchronous activity in the network., Comment: 6 pages, 5 figures

The explosion of data on animal behavior in more natural contexts highlights the fact that these behaviors exhibit correlations across many time scales. But there are major challenges in analyzing these data: records of behavior in single animals have fewer independent samples than one might expect; in pooling data from multiple animals, individual differences can mimic long-ranged temporal correlations; conversely long-ranged correlations can lead to an over-estimate of individual differences. We suggest an analysis scheme that addresses these problems directly, apply this approach to data on the spontaneous behavior of walking flies, and find evidence for scale invariant correlations over nearly three decades in time, from seconds to one hour. Three different measures of correlation are consistent with a single underlying scaling field of dimension $\Delta = 0.180\pm 0.005$.

Organisms and algorithms learn probability distributions from previous observations, either over evolutionary time or on the fly. In the absence of regularities, estimating the underlying distribution from data would require observing each possible outcome many times. Here we show that two conditions allow us to escape this infeasible requirement. First, the mutual information between two halves of the system should be consistently sub-extensive. Second, this shared information should be compressible, so that it can be represented by a number of bits proportional to the information rather than to the entropy. Under these conditions, a distribution can be described with a number of parameters that grows linearly with system size. These conditions are borne out in natural images and in models from statistical physics, respectively., Comment: This paper has been combined with work discussed in arXiv:2112.14334v1, and extended. The longer paper is now available in arXiv:2112.14334v2

Many organisms, from flies to humans, use visual signals to estimate their motion through the world. To explore the motion estimation problem, we have constructed a camera/gyroscope system that allows us to sample, at high temporal resolution, the joint distribution of input images and rotational motions during a long walk in the woods. From these data we construct the optimal estimator of velocity based on spatial and temporal derivatives of image intensity in small patches of the visual world. Over the bulk of the naturally occurring dynamic range, the optimal estimator exhibits the same systematic errors seen in neural and behavioral responses, including the confounding of velocity and contrast. These results suggest that apparent errors of sensory processing may reflect an optimal response to the physical signals in the environment.

In large neuronal networks, it is believed that functions emerge through the collective behavior of many interconnected neurons. Recently, the development of experimental techniques that allow simultaneous recording of calcium concentration from a large fraction of all neurons in Caenorhabditis elegans - a nematode with 302 neurons - creates the opportunity to ask if such emergence is universal, reaching down to even the smallest brains. Here, we measure the activity of 50+ neurons in C. elegans, and analyze the data by building the maximum entropy model that matches the mean activity and pairwise correlations among these neurons. To capture the graded nature of the cells' responses, we assign each cell multiple states. These models, which are equivalent to a family of Potts glasses, successfully predict higher statistical structure in the network. In addition, these models exhibit signatures of collective behavior: the state of single cells can be predicted from the state of the rest of the network; the network, despite being sparse in a way similar to the structural connectome, distributes its response globally when locally perturbed; the distribution over network states has multiple local maxima, as in models for memory; and the parameters that describe the real network are close to a critical surface in this family of models., Comment: 14 pages, 12 figures

Theoretical physics is the search for simple and universal mathematical descriptions of the natural world. In contrast, much of modern biology is an exploration of the complexity and diversity of life. For many, this contrast is prima facie evidence that theory, in the sense that physicists use the word, is impossible in a biological context. For others, this contrast serves to highlight a grand challenge. I'm an optimist, and believe (along with many colleagues) that the time is ripe for the emergence of a more unified theoretical physics of biological systems, building on successes in thinking about particular phenomena. In this essay I try to explain the reasons for my optimism, through a combination of historical and modern examples., Comment: Based on a lecture at the 2014 conference on Theory in Biology, at the Simons Foundation

The activity of a neural network is defined by patterns of spiking and silence from the individual neurons. Because spikes are (relatively) sparse, patterns of activity with increasing numbers of spikes are less probable, but with more spikes the number of possible patterns increases. This tradeoff between probability and numerosity is mathematically equivalent to the relationship between entropy and energy in statistical physics. We construct this relationship for populations of up to N=160 neurons in a small patch of the vertebrate retina, using a combination of direct and model-based analyses of experiments on the response of this network to naturalistic movies. We see signs of a thermodynamic limit, where the entropy per neuron approaches a smooth function of the energy per neuron as N increases. The form of this function corresponds to the distribution of activity being poised near an unusual kind of critical point. Networks with more or less correlation among neurons would not reach this critical state. We suggest further tests of criticality, and give a brief discussion of its functional significance.

If we have a system of binary variables and we measure the pairwise correlations among these variables, then the least structured or maximum entropy model for their joint distribution is an Ising model with pairwise interactions among the spins. Here we consider inhomogeneous systems in which we constrain (for example) not the full matrix of correlations, but only the distribution from which these correlations are drawn. In this sense, what we have constructed is an inverse spin glass: rather than choosing coupling constants at random from a distribution and calculating correlations, we choose the correlations from a distribution and infer the coupling constants. We argue that such models generate a block structure in the space of couplings, which provides an explicit solution of the inverse problem. This allows us to generate a phase diagram in the space of (measurable) moments of the distribution of correlations. We expect that these ideas will be most useful in building models for systems that are nonequilibrium statistical mechanics problems, such as networks of real neurons.

Spatial patterns in the early fruit fly embryo emerge from a network of interactions among transcription factors, the gap genes, driven by maternal inputs. Such networks can exhibit many qualitatively different behaviors, separated by critical surfaces. At criticality, we should observe strong correlations in the fluctuations of different genes around their mean expression levels, a slowing of the dynamics along some but not all directions in the space of possible expression levels, correlations of expression fluctuations over long distances in the embryo, and departures from a Gaussian distribution of these fluctuations. Analysis of recent experiments on the gap genes shows that all these signatures are observed, and that the different signatures are related in ways predicted by theory. While there might be other explanations for these individual phenomena, the confluence of evidence suggests that this genetic network is tuned to criticality.

Recent work emphasizes that the maximum entropy principle provides a bridge between statistical mechanics models for collective behavior in neural networks and experiments on networks of real neurons. Most of this work has focused on capturing the measured correlations among pairs of neurons. Here we suggest an alternative, constructing models that are consistent with the distribution of global network activity, i.e. the probability that K out of N cells in the network generate action potentials in the same small time bin. The inverse problem that we need to solve in constructing the model is analytically tractable, and provides a natural "thermodynamics" for the network in the limit of large N. We analyze the responses of neurons in a small patch of the retina to naturalistic stimuli, and find that the implied thermodynamics is very close to an unusual critical point, in which the entropy (in proper units) is exactly equal to the energy.

The concept of feature selectivity in sensory signal processing can be formalized as dimensionality reduction: in a stimulus space of very high dimensions, neurons respond only to variations within some smaller, relevant subspace. But if neural responses exhibit invariances, then the relevant subspace typically cannot be reached by a Euclidean projection of the original stimulus. We argue that, in several cases, we can make progress by appealing to the simplest nonlinear construction, identifying the relevant variables as quadratic forms, or "stimulus energies." Natural examples include non-phase-locked cells in the auditory system, complex cells in visual cortex, and motion-sensitive neurons in the visual system. Generalizing the idea of maximally informative dimensions, we show that one can search for the kernels of the relevant quadratic forms by maximizing the mutual information between the stimulus energy and the arrival times of action potentials. Simple implementations of this idea successfully recover the underlying properties of model neurons even when the number of parameters in the kernel is comparable to the number of action potentials and stimuli are completely natural. We explore several generalizations that allow us to incorporate plausible structure into the kernel and thereby restrict the number of parameters. We hope that this approach will add significantly to the set of tools available for the analysis of neural responses to complex, naturalistic stimuli., Comment: 16 pages, 9 figures

The inverse problem of statistical mechanics involves finding the minimal Hamiltonian that is consistent with some observed set of correlation functions. This problem has received renewed interest in the analysis of biological networks; in particular, several such networks have been described successfully by maximum entropy models consistent with pairwise correlations. These correlations are usually weak in an absolute sense (e.g., correlation coefficients ~ 0.1 or less), and this is sometimes taken as evidence against the existence of interesting collective behavior in the network. If correlations are weak, it should be possible to capture their effects in perturbation theory, so we develop an expansion for the entropy of Ising systems in powers of the correlations, carrying this out to fourth order. We then consider recent work on networks of neurons [Schneidman et al., Nature 440, 1007 (2006); Tkacik et al., arXiv:0912.5409 [q-bio.NC] (2009)], and show that even though all pairwise correlations are weak, the fact that these correlations are widespread means that their impact on the network as a whole is not captured in the leading orders of perturbation theory. More positively, this means that recent successes of maximum entropy approaches are not simply the result of correlations being weak.

Many of life's most fascinating phenomena emerge from interactions among many elements--many amino acids determine the structure of a single protein, many genes determine the fate of a cell, many neurons are involved in shaping our thoughts and memories. Physicists have long hoped that these collective behaviors could be described using the ideas and methods of statistical mechanics. In the past few years, new, larger scale experiments have made it possible to construct statistical mechanics models of biological systems directly from real data. We review the surprising successes of this "inverse" approach, using examples form families of proteins, networks of neurons, and flocks of birds. Remarkably, in all these cases the models that emerge from the data are poised at a very special point in their parameter space--a critical point. This suggests there may be some deeper theoretical principle behind the behavior of these diverse systems., Comment: 21 pages

Recent work has shown that probabilistic models based on pairwise interactions-in the simplest case, the Ising model-provide surprisingly accurate descriptions of experiments on real biological networks ranging from neurons to genes. Finding these models requires us to solve an inverse problem: given experimentally measured expectation values, what are the parameters of the underlying Hamiltonian? This problem sits at the intersection of statistical physics and machine learning, and we suggest that more efficient solutions are possible by merging ideas from the two fields. We use a combination of recent coordinate descent algorithms with an adaptation of the histogram Monte Carlo method, and implement these techniques to take advantage of the sparseness found in data on real neurons. The resulting algorithm learns the parameters of an Ising model describing a network of forty neurons within a few minutes. This opens the possibility of analyzing much larger data sets now emerging, and thus testing hypotheses about the collective behaviors of these networks.

Ising models with pairwise interactions are the least structured, or maximum-entropy, probability distributions that exactly reproduce measured pairwise correlations between spins. Here we use this equivalence to construct Ising models that describe the correlated spiking activity of populations of 40 neurons in the retina, and show that pairwise interactions account for observed higher-order correlations. By first finding a representative ensemble for observed networks we can create synthetic networks of 120 neurons, and find that with increasing size the networks operate closer to a critical point and start exhibiting collective behaviors reminiscent of spin glasses., Comment: 4 pages, 3 figures

Entropy and information provide natural measures of correlation among elements in a network. We construct here the information theoretic analog of connected correlation functions: irreducible $N$--point correlation is measured by a decrease in entropy for the joint distribution of $N$ variables relative to the maximum entropy allowed by all the observed $N-1$ variable distributions. We calculate the ``connected information'' terms for several examples, and show that it also enables the decomposition of the information that is carried by a population of elements about an outside source., Comment: 4 pages, 3 figures

We all are fascinated by the phenomena of intelligent behavior, as generated both by our own brains and by the brains of other animals. As physicists we would like to understand if there are some general principles that govern the structure and dynamics of the neural circuits that underlie these phenomena. At the molecular level there is an extraordinary universality, but these mechanisms are surprisingly complex. This raises the question of how the brain selects from these diverse mechanisms and adapts to compute "the right thing" in each context. One approach is to ask what problems the brain really solves. There are several examples - from the ability of the visual system to count photons on a dark night to our gestalt recognition of statistical tendencies toward symmetry in random patterns - where the performance of the system in fact approaches some fundamental physical or statistical limits. This suggests that some sort of optimization principles may be at work, and there are examples where these principles have been formulated clearly and generated predictions which are confirmed in new experiments; a central theme in this work is the matching of the coding and computational strategies of the brain to the statistical structure of the world around us. Extension of these principles to the problem of learning leads us into interesting theoretical questions about how to measure the complexity of the data from which we learn and the complexity of the models that we use in learning, as well as opening some new opportunities for experiment. This combination of theoretical and experimental work gives us some new (if still speculative) perspectives on classical problems and controversies in cognition., Comment: Based on lectures at Les Houches Session LXXV, July 2001. To appear in Physics of Biomolecules and Cells, H. Flyvbjerg et al., eds. (EDP Sciences, Les Ulis; Springer-Verlag, Berlin, 2002). Revised version corrects minor typographical errors

We define {\em predictive information} $I_{\rm pred} (T)$ as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times $T$: $I_{\rm pred} (T)$ can remain finite, grow logarithmically, or grow as a fractional power law. If the time series allows us to learn a model with a finite number of parameters, then $I_{\rm pred} (T)$ grows logarithmically with a coefficient that counts the dimensionality of the model space. In contrast, power--law growth is associated, for example, with the learning of infinite parameter (or nonparametric) models such as continuous functions with smoothness constraints. There are connections between the predictive information and measures of complexity that have been defined both in learning theory and in the analysis of physical systems through statistical mechanics and dynamical systems theory. Further, in the same way that entropy provides the unique measure of available information consistent with some simple and plausible conditions, we argue that the divergent part of $I_{\rm pred} (T)$ provides the unique measure for the complexity of dynamics underlying a time series. Finally, we discuss how these ideas may be useful in different problems in physics, statistics, and biology., Comment: 53 pages, 3 figures, 98 references, LaTeX2e

Many processes in biology, from the regulation of gene expression in bacteria to memory in the brain, involve switches constructed from networks of biochemical reactions. Crucial molecules are present in small numbers, raising questions about noise and stability. Analysis of noise in simple reaction schemes indicates that switches stable for years and switchable in milliseconds can be built from fewer than one hundred molecules. Prospects for direct tests of this prediction, as well as implications, are discussed.

We define the relevant information in a signal $x\in X$ as being the information that this signal provides about another signal $y\in \Y$. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal $x$ requires more than just predicting $y$, it also requires specifying which features of $\X$ play a role in the prediction. We formalize this problem as that of finding a short code for $\X$ that preserves the maximum information about $\Y$. That is, we squeeze the information that $\X$ provides about $\Y$ through a `bottleneck' formed by a limited set of codewords $\tX$. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure $d(x,\x)$ emerges from the joint statistics of $\X$ and $\Y$. This approach yields an exact set of self consistent equations for the coding rules $X \to \tX$ and $\tX \to \Y$. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.

Observations on the past provide some hints about what will happen in the future, and this can be quantified using information theory. The ``predictive information'' defined in this way has connections to measures of complexity that have been proposed both in the study of dynamical systems and in mathematical statistics. In particular, the predictive information diverges when the observed data stream allows us to learn an increasingly precise model for the dynamics that generate the data, and the structure of this divergence measures the complexity of the model. We argue that divergent contributions to the predictive information provide the only measure of complexity or richness that is consistent with certain plausible requirements.

We construct a model that predicts the statistical properties of spike trains generated by a sensory neuron. The model describes the combined effects of the neuron's intrinsic properties, the noise in the surrounding, and the external driving stimulus. We show that the spike trains exhibit universal statistical behavior over short times, modulated by a strongly stimulus-dependent behavior over long times. These predictions are confirmed in experiments on H1, a motion-sensitive neuron in the fly visual system., Comment: 7 pages, 4 figures