Search

Transformer-based models have demonstrated exceptional performance across diverse domains, becoming the state-of-the-art solution for addressing sequential machine learning problems. Even though we have a general understanding of the fundamental components in the transformer architecture, little is known about how they operate or what are their expected dynamics. Recently, there has been an increasing interest in exploring the relationship between attention mechanisms and Hopfield networks, promising to shed light on the statistical physics of transformer networks. However, to date, the dynamical regimes of transformer-like models have not been studied in depth. In this paper, we address this gap by using methods for the study of asymmetric Hopfield networks in nonequilibrium regimes --namely path integral methods over generating functionals, yielding dynamics governed by concurrent mean-field variables. Assuming 1-bit tokens and weights, we derive analytical approximations for the behavior of large self-attention neural networks coupled to a softmax output, which become exact in the large limit size. Our findings reveal nontrivial dynamical phenomena, including nonequilibrium phase transitions associated with chaotic bifurcations, even for very simple configurations with a few encoded features and a very short context window. Finally, we discuss the potential of our analytic approach to improve our understanding of the inner workings of transformer models, potentially reducing computational training costs and enhancing model interpretability.

Pressure evolution of the crystal structure and magnetism of the honeycomb $\alpha$-RuBr$_3$ is studied using high-pressure x-ray diffraction, magnetometry, and density-functional band-structure calculations. Hydrostatic compression transforms antiferromagnetic $\alpha$-RuBr$_3$ ($R\bar 3$) into paramagnetic $\alpha'$-RuBr$_3$ ($P\bar 1$) where short Ru-Ru bonds cause magnetism collapse above 1.3 GPa at 0 K and 2.5 GPa at 295 K. Below this critical pressure, the N\'eel temperature of $\alpha$-RuBr$_3$ increases with the slope of 1.8 K/GPa. Pressure tunes $\alpha$-RuBr$_3$ away from the Kitaev limit, whereas increased third-neighbor in-plane coupling and interlayer coupling lead to a further stabilization of the collinear zigzag state. Both $\alpha$- and $\alpha'$-RuBr$_3$ are metastable at ambient pressure, but their transformation into the thermodynamically stable $\beta$-polymorph is kinetically hindered at room temperature., Comment: 8 pages + Supplemental Material: published version

Bayesian theories of biological and brain function speculate that Markov blankets (a conditional independence separating a system from external states) play a key role for facilitating inference-like behaviour in living systems. Although it has been suggested that Markov blankets are commonplace in sparsely connected, nonequilibrium complex systems, this has not been studied in detail. Here, we show in two different examples (a pair of coupled Lorenz systems and a nonequilibrium Ising model) that sparse connectivity does not guarantee Markov blankets in the steady-state density of nonequilibrium systems. Conversely, in the nonequilibrium Ising model explored, the more distant from equilibrium the system appears to be correlated with the distance from displaying a Markov blanket. These result suggests that further assumptions might be needed in order to assume the presence of Markov blankets in the kind of nonequilibrium processes describing the activity of living systems.

We extend previous mean-field approaches for non-equilibrium neural network models to estimate correlations in the system. This offers a powerful tool for approximating the system dynamics as well as a fast method to infer network parameters from observations. We develop our method in an asymmetric kinetic Ising model and test its performance on 1) synthetic data generated by an asymmetric version of the Sherrington Kirkpatric model and 2) recordings of in vitro neuron spiking activity from the mouse somatosensory cortex. We find that our mean-field method outperforms previous ones in estimating networks correlations and successfully reconstructs network dynamics from data near a phase transition showing large fluctuations.

An exploratory study of the time-like pion electromagnetic form factor in a Poincar\'e-covariant bound state formalism in the isospin symmetric limit is presented. Starting from a quark interaction kernel representing gluon-intermediated interactions for valence-type quarks, non-valence effects are included by introducing pions as explicit degrees of freedom. The two most important qualitative aspects are, in view of the presented study, the opening of the dominant $\rho$-meson decay channel and the presence of a multi-particle branch cut setting in when the two-pion threshold is crossed. Based on a recent respective computation of the quark-photon vertex, the pion electromagnetic form factor for space-like and time-like kinematics is calculated. The obtained results for its absolute value and its phase compare favorably to the available experimental data, and they are analyzed in detail by confronting them to the expectations based on an isospin-symmetric version of a vector-meson dominance model., Comment: 14 pages, 8 figures. References added

Motivated by the study of physical models associated with General Relativity, we review some finite-dimensional, geometric and covariant formulations that allow us to characterize in a simple manner the symmetries for classical field theory by implementing an appropriate fibre-bundle structure, either at the Lagrangian, the multisymplectic or the polysymplectic levels. In particular, we are able to formulate Noether's theorems by means of the covariant momentum maps and to systematically introduce a covariant Poisson-Hamiltonian framework. Also, by focusing on the space plus time decomposition for a generic classical field theory and its relation to these geometric formulations, we are able to successfully recover the gauge content and the true local degrees of freedom for the theory. In order to illustrate the relevance of these geometric frameworks, we center our attention to the analysis of a model for $3$-dimensional theory of General Relativity that involves an arbitrary Immirzi-like parameter. At the Lagrangian level, we reproduce the field equations of the system which for this model turn out to be equivalent to the vanishing torsion condition and the Einstein equations. We also concentrate on the analysis of the gauge symmetries of the system in order to obtain the Lagrangian covariant momentum map associated with the theory and, consequently, its corresponding Noether currents. Next, we aim our attention to describing how the gauge symmetries of the model yield covariant canonical transformations on the covariant multimomenta phase-space, thus giving rise to the existence of a covariant momentum map. Besides, we analyze the physical system under consideration within the De Donder-Weyl canonical theory implemented at the polysymplectic level, thus establishing a relation from the covariant momentum map to the conserved currents of the theory., Comment: 55 pages, no figures

We analyze the inherent symmetries associated to the non-Abelian topological BF theory from the geometric and covariant perspectives of the Lagrangian and the multisymplectic formalisms. At the Lagrangian level, we classify the symmetries of the theory as natural and Noether symmetries and construct the associated Noether currents, while at the multisymplectic level the symmetries of the theory arise as covariant canonical transformations. These transformations allowed us to build within the multisymplectic approach, in a complete covariant way, the momentum maps which are analogous to the conserved Noether currents. The covariant momentum maps are fundamental to recover, after the space plus time decomposition of the background manifold, not only the extended Hamiltonian of the BF theory but also the generators of the gauge transformations which arise in the instantaneous Dirac-Hamiltonian analysis of the first-class constraint structure that characterizes the BF model under study. To the best of our knowledge, this is the first non-trivial physical model associated to General Relativity for which both natural and Noether symmetries have been analyzed at the multisymplectic level. Our study shed some light on the understanding of the manner in which the generators of gauge transformations may be recovered from the multisymplectic formalism for field theory., Comment: 28 pages, no figures

The polysymplectic formulation of the CMPR action, which is a BF-type formulation of General Relativity that involves an arbitrary Immirzi parameter, is performed. We implement a particular scheme within this covariant Hamiltonian approach to analyze the constraints that characterize the CMPR model. By means of the privileged $(n-1)$-forms and the Poisson-Gerstenhaber bracket, inherent to the polysymplectic framework, the BF field equations associated to the CMPR action are obtained and, in consequence, the Einstein equations naturally emerge by solving the simplicity constraints of the theory. Further, from the polysymplectic analysis of the CMPR action the De Donder-Weyl Hamiltonian formulation of the Holst action is recovered, which is consistent with the Lagrangian analysis of this model as reported in the literature., Comment: 19 pages, no figures