Search

In-context learning (ICL), the remarkable ability to solve a task from only input exemplars, is often assumed to be a unique hallmark of Transformer models. By examining commonly employed synthetic ICL tasks, we demonstrate that multi-layer perceptrons (MLPs) can also learn in-context. Moreover, MLPs, and the closely related MLP-Mixer models, learn in-context competitively with Transformers given the same compute budget in this setting. We further show that MLPs outperform Transformers on a series of classical tasks from psychology designed to test relational reasoning, which are closely related to in-context classification. These results underscore a need for studying in-context learning beyond attention-based architectures, while also challenging strong prior arguments about MLPs' limited ability to solve relational tasks. Altogether, our results highlight the unexpected competence of MLPs, and support the growing interest in all-MLP alternatives to task-specific architectures., Comment: 30 pages, 10 figures, code available at https://github.com/wtong98/mlp-icl

Understanding how feature learning affects generalization is among the foremost goals of modern deep learning theory. Here, we study how the ability to learn representations affects the generalization performance of a simple class of models: deep Bayesian linear neural networks trained on unstructured Gaussian data. By comparing deep random feature models to deep networks in which all layers are trained, we provide a detailed characterization of the interplay between width, depth, data density, and prior mismatch. We show that both models display sample-wise double-descent behavior in the presence of label noise. Random feature models can also display model-wise double-descent if there are narrow bottleneck layers, while deep networks do not show these divergences. Random feature models can have particular widths that are optimal for generalization at a given data density, while making neural networks as wide or as narrow as possible is always optimal. Moreover, we show that the leading-order correction to the kernel-limit learning curve cannot distinguish between random feature models and deep networks in which all layers are trained. Taken together, our findings begin to elucidate how architectural details affect generalization performance in this simple class of deep regression models., Comment: 35 pages, 7 figures. v2: minor typos corrected and references added; published in PRE

We derive the equations of motion for the planar somersault, which consist of two additive terms. The first is the dynamic phase that is proportional to the angular momentum, and the second is the geometric phase that is independent of angular momentum and depends solely on the details of the shape change. Next, we import digitised footage of an elite athlete performing 3.5 forward somersaults off the 3m springboard, and use the data to validate our model. We show that reversing and reordering certain sections of the digitised dive can maximise the geometric phase without affecting the dynamic phase, thereby increasing the overall rotation achieved. Finally, we propose a theoretical planar somersault consisting of four shape changing states, where the optimisation lies in finding the shape change strategy that maximises the overall rotation of the dive. This is achieved by balancing the rotational contributions from the dynamic and geometric phases, in which we show the geometric phase plays a small but important role in the optimisation process., Comment: 21 pages, 14 figures, work from PhD thesis of William Tong

We present the mathematical framework of an athlete modelled as a system of coupled rigid bodies to simulate platform and springboard diving. Euler's equations of motion are generalised to non-rigid bodies, and are then used to innovate a new dive sequence that in principle can be performed by real world athletes. We begin by assuming shape changes are instantaneous so that the equations of motion simplify enough to be solved analytically, and then use this insight to present a new dive (513XD) consisting of 1.5 somersaults and 5 twists using realistic shape changes. Finally, we demonstrate the phenomenon of converting pure somersaulting motion into pure twisting motion by using a sequence of impulsive shape changes, which may have applications in other fields such as space aeronautics., Comment: 26 pages, 17 figures, work from PhD thesis of William Tong

A complete description of twisting somersaults is given using a reduction to a time-dependent Euler equation for non-rigid body dynamics. The central idea is that after reduction the twisting motion is apparent in a body frame, while the somersaulting (rotation about the fixed angular momentum vector in space) is recovered by a combination of dynamic and geometric phase. In the simplest "kick-model" the number of somersaults $m$ and the number of twists $n$ are obtained through a rational rotation number $W = m/n$ of a (rigid) Euler top. This rotation number is obtained by a slight modification of Montgomery's formula [9] for how much the rigid body has rotated. Using the full model with shape changes that take a realistic time we then derive the master twisting-somersault formula: An exact formula that relates the airborne time of the diver, the time spent in various stages of the dive, the numbers $m$ and $n$, the energy in the stages, and the angular momentum by extending a geometric phase formula due to Cabrera [3]. Numerical simulations for various dives agree perfectly with this formula where realistic parameters are taken from actual observations., Comment: 16 pages, 6 figures, work from PhD thesis of William Tong

We present and analyse a simple model for the twisting somersault. The model is a rigid body with a rotor attached which can be switched on and off. This makes it simple enough to devise explicit analytical formulas whilst still maintaining sufficient complexity to preserve the shape-changing dynamics essential for twisting somersaults in springboard and platform diving. With `rotor on' and with `rotor off' the corresponding Euler-type equations can be solved, and the essential quantities characterising the dynamics, such as the periods and rotation numbers, can be computed in terms of complete elliptic integrals. Thus we arrive at explicit formulas for how to achieve a dive with m somersaults and n twists in a given total time. This can be thought of as a special case of a geometric phase formula due to Cabrera 2007., Comment: 15 pages, 6 figures

A pentagon in the plane with fixed side-lengths has a two-dimensional shape space. Considering the pentagon as a mechanical system with point masses at the corners we answer the question of how much the pentagon can rotate with zero angular momentum. We show that the shape space of the equilateral pentagon has genus 4 and find a fundamental region by discrete symmetry reduction with respect to symmetry group D_5. The amount of rotation \Delta \theta for a loop in shape space at zero angular momentum is interpreted as a geometric phase and is obtained as an integral of a function B over the region of shape space enclosed by the loop. With a simple variational argument we determine locally optimal loops as the zero contours of the function B. The resulting shape change is represented as a Fourier series, and the global maximum of \Delta \theta \approx 45\degree is found for a loop around the regular pentagram. We also show that restricting allowed shapes to convex pentagons the optimal loop is the boundary of the convex region and gives \Delta \theta \approx 19\degree., Comment: 29 pages