1. RotRNN: Modelling Long Sequences with Rotations
- Author
-
Biegun, Kai, Dolga, Rares, Cunningham, Jake, and Barber, David
- Subjects
Computer Science - Machine Learning ,Statistics - Machine Learning - Abstract
Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, their empirical performance is not well understood and they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN -- a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple and efficient model with a robust normalisation procedure, and a practical implementation that remains faithful to its theoretical derivation. RotRNN also achieves competitive performance to state-of-the-art linear recurrent models on several long sequence modelling datasets., Comment: Next Generation of Sequence Modeling Architectures Workshop at ICML 2024
- Published
- 2024