Search

This paper introduces BreakGPT, a novel large language model (LLM) architecture adapted specifically for time series forecasting and the prediction of sharp upward movements in asset prices. By leveraging both the capabilities of LLMs and Transformer-based models, this study evaluates BreakGPT and other Transformer-based models for their ability to address the unique challenges posed by highly volatile financial markets. The primary contribution of this work lies in demonstrating the effectiveness of combining time series representation learning with LLM prediction frameworks. We showcase BreakGPT as a promising solution for financial forecasting with minimal training and as a strong competitor for capturing both local and global temporal dependencies.

We investigate the influence of the helical compactification of spatial dimension on the local properties of the vacuum state for a charged scalar field with general curvature coupling parameter. A general background geometry is considered with rotational symmetry in the subspace with the coordinates appearing in the helical periodicity condition. It is shown that by a coordinate transformation the problem is reduced to the problem with standard quasiperiodicity condition in the same local geometry and with the effective compactification radius determined by the length of the compact dimension and the helicity parameter. As an application of the general procedure we have considered locally de Sitter spacetime with a helical compact dimension. By using the Hadamard function for the Bunch-Davies vacuum state, the vacuum expectation values of the field squared, current density, and energy-momentum tensor are studied. The topological contributions are explicitly separated and their asymptotics are described at early and late stages of cosmological expansion. An important difference, compared to the problem with quasiperiodic conditions, is the appearance of the nonzero off-diagonal component of the energy-momentum tensor and of the component of the current density along the uncompact dimension., Comment: 13 pages

The evolution of dynamic processes in graphene-family materials are of great interest for both scientific purposes and technical applications. Scanning electron microscopy and transmission electron microscopy outstand among the techniques that allow both observing and controlling such dynamic processes in real time. On the other hand, functionalized graphene oxide emerges as a favorable candidate from graphene-family materials for such an investigation due to its distinctive properties, that encompass a large surface area, robust thermal stability, and noteworthy electrical and mechanical properties after its reduction. Here, we report on studies of surface structure and adsorption dynamics of L-Cysteine on electrochemically exfoliated graphene oxides basal plane. We show that electron beam irradiation prompts an amorphization of functionalized graphene oxide along with the formation of micropatterns of controlled geometry composed of L-Cysteine-Graphene oxide nanostructures. The controlled growth and predetermined arrangement of micropatterns as well as controlled structure disorder induced by e beam amorphization, in its turn potentially offering tailored properties and functionalities paving the way for potential applications in nanotechnology, sensor development, and surface engineering. Our findings demonstrate that graphene oxide can cover L-Cysteine in such a way to provide a control on the positioning of emerging microstructures about 10-20 um in diameter. Besides, Raman and SAED measurement analyses yield above 50% amorphization in a material. The results of our studies demonstrate that such a technique enables the direct creation of micropatterns of L-Cysteine-Graphene oxide eliminating the need for complicated mask patterning procedures.

A fundamental requirement for photonic technologies is the ability to control the confinement and propagation of light. Widely utilized platforms include two-dimensional (2D) optical microcavities in which electromagnetic waves are confined between either metallic or distributed Bragg reflectors. Recently, transition metal dichalcogenides hosting tightly bound excitons with high optical quality have emerged as promising atomically thin mirrors. In this work, we propose and experimentally demonstrate a sub-wavelength 2D nano-cavity using two atomically thin mirrors with degenerate resonances. Angle-resolved measurements show a flat band, which sets this system apart from conventional photonic cavities. Remarkably, we demonstrate how the excitonic nature of the mirrors enables the formation of chiral and tunable optical modes upon the application of an external magnetic field. Moreover, we show the electrical tunability of the confined mode. Our work demonstrates a mechanism for confining light with high-quality excitonic materials, opening perspectives for spin-photon interfaces, and chiral cavity electrodynamics., Comment: Main text: 10 pages, 4 figures. Supplementary Material: 7 pages, 11 figures

The Heavy Photon Search experiment (HPS) at the Thomas Jefferson National Accelerator Facility searches for electro-produced dark photons. We report results from the 2016 Engineering Run consisting of 10608/nb of data for both the prompt and displaced vertex searches. A search for a prompt resonance in the $e^+e^-$ invariant mass distribution between 39 and 179 MeV showed no evidence of dark photons above the large QED background, limiting the coupling of {\epsilon}^2 {\geq} 10^-5, in agreement with previous searches. The search for displaced vertices showed no evidence of excess signal over background in the masses between 60 and 150 MeV, but had insufficient luminosity to limit canonical heavy photon production. This is the first displaced vertex search result published by HPS. HPS has taken high-luminosity data runs in 2019 and 2021 that will explore new dark photon phase space., Comment: 28 pages, 46 figures

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data., Comment: 54 pages. In Proceedings of Neural Information Processing Systems (NeurIPS) 2022

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

General perception systems such as Perceivers can process arbitrary modalities in any combination and are able to handle up to a few hundred thousand inputs. They achieve this generality by using exclusively global attention operations. This however hinders them from scaling up to the inputs sizes required to process raw high-resolution images or video. In this paper, we show that some degree of locality can be introduced back into these models, greatly improving their efficiency while preserving their generality. To scale them further, we introduce a self-supervised approach that enables learning dense low-dimensional positional embeddings for very large signals. We call the resulting model a Hierarchical Perceiver (HiP). In sum our contributions are: 1) scaling Perceiver-type models to raw high-resolution images and audio+video, 2) showing the feasibility of learning 1M+ positional embeddings from scratch using masked auto-encoding, 3) demonstrating competitive performance on raw data from ImageNet, AudioSet, PASCAL VOC, ModelNet40 and Kinetics datasets with the same exact, unchanged model and without specialized preprocessing or any tokenization.

The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input. For these models, parameter count and computational requirement form two independent axes along which an increase leads to better performance. In this work we derive and justify scaling laws defined on these two variables which generalize those known for standard language models and describe the performance of a wide range of routing architectures trained via three different techniques. Afterwards we provide two applications of these laws: first deriving an Effective Parameter Count along which all models scale at the same rate, and then using the scaling coefficients to give a quantitative comparison of the three routing techniques considered. Our analysis derives from an extensive evaluation of Routing Networks across five orders of magnitude of size, including models with hundreds of experts and hundreds of billions of parameters., Comment: Fixing typos and affiliation clarity

Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms., Comment: 120 pages

We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale., Comment: Fix incorrect reported numbers in Table 14

Beam search is the go-to method for decoding auto-regressive machine translation models. While it yields consistent improvements in terms of BLEU, it is only concerned with finding outputs with high model likelihood, and is thus agnostic to whatever end metric or score practitioners care about. Our aim is to establish whether beam search can be replaced by a more powerful metric-driven search technique. To this end, we explore numerous decoding algorithms, including some which rely on a value function parameterised by a neural network, and report results on a variety of metrics. Notably, we introduce a Monte-Carlo Tree Search (MCTS) based method and showcase its competitiveness. We provide a blueprint for how to use MCTS fruitfully in language applications, which opens promising future directions. We find that which algorithm is best heavily depends on the characteristics of the goal metric; we believe that our extensive experiments and analysis will inform further research in this area., Comment: 23 pages

Precipitation nowcasting, the high-resolution forecasting of precipitation up to two hours ahead, supports the real-world socio-economic needs of many sectors reliant on weather-dependent decision-making. State-of-the-art operational nowcasting methods typically advect precipitation fields with radar-based wind estimates, and struggle to capture important non-linear events such as convective initiations. Recently introduced deep learning methods use radar to directly predict future rain rates, free of physical constraints. While they accurately predict low-intensity rainfall, their operational utility is limited because their lack of constraints produces blurry nowcasts at longer lead times, yielding poor performance on more rare medium-to-heavy rain events. To address these challenges, we present a Deep Generative Model for the probabilistic nowcasting of precipitation from radar. Our model produces realistic and spatio-temporally consistent predictions over regions up to 1536 km x 1280 km and with lead times from 5-90 min ahead. In a systematic evaluation by more than fifty expert forecasters from the Met Office, our generative model ranked first for its accuracy and usefulness in 88% of cases against two competitive methods, demonstrating its decision-making value and ability to provide physical insight to real-world experts. When verified quantitatively, these nowcasts are skillful without resorting to blurring. We show that generative nowcasting can provide probabilistic predictions that improve forecast value and support operational utility, and at resolutions and lead times where alternative methods struggle., Comment: 46 pages, 17 figures, 2 tables

Semantically meaningful information content in perceptual signals is usually unevenly distributed. In speech signals for example, there are often many silences, and the speed of pronunciation can vary considerably. In this work, we propose slow autoencoders (SlowAEs) for unsupervised learning of high-level variable-rate discrete representations of sequences, and apply them to speech. We show that the resulting event-based representations automatically grow or shrink depending on the density of salient information in the input signals, while still allowing for faithful signal reconstruction. We develop run-length Transformers (RLTs) for event-based representation modelling and use them to construct language models in the speech domain, which are able to generate grammatical and semantically coherent utterances and continuations., Comment: 26 pages, 15 figures, samples can be found at https://vdrl.github.io/

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets