Search

We present OLMo 2, the next generation of our fully open language models. OLMo 2 includes dense autoregressive models with improved architecture and training recipe, pretraining data mixtures, and instruction tuning recipes. Our modified model architecture and training recipe achieve both better training stability and improved per-token efficiency. Our updated pretraining data mixture introduces a new, specialized data mix called Dolmino Mix 1124, which significantly improves model capabilities across many downstream task benchmarks when introduced via late-stage curriculum training (i.e. specialized data during the annealing phase of pretraining). Finally, we incorporate best practices from T\"ulu 3 to develop OLMo 2-Instruct, focusing on permissive data and extending our final-stage reinforcement learning with verifiable rewards (RLVR). Our OLMo 2 base models sit at the Pareto frontier of performance to compute, often matching or outperforming open-weight only models like Llama 3.1 and Qwen 2.5 while using fewer FLOPs and with fully transparent training data, code, and recipe. Our fully open OLMo 2-Instruct models are competitive with or surpassing open-weight only models of comparable size, including Qwen 2.5, Llama 3.1 and Gemma 2. We release all OLMo 2 artifacts openly -- models at 7B and 13B scales, both pretrained and post-trained, including their full training data, training code and recipes, training logs and thousands of intermediate checkpoints. The final instruction model is available on the Ai2 Playground as a free research demo., Comment: Model demo available at playground.allenai.org

Some galaxy clusters contain non-thermal synchrotron emitting plasma permeating the intracluster medium (ICM). The spectral properties of this radio emission are not well characterized at decameter wavelengths ({\nu} < 30 MHz), primarily due to the severe corrupting effects of the ionosphere. Using a recently developed calibration strategy, we present LOFAR images below 30 MHz of the low mass galaxy cluster Abell 655, which was serendipitously detected in an observation of the bright calibrator 3C 196. We combine this observation with LOFAR data at 144 MHz, and new Band 4 Giant Metrewave Radio Telescope observations centered at 650 MHz. In the 15-30 MHz LOFAR image, diffuse emission is seen with a physical extent of about 700 kpc. We argue that the diffuse emission detected in this galaxy cluster likely has multiple origins. At higher frequencies (650 MHz), the diffuse emission resembles a radio halo, while at lower frequencies the emission seems to consist of several components and bar-like structures. It suggests that most low-frequency emission in this cluster comes from re-energized fossil plasma from old AGN outbursts, coexisting with the radio halo component. By counting the number of cluster radio detections in the decameter band, we estimate that around a quarter of the Planck clusters host re-energised fossil plasma that is detectable in the decameter band with LOFAR., Comment: Accepted for publication in A&A. 11 pages, 3 tables, 8 figures

We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: first use model and data size to predict a task-specific loss, and then use this task loss to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks written in ranked classification format, we can predict the accuracy of both target models within 2 points of absolute error. We have higher prediction error on four other tasks (average absolute error 6.9) and find that these are often tasks with higher variance in task metrics. We also find that using less compute to train fewer ladder models tends to deteriorate predictions. Finally, we empirically show that our design choices and the two-step approach lead to superior performance in establishing scaling laws.

The Perseus cluster is the brightest X-ray cluster in the sky and is known as a cool-core galaxy cluster. Being a very nearby cluster, it has been extensively studied. This has provided a comprehensive view of the physical processes that operate in the intracluster medium (ICM), including feedback from the AGN 3C84 and measurements of ICM turbulence. Additionally, the Perseus cluster contains a central radio mini-halo. This diffuse radio source traces cosmic ray electrons (re-)accelerated in-situ in the ICM. Here we report on LOFAR high-band antenna 120-168 MHz observations of the Perseus cluster that probe a range of four orders of magnitude in angular scales. In our 0.3 arcsec resolution image, we find that the northern extension of the 3C84 lobe consists of several narrow 1.5-3 kpc parallel strands of emission. In addition, we detect steep-spectrum filaments associated with a previous outburst of the central AGN radio emission filling two known X-ray ghost cavities. At 7 arcsec resolution, our images show a complex structured radio mini-halo, with several edges and filaments. At resolutions of 26 arcsec and 80 arcsec, we discover diffuse radio emission with a 1.1 Mpc extent. We classify this emission as a giant radio halo and its properties are distinct from the inner mini-halo. We also detect two diffuse sources at projected cluster centric radii of 0.7 and 1.0 Mpc. Finally, we observe a 0.9 Mpc long trail of radio emission from the cluster member galaxy IC310, connecting it with the giant radio halo. Together with other recent studies of relaxed clusters, our LOFAR observations indicate that cluster-wide radio emission could be (more) common in cool-core clusters. In the case of the Perseus cluster, a past off-axis merger event that preserved the cool core might have generated enough turbulence to produce an extended radio halo observable at low frequencies., Comment: accepted for publication in A&A, 18 pages, 19 figures

Today's most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog., Comment: Updated with ablations and more technical details

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE training, analyze routing in our model showing high specialization, and open-source all aspects of our work: model weights, training data, code, and logs., Comment: 61 pages (24 main), 36 figures, 14 tables

We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-Baseline enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation., Comment: Project page: https://www.datacomp.ai/dclm/

The largely unexplored decameter radio band (10-30 MHz) provides a unique window for studying a range of astronomical topics, such as auroral emission from exoplanets, inefficient cosmic ray acceleration mechanisms, fossil radio plasma, and free-free absorption. The scarcity of low-frequency studies is mainly due to the severe perturbing effects of the ionosphere. Here we present a calibration strategy that can correct for the ionosphere in the decameter band. We apply this to an observation from the Low Frequency Array (LOFAR) between 16 to 30 MHz . The resulting image covers 330 square degrees of sky at a resolution of 45", reaching a sensitivity of 12 mJy/beam. Residual ionospheric effects cause additional blurring ranging between 60 to 100". This represents an order of magnitude improvement in terms of sensitivity and resolution compared to previous decameter band observations. In the region we surveyed, we have identified four fossil plasma sources. These rare sources are believed to contain old, possibly re-energised, radio plasma originating from previous outbursts of active galactic nuclei. At least three of them are situated near the center of low-mass galaxy clusters. Notably, two of these sources display the steepest radio spectral index among all the sources detected at 23 MHz. This indicates that fossil plasma sources constitute the primary population of steep-spectrum sources at these frequencies, emphasising the large discovery potential of ground-based decameter observations., Comment: 19 pages, 9 figures, 1 table. Submitted version, full version is published by Nature Astronomy

Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, we have built OLMo, a competitive, truly Open Language Model, to enable the scientific study of language models. Unlike most prior efforts that have only released model weights and inference code, we release OLMo alongside open training data and training and evaluation code. We hope this release will empower the open research community and inspire a new wave of innovation.

Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation., Comment: Accepted at ACL 2024; Dataset: https://hf.co/datasets/allenai/dolma; Code: https://github.com/allenai/dolma

Creativity is a critical skill that professional software engineers leverage to tackle difficult problems. In higher education, multiple efforts have been made to spark creative skills of engineering students. However, creativity is a vague concept that is open to interpretation. Furthermore, studies have shown that there is a gap in perception and implementation of creativity between industry and academia. To better understand the role of creativity in software engineering (SE), we interviewed 33 professionals via four focus groups and 10 SE students. Our results reveal 45 underlying topics related to creativity. When comparing the perception of students versus professionals, we discovered fundamental differences, grouped into five themes: the creative environment, application of techniques, creative collaboration, nature vs nurture, and the perceived value of creativity. As our aim is to use these findings to install and further encourage creative problem solving in higher education, we have included a list of implications for educational practice.

Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training. Implicitly or explicitly, this data is composed of domains--varying distributions of language. We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains, instead of assuming perplexity on one distribution extrapolates to others. We include two new datasets of the top 100 subreddits (e.g., r/depression on Reddit) and programming languages (e.g., Java on GitHub), both sources common in contemporary LMs. With our benchmark, we release 6 baseline 1B LMs carefully controlled to provide fair comparisons about which pretraining corpus is best and code for others to apply those controls to their own experiments. Our case studies demonstrate how the fine-grained results from Paloma surface findings such as that models pretrained without data beyond Common Crawl exhibit anomalous gaps in LM fit to many domains or that loss is dominated by the most frequently occurring strings in the vocabulary., Comment: Conference: NeurIPS 2024, Project Page: https://paloma.allen.ai/

The success of large language models has shifted the evaluation paradigms in natural language processing (NLP). The community's interest has drifted towards comparing NLP models across many tasks, domains, and datasets, often at an extreme scale. This imposes new engineering challenges: efforts in constructing datasets and models have been fragmented, and their formats and interfaces are incompatible. As a result, it often takes extensive (re)implementation efforts to make fair and controlled comparisons at scale. Catwalk aims to address these issues. Catwalk provides a unified interface to a broad range of existing NLP datasets and models, ranging from both canonical supervised training and fine-tuning, to more modern paradigms like in-context learning. Its carefully-designed abstractions allow for easy extensions to many others. Catwalk substantially lowers the barriers to conducting controlled experiments at scale. For example, we finetuned and evaluated over 64 models on over 86 datasets with a single command, without writing any code. Maintained by the AllenNLP team at the Allen Institute for Artificial Intelligence (AI2), Catwalk is an ongoing open-source effort: https://github.com/allenai/catwalk., Comment: technical report, work in progress

Large text corpora are the backbone of language models. However, we have a limited understanding of the content of these corpora, including general statistics, quality, social factors, and inclusion of evaluation data (contamination). In this work, we propose What's In My Big Data? (WIMBD), a platform and a set of sixteen analyses that allow us to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities -- count and search -- at scale, which allows us to analyze more than 35 terabytes on a standard compute node. We apply WIMBD to ten different corpora used to train popular language models, including C4, The Pile, and RedPajama. Our analysis uncovers several surprising and previously undocumented findings about these corpora, including the high prevalence of duplicate, synthetic, and low-quality content, personally identifiable information, toxic language, and benchmark contamination. For instance, we find that about 50% of the documents in RedPajama and LAION-2B-en are duplicates. In addition, several datasets used for benchmarking models trained on such corpora are contaminated with respect to important benchmarks, including the Winograd Schema Challenge and parts of GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them., Comment: Published at ICLR 2024 spotlight

This study explores the influence of an interdisciplinary intervention on creative problem-solving skills. Literature deems such skills as vital for software engineering (SE) students in higher education. 39 SE students and graphic design (GD) students were randomly paired to work on an open-ended creative coding assignment in p5.js, an online JS-based Processing editor that makes it easy for novices to quickly and easily code visual webpages. Three categories were formed: the test group SE+GD (18 students), and control groups SE+SE (10) and GD+GD (11). A mixed methods approach was taken to gather and interpret results: Amabile's Consensual Assessment Technique provided a global creativity score for the finished product, the Creative Programming Problem Solving Test assessed three dimensions of the creative process (Ability, Mindset, Interaction), and 9 semi-structured follow-up interviews provided context and revealed underlying themes. The results indicate that, while the creativity of the end product initially takes a hit, the SE+GD groups' socio-interactive creativity levels increased. We also observed fixed mindsets towards creativity ("design students are more creative than we") that call for future work.

Feedback beyond the grade is an important part of the learning process. However, because of the large student groups, many teachers in higher education are faced with practicalities such as the limited time to prepare and communicate the feedback to individual students. We have set up an experiment, titled Pro-f-quiz, in which over two years 236 students participated and in which the feedback is communicated through an online quiz that activates the students to reflect upon their solution. The system can be set up in a very limited time compared to booking individual time slots. The results show that approximately 85% of the students appreciate the approach with 60% indicating that they reflect more intensively about their work than when the feedback is transmitted traditionally. Moreover, the grade of the students participating in the project was substantially higher than students not participating.

Large language models (LLMs) enable unparalleled few- and zero-shot reasoning capabilities but at a high computational footprint. A growing assortment of methods for compression promises to reduce the computational burden of LLMs in deployment, but so far, only quantization approaches have been demonstrated to be effective for LLM compression while maintaining zero-shot performance. A critical step in the compression process, the pretrain-then-finetune paradigm, has largely been overlooked when adapting existing pruning strategies to LLMs or proposing new ones. In this work, we show that embarrassingly simple layer pruning coupled with an extended language model pretraining as the finetuning phase produces state-of-the-art results against structured and even semi-structured compression of models at a 7B scale while being more inference efficient. We call this method LayerChop, where we deterministically remove layers from a model followed by task-agnostic finetuning of the remaining weights by continued self-supervised pretraining. At this scale, we also show how distillation, which has been super effective in task-agnostic compression of smaller BERT-style models, becomes inefficient against our simple pruning technique., Comment: 13 pages, 6 figures, 6 tables

Recently introduced language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters. Nevertheless, these methods still often trail behind full model finetuning. In this work, we investigate if a dedicated continued pretraining stage could improve "promptability", i.e., zero-shot performance with natural language prompts or few-shot performance with prompt tuning. We reveal settings where existing continued pretraining methods lack promptability. We also identify current methodological gaps, which we fill with thorough large-scale experiments. We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero- and few-shot settings compared to existing methods, up to 31% relative. On the other hand, we find that continued pretraining using MAML-style meta-learning, a method that directly optimizes few-shot promptability, yields subpar performance. We validate our findings with two prompt tuning methods, and, based on our results, we provide concrete recommendations to optimize promptability for different use cases., Comment: EMNLP 2022