Search

Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce T\"ULU 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. T\"ULU 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With T\"ULU 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. In addition to the T\"ULU 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the T\"ULU 3 approach to more domains.

Learning from human feedback has enabled the alignment of language models (LMs) with human preferences. However, directly collecting human preferences can be expensive, time-consuming, and can have high variance. An appealing alternative is to distill preferences from LMs as a source of synthetic annotations as they are more consistent, cheaper, and scale better than human annotation; however, they are also prone to biases and errors. In this work, we introduce a routing framework that combines inputs from humans and LMs to achieve better annotation quality, while reducing the total cost of human annotation. The crux of our approach is to identify preference instances that will benefit from human annotations. We formulate this as an optimization problem: given a preference dataset and an evaluation metric, we train a performance prediction model to predict a reward model's performance on an arbitrary combination of human and LM annotations and employ a routing strategy that selects a combination that maximizes predicted performance. We train the performance prediction model on MultiPref, a new preference dataset with 10K instances paired with human and LM labels. We show that the selected hybrid mixture of LM and direct human preferences using our routing framework achieves better reward model performance compared to using either one exclusively. We simulate selective human preference collection on three other datasets and show that our method generalizes well to all three. We analyze features from the routing model to identify characteristics of instances that can benefit from human feedback, e.g., prompts with a moderate safety concern or moderate intent complexity. We release the dataset, annotation platform, and source code used in this study to foster more efficient and accurate preference collection in the future., Comment: Code in https://github.com/allenai/hybrid-preferences, MultiPref dataset in https://huggingface.co/datasets/allenai/multipref, Updated related work

Reward models (RMs) have driven the state-of-the-art performance of LLMs today by enabling the integration of human feedback into the language modeling process. However, RMs are primarily trained and evaluated in English, and their capabilities in multilingual settings remain largely understudied. In this work, we conduct a systematic evaluation of several reward models in multilingual settings. We first construct the first-of-its-kind multilingual RM evaluation benchmark, M-RewardBench, consisting of 2.87k preference instances for 23 typologically diverse languages, that tests the chat, safety, reasoning, and translation capabilities of RMs. We then rigorously evaluate a wide range of reward models on M-RewardBench, offering fresh insights into their performance across diverse languages. We identify a significant gap in RMs' performances between English and non-English languages and show that RM preferences can change substantially from one language to another. We also present several findings on how different multilingual aspects impact RM performance. Specifically, we show that the performance of RMs is improved with improved translation quality. Similarly, we demonstrate that the models exhibit better performance for high-resource languages. We release M-RewardBench dataset and the codebase in this study to facilitate a better understanding of RM evaluation in multilingual settings., Comment: 16 pages, 6 figures, 10 tables. Website: https://m-rewardbench.github.io/ , Updated results with latest models. Added more author information

One of the most intriguing 6G vertical markets is precision agriculture, where communications, sensing, control, and robotics technologies are used to improve agricultural outputs and decrease environmental impact. Ambient IoT (A-IoT), which uses a network of devices that harvest ambient energy to enable communications, is expected to play an important role in agricultural use cases due to its low costs, simplicity, and battery-free (or battery-assisted) operation. In this paper, we review the use cases of precision agriculture and discuss the challenges. We discuss how A-IoT can be used for precision agriculture and compare it with other ambient energy source technologies. We also discuss research directions related to both A-IoT and precision agriculture., Comment: 7 pages, 4 figures and 2 tables

Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the dominance of English training data, raising concerns about potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub that fills the resource gap by providing standardized corpora in nearly 1,000 SEA languages across three modalities. Through our SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous languages across 13 tasks, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in SEA., Comment: https://seacrowd.github.io/ Accepted in EMNLP 2024

Public education funding is a complex matter. It is widely understood that dollars for public schools come from federal, state, and local sources, but the revenue stream from each source is calculated in complex formulas and criteria that can make it hard for anyone to understand the system fully. Calculating aid based on some measure of the number of students that a school district serves is the primary modus operandi by state education departments. Some states determine aid based on average daily attendance, and others use enrollment counts. Each approach has trade-offs. The goal of this article is to explain various school funding protections (also known as "hold harmless" provisions) that states take--which give additional funds to school districts that otherwise would not have received them--in order to provide a better understanding of the various types of funding protections and the trade-offs they present to policymakers and citizens. The first part of the article examines declining enrollment provisions and their trade-offs, while the second part does the same for funding guarantees. A follow-up discussion with suggestions for using either policy more effectively is provided.

Machine learning (ML) offers a promising solution to pathloss prediction. However, its effectiveness can be degraded by the limited availability of data. To alleviate these challenges, this paper introduces a novel simulation-enhanced data augmentation method for ML pathloss prediction. Our method integrates synthetic data generated from a cellular coverage simulator and independently collected real-world datasets. These datasets were collected through an extensive measurement campaign in different environments, including farms, hilly terrains, and residential areas. This comprehensive data collection provides vital ground truth for model training. A set of channel features was engineered, including geographical attributes derived from LiDAR datasets. These features were then used to train our prediction model, incorporating the highly efficient and robust gradient boosting ML algorithm, CatBoost. The integration of synthetic data, as demonstrated in our study, significantly improves the generalizability of the model in different environments, achieving a remarkable improvement of approximately 12dB in terms of mean absolute error for the best-case scenario. Moreover, our analysis reveals that even a small fraction of measurements added to the simulation training set, with proper data balance, can significantly enhance the model's performance., Comment: 6 pages, 5 figures, Accepted at ICC 2024

We introduce calamanCy, an open-source toolkit for constructing natural language processing (NLP) pipelines for Tagalog. It is built on top of spaCy, enabling easy experimentation and integration with other frameworks. calamanCy addresses the development gap by providing a consistent API for building NLP applications and offering general-purpose multitask models with out-of-the-box support for dependency parsing, parts-of-speech (POS) tagging, and named entity recognition (NER). calamanCy aims to accelerate the progress of Tagalog NLP by consolidating disjointed resources in a unified framework. The calamanCy toolkit is available on GitHub: https://github.com/ljvmiranda921/calamanCy., Comment: To be published in The Third Workshop for NLP-OSS at EMNLP 2023

We present the development of a Named Entity Recognition (NER) dataset for Tagalog. This corpus helps fill the resource gap present in Philippine languages today, where NER resources are scarce. The texts were obtained from a pretraining corpora containing news reports, and were labeled by native speakers in an iterative fashion. The resulting dataset contains ~7.8k documents across three entity types: Person, Organization, and Location. The inter-annotator agreement, as measured by Cohen's $\kappa$, is 0.81. We also conducted extensive empirical evaluation of state-of-the-art methods across supervised and transfer learning settings. Finally, we released the data and processing code publicly to inspire future work on Tagalog NLP., Comment: To be published in The First Workshop for Southeast Asian Language Processing 2023 at IJCNLP-AACL

On 26 September 2022, the Double Asteroid Redirection Test (DART) spacecraft impacted Dimorphos, the satellite of binary near-Earth asteroid (65803) Didymos. This demonstrated the efficacy of a kinetic impactor for planetary defense by changing the orbital period of Dimorphos by 33 minutes (Thomas et al. 2023). Measuring the period change relied heavily on a coordinated campaign of lightcurve photometry designed to detect mutual events (occultations and eclipses) as a direct probe of the satellite's orbital period. A total of 28 telescopes contributed 224 individual lightcurves during the impact apparition from July 2022 to February 2023. We focus here on decomposable lightcurves, i.e. those from which mutual events could be extracted. We describe our process of lightcurve decomposition and use that to release the full data set for future analysis. We leverage these data to place constraints on the post-impact evolution of ejecta. The measured depths of mutual events relative to models showed that the ejecta became optically thin within the first ~1 day after impact, and then faded with a decay time of about 25 days. The bulk magnitude of the system showed that ejecta no longer contributed measurable brightness enhancement after about 20 days post-impact. This bulk photometric behavior was not well represented by an HG photometric model. An HG1G2 model did fit the data well across a wide range of phase angles. Lastly, we note the presence of an ejecta tail through at least March 2023. Its persistence implied ongoing escape of ejecta from the system many months after DART impact., Comment: 52 pages, 5 tables, 9 figures, accepted to PSJ

Dual-functional radar-communication (DFRC) is a promising technology where radar and communication functions operate on the same spectrum and hardware. In this paper, we propose an algorithm for designing constant modulus waveforms for DFRC systems. Particularly, we jointly optimize the correlation properties and the spatial beam pattern. For communication, we employ constructive interference-based block-level precoding (CI-BLP) to exploit distortion due to multi-user and radar transmission. We propose a majorization-minimization (MM)-based solution to the formulated problem. To accelerate convergence, we propose an improved majorizing function that leverages a novel diagonal matrix structure. We then evaluate the performance of the proposed algorithm through rigorous simulations. Simulation results demonstrate the effectiveness of the proposed approach and the proposed majorizer., Comment: Accepted to IEEE International Conference on Communications (ICC) 2024

We investigate the propagation of acoustic singular surfaces, specifically, linear shock waves and nonlinear acceleration waves, in a class of inhomogeneous gases whose ambient mass density varies exponentially. Employing the mathematical tools of singular surface theory, we first determine the evolution of both the jump amplitudes and the locations/velocities of their associated wave-fronts, along with a variety of related analytical results. We then turn to what have become known as Krylov subspace spectral (KSS) methods to numerically simulate the evolution of the full waveforms under consideration. These are not only performed quite efficiently, since KSS allows the use of `large' CFL numbers, but also quite accurately, in the sense of capturing theoretically-predicted features of the solution profiles more faithfully than other time-stepping methods, since KSS customizes the computation of the components of the solution corresponding to the different frequencies involved. The presentation concludes with a listing of possible, acoustics-related, follow-on studies.

Accurate extraction of mitral valve shape from clinical tomographic images acquired in patients has proven useful for planning surgical and interventional mitral valve treatments. However, manual extraction of the mitral valve shape is laborious, and the existing automatic extraction methods have not been sufficiently accurate. In this paper, we propose a fully automated method of extracting mitral valve shape from computed tomography (CT) images for the all phases of the cardiac cycle. This method extracts the mitral valve shape based on DenseNet using both the original CT image and the existence probability maps of the mitral valve area inferred by U-Net as input. A total of 1585 CT images from 204 patients with various cardiac diseases including mitral regurgitation (MR) were collected and manually annotated for mitral valve region. The proposed method was trained and evaluated by 10-fold cross validation using the collected data and was compared with the method without the existence probability maps. The mean error of shape extraction error in the proposed method is 0.88 mm, which is an improvement of 0.32 mm compared with the method without the existence probability maps., Comment: 15 pages, 6 figure, 3 table. changed title, modified taipo

Emerging wireless applications are requiring ever more accurate location-positioning from sensor measurements. In this paper, we develop sensor selection strategies for 3D wireless positioning based on time of arrival (TOA) and received signal strength (RSS) measurements to handle two distinct scenarios: (i) known approximated target location, for which we conduct dynamic sensor selection to minimize the positioning error; and (ii) unknown approximated target location, in which the worst-case positioning error is minimized via robust sensor selection. We derive expressions for the Cram\'er-Rao lower bound (CRLB) as a performance metric to quantify the positioning accuracy resulted from selected sensors. For dynamic sensor selection, two greedy selection strategies are proposed, each of which exploits properties revealed in the derived CRLB expressions. These selection strategies are shown to strike an efficient balance between computational complexity and performance suboptimality. For robust sensor selection, we show that the conventional convex relaxation approach leads to instability, and then develop three algorithms based on (i) iterative convex optimization (ICO), (ii) difference of convex functions programming (DCP), and (iii) discrete monotonic optimization (DMO). Each of these strategies exhibits a different tradeoff between computational complexity and optimality guarantee. Simulation results show that the proposed sensor selection strategies provide significant improvements in terms of accuracy and/or complexity compared to existing sensor selection methods., Comment: This paper has been accepted to IEEE Transactions on Vehicular Technology for future publication

The preferential siting of the locations of monitors of hazardous environmental fields can lead to the serious underestimation of the impacts of those fields. In particular, human health effects can be severely underestimated when standard statistical are applied without appropriate adjustment. This report describes an extensive analysis of the siting of monitors for a network that measures air pollution PM10 in California's South Coast Air Basin SOCAB. That analysis uses EPA data collected during the 1986 to 2019 period. Background descriptions, including those published by the US EPA are provided. The analysis uses a very general and fast Monte Carlo test for preferential sampling developed by Dr Joe Watson, which confirms that the sites were preferentially sited, as would be expected, given the intended purpose of the network to detect noncompliance with air quality standards. Our findings demonstrate both the value of that algorithm for application where where such background knowledge is not available, and hence to situations in which standard statistical tools require modification., Comment: 65 pages and 26 figures

This work is motivated by the ECMWF CAMS reanalysis data, a valuable resource for researchers in environmental-related areas, as they contain the most updated atmospheric composition information on a global scale. Unlike observational data obtained from monitoring equipment, such reanalysis data are produced by computers via a 4D-Var data assimilation mechanism, thus their stochastic property remains largely unclear. Such lack of knowledge in turn limits their utility scope and hinders them from wider and more flexible statistical usages, especially spatio-temporal modelling except for uncertainty quantification and data fusion. Therefore, this paper studies the stochastic property of these reanalysis outputs data. We used measure theory and proved the tangible existence of spatial and temporal stochasticity associated with these reanalysis data and revealed that they are essentially realisations from digitised versions of real-world hidden spatial and/or temporal stochastic processes. This means we can treat the reanalysis outputs data the same as observational data in practice and thus ensures more flexible spatio-temporal stochastic methodologies apply to them. We also objectively analysed different types of errors in the reanalysis data and deciphered their mutual dependence/independence, which together give clear and definite guidance on the modelling of error terms. The results of this study also serve as a solid stepping stone for spatio-temporal modellers and environmental AI researchers to embark on their research directly with these reanalysis outputs data using stochastic models.