Search

Data curation is a field with origins in librarianship and archives, whose scholarship and thinking on data issues go back centuries, if not millennia. The field of machine learning is increasingly observing the importance of data curation to the advancement of both applications and fundamental understanding of machine learning models - evidenced not least by the creation of the Datasets and Benchmarks track itself. This work provides an analysis of dataset development practices at NeurIPS through the lens of data curation. We present an evaluation framework for dataset documentation, consisting of a rubric and toolkit developed through a literature review of data curation principles. We use the framework to assess the strengths and weaknesses in current dataset development practices of 60 datasets published in the NeurIPS Datasets and Benchmarks track from 2021-2023. We summarize key findings and trends. Results indicate greater need for documentation about environmental footprint, ethical considerations, and data management. We suggest targeted strategies and resources to improve documentation in these areas and provide recommendations for the NeurIPS peer-review process that prioritize rigorous data curation in ML. Finally, we provide results in the format of a dataset that showcases aspects of recommended data curation practices. Our rubric and results are of interest for improving data curation practices broadly in the field of ML as well as to data curation and science and technology studies scholars studying practices in ML. Our aim is to support continued improvement in interdisciplinary research on dataset practices, ultimately improving the reusability and reproducibility of new datasets and benchmarks, enabling standardized and informed human oversight, and strengthening the foundation of rigorous and responsible ML research., Comment: Accepted in NeurIPS Datasets & Benchmarks track 2024

Recent advancements in large language models (LLMs) have significantly enhanced their ability to understand both natural language and code, driving their use in tasks like natural language-to-code (NL2Code) and code summarization. However, LLMs are prone to hallucination-outputs that stray from intended meanings. Detecting hallucinations in code summarization is especially difficult due to the complex interplay between programming and natural languages. We introduce a first-of-its-kind dataset with $\sim$10K samples, curated specifically for hallucination detection in code summarization. We further propose a novel Entity Tracing Framework (ETF) that a) utilizes static program analysis to identify code entities from the program and b) uses LLMs to map and verify these entities and their intents within generated code summaries. Our experimental analysis demonstrates the effectiveness of the framework, leading to a 0.73 F1 score. This approach provides an interpretable method for detecting hallucinations by grounding entities, allowing us to evaluate summary accuracy., Comment: 11 pages, 6 Figures, 5 Tables

This paper describes NLIP Lab's multilingual machine translation system for the WAT24 shared task on multilingual Indic MT task for 22 scheduled languages belonging to 4 language families. We explore pre-training for Indic languages using alignment agreement objectives. We utilize bi-lingual dictionaries to substitute words from source sentences. Furthermore, we fine-tuned language direction-specific multilingual translation models using small and high-quality seed data. Our primary submission is a 243M parameters multilingual translation model covering 22 Indic languages. In the IN22-Gen benchmark, we achieved an average chrF++ score of 46.80 and 18.19 BLEU score for the En-Indic direction. In the Indic-En direction, we achieved an average chrF++ score of 56.34 and 30.82 BLEU score. In the In22-Conv benchmark, we achieved an average chrF++ score of 43.43 and BLEU score of 16.58 in the En-Indic direction, and in the Indic-En direction, we achieved an average of 52.44 and 29.77 for chrF++ and BLEU respectively. Our model\footnote{Our code and models are available at \url{https://github.com/maharajbrahma/WAT2024-MultiIndicMT}} is competitive with IndicTransv1 (474M parameter model)., Comment: WMT 24 WAT Shared Task IndicMultiMT (Best System)

We investigate the universe's evolution within the framework of Rastall gravity, which is an extension of the standard $\Lambda$CDM model. Utilizing a linear parametrization of the Equation of State (EoS) in a Friedmann-Lema\^{\i}tre-Robertson-Walker (FLRW) background, we constrain the model parameters through analysis of cosmic chronometers (CC), Pantheon, Gold, Gamma Ray Burst (GRB), and Baryon Acoustic Oscillations (BAO) datasets, as well as their joint analysis, under $1\sigma$ and $2\sigma$ confidence levels, considering the Rastall parameter $\lambda$. The constrained parameters are then used to compare our model with the standard $\Lambda$CDM model. Our findings include a detailed examination of the model's physical interpretations and demonstrate the potential for an accelerating universe expansion in later times, aligning with the observed behavior of dark energy., Comment: 21 pages, 19 figures, accepted for publication in Physica Scripta

The EHT observation revealed event horizon-scale images of the supermassive black holes Sgr A* and M87* and these results are consistent with the shadow of a Kerr black hole as predicted by general relativity. However, Kerr-like rotating black holes in modified gravity theories can not ruled out, as they provide a crucial testing ground for these theories through EHT observations. It motivates us to investigate the Bumblebee theory, a vector-tensor extension of the Einstein-Maxwell theory that permits spontaneous symmetry breaking, resulting in the field acquiring a vacuum expectation value and introducing Lorentz violation. We present rotating black holes within this bumblebee gravity model, which includes an additional parameter $\ell$ alongside the mass $M$ and spin parameter $a$ - namely RBHBG. Unlike the Kerr black hole, an extremal RBHBG, for $\ell<0$, refers to a black hole with angular momentum $a>M$. We derive an analytical formula necessary for the shadow of our rotating black holes, then visualize them with varying parameters $a$ and $\ell$, and also estimate the black hole parameters using shadow observables viz. shadow radius $R_s$, distortion $\delta_s$, shadow area $A$ and oblateness $D$ using two well-known techniques. We find that $\ell$ incrementally increases the shadow size and causes more significant deformation while decreasing the event horizon area. Remarkably, an increase in $\ell$ enlarges the shadow radius irrespective of spin or inclination angle $\theta_0$., Comment: 17 Pages, 13 Figures and 6 Tables

In this paper, we describe our system for the WMT 24 shared task of Low-Resource Indic Language Translation. We consider eng $\leftrightarrow$ {as, kha, lus, mni} as participating language pairs. In this shared task, we explore the finetuning of a pre-trained model motivated by the pre-trained objective of aligning embeddings closer by alignment augmentation \cite{lin-etal-2020-pre} for 22 scheduled Indian languages. Our primary system is based on language-specific finetuning on a pre-trained model. We achieve chrF2 scores of 50.6, 42.3, 54.9, and 66.3 on the official public test set for eng$\rightarrow$as, eng$\rightarrow$kha, eng$\rightarrow$lus, eng$\rightarrow$mni respectively. We also explore multilingual training with/without language grouping and layer-freezing. Our code, models, and generated translations are available here: https://github.com/pramitsahoo/WMT2024-LRILT., Comment: WMT2024 INDICMT Shared Task

We provide a statistical analysis of the recent controversy between Vladimir Kramnik (ex chess world champion) and Hikaru Nakamura. Hikaru Nakamura is a chess prodigy and a five-time United States chess champion. Kramnik called into question Nakamura's 45.5 out of 46 win streak in an online blitz contest at chess.com. We assess the weight of evidence using a priori assessment of Viswanathan Anand and the streak evidence. Based on this evidence, we show that Nakamura has a 99.6 percent chance of not cheating. We study the statistical fallacies prevalent in both their analyses. On the one hand Kramnik bases his argument on the probability of such a streak is very small. This falls precisely into the Prosecutor's Fallacy. On the other hand, Nakamura tries to refute the argument using a cherry-picking argument. This violates the likelihood principle. We conclude with a discussion of the relevant statistical literature on the topic of fraud detection and the analysis of streaks in sports data.

Large Language Models (LLMs) are increasingly integrated into diverse applications. The rapid evolution of LLMs presents opportunities for developers to enhance applications continuously. However, this constant adaptation can also lead to performance regressions during model migrations. While several interactive tools have been proposed to streamline the complexity of prompt engineering, few address the specific requirements of regression testing for LLM Migrations. To bridge this gap, we introduce RETAIN (REgression Testing guided LLM migrAtIoN), a tool designed explicitly for regression testing in LLM Migrations. RETAIN comprises two key components: an interactive interface tailored to regression testing needs during LLM migrations, and an error discovery module that facilitates understanding of differences in model behaviors. The error discovery module generates textual descriptions of various errors or differences between model outputs, providing actionable insights for prompt refinement. Our automatic evaluation and empirical user studies demonstrate that RETAIN, when compared to manual evaluation, enabled participants to identify twice as many errors, facilitated experimentation with 75% more prompts, and achieves 12% higher metric scores in a given time frame., Comment: Preprint

Recent advancements in retrieval-augmented generation (RAG) have demonstrated impressive performance in the question-answering (QA) task. However, most previous works predominantly focus on text-based answers. While some studies address multimodal data, they still fall short in generating comprehensive multimodal answers, particularly for explaining concepts or providing step-by-step tutorials on how to accomplish specific goals. This capability is especially valuable for applications such as enterprise chatbots and settings such as customer service and educational systems, where the answers are sourced from multimodal data. In this paper, we introduce a simple and effective framework named MuRAR (Multimodal Retrieval and Answer Refinement). MuRAR enhances text-based answers by retrieving relevant multimodal data and refining the responses to create coherent multimodal answers. This framework can be easily extended to support multimodal answers in enterprise chatbots with minimal modifications. Human evaluation results indicate that multimodal answers generated by MuRAR are more useful and readable compared to plain text answers., Comment: Preprint

The development of conversational AI assistants is an iterative process with multiple components. As such, the evaluation and continual improvement of these assistants is a complex and multifaceted problem. This paper introduces the challenges in evaluating and improving a generative AI assistant for enterprises, which is under active development, and how we address these challenges. We also share preliminary results and discuss lessons learned., Comment: Accepted to DaSH Workshop at NAACL 2024

Mental disorders pose a global challenge, aggravated by the shortage of qualified mental health professionals. Mental disorder prediction from social media posts by current LLMs is challenging due to the complexities of sequential text data and the limited context length of language models. Current language model-based approaches split a single data instance into multiple chunks to compensate for limited context size. The predictive model is then applied to each chunk individually, and the most voted output is selected as the final prediction. This results in the loss of inter-post dependencies and important time variant information, leading to poor performance. We propose a novel framework which first compresses the large sequence of chronologically ordered social media posts into a series of numbers. We then use this time variant representation for mental disorder classification. We demonstrate the generalization capabilities of our framework by outperforming the current SOTA in three different mental conditions: depression, self-harm, and anorexia, with an absolute improvement of 5% in the F1 score. We investigate the situation where current data instances fall within the context length of language models and present empirical results highlighting the importance of temporal properties of textual data. Furthermore, we utilize the proposed framework for a cross-domain study, exploring commonalities across disorders and the possibility of inter-domain data usage., Comment: RK and KM contributed equally to this work, 15 pages, 5 figures, 9 table

Studies of dataset development in machine learning call for greater attention to the data practices that make model development possible and shape its outcomes. Many argue that the adoption of theory and practices from archives and data curation fields can support greater fairness, accountability, transparency, and more ethical machine learning. In response, this paper examines data practices in machine learning dataset development through the lens of data curation. We evaluate data practices in machine learning as data curation practices. To do so, we develop a framework for evaluating machine learning datasets using data curation concepts and principles through a rubric. Through a mixed-methods analysis of evaluation results for 25 ML datasets, we study the feasibility of data curation principles to be adopted for machine learning data work in practice and explore how data curation is currently performed. We find that researchers in machine learning, which often emphasizes model development, struggle to apply standard data curation principles. Our findings illustrate difficulties at the intersection of these fields, such as evaluating dimensions that have shared terms in both fields but non-shared meanings, a high degree of interpretative flexibility in adapting concepts without prescriptive restrictions, obstacles in limiting the depth of data curation expertise needed to apply the rubric, and challenges in scoping the extent of documentation dataset creators are responsible for. We propose ways to address these challenges and develop an overall framework for evaluation that outlines how data curation concepts and methods can inform machine learning data practices., Comment: In ACM Conference on Fairness, Accountability, and Transparency 2024. ACM, Rio de Janeiro, Brazil

This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose $200+$ concrete research questions.

Caseworkers in the child welfare (CW) sector use predictive decision-making algorithms built on risk assessment (RA) data to guide and support CW decisions. Researchers have highlighted that RAs can contain biased signals which flatten CW case complexities and that the algorithms may benefit from incorporating contextually rich case narratives, i.e. - casenotes written by caseworkers. To investigate this hypothesized improvement, we quantitatively deconstructed two commonly used RAs from a United States CW agency. We trained classifier models to compare the predictive validity of RAs with and without casenote narratives and applied computational text analysis on casenotes to highlight topics uncovered in the casenotes. Our study finds that common risk metrics used to assess families and build CWS predictive risk models (PRMs) are unable to predict discharge outcomes for children who are not reunified with their birth parent(s). We also find that although casenotes cannot predict discharge outcomes, they contain contextual case signals. Given the lack of predictive validity of RA scores and casenotes, we propose moving beyond quantitative risk assessments for public sector algorithms and towards using contextual sources of information such as narratives to study public sociotechnical systems.

This follow-on work investigates the effect of temperature on the frequency limit of 2-dimensional electron gas (2DEG) Hall-effect sensors., Comment: 4 pages

We investigate the status of the gravitational arrow of time in the case of a spherical collapse of a fluid that conducts heat and radiates energy. In particular, we examine the results obtained by W. B. Bonnor in his 1985 paper where he found that the gravitational arrow of time was opposite to the thermodynamic arrow of time. The measure of gravitational epoch function $P$ used by Bonnor was given by the ratio of the Weyl square to the Ricci square. In this paper, we have assumed the measure of gravitational entropy $P_{1}$ to be given by the ratio of the Weyl scalar to the Kretschmann scalar. Our analysis indicates that Bonnor's result seems to be validated, i.e., the gravitational arrow and the thermodynamic arrow of time point in opposite directions. This strengthens the opinion that the Weyl proposal of gravitational entropy applies only to the universe as a whole (provided that we exclude the white holes)., Comment: 19 pages, accepted for publication in CQG

The Milky Way galaxy is estimated to be home to ten million to a billion stellar-mass black holes (BHs). Accurately determining this number and distribution of BH masses can provide crucial information about the processes involved in BH formation, the possibility of the existence of primordial BHs, and interpreting gravitational wave (GW) signals detected in LIGO-VIRGO-KAGRA. Sahu et al. recently confirmed one isolated stellar-mass BH in our galaxy using astrometric microlensing. This work proposes a novel method to identify such BHs using the gravitational analog of the Gertsenshtein-Zel'dovich (GZ) effect. We explicitly demonstrate the generation of GWs when a kilohertz(kHz) electromagnetic (EM) pulse from a pulsar is intervened by a spherically symmetric compact object situated between the pulsar and Earth. Specifically, we show that the curvature of spacetime acts as the catalyst, akin to the magnetic field in the GZ effect. Using the covariant semi-tetrad formalism, we quantify the GW generated from the EM pulse through the Regge-Wheeler tensor and express the amplitude of the generated GW in terms of the EM energy and flux. We demonstrate how GW detectors can detect stellar-mass BHs by considering known pulsars within our galaxy. This approach has a distinct advantage in detecting stellar mass BHs at larger distances since the GW amplitude falls as $1/r$., Comment: 24 pages; 2 figures; comments welcome

This paper investigates the flat Friedmann-Lema$\hat{\imath}$tre-Robertson-Walker (FLRW) cosmological model using a suitable parameterization represented as a differential equation concerning the energy density of the scalar field, $\rho_{\phi}$, in the context of $f(R,T)$ gravity theory. This parameterization is known as the Energy Density Scalar Field Differential Equation (EDSFD) parametrization. It results in a solution of the Hubble parameter containing four model parameters, namely, $\Omega_{m0},\Omega_{\phi 0}, H_0,$ and $\alpha$. To constrain the model parameters, $77$ data points from the Hubble dataset, $1048$ points from the Pantheon dataset, and $6$ data points from BAO are used. Using the constrained values, we analyze and compare our model with the standard $\Lambda$CDM model. The evolution of the physical parameters, which includes the deceleration parameter, density parameter, Equation of State (EoS) for Dark Energy, and $Om(z)$ diagnostic, are discussed., Comment: 16 pages, 18 figures

We develop a static charged stellar model in $f(R,T)$ gravity where the modification is assumed to be linear in $T$ which is the trace of the energy momentum tensor. The exterior spacetime of the charged object is described by the Reissner-Nordstr\"om metric. The interior solution is obtained by invoking the Buchdahl-Vaidya-Tikekar ansatz, for the metric potential $g_{rr}$, which has a clear geometric interpretation. A detailed physical analysis of the model clearly shows distinct physical features of the resulting stellar configuration under such a modification. We find the maximum compactness bound for such a class of compact stars which is a generalization of the Buchdahl bound for a charged sphere described in $f(R,T)$ gravity. Our result shows physical behaviour that is distinct from general relativity.

Artificial Intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increases in capabilities and autonomy may soon massively amplify AI's impact, with risks that include large-scale social harms, malicious uses, and an irreversible loss of human control over autonomous AI systems. Although researchers have warned of extreme risks from AI, there is a lack of consensus about how exactly such risks arise, and how to manage them. Society's response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems. In this short consensus paper, we describe extreme risks from upcoming, advanced AI systems. Drawing on lessons learned from other safety-critical technologies, we then outline a comprehensive plan combining technical research and development with proactive, adaptive governance mechanisms for a more commensurate preparation., Comment: Published in Science: https://www.science.org/doi/10.1126/science.adn0117

We demonstrate that LLMs may learn indicators of document usefulness and modulate their updates accordingly. We introduce random strings ("tags") as indicators of usefulness in a synthetic fine-tuning dataset. Fine-tuning on this dataset leads to implicit meta-learning (IML): in further fine-tuning, the model updates to make more use of text that is tagged as useful. We perform a thorough empirical investigation of this phenomenon, finding (among other things) that (i) it occurs in both pretrained LLMs and those trained from scratch, as well as on a vision task, and (ii) larger models and smaller batch sizes tend to give more IML. We also use probing to examine how IML changes the way models store knowledge in their parameters. Finally, we reflect on what our results might imply about capabilities, risks, and controllability of future AI systems. Our code can be found at https://github.com/krasheninnikov/internalization.

Exact solutions describing rotating black holes can provide significant opportunities for testing modified theories of gravity, which are motivated by the challenges posed by dark energy and dark matter. Starting with a spherical Kiselev black hole as a seed metric, we construct rotating Kiselev black holes within the $f(R,T)$ gravity framework using the revised Newman-Janis algorithm - the $f(R,T)$ gravity-motivated rotating Kiselev black holes (FRKBH), which encompasses, as exceptional cases, Kerr ($K=0$) and Kerr-Newman ($K=Q^2$) black holes. These solutions give rise to distinct classes of black holes surrounded by fluids while considering specific values of the equation-of-state parameter, $w$, for viable choices for the $f(R,T)$ function. From the parameter space or domain of existence of black holes defined by $a$ and $\gamma$ for FKRBH, we discover that when $a_1a_2$), we encounter two distinct critical values $\gamma=\gamma_{E1}, \; \gamma_{E2}$ with $\gamma_{E1}>\gamma_{E2}$ (or $\gamma=\gamma_{E3},\; \gamma_{E4}$ with $\gamma_{E3}>\gamma_{E4}$. We delve into the horizon and global structure of FKRBH spacetimes and examine their dependence on parameters $w$ and $\gamma$. This exploration is motivated by the remarkable effects of $f(R,T)$ gravity, which gives rise to diverse and intricate spacetime structures within the domain where black holes exist., Comment: 17 Pages, 11 Figures, 1 Table

We propose a neural network-based approach to calculate the value of a chess square-piece combination. Our model takes a triplet (Color, Piece, Square) as an input and calculates a value that measures the advantage/disadvantage of having this piece on this square. Our methods build on recent advances in chess AI, and can accurately assess the worth of positions in a game of chess. The conventional approach assigns fixed values to pieces $(\symking=\infty, \symqueen=9, \symrook=5, \symbishop=3, \symknight=3, \sympawn=1)$. We enhance this analysis by introducing marginal valuations. We use deep Q-learning to estimate the parameters of our model. We demonstrate our method by examining the positioning of Knights and Bishops, and also provide valuable insights into the valuation of pawns. Finally, we conclude by suggesting potential avenues for future research.

This is an important and natural question as the spacetime shear, inhomogeneity and tidal effects are all intertwined via the Einstein field equations. However, as we show in this paper, such scenarios are possible for limited classes of equations of state that are solutions to a highly non-linear and fourth order differential equation. To show this, we use a covariant semitetrad spacetime decomposition and present a novel geometrical classification of shear-free Locally Rotationally Symmetric (LRS-II) perfect fluid self-gravitating systems, in terms of the covariantly defined fluid acceleration and the fluid expansion. Noteworthily, we deduce the governing differential equation that gives the possible limited equations of state of matter., Comment: 8 pages, 2 figures

In this work, we experimentally investigate the frequency limit of Hall effect sensor designs based on a 2 dimensional electron gas (2DEG) gallium arsenide/aluminum gallium arsenide (GaAs/AlGaAs) heterostructure. The frequency limit is measured and compared for four GaAs/AlGaAs Hall effect sensor designs where the Ohmic contact length (contact geometry) is varied across the four devices. By varying the geometry, the trade-off in sensitivity and frequency limit is explored and the underlying causes of the frequency limit from the resistance and capacitance perspective is investigated. Current spinning, the traditional method to remove offset noise, imposes a practical frequency limit on Hall effect sensors. The frequency limit of the Hall effect sensor, without current spinning, is significantly higher. Wide-frequency Hall effect sensors can measure currents in power electronics that operate at higher frequencies is one such application., Comment: Hall effect sensors, magnetic sensing, frequency limit, 2DEGs