Search

Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better on specialized tasks. Inspired by this insight, we developed INDUS, a comprehensive suite of LLMs tailored for the closely-related domains of Earth science, biology, physics, heliophysics, planetary sciences and astrophysics, and trained using curated scientific corpora drawn from diverse data sources. The suite of models include: (1) an encoder model trained using domain-specific vocabulary and corpora to address NLP tasks, (2) a contrastive-learning based text embedding model trained using a diverse set of datasets to address information retrieval tasks and (3) smaller versions of these models created using knowledge distillation for applications which have latency or resource constraints. We also created three new scientific benchmark datasets, CLIMATE-CHANGE NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR (IR) to accelerate research in these multi-disciplinary fields. We show that our models outperform both general-purpose (RoBERTa) and domain-specific (SCIBERT) encoders on these new tasks as well as existing tasks in the domains of interest. Furthermore, we demonstrate the use of these models in two industrial settings -- as a retrieval model for large-scale vector search applications and in automatic content tagging systems., Comment: EMNLP 2024 (Industry Track)

The paper presents a novel Auto Language Prediction Dictionary Capsule (ALPDC) framework for language prediction and machine translation. The model uses a combination of neural networks and symbolic representations to predict the language of a given input text and then translate it to a target language using pre-built dictionaries. This research work also aims to translate the text of various languages to its literal meaning in English. The proposed model achieves state-of-the-art results on several benchmark datasets and significantly improves translation accuracy compared to existing methods. The results show the potential of the proposed method for practical use in multilingual communication and natural language processing tasks., Comment: 21 Pages

It is widely believed that the origin of a significant cause for the voltage and capacity fading observed in lithium (Li)-ion batteries is related to structural modifications occurring in the cathode material during the Li-ion insertion/de-insertion process. The Li-ion insertion/de-insertion mechanism and the resulting structural changes are known to exert a severe strain on the lattice, and consequently leading to performance degradation. Here, with a view to shed more light on the effect of such strain on the structural properties of the cathode material, we have systematically investigated the pressure dependence of structural and transport properties of an LixCoO2 single crystal, grown using 5% excess Li in the precursors. Ambient pressure synchrotron diffraction on these crystals reveals that, the excess Li during the growth, has facilitated the stabilization of a layered rhombohedral phase (space group R3m) as well as a disordered rock-salt phase (space group Fm3m). The volume fraction of the rhombohedral and cubic phase is 60:40, respectively, which remains unchanged up to 10.6 GPa. No structural phase transition has been observed up to 10.6 GPa. An increase in resistance with a decrease in temperature has revealed the semi-metallic nature of the sample. Further, the application of hydrostatic pressure up to 2.8 GPa shows the enhancement of semi-metallic nature. The obtained experimental results can be qualitatively explained via density functional theory (DFT) and thermodynamics modelling. The calculated density of states was reduced, and the activation energy was increased by applied pressure. Our investigations indicate a significant phase stability of the mixed phase crystals under externally applied high pressure and thus suggest the possible use of such mixed phase materials as a cathode in lithium-ion batteries., Comment: 24 pages and 10 figures

Accurate detection of inundated water extents during flooding events is crucial in emergency response decisions and aids in recovery efforts. Satellite Remote Sensing data provides a global framework for detecting flooding extents. Specifically, Sentinel-1 C-Band Synthetic Aperture Radar (SAR) imagery has proven to be useful in detecting water bodies due to low backscatter of water features in both co-polarized and cross-polarized SAR imagery. However, increased backscatter can be observed in certain flooded regions such as presence of infrastructure and trees - rendering simple methods such as pixel intensity thresholding and time-series differencing inadequate. Machine Learning techniques has been leveraged to precisely capture flood extents in flooded areas with bumps in backscatter but needs high amounts of labelled data to work desirably. Hence, we created a labeled known water body extent and flooded area extents during known flooding events covering about 36,000 sq. kilometers of regions within mainland U.S and Bangladesh. Further, We also leveraged citizen science by open-sourcing the dataset and hosting an open competition based on the dataset to rapidly prototype flood extent detection using community generated models. In this paper we present the information about the dataset, the data processing pipeline, a baseline model and the details about the competition, along with discussion on winning approaches. We believe the dataset adds to already existing datasets based on Sentinel-1C SAR data and leads to more robust modeling of flood extents. We also hope the results from the competition pushes the research in flood extent detection further., Comment: 13 pages in AGU format, 7 figures

Significant progress in the development of highly adaptable and reusable Artificial Intelligence (AI) models is expected to have a significant impact on Earth science and remote sensing. Foundation models are pre-trained on large unlabeled datasets through self-supervision, and then fine-tuned for various downstream tasks with small labeled datasets. This paper introduces a first-of-a-kind framework for the efficient pre-training and fine-tuning of foundational models on extensive geospatial data. We have utilized this framework to create Prithvi, a transformer-based geospatial foundational model pre-trained on more than 1TB of multispectral satellite imagery from the Harmonized Landsat-Sentinel 2 (HLS) dataset. Our study demonstrates the efficacy of our framework in successfully fine-tuning Prithvi to a range of Earth observation tasks that have not been tackled by previous work on foundation models involving multi-temporal cloud gap imputation, flood mapping, wildfire scar segmentation, and multi-temporal crop segmentation. Our experiments show that the pre-trained model accelerates the fine-tuning process compared to leveraging randomly initialized weights. In addition, pre-trained Prithvi compares well against the state-of-the-art, e.g., outperforming a conditional GAN model in multi-temporal cloud imputation by up to 5pp (or 5.7%) in the structural similarity index. Finally, due to the limited availability of labeled data in the field of Earth observation, we gradually reduce the quantity of available labeled data for refining the model to evaluate data efficiency and demonstrate that data can be decreased significantly without affecting the model's accuracy. The pre-trained 100 million parameter model and corresponding fine-tuning workflows have been released publicly as open source contributions to the global Earth sciences community through Hugging Face.

Reverse engineering of FPGA based designs from the flattened LUT level netlist to high level RTL helps in verification of the design or in understanding legacy designs. We focus on flattened netlists for FPGA devices from Xilinx 7 series and Zynq 7000. We propose a design element grouping algorithm that makes use of the location information of the elements on the physical device after place and route. The proposed grouping algorithm gives clusters with average NMI of 0.73 for groupings including all element types. The benchmarks chosen include a range of designs from communication, arithmetic units, processors and DSP processing units., Comment: Paper accepted into proceedings of VLSID2023 conference

Reverse engineering of FPGA designs from bitstreams to RTL models aids in understanding the high level functionality of the design and for validating and reconstructing legacy designs. Fast carry-chains are commonly used in synthesis of operators in FPGA designs. We propose a method to detect word-level structures by analyzing these carry-chains in LUT (Look-Up Table) level netlists. We also present methods to adapt existing techniques to identify combinational operations and sequential modules in ASIC netlists to LUT netlists. All developed and adapted techniques are consolidated into an integrated tool-chain to aid in reverse engineering of word-level designs from LUT-level netlists. When evaluated on a set of real-world designs, the tool-chain infers 34\% to 100\% of the elements in the netlist to be part of a known word-level operation or a known sequential module., Comment: 8 pages, 6 figures, Accepted to appear in ISQED 2023 conference