Search

Building semantic 3D maps is valuable for searching for objects of interest in offices, warehouses, stores, and homes. We present a mapping system that incrementally builds a Language-Embedded Gaussian Splat (LEGS): a detailed 3D scene representation that encodes both appearance and semantics in a unified representation. LEGS is trained online as a robot traverses its environment to enable localization of open-vocabulary object queries. We evaluate LEGS on 4 room-scale scenes where we query for objects in the scene to assess how LEGS can capture semantic meaning. We compare LEGS to LERF and find that while both systems have comparable object query success rates, LEGS trains over 3.5x faster than LERF. Results suggest that a multi-camera setup and incremental bundle adjustment can boost visual reconstruction quality in constrained robot trajectories, and suggest LEGS can localize open-vocabulary and long-tail object queries with up to 66% accuracy.

We study the well-known coded caching problem in an online learning framework, wherein requests arrive sequentially, and an online policy can update the cache contents based on the history of requests seen thus far. We introduce a caching policy based on the Follow-The-Perturbed-Leader principle and show that for any time horizon T and any request sequence, it achieves a sub-linear regret of \mathcal{O}(\sqrt(T) ) with respect to an oracle that knows the request sequence beforehand. Our study marks the first examination of adversarial regret in the coded caching setup. Furthermore, we also address the issue of switching cost by establishing an upper bound on the expected number of cache updates made by our algorithm under unrestricted switching and also provide an upper bound on the regret under restricted switching when cache updates can only happen in a pre-specified subset of timeslots. Finally, we validate our theoretical insights with numerical results using a real-world dataset

Differential Dynamic Programming (DDP) is one of the indirect methods for solving an optimal control problem. Several extensions to DDP have been proposed to add stagewise state and control constraints, which can mainly be classified as augmented lagrangian methods, active set methods, and barrier methods. In this paper, we use an interior point method, which is a type of barrier method, to incorporate arbitrary stagewise equality and inequality state and control constraints. We also provide explicit update formulas for all the involved variables. Finally, we apply this algorithm to example systems such as the inverted pendulum, a continuously stirred tank reactor, car parking, and obstacle avoidance.

Characterizing materials with electron micrographs poses significant challenges for automated labeling due to the complex nature of nanomaterial structures. To address this, we introduce a fully automated, end-to-end pipeline that leverages recent advances in Generative AI. It is designed for analyzing and understanding the microstructures of semiconductor materials with effectiveness comparable to that of human experts, contributing to the pursuit of Artificial General Intelligence (AGI) in nanomaterial identification. Our approach utilizes Large MultiModal Models (LMMs) such as GPT-4V, alongside text-to-image models like DALLE-3. We integrate a GPT-4 guided Visual Question Answering (VQA) method to analyze nanomaterial images, generate synthetic nanomaterial images via DALLE-3, and employ in-context learning with few-shot prompting in GPT-4V for accurate nanomaterial identification. Our method surpasses traditional techniques by enhancing the precision of nanomaterial identification and optimizing the process for high-throughput screening., Comment: Published at Deployable AI (DAI) Workshop at AAAI-2024

Shot-noise measures the correlations of fluctuations of current for a voltage applied much larger than the temperature and reveals aspects of correlations in fermions beyond those revealed in the conductivity. Recent measurements of shot-noise in the quantum-critical region of the heavy-fermion compound YbRh$_2$Si$_2$ (YRS) have presented a conceptual challenge to old theory and those devised following the experiments. Since the measured resistivity and the specific heat in YRS follow the predictions of marginal Fermi liquid (MFL) theory, we use it to calculate noise using the method developed by Nagaev. We get fair agreement with the magnitude and temperature dependence in the experiments using parameters from resistivity measurements. To achieve this, we find it necessary that the collisions between fermions by exchanging the MFL fluctuations conserve energy but lose momentum through Umklapp scattering and that the fermions and their fluctuations are locally in mutual equilibrium. %and that the self-energy rides the local chemical potential. At low temperatures, impurity scattering determines the noise and at high temperatures the MFL scattering. We show that the noise for MFL scattering for high T alone is the same as the Johnson-Nyquist noise, which in this case is temperature independent. Therefore the Fano factor crosses over to $0$ at high temperatures independent of the voltage applied.

Given a set $S \subseteq \mathbb{R}^d$, an empty polytope has vertices in $S$ but contains no other point of $S$. Empty polytopes are closely related to so-called Helly numbers, which extend Helly's theorem to more general point sets in $\mathbb{R}^d$. We improve bounds on the number of vertices in empty polytopes in exponential lattices, arithmetic congruence sets, and 2-syndetic sets. We also study hollow polytopes, which have vertices in $S$ and no points of $S$ in their interior. We obtain bounds on the number of vertices in hollow polytopes under certain conditions, such as the vertices being in general position. Finally, we obtain relatively tight asymptotic bounds for polytopes which do not contain lattice segments of large length., Comment: 13 pages, 2 figures

The rise of powerful multimodal LLMs has enhanced the viability of building web agents which can, with increasing levels of autonomy, assist users to retrieve information and complete tasks on various human-computer interfaces. It is hence necessary to build challenging benchmarks that span a wide-variety of use cases reflecting real-world usage. In this work, we present WebQuest, a multi-page question-answering dataset that requires reasoning across multiple related web pages. In contrast to existing UI benchmarks that focus on multi-step web navigation and task completion, our dataset evaluates information extraction, multimodal retrieval and composition of information from many web pages. WebQuest includes three question categories: single-screen QA, multi-screen QA, and QA based on navigation traces. We evaluate leading proprietary multimodal models like GPT-4V, Gemini Flash, Claude 3, and open source models like InstructBLIP, PaliGemma on our dataset, revealing a significant gap between single-screen and multi-screen reasoning. Finally, we investigate inference time techniques like Chain-of-Thought prompting to improve model capabilities on multi-screen reasoning.

Full coherent control and generation of superpositions of the quantum harmonic oscillator are not only of fundamental interest but are crucial for applications in quantum simulations, quantum-enhanced metrology and continuous-variable quantum computation. The extension of such superpositions to nonclassical states increases their power as a resource for such applications. Here, we create arbitrary superpositions of nonclassical and non-Gaussian states of a quantum harmonic oscillator using the motion of a trapped ion coupled to its internal spin states. We interleave spin-dependent nonlinear bosonic interactions and mid-circuit measurements of the spin that preserve the coherence of the oscillator. These techniques enable the creation of superpositions between squeezed, trisqueezed, and quadsqueezed states, which have never been demonstrated before, with independent control over the complex-valued squeezing parameter and the probability amplitude of each constituent, as well as their spatial separation. We directly observe the nonclassical nature of these states in the form of Wigner negativity following a full state reconstruction. Our methods apply to any system where a quantum harmonic oscillator is coupled to a spin.

The current technology landscape lacks a foundational AI model for solving process engineering calculations. In this work, we introduce a novel autonomous agent framework leveraging Retrieval-Augmented Instruction-Tuning (RAIT) to enhance open, customizable small code language models (SLMs) for these calculations. By combining instruction tuned code SLMs with Retrieval-Augmented Code Generation (RACG) using external tools, the agent generates, debugs, and optimizes code from natural language specifications. Our approach addresses the limitations of the current lack of a foundational AI model for specialized process engineering tasks and offers benefits of explainability, knowledge editing, and cost-effectiveness. Additionally, we curate custom datasets of chemical and process engineering problems and solutions to overcome data scarcity. Experimental results show that our framework matches the performance of large-scale proprietary models on benchmark datasets, proving its effectiveness and usability., Comment: Accepted for publication at ML4CCE workshop at ECML PKDD 2024. Please find the link: https://ml4cce-ecml.com/#agenda

We present a novel framework for analyzing and interpreting electron microscopy images in semiconductor manufacturing using vision-language instruction tuning. The framework employs a unique teacher-student approach, leveraging pre-trained multimodal large language models such as GPT-4 to generate instruction-following data for zero-shot visual question answering (VQA) and classification tasks, customizing smaller multimodal models (SMMs) for microscopy image analysis, resulting in an instruction-tuned language-and-vision assistant. Our framework merges knowledge engineering with machine learning to integrate domain-specific expertise from larger to smaller multimodal models within this specialized field, greatly reducing the need for extensive human labeling. Our study presents a secure, cost-effective, and customizable approach for analyzing microscopy images, addressing the challenges of adopting proprietary models in semiconductor manufacturing., Comment: Paper published at AAAI 2024 Spring Symposium Series

Semiconductors, crucial to modern electronics, are generally under-researched in foundational models. It highlights the need for research to enhance the semiconductor device technology portfolio and aid in high-end device fabrication. In this paper, we introduce sLAVA, a small-scale vision-language assistant tailored for semiconductor manufacturing, with a focus on electron microscopy image analysis. It addresses challenges of data scarcity and acquiring high-quality, expert-annotated data. We employ a teacher-student paradigm, using a foundational vision language model like GPT-4 as a teacher to create instruction-following multimodal data for customizing the student model, sLAVA, for electron microscopic image analysis tasks on consumer hardware with limited budgets. Our approach allows enterprises to further fine-tune the proposed framework with their proprietary data securely within their own infrastructure, protecting intellectual property. Rigorous experiments validate that our framework surpasses traditional methods, handles data shifts, and enables high-throughput screening., Comment: Paper published at ICML 2024 Workshop on Foundation Models in the Wild

In the field of chemistry, the objective is to create novel molecules with desired properties, facilitating accurate property predictions for applications such as material design and drug screening. However, existing graph deep learning methods face limitations that curb their expressive power. To address this, we explore the integration of vast molecular domain knowledge from Large Language Models (LLMs) with the complementary strengths of Graph Neural Networks (GNNs) to enhance performance in property prediction tasks. We introduce a Multi-Modal Fusion (MMF) framework that synergistically harnesses the analytical prowess of GNNs and the linguistic generative and predictive abilities of LLMs, thereby improving accuracy and robustness in predicting molecular properties. Our framework combines the effectiveness of GNNs in modeling graph-structured data with the zero-shot and few-shot learning capabilities of LLMs, enabling improved predictions while reducing the risk of overfitting. Furthermore, our approach effectively addresses distributional shifts, a common challenge in real-world applications, and showcases the efficacy of learning cross-modal representations, surpassing state-of-the-art baselines on benchmark datasets for property prediction tasks., Comment: Paper Accepted at Workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models at NeurIPS 2023

Spatio-temporal forecasting plays a crucial role in various sectors such as transportation systems, logistics, and supply chain management. However, existing methods are limited by their ability to handle large, complex datasets. To overcome this limitation, we introduce a hybrid approach that combines the strengths of open-source large and small-scale language models (LLMs and LMs) with traditional forecasting methods. We augment traditional methods with dynamic prompting and a grouped-query, multi-head attention mechanism to more effectively capture both intra-series and inter-series dependencies in evolving nonlinear time series data. In addition, we facilitate on-premises customization by fine-tuning smaller open-source LMs for time series trend analysis utilizing descriptions generated by open-source large LMs on consumer-grade hardware using Low-Rank Adaptation with Activation Memory Reduction (LoRA-AMR) technique to reduce computational overhead and activation storage memory demands while preserving inference latency. We combine language model processing for time series trend analysis with traditional time series representation learning method for cross-modal integration, achieving robust and accurate forecasts. The framework effectiveness is demonstrated through extensive experiments on various real-world datasets, outperforming existing methods by significant margins in terms of forecast accuracy., Comment: Paper published at the Deployable AI (DAI) workshop at AAAI-2024

In the chemical and process industries, Process Flow Diagrams (PFDs) and Piping and Instrumentation Diagrams (P&IDs) are critical for design, construction, and maintenance. Recent advancements in Generative AI, such as Large Multimodal Models (LMMs) like GPT4 (Omni), have shown promise in understanding and interpreting process diagrams for Visual Question Answering (VQA). However, proprietary models pose data privacy risks, and their computational complexity prevents knowledge editing for domain-specific customization on consumer hardware. To overcome these challenges, we propose a secure, on-premises enterprise solution using a hierarchical, multi-agent Retrieval Augmented Generation (RAG) framework for open-domain question answering (ODQA) tasks, offering enhanced data privacy, explainability, and cost-effectiveness. Our novel multi-agent framework employs introspective and specialized sub-agents using open-source, small-scale multimodal models with the ReAct (Reason+Act) prompting technique for PFD and P&ID analysis, integrating multiple information sources to provide accurate and contextually relevant answers. Our approach, supported by iterative self-correction, aims to deliver superior performance in ODQA tasks. We conducted rigorous experimental studies, and the empirical results validated the proposed approach effectiveness., Comment: Our paper is accepted for publication at ML4CCE workshop at ECML PKDD 2024

Characterizing materials with electron micrographs is a crucial task in fields such as semiconductors and quantum materials. The complex hierarchical structure of micrographs often poses challenges for traditional classification methods. In this study, we propose an innovative backbone architecture for analyzing electron micrographs. We create multi-modal representations of the micrographs by tokenizing them into patch sequences and, additionally, representing them as vision graphs, commonly referred to as patch attributed graphs. We introduce the Hierarchical Network Fusion (HNF), a multi-layered network structure architecture that facilitates information exchange between the multi-modal representations and knowledge integration across different patch resolutions. Furthermore, we leverage large language models (LLMs) to generate detailed technical descriptions of nanomaterials as auxiliary information to assist in the downstream task. We utilize a cross-modal attention mechanism for knowledge fusion across cross-domain representations(both image-based and linguistic insights) to predict the nanomaterial category. This multi-faceted approach promises a more comprehensive and accurate representation and classification of micrographs for nanomaterial identification. Our framework outperforms traditional methods, overcoming challenges posed by distributional shifts, and facilitating high-throughput screening., Comment: Our paper is published at the workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models at NeurIPS 2023

Spatio-temporal forecasting is crucial in transportation, logistics, and supply chain management. However, current methods struggle with large, complex datasets. We propose a dynamic, multi-modal approach that integrates the strengths of traditional forecasting methods and instruction tuning of small language models for time series trend analysis. This approach utilizes a mixture of experts (MoE) architecture with parameter-efficient fine-tuning (PEFT) methods, tailored for consumer hardware to scale up AI solutions in low resource settings while balancing performance and latency tradeoffs. Additionally, our approach leverages related past experiences for similar input time series to efficiently handle both intra-series and inter-series dependencies of non-stationary data with a time-then-space modeling approach, using grouped-query attention, while mitigating the limitations of traditional forecasting techniques in handling distributional shifts. Our approach models predictive uncertainty to improve decision-making. Our framework enables on-premises customization with reduced computational and memory demands, while maintaining inference speed and data privacy/security. Extensive experiments on various real-world datasets demonstrate that our framework provides robust and accurate forecasts, significantly outperforming existing methods., Comment: Published at the ICLR 2024 Workshop on Practical ML for Low Resource Settings(PML4LRS)

Characterizing materials using electron micrographs is crucial in areas such as semiconductors and quantum materials. Traditional classification methods falter due to the intricatestructures of these micrographs. This study introduces an innovative architecture that leverages the generative capabilities of zero-shot prompting in Large Language Models (LLMs) such as GPT-4(language only), the predictive ability of few-shot (in-context) learning in Large Multimodal Models (LMMs) such as GPT-4(V)ision, and fuses knowledge across image based and linguistic insights for accurate nanomaterial category prediction. This comprehensive approach aims to provide a robust solution for the automated nanomaterial identification task in semiconductor manufacturing, blending performance, efficiency, and interpretability. Our method surpasses conventional approaches, offering precise nanomaterial identification and facilitating high-throughput screening., Comment: Published at Deployable AI (DAI) Workshop at AAAI-2024

We present the Process Engineering Operations Assistant (PEOA), an AI-driven framework designed to solve complex problems in the chemical and process industries. The framework employs a modular architecture orchestrated by a meta-agent, which serves as the central coordinator, managing an action generator and instruction-tuned small-scale language models (expert models). The action generator decomposes complex problems into sub-tasks and identifies suitable expert models to execute each, delivering precise solutions for multi-step problem-solving. Key techniques include advanced knowledge modeling using property graphs for improved information retrieval, facilitating more accurate and contextually relevant solutions. Additionally, the framework utilizes a teacher-student transfer-learning approach with GPT-4 (Omni) to fine-tune the action generator and expert models for domain adaptation, alongside an iterative problem-solving mechanism with sophisticated error handling. Custom datasets were developed to evaluate the framework against leading proprietary language models on various engineering tasks. The results demonstrate the framework effectiveness in automating calculations, accelerating prototyping, and providing AI-augmented decision support for industrial processes, marking a significant advancement in process engineering capabilities., Comment: Accepted for Publication by Association for the Advancement of Artificial Intelligence, Fall Symposium Series

Semiconductor imaging and analysis are critical yet understudied in deep learning, limiting our ability for precise control and optimization in semiconductor manufacturing. We introduce a small-scale multimodal framework for analyzing semiconductor electron microscopy images (MAEMI) through vision-language instruction tuning. We generate a customized instruction-following dataset using large multimodal models on microscopic image analysis. We perform knowledge transfer from larger to smaller models through knowledge distillation, resulting in improved accuracy of smaller models on visual question answering (VQA) tasks. This approach eliminates the need for expensive, human expert-annotated datasets for microscopic image analysis tasks. Enterprises can further finetune MAEMI on their intellectual data, enhancing privacy and performance on low-cost consumer hardware. Our experiments show that MAEMI outperforms traditional methods, adapts to data distribution shifts, and supports high-throughput screening., Comment: Our paper is published at ICML 2024 Workshop ML for Life and Material Science: From Theory to Industry Applications, Vienna, Austria

Question difficulty estimation remains a multifaceted challenge in educational and assessment settings. Traditional approaches often focus on surface-level linguistic features or learner comprehension levels, neglecting the intricate interplay of factors contributing to question complexity. This paper presents a novel framework for domain-specific question difficulty estimation, leveraging a suite of NLP techniques and knowledge graph analysis. We introduce four key parameters: Topic Retrieval Cost, Topic Salience, Topic Coherence, and Topic Superficiality, each capturing a distinct facet of question complexity within a given subject domain. These parameters are operationalized through topic modelling, knowledge graph analysis, and information retrieval techniques. A model trained on these features demonstrates the efficacy of our approach in predicting question difficulty. By operationalizing these parameters, our framework offers a novel approach to question complexity estimation, paving the way for more effective question generation, assessment design, and adaptive learning systems across diverse academic disciplines., Comment: 14 pages, 6 figures

Forecasting the behaviour of complex dynamical systems such as interconnected sensor networks characterized by high-dimensional multivariate time series(MTS) is of paramount importance for making informed decisions and planning for the future in a broad spectrum of applications. Graph forecasting networks(GFNs) are well-suited for forecasting MTS data that exhibit spatio-temporal dependencies. However, most prior works of GFN-based methods on MTS forecasting rely on domain-expertise to model the nonlinear dynamics of the system, but neglect the potential to leverage the inherent relational-structural dependencies among time series variables underlying MTS data. On the other hand, contemporary works attempt to infer the relational structure of the complex dependencies between the variables and simultaneously learn the nonlinear dynamics of the interconnected system but neglect the possibility of incorporating domain-specific prior knowledge to improve forecast accuracy. To this end, we propose a hybrid architecture that combines explicit prior knowledge with implicit knowledge of the relational structure within the MTS data. It jointly learns intra-series temporal dependencies and inter-series spatial dependencies by encoding time-conditioned structural spatio-temporal inductive biases to provide more accurate and reliable forecasts. It also models the time-varying uncertainty of the multi-horizon forecasts to support decision-making by providing estimates of prediction uncertainty. The proposed architecture has shown promising results on multiple benchmark datasets and outperforms state-of-the-art forecasting methods by a significant margin. We report and discuss the ablation studies to validate our forecasting architecture., Comment: Paper accepted at ML4IoT Workshop, International Conference on Learning Representations(ICLR) 2023

Accurately predicting the behavior of complex dynamical systems, characterized by high-dimensional multivariate time series(MTS) in interconnected sensor networks, is crucial for informed decision-making in various applications to minimize risk. While graph forecasting networks(GFNs) are ideal for forecasting MTS data that exhibit spatio-temporal dependencies, prior works rely solely on the domain-specific knowledge of time-series variables inter-relationships to model the nonlinear dynamics, neglecting inherent relational structural dependencies among the variables within the MTS data. In contrast, contemporary works infer relational structures from MTS data but neglect domain-specific knowledge. The proposed hybrid architecture addresses these limitations by combining both domain-specific knowledge and implicit knowledge of the relational structure underlying the MTS data using Knowledge-Based Compositional Generalization. The hybrid architecture shows promising results on multiple benchmark datasets, outperforming state-of-the-art forecasting methods. Additionally, the architecture models the time varying uncertainty of multi-horizon forecasts., Comment: Paper is accepted at Knowledge-Based Compositional Generalization Workshop, International Joint Conferences on Artificial Intelligence(IJCAI-23)

Digital Twin technology creates virtual replicas of physical objects, processes, or systems by replicating their properties, data, and behaviors. This advanced technology offers a range of intelligent functionalities, such as modeling, simulation, and data-driven decision-making, that facilitate design optimization, performance estimation, and monitoring operations. Forecasting plays a pivotal role in Digital Twin technology, as it enables the prediction of future outcomes, supports informed decision-making, minimizes risks, driving improvements in efficiency, productivity, and cost reduction. Recently, Digital Twin technology has leveraged Graph forecasting techniques in large-scale complex sensor networks to enable accurate forecasting and simulation of diverse scenarios, fostering proactive and data-driven decision making. However, existing Graph forecasting techniques lack scalability for many real-world applications. They have limited ability to adapt to non-stationary environments, retain past knowledge, lack a mechanism to capture the higher order spatio-temporal dynamics, and estimate uncertainty in model predictions. To surmount the challenges, we introduce a hybrid architecture that enhances the hypergraph representation learning backbone by incorporating fast adaptation to new patterns and memory-based retrieval of past knowledge. This balance aims to improve the slowly-learned backbone and achieve better performance in adapting to recent changes. In addition, it models the time-varying uncertainty of multi-horizon forecasts, providing estimates of prediction uncertainty. Our forecasting architecture has been validated through ablation studies and has demonstrated promising results across multiple benchmark datasets, surpassing state-ofthe-art forecasting methods by a significant margin., Comment: Paper accepted at AI for Digital Twins and Cyber-Physical Applications Workshop, International Joint Conferences on Artificial Intelligence(IJCAI-23). arXiv admin note: text overlap with arXiv:2408.12409

Google Vizier has performed millions of optimizations and accelerated numerous research and production systems at Google, demonstrating the success of Bayesian optimization as a large-scale service. Over multiple years, its algorithm has been improved considerably, through the collective experiences of numerous research efforts and user feedback. In this technical report, we discuss the implementation details and design choices of the current default algorithm provided by Open Source Vizier. Our experiments on standardized benchmarks reveal its robustness and versatility against well-established industry baselines on multiple practical modes., Comment: Google DeepMind Technical Report. Code can be found in https://github.com/google/vizier

Anomaly detection is fundamental yet, challenging problem with practical applications in industry. The current approaches neglect the higher-order dependencies within the networks of interconnected sensors in the high-dimensional time series(multisensor data) for anomaly detection. To this end, we present a self-adapting anomaly detection framework for joint learning of (a) discrete hypergraph structure and (b) modeling the temporal trends and spatial relations among the interdependent sensors using the hierarchical encoder-decoder architecture to overcome the challenges. The hypergraph representation learning-based framework exploits the relational inductive biases in the hypergraph-structured data to learn the pointwise single-step-ahead forecasts through the self-supervised autoregressive task and predicts the anomalies based on the forecast error. Furthermore, our framework incentivizes learning the anomaly-diagnosis ontology through a differentiable approach. It derives the anomaly information propagation-based computational hypergraphs for root cause analysis and provides recommendations through an offline, optimal predictive control policy to remedy an anomaly. We conduct extensive experiments to evaluate the proposed method on the benchmark datasets for fair and rigorous comparison with the popular baselines. The proposed method outperforms the baseline models and achieves SOTA performance. We report the ablation studies to support the efficacy of the framework., Comment: 16 pages, 10 figure, Accepted at IEEE International Conference on Big Data 2022, Osaka, Japan

Material characterization using electron micrographs is a crucial but challenging task with applications in various fields, such as semiconductors, quantum materials, batteries, etc. The challenges in categorizing electron micrographs include but are not limited to the complexity of patterns, high level of detail, and imbalanced data distribution(long-tail distribution). Existing methods have difficulty in modeling the complex relational structure in electron micrographs, hindering their ability to effectively capture the complex relationships between different spatial regions of micrographs. We propose a hypergraph neural network(HgNN) backbone architecture, a conceptually alternative approach, to better model the complex relationships in electron micrographs and improve material characterization accuracy. By utilizing cost-effective GPU hardware, our proposed framework outperforms popular baselines. The results of the ablation studies demonstrate that the proposed framework is effective in achieving state-of-the-art performance on benchmark datasets and efficient in terms of computational and memory requirements for handling large-scale electron micrograph-based datasets., Comment: 21 pages, Accepted in PML4DC Workshop at International Conference on Learning Representations (ICLR) 2023

Planning remains one of the last standing bastions for large language models (LLMs), which now turn their attention to search. Most of the literature uses the language models as world models to define the search space, forgoing soundness for the sake of flexibility. A recent work, Thought of Search (ToS), proposed defining the search space with code, having the language models produce that code. ToS requires a human in the loop, collaboratively producing a sound successor function and goal test. The result, however, is worth the effort: all the tested datasets were solved with 100% accuracy. At the same time LLMs have demonstrated significant progress in code generation and refinement for complex reasoning tasks. In this work, we automate ToS (AutoToS), completely taking the human out of the loop of solving planning problems. AutoToS guides the language model step by step towards the generation of sound and complete search components, through feedback from both generic and domain specific unit tests. We achieve 100% accuracy, with minimal feedback iterations, using LLMs of various sizes on all evaluated domains.

Characterization of materials via electron micrographs is an important and challenging task in several materials processing industries. Classification of electron micrographs is complex due to the high intra-class dissimilarity, high inter-class similarity, and multi-spatial scales of patterns. However, existing methods are ineffective in learning complex image patterns. We propose an effective end-to-end electron micrograph representation learning-based framework for nanomaterial identification to overcome the challenges. We demonstrate that our framework outperforms the popular baselines on the open-source datasets in nanomaterials-based identification tasks. The ablation studies are reported in great detail to support the efficacy of our approach., Comment: 12 pages, 10 figures, Accepted in a ACM SIGKDD 2022 Workshop on Machine Learning for Materials

The explosion of machine learning model size has led to its execution on distributed clusters at a very large scale. Many works have tried to optimize the process of producing collective algorithms and running collective communications, which act as a bottleneck to distributed machine learning. However, different works use their own collective algorithm representation, pushing away from co-optimizing collective communication and the rest of the workload. The lack of a standardized collective algorithm representation has also hindered interoperability between collective algorithm producers and consumers. Additionally, tool-specific conversions and modifications have to be made for each pair of tools producing and consuming collective algorithms which adds to engineering efforts. In this position paper, we propose a standardized workflow leveraging a common collective algorithm representation. Upstream producers and downstream consumers converge to a common representation format based on Chakra Execution Trace, a commonly used graph based representation of distributed machine learning workloads. Such a common representation enables us to view collective communications at the same level as workload operations and decouple producer and consumer tools, enhance interoperability, and relieve the user from the burden of having to focus on downstream implementations. We provide a proof-of-concept of this standardized workflow by simulating collective algorithms generated by the MSCCLang domain-specific language through the ASTRA-sim distributed machine learning simulator using various network configurations.

Time series modeling is crucial for many applications, however, it faces challenges such as complex spatio-temporal dependencies and distribution shifts in learning from historical context to predict task-specific outcomes. To address these challenges, we propose a novel approach using an agentic Retrieval-Augmented Generation (RAG) framework for time series analysis. The framework leverages a hierarchical, multi-agent architecture where the master agent orchestrates specialized sub-agents and delegates the end-user request to the relevant sub-agent. The sub-agents utilize smaller, pre-trained language models (SLMs) customized for specific time series tasks through fine-tuning using instruction tuning and direct preference optimization, and retrieve relevant prompts from a shared repository of prompt pools containing distilled knowledge about historical patterns and trends to improve predictions on new data. Our proposed modular, multi-agent RAG approach offers flexibility and achieves state-of-the-art performance across major time series tasks by tackling complex challenges more effectively than task-specific customized methods across benchmark datasets., Comment: Paper was accepted for Undergraduate Consortium at ACM KDD, 2024. Please find the link: https://kdd2024.kdd.org/undergraduate-consortium/

Pre-trained large language models (PLLMs) like OpenAI ChatGPT and Google Gemini face challenges such as inaccurate factual recall, hallucinations, biases, and future data leakage for temporal Knowledge Graph (tKG) forecasting. To address these issues, we introduce sLA-tKGF (small-scale language assistant for tKG forecasting), which utilizes Retrieval-Augmented Generation (RAG) aided, custom-trained small-scale language models through a tabula rasa approach from scratch for effective tKG forecasting. Our framework constructs knowledge-infused prompts with relevant historical data from tKGs, web search results, and PLLMs-generated textual descriptions to understand historical entity relationships prior to the target time. It leverages these external knowledge-infused prompts for deeper understanding and reasoning of context-specific semantic and temporal information to zero-shot prompt small-scale language models for more accurate predictions of future events within tKGs. It reduces hallucinations and mitigates distributional shift challenges through comprehending changing trends over time. As a result, it enables more accurate and contextually grounded forecasts of future events while minimizing computational demands. Rigorous empirical studies demonstrate our framework robustness, scalability, and state-of-the-art (SOTA) performance on benchmark datasets with interpretable and trustworthy tKG forecasting., Comment: Paper was accepted at ACM KDD -2024 -- Undergraduate Consortium. Please find the link: https://kdd2024.kdd.org/undergraduate-consortium/

High-performance computing (HPC) is essential for tackling complex computational problems across various domains. As the scale and complexity of HPC applications continue to grow, the need for scalable systems and software architectures becomes paramount. This paper provides a comprehensive overview of architecture for HPC on premise focusing on both hardware and software aspects and details the associated challenges in building the HPC cluster on premise. It explores design principles, challenges, and emerging trends in building scalable HPC systems and software, addressing issues such as parallelism, memory hierarchy, communication overhead, and fault tolerance on various cloud platforms. By synthesizing research findings and technological advancements, this paper aims to provide insights into scalable solutions for meeting the evolving demands of HPC applications on cloud., Comment: 6 Pages

Molecule design is a multifaceted approach that leverages computational methods and experiments to optimize molecular properties, fast-tracking new drug discoveries, innovative material development, and more efficient chemical processes. Recently, text-based molecule design has emerged, inspired by next-generation AI tasks analogous to foundational vision-language models. Our study explores the use of knowledge-augmented prompting of large language models (LLMs) for the zero-shot text-conditional de novo molecular generation task. Our approach uses task-specific instructions and a few demonstrations to address distributional shift challenges when constructing augmented prompts for querying LLMs to generate molecules consistent with technical descriptions. Our framework proves effective, outperforming state-of-the-art (SOTA) baseline models on benchmark datasets., Comment: Paper was accepted at R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Foundation Models, NeurIPS-2023. Please find the links: https://sites.google.com/view/r0-fomo/accepted-papers?authuser=0 and https://neurips.cc/virtual/2023/workshop/66517

NLP-assisted solutions have gained considerable traction to support qualitative data analysis. However, there does not exist a unified evaluation framework that can account for the many different settings in which qualitative researchers may employ them. In this paper, we take a first step in this direction by proposing an evaluation framework to study the way in which different tools may result in different outcomes depending on the collaboration strategy employed. Specifically, we study the impact of synchronous vs. asynchronous collaboration using two different NLP-assisted qualitative research tools and present a comprehensive analysis of significant differences in the consistency, cohesiveness, and correctness of their outputs.

National digital identity verification systems have played a critical role in the effective distribution of goods and services, particularly, in developing countries. Due to the cost involved in deploying and maintaining such systems, combined with a lack of in-house technical expertise, governments seek to outsource this service to third-party cloud service providers to the extent possible. This leads to increased concerns regarding the privacy of users' personal data. In this work, we propose a practical privacy-preserving digital identity (ID) verification protocol where the third-party cloud services process the identity data encrypted using a (single-key) Fully Homomorphic Encryption (FHE) scheme such as BFV. Though the role of a trusted entity such as government is not completely eliminated, our protocol does significantly reduces the computation load on such parties. A challenge in implementing a privacy-preserving ID verification protocol using FHE is to support various types of queries such as exact and/or fuzzy demographic and biometric matches including secure age comparisons. From a cryptographic engineering perspective, our main technical contribution is a user data encoding scheme that encodes demographic and biometric user data in only two BFV ciphertexts and yet facilitates us to outsource various types of ID verification queries to a third-party cloud. Our encoding scheme also ensures that the only computation done by the trusted entity is a query-agnostic "extended" decryption. This is in stark contrast with recent works that outsource all the non-arithmetic operations to a trusted server. We implement our protocol using the Microsoft SEAL FHE library and demonstrate its practicality., Comment: This work was presented (without proceedings) at the Turing Trustworthy Digital Identity International Conference 2022 at The Alan Turing Institute, London, UK, on Sep. 16, 2022

Battery life estimation is critical for optimizing battery performance and guaranteeing minimal degradation for better efficiency and reliability of battery-powered systems. The existing methods to predict the Remaining Useful Life(RUL) of Lithium-ion Batteries (LiBs) neglect the relational dependencies of the battery parameters to model the nonlinear degradation trajectories. We present the Battery GraphNets framework that jointly learns to incorporate a discrete dependency graph structure between battery parameters to capture the complex interactions and the graph-learning algorithm to model the intrinsic battery degradation for RUL prognosis. The proposed method outperforms several popular methods by a significant margin on publicly available battery datasets and achieves SOTA performance. We report the ablation studies to support the efficacy of our approach., Comment: Accepted in Workshop on Graph Learning for Industrial Applications : Finance, Crime Detection, Medicine, and Social Media (NeurIPS 2022)

Scaling the number of entangled nodes in a quantum network is a challenge with significant implications for quantum computing, clock synchronisation, secure communications, and quantum sensing. In a quantum network, photons interact with matter qubits at different nodes, flexibly enabling the creation of remote entanglement between them. Multipartite entanglement among multiple nodes will be crucial for many proposed quantum network applications, including quantum computational tasks and quantum metrology. To date, experimental efforts have primarily focused on generating bipartite entanglement between nodes, which is widely regarded as the fundamental quantum resource for quantum networks. However, relying exclusively on bipartite entanglement to form more complex multipartite entanglement introduces several challenges. These include the need for ancillary qubits, extensive local entangling operations which increases the preparation latency, and increasingly stringent requirements on coherence times as the number of nodes grows. Here, we analyse various schemes that achieve multipartite entanglement between nodes in a single step, bypassing the need for multiple rounds of bipartite entanglement. We demonstrate that different schemes can produce distinct multipartite entangled states, with varying fidelity and generation rates. Additionally, we discuss the applicability of these schemes across different experimental platforms, highlighting their primary advantages and disadvantages.

This study develops a real-time framework for estimating the risk of near-misses by using high-fidelity two-dimensional (2D) risk indicator time-to-collision (TTC), which is calculated from high-resolution data collected by autonomous vehicles (AVs). The framework utilizes extreme value theory (EVT) to derive near-miss risk based on observed TTC data. Most existing studies employ a generalized extreme value (GEV) distribution for specific sites and conflict types and often overlook individual vehicle dynamics heterogeneity. This framework is versatile across various highway geometries and can encompass vehicle dynamics and fidelity by incorporating covariates such as speed, acceleration, steering angle, and heading. This makes the risk estimation framework suitable for dynamic, real-world traffic environments. The dataset for this study is derived from Waymo perception data, encompassing six sites across three cities: San Francisco, Phoenix, and Los Angeles. Vehicle trajectory data were extracted from the dataset, and near-miss frequencies were calculated using high-fidelity 2D TTC. The crash risk was derived from observed near misses using four hierarchical Bayesian GEV models, explicitly focusing on conflicting pairs as block minima (BM), which revealed that crash risk varies across pairs.The proposed framework is efficient using a hierarchical Bayesian structure random parameter (HBSRP) model, offering superior statistical performance and flexibility by accounting for unobserved heterogeneity across sites. The study identifies and quantifies that the most hazardous conditions involve conflicting vehicle speeds and rapid acceleration and deceleration, significantly increasing crash risk in urban arterials., Comment: 26 pages, 13 figures

We propose an approach to use linearly polarized light to imprint superconducting vortices. Within the framework of the generalized time-dependent Ginzburg-Landau equations we demonstrate the induction of the coherent vortex pairs that are moving in phase with electormagnetic wave oscillations. The overall vorticity of the superconductor remain zero throughout the cycle. Our results uncover rich multiscale dynamics of SC vorticity and suggest new optical applications for various types of structured light. In departure from classical laser printing, the laser printing proposed here can be viewed as quantum print where we induce quantum excitations in the SC liquid.

An extensive experimental investigation on the structural, static magnetic, and non-equilibrium dynamical properties of polycrystalline Mn$_2$CuGe Heusler alloy using powder X-ray diffraction, DC magnetization, magnetic relaxation, magnetic memory effect, and specific heat measurements is presented. Structural studies reveal that the alloy crystallizes in a mixed hexagonal crystal structure (space groups P3c1 (no. 158) and P6$_3$/mmc (no. 194)) with lattice parameters a = b = 7.18(4) $\mathring{A}$ and c = 13.12(4) $\mathring{A}$ for the majority phase. The DC magnetization analysis reveals a paramagnetic to ferrimagnetic phase transition around T$_C$ $\approx$ 682 K with a compensation of magnetization at $\approx$ 250 K, and a spin-glass transition around T$_P$ $\approx$ 25.6 K. The N\'eel theory of ferrimagnets supports the ferrimagnetic nature of the studied alloy and the estimated T$_C$ ($\approx$ 687 K) from this theory is consistent with that obtained from the DC magnetization data. A detailed study of non-equilibrium spin dynamics via magnetic relaxation and memory effect experiments shows the evolution of the system through a number of intermediate states and striking magnetic memory effect. Furthermore, heat capacity measurements suggest a large electronic contribution to the specific heat capacity suggesting strong spin fluctuations, due to competing magnetic interactions. All the observations render a spin-glass behavior in Mn$_2$CuGe, attributed to the magnetic frustration possibly arising out of the competing ferromagnetic and antiferromagnetic interactions., Comment: 10 pages, 12 figures, 1 table

Do different generative image models secretly learn similar underlying representations? We investigate this by measuring the latent space similarity of four different models: VAEs, GANs, Normalizing Flows (NFs), and Diffusion Models (DMs). Our methodology involves training linear maps between frozen latent spaces to "stitch" arbitrary pairs of encoders and decoders and measuring output-based and probe-based metrics on the resulting "stitched'' models. Our main findings are that linear maps between latent spaces of performant models preserve most visual information even when latent sizes differ; for CelebA models, gender is the most similarly represented probe-able attribute. Finally we show on an NF that latent space representations converge early in training.

An important aspect of 5G networks is the development of Radio Access Network (RAN) slicing, a concept wherein the virtualized infrastructure of wireless networks is subdivided into slices (or enterprises), tailored to fulfill specific use-cases. A key focus in this context is the efficient radio resource allocation to meet various enterprises' service-level agreements (SLAs). In this work, we introduce a channel-aware and SLA-aware RAN slicing framework for massive multiple input multiple output (MIMO) networks where resource allocation extends to incorporate the spatial dimension available through beamforming. Essentially, the same time-frequency resource block (RB) can be shared across multiple users through multiple antennas. Notably, certain enterprises, particularly those operating critical infrastructure, necessitate dedicated RB allocation, denoted as private networks, to ensure security. Conversely, some enterprises would allow resource sharing with others in the public network to maintain network performance while minimizing capital expenditure. Building upon this understanding, the proposed scheduler comprises scheduling schemes under both scenarios: where different slices share the same set of RBs, and where they require exclusivity of allocated RBs. We validate the efficacy of our proposed schedulers through simulation by utilizing a channel data set collected from a real-world massive MIMO testbed. Our assessments demonstrate that resource sharing across slices using our approach can lead up to 60.9% reduction in RB usage compared to other approaches. Moreover, our proposed schedulers exhibit significantly enhanced operational efficiency, with significantly faster running time compared to exhaustive greedy approaches while meeting the stringent 5G sub-millisecond-level latency requirement.

The central challenge of quantum computing is implementing high-fidelity quantum gates at scale. However, many existing approaches to qubit control suffer from a scale-performance trade-off, impeding progress towards the creation of useful devices. Here, we present a vision for an electronically controlled trapped-ion quantum computer that alleviates this bottleneck. Our architecture utilizes shared current-carrying traces and local tuning electrodes in a microfabricated chip to perform quantum gates with low noise and crosstalk regardless of device size. To verify our approach, we experimentally demonstrate low-noise site-selective single- and two-qubit gates in a seven-zone ion trap that can control up to 10 qubits. We implement electronic single-qubit gates with 99.99916(7)% fidelity, and demonstrate consistent performance with low crosstalk across the device. We also electronically generate two-qubit maximally entangled states with 99.97(1)% fidelity and long-term stable performance over continuous system operation. These state-of-the-art results validate the path to directly scaling these techniques to large-scale quantum computers based on electronically controlled trapped-ion qubits.

We present the analysis of optical photometric and spectroscopic observations of two non-magnetic cataclysmic variables, namely CRTS J080846.2+313106 and V416 Dra. CRTS J080846.2+313106 has been found to vary with a period of 4.9116$\pm$0.0003 h, which was not found in earlier studies and is provisionally suggested as the orbital period of the system. In both long-period systems, the observed dominant signal at second harmonic of the orbital frequency and the orbital modulation during quiescence are suggestive of ellipsoidal variation from changing aspects of the secondary, with an additional contribution from the accretion stream or hotspot. However, during the outburst, the hotspot itself is overwhelmed by the increased brightness, which is possibly associated with the accretion disc. The mid-eclipse phase for V416 Dra occurs earlier and the width of the eclipse is greater during outbursts compared to quiescence, suggesting an increased accretion disc radius during outbursts. Furthermore, the investigation of accretion disc eclipse in V416 Dra implies that a total disc eclipse is possible during quiescence, whereas the disc seems to be partially obscured during outbursts, which further signifies that the disc may grow in size as the outburst progresses. Optical spectra of CRTS J080846.2+313106 and V416 Dra are typical of dwarf novae during quiescence, and they both show a significant contribution from the M2-4V secondary. The light curve patterns, orbital periods, and spectra observed in both systems look remarkably similar, and seem to resemble the characteristics of U Gem-type dwarf novae., Comment: 14 pages, 11 Figures, and 3 Tables, Accepted for publication in A&A

Models for low-latency, streaming applications could benefit from the knowledge capacity of larger models, but edge devices cannot run these models due to resource constraints. A possible solution is to transfer hints during inference from a large model running remotely to a small model running on-device. However, this incurs a communication delay that breaks real-time requirements and does not guarantee that both models will operate on the same data at the same time. We propose knowledge boosting, a novel technique that allows a large model to operate on time-delayed input during inference, while still boosting small model performance. Using a streaming neural network that processes 8 ms chunks, we evaluate different speech separation and enhancement tasks with communication delays of up to six chunks or 48 ms. Our results show larger gains where the performance gap between the small and large models is wide, demonstrating a promising method for large-small model collaboration for low-latency applications. Code, dataset, and audio samples available at https://knowledgeboosting.cs.washington.edu/., Comment: Accepted by Interspeech 2024

We address the challenge of online convex optimization where the objective function's gradient exhibits sparsity, indicating that only a small number of dimensions possess non-zero gradients. Our aim is to leverage this sparsity to obtain useful estimates of the objective function's gradient even when the only information available is a limited number of function samples. Our motivation stems from distributed queueing systems like microservices-based applications, characterized by request-response workloads. Here, each request type proceeds through a sequence of microservices to produce a response, and the resource allocation across the collection of microservices is controlled to balance end-to-end latency with resource costs. While the number of microservices is substantial, the latency function primarily reacts to resource changes in a few, rendering the gradient sparse. Our proposed method, CONGO (Compressive Online Gradient Optimization), combines simultaneous perturbation with compressive sensing to estimate gradients. We establish analytical bounds on the requisite number of compressive sensing samples per iteration to maintain bounded bias of gradient estimates, ensuring sub-linear regret. By exploiting sparsity, we reduce the samples required per iteration to match the gradient's sparsity, rather than the problem's original dimensionality. Numerical experiments and real-world microservices benchmarks demonstrate CONGO's superiority over multiple stochastic gradient descent approaches, as it quickly converges to performance comparable to policies pre-trained with workload awareness., Comment: 28 pages, 7 figures

Sample Return Capsules (SRCs) entering Earth's atmosphere at hypervelocity from interplanetary space are a valuable resource for studying meteor phenomena. The 24 September 2023 arrival of the OSIRIS-REx (Origins, Spectral Interpretation, Resource Identification, and Security-Regolith Explorer) SRC provided an unprecedented chance for geophysical observations of a well-characterized source with known parameters, including timing and trajectory. A collaborative effort involving researchers from 16 institutions executed a carefully planned geophysical observational campaign at strategically chosen locations, deploying over 400 ground-based sensors encompassing infrasound, seismic, distributed acoustic sensing (DAS), and GPS technologies. Additionally, balloons equipped with infrasound sensors were launched to capture signals at higher altitudes. This campaign (the largest of its kind so far) yielded a wealth of invaluable data anticipated to fuel scientific inquiry for years to come. The success of the observational campaign is evidenced by the near-universal detection of signals across instruments, both proximal and distal. This paper presents a comprehensive overview of the collective scientific effort, field deployment, and preliminary findings. The early findings have the potential to inform future space missions and terrestrial campaigns, contributing to our understanding of meteoroid interactions with planetary atmospheres. Furthermore, the dataset collected during this campaign will improve entry and propagation models as well as augment the study of atmospheric dynamics and shock phenomena generated by meteoroids and similar sources., Comment: 87 pages, 14 figures

Distributed quantum computing (DQC) combines the computing power of multiple networked quantum processing modules, enabling the execution of large quantum circuits without compromising on performance and connectivity. Photonic networks are well-suited as a versatile and reconfigurable interconnect layer for DQC; remote entanglement shared between matter qubits across the network enables all-to-all logical connectivity via quantum gate teleportation (QGT). For a scalable DQC architecture, the QGT implementation must be deterministic and repeatable; until now, there has been no demonstration satisfying these requirements. We experimentally demonstrate the distribution of quantum computations between two photonically interconnected trapped-ion modules. The modules are separated by $\sim$ 2 m, and each contains dedicated network and circuit qubits. By using heralded remote entanglement between the network qubits, we deterministically teleport a controlled-Z gate between two circuit qubits in separate modules, achieving 86% fidelity. We then execute Grover's search algorithm - the first implementation of a distributed quantum algorithm comprising multiple non-local two-qubit gates - and measure a 71% success rate. Furthermore, we implement distributed iSWAP and SWAP circuits, compiled with 2 and 3 instances of QGT, respectively, demonstrating the ability to distribute arbitrary two-qubit operations. As photons can be interfaced with a variety of systems, this technique has applications extending beyond trapped-ion quantum computers, providing a viable pathway towards large-scale quantum computing for a range of physical platforms., Comment: 16 pages, 7 figures, 2 tables