Search

State-of-the-art multimodal web agents, powered by Multimodal Large Language Models (MLLMs), can autonomously execute many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). Current strategies for building web agents rely on (i) the generalizability of underlying MLLMs and their steerability via prompting, and (ii) large-scale fine-tuning of MLLMs on web-related tasks. However, web agents still struggle to automate tasks on unseen websites and domains, limiting their applicability to enterprise-specific and proprietary platforms. Beyond generalization from large-scale pre-training and fine-tuning, we propose building agents for few-shot adaptability using human demonstrations. We introduce the AdaptAgent framework that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). Our experiments on two popular benchmarks -- Mind2Web & VisualWebArena -- show that using in-context demonstrations (for proprietary models) or meta-adaptation demonstrations (for meta-learned open-weights models) boosts task success rate by 3.36% to 7.21% over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03% to 65.75%. Furthermore, our additional analyses (a) show the effectiveness of multimodal demonstrations over text-only ones, (b) shed light on the influence of different data selection strategies during meta-learning on the generalization of the agent, and (c) demonstrate the effect of number of few-shot examples on the web agent's success rate. Overall, our results unlock a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning, emphasizing few-shot adaptability., Comment: 18 pages, 3 figures, an abridged version to appear in NeurIPS 2024 AFM Workshop

The rapid development in large language models (LLMs) has transformed the landscape of natural language processing and understanding (NLP/NLU), offering significant benefits across various domains. However, when applied to scientific research, these powerful models exhibit critical failure modes related to scientific integrity and trustworthiness. Existing general-purpose LLM guardrails are insufficient to address these unique challenges in the scientific domain. We provide comprehensive guidelines for deploying LLM guardrails in the scientific domain. We identify specific challenges -- including time sensitivity, knowledge contextualization, conflict resolution, and intellectual property concerns -- and propose a guideline framework for the guardrails that can align with scientific needs. These guardrail dimensions include trustworthiness, ethics & bias, safety, and legal aspects. We also outline in detail the implementation strategies that employ white-box, black-box, and gray-box methodologies that can be enforced within scientific contexts.

Financial intelligence generation from vast data sources has typically relied on traditional methods of knowledge-graph construction or database engineering. Recently, fine-tuned financial domain-specific Large Language Models (LLMs), have emerged. While these advancements are promising, limitations such as high inference costs, hallucinations, and the complexity of concurrently analyzing high-dimensional financial data, emerge. This motivates our invention FISHNET (Financial Intelligence from Sub-querying, Harmonizing, Neural-Conditioning, Expert swarming, and Task planning), an agentic architecture that accomplishes highly complex analytical tasks for more than 98,000 regulatory filings that vary immensely in terms of semantics, data hierarchy, or format. FISHNET shows remarkable performance for financial insight generation (61.8% success rate over 5.0% Routing, 45.6% RAG R-Precision). We conduct rigorous ablations to empirically prove the success of FISHNET, each agent's importance, and the optimized performance of assembling all agents. Our modular architecture can be leveraged for a myriad of use-cases, enabling scalability, flexibility, and data integrity that are critical for financial tasks., Comment: Accepted at the 5th ACM International Conference on AI in Finance (ICAIF '24)

A significant limitation of current smartphone-based eye-tracking algorithms is their low accuracy when applied to video-type visual stimuli, as they are typically trained on static images. Also, the increasing demand for real-time interactive applications like games, VR, and AR on smartphones requires overcoming the limitations posed by resource constraints such as limited computational power, battery life, and network bandwidth. Therefore, we developed two new smartphone eye-tracking techniques for video-type visuals by combining Convolutional Neural Networks (CNN) with two different Recurrent Neural Networks (RNN), namely Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU). Our CNN+LSTM and CNN+GRU models achieved an average Root Mean Square Error of 0.955cm and 1.091cm, respectively. To address the computational constraints of smartphones, we developed an edge intelligence architecture to enhance the performance of smartphone-based eye tracking. We applied various optimisation methods like quantisation and pruning to deep learning models for better energy, CPU, and memory usage on edge devices, focusing on real-time processing. Using model quantisation, the model inference time in the CNN+LSTM and CNN+GRU models was reduced by 21.72% and 19.50%, respectively, on edge devices.

Gamma-ray bursts (GRBs) are intense, short-lived bursts of gamma-ray radiation observed up to a high redshift ($z \sim 10$) due to their luminosities. Thus, they can serve as cosmological tools to probe the early Universe. However, we need a large sample of high$-z$ GRBs, currently limited due to the difficulty in securing time at the large aperture Telescopes. Thus, it is painstaking to determine quickly whether a GRB is high$z$ or low$-z$, which hampers the possibility of performing rapid follow-up observations. Previous efforts to distinguish between high$-$ and low$-z$ GRBs using GRB properties and machine learning (ML) have resulted in limited sensitivity. In this study, we aim to improve this classification by employing an ensemble ML method on 251 GRBs with measured redshifts and plateaus observed by the Neil Gehrels Swift Observatory. Incorporating the plateau phase with the prompt emission, we have employed an ensemble of classification methods to enhance the sensitivity unprecedentedly. Additionally, we investigate the effectiveness of various classification methods using different redshift thresholds, $z_{threshold}$=$z_t$ at $z_{t}=$ 2.0, 2.5, 3.0, and 3.5. We achieve a sensitivity of 87\% and 89\% with a balanced sampling for both $z_{t}=3.0$ and $z_{t}=3.5$, respectively, representing a 9\% and 11\% increase in the sensitivity over Random Forest used alone. Overall, the best results are at $z_{t} = 3.5$, where the difference between the sensitivity of the training set and the test set is the smallest. This enhancement of the proposed method paves the way for new and intriguing follow-up observations of high$-z$ GRBs., Comment: 36 pages, 14 Figures (6 Figures with single panel, 4 Figures with 8 panels, 2 Figures with 4 panels, 1 Figure with 12 panels, 1 Figure with 2 panels), 6 Tables, submitted to ApJS

In recent years, photon routing has garnered considerable research activity due to its key applications in quantum networking and optical communications. This paper studies the single photon routing scheme in many-emitter disordered chiral waveguide quantum electrodynamics (wQED) ladders. The wQED ladder consists of two one-dimensional lossless waveguides simultaneously and chirally coupled with a chain of dipole-dipole interacting two-level quantum emitters (QEs) or atoms. In particular, we analyze how a departure from the periodic placement of the QEs due to temperature-induced position disorder can impact the routing probability. This involves analyzing how the interplay between the collective atomic effects originating from the dipole-dipole interaction and disorder in the atomic location leading to single-photon localization can change the routing probabilities. As for some key results, we find that the routing probability exhibits a considerable improvement (more than $90\%$ value) for periodic and disordered wQED ladders when considering lattices consisting of twenty QEs. This robustness of collective effects against spontaneous emission loss and weak disorders is further confirmed by examining the routing efficiency and localization length for up to twenty QE chains. These results may find applications in quantum networking and distributed quantum computing under the realistic conditions of imperfect emitter trappings., Comment: 10 pages, 6 figures

Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better on specialized tasks. Inspired by this insight, we developed INDUS, a comprehensive suite of LLMs tailored for the closely-related domains of Earth science, biology, physics, heliophysics, planetary sciences and astrophysics, and trained using curated scientific corpora drawn from diverse data sources. The suite of models include: (1) an encoder model trained using domain-specific vocabulary and corpora to address NLP tasks, (2) a contrastive-learning based text embedding model trained using a diverse set of datasets to address information retrieval tasks and (3) smaller versions of these models created using knowledge distillation for applications which have latency or resource constraints. We also created three new scientific benchmark datasets, CLIMATE-CHANGE NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR (IR) to accelerate research in these multi-disciplinary fields. We show that our models outperform both general-purpose (RoBERTa) and domain-specific (SCIBERT) encoders on these new tasks as well as existing tasks in the domains of interest. Furthermore, we demonstrate the use of these models in two industrial settings -- as a retrieval model for large-scale vector search applications and in automatic content tagging systems., Comment: EMNLP 2024 (Industry Track)

Hallucination continues to be one of the most critical challenges in the institutional adoption journey of Large Language Models (LLMs). While prior studies have primarily focused on the post-generation analysis and refinement of outputs, this paper centers on the effectiveness of queries in eliciting accurate responses from LLMs. We present HalluciBot, a model that estimates the query's propensity to hallucinate before generation, without invoking any LLMs during inference. HalluciBot can serve as a proxy reward model for query rewriting, offering a general framework to estimate query quality based on accuracy and consensus. In essence, HalluciBot investigates how poorly constructed queries can lead to erroneous outputs - moreover, by employing query rewriting guided by HalluciBot's empirical estimates, we demonstrate that 95.7% output accuracy can be achieved for Multiple Choice questions. The training procedure for HalluciBot consists of perturbing 369,837 queries n times, employing n+1 independent LLM agents, sampling an output from each query, conducting a Multi-Agent Monte Carlo simulation on the sampled outputs, and training an encoder classifier. The idea of perturbation is the outcome of our ablation studies that measures the increase in output diversity (+12.5 agreement spread) by perturbing a query in lexically different but semantically similar ways. Therefore, HalluciBot paves the way to ratiocinate (76.0% test F1 score, 46.6% in saved computation on hallucinatory queries), rewrite (+30.2% positive class transition from hallucinatory to non-hallucinatory), rank (+50.6% positive class transition from hallucinatory to non-hallucinatory), and route queries to effective pipelines.

We develop a new neural network architecture that strictly enforces constitutive constraints such as polyconvexity, frame-indifference, and the symmetry of the stress and material stiffness. Additionally, we show that the accuracy of the stress and material stiffness predictions is significantly improved for this neural network by using a Sobolev minimization strategy that includes derivative terms. Using our neural network, we model the constitutive behavior of fibrous-type discrete network material. With Sobolev minimization, we obtain a normalized mean square error of 0.15% for the strain energy density, 0.815% averaged across the components of the stress, and 5.4% averaged across the components of the stiffness tensor. This machine-learned constitutive model was deployed in a finite element simulation of a facet capsular ligament. The displacement fields and stress-strain curves were compared to a multiscale simulation that required running on a GPU-based supercomputer. The new approach maintained upward of 85% accuracy in stress up to 70% strain while reducing the computation cost by orders of magnitude., Comment: Data and code can be found at https://zenodo.org/doi/10.5281/zenodo.11205879 and https://github.com/LACES-LAB/ML-biotissues

As highly automated vehicles reach higher deployment rates, they find themselves in increasingly dangerous situations. Knowing that the consequence of a crash is significant for the health of occupants, bystanders, and properties, as well as to the viability of autonomy and adjacent businesses, we must search for more efficacious ways to comprehensively and reliably train autonomous vehicles to better navigate the complex scenarios with which they struggle. We therefore introduce a taxonomy of potentially adversarial elements that may contribute to poor performance or system failures as a means of identifying and elucidating lesser-seen risks. This taxonomy may be used to characterize failures of automation, as well as to support simulation and real-world training efforts by providing a more comprehensive classification system for events resulting in disengagement, collision, or other negative consequences. This taxonomy is created from and tested against real collision events to ensure comprehensive coverage with minimal class overlap and few omissions. It is intended to be used both for the identification of harm-contributing adversarial events and in the generation thereof (to create extreme edge- and corner-case scenarios) in training procedures., Comment: 18 pages total, 4 pages of references, initial page left blank for IEEE submission statement. Includes 4 figures and 2 tables. Written using IEEEtran document class

We theoretically study the single photon transport properties in periodic and position-disordered Jaynes-Cummings (or JC) arrays of waveguide-coupled microtoroidal ring resonators, each interacting with a single two-level quantum emitter. Employing the real-space formalism of quantum optics, we focus on various parameter regimes of cavity quantum electrodynamics (cQED) to gain better control of single photon propagation in such a many-body quantum optical setting. As for some of the key findings, we observe that the periodic setting leads to the formation of the band structure in the photon transmission spectra, which is most evident in the strong coupling regime of cQCD. However, under the resonant conditions with no losses, the application of Bloch's theorem indicates that the width of forbidden gaps can be altered by tuning the emitter-cavity coupling to small values. Moreover, in the disordered case, we find that the single photon transmission curves show the disappearance of band formation. However, spectral features originating from cQED interactions observed for single atom-cavity problem remain robust against weak-disordered conditions. The results of this work may find application in the study of quantum many-body effects in the optical domain as well as in different areas of quantum computation and quantum networking., Comment: 12 pages, 5 figures

We investigate the deconfinement transition driven by excitations in long-range spin models. At low temperatures, these models exhibit a confined phase where domain-wall (or kinks) are localized. As temperature increases, kinks interact and propagate, leading to a transition to a de-confined phase. This transition is influenced by the interplay between thermal energy and interaction effects, resulting in extended, de-confined regions. Although kinks density is dynamically stable, non-equilibrium changes in their fluctuations characterize the transition. Our findings provide insights into the mechanisms of confinement and deconfinement in long-range spin models, with implications for both condensed matter physics and lattice gauge theories. Bridging these fields, this study sheds light on the universal aspects of confinement and opens avenues for further exploration and experimental verification., Comment: Main Text : 6 pages, 5 figures/ Supplemental material: 5 pages, 4 figures

Steganography, or hiding messages in plain sight, is a form of information hiding that is most commonly used for covert communication. As modern steganographic mediums include images, text, audio, and video, this communication method is being increasingly used by bad actors to propagate malware, exfiltrate data, and discreetly communicate. Current protection mechanisms rely upon steganalysis, or the detection of steganography, but these approaches are dependent upon prior knowledge, such as steganographic signatures from publicly available tools and statistical knowledge about known hiding methods. These dependencies render steganalysis useless against new or unique hiding methods, which are becoming increasingly common with the application of deep learning models. To mitigate the shortcomings of steganalysis, this work focuses on a deep learning sanitization technique called SUDS that is not reliant upon knowledge of steganographic hiding techniques and is able to sanitize universal and dependent steganography. SUDS is tested using least significant bit method (LSB), dependent deep hiding (DDH), and universal deep hiding (UDH). We demonstrate the capabilities and limitations of SUDS by answering five research questions, including baseline comparisons and an ablation study. Additionally, we apply SUDS to a real-world scenario, where it is able to increase the resistance of a poisoned classifier against attacks by 1375%., Comment: Accepted to European Conference on Artificial Intelligence (ECAI) 2023

We introduce Augmented Math, a machine learning-based approach to authoring AR explorable explanations by augmenting static math textbooks without programming. To augment a static document, our system first extracts mathematical formulas and figures from a given document using optical character recognition (OCR) and computer vision. By binding and manipulating these extracted contents, the user can see the interactive animation overlaid onto the document through mobile AR interfaces. This empowers non-technical users, such as teachers or students, to transform existing math textbooks and handouts into on-demand and personalized explorable explanations. To design our system, we first analyzed existing explorable math explanations to identify common design strategies. Based on the findings, we developed a set of augmentation techniques that can be automatically generated based on the extracted content, which are 1) dynamic values, 2) interactive figures, 3) relationship highlights, 4) concrete examples, and 5) step-by-step hints. To evaluate our system, we conduct two user studies: preliminary user testing and expert interviews. The study results confirm that our system allows more engaging experiences for learning math concepts., Comment: UIST 2023