Search

Conformal Prediction is a robust framework that ensures reliable coverage across machine learning tasks. Although recent studies have applied conformal prediction to graph neural networks, they have largely emphasized post-hoc prediction set generation. Improving conformal prediction during the training stage remains unaddressed. In this work, we tackle this challenge from a denoising perspective by introducing SparGCP, which incorporates graph sparsification and a conformal prediction-specific objective into GNN training. SparGCP employs a parameterized graph sparsification module to filter out task-irrelevant edges, thereby improving conformal prediction efficiency. Extensive experiments on real-world graph datasets demonstrate that SparGCP outperforms existing methods, reducing prediction set sizes by an average of 32\% and scaling seamlessly to large networks on commodity GPUs.

Conformal prediction has become increasingly popular for quantifying the uncertainty associated with machine learning models. Recent work in graph uncertainty quantification has built upon this approach for conformal graph prediction. The nascent nature of these explorations has led to conflicting choices for implementations, baselines, and method evaluation. In this work, we analyze the design choices made in the literature and discuss the tradeoffs associated with existing methods. Building on the existing implementations for existing methods, we introduce techniques to scale existing methods to large-scale graph datasets without sacrificing performance. Our theoretical and empirical results justify our recommendations for future scholarship in graph conformal prediction.

The development of large language models (LLMs) has expanded to multi-modal systems capable of processing text, images, and speech within a unified framework. Training these models demands significantly larger datasets and computational resources compared to text-only LLMs. To address the scaling challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal transformer architecture that significantly reduces pretraining computational costs. MoT decouples non-embedding parameters of the model by modality -- including feed-forward networks, attention matrices, and layer normalization -- enabling modality-specific processing with global self-attention over the full input sequence. We evaluate MoT across multiple settings and model scales. In the Chameleon 7B setting (autoregressive text-and-image generation), MoT matches the dense baseline's performance using only 55.8\% of the FLOPs. When extended to include speech, MoT reaches speech performance comparable to the dense baseline with only 37.2\% of the FLOPs. In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image generation metrics. System profiling further highlights MoT's practical benefits, achieving dense baseline image quality in 47.2\% of the wall-clock time and text quality in 75.6\% of the wall-clock time (measured on AWS p4de.24xlarge instances with NVIDIA A100 GPUs).

The widespread availability of maps has enabled the development of numerous location-based applications, including navigation, ride-sharing, fitness tracking, gaming, robotics, and augmented reality. Today, the maps that power these services are predominantly controlled by a few large corporations and mostly cover outdoor spaces. As the use of these applications expands and indoor localization technologies advance, we are seeing the need for a scalable, federated location management system that can extend into private spaces. We introduce OpenFLAME (Open Federated Localization and Mapping Engine), the first federated and decentralized localization service. OpenFLAME links servers that handle localization for specific regions, providing applications with a seamless global view. Creating a federated localization system poses challenges, such as discovering the appropriate servers for a region and integrating services managed by independent providers. To address these issues and ensure scalability, we leverage Domain Name System (DNS) for service discovery and implement map abstractions to retrieve and merge locations across different maps. Our trace-driven study demonstrates that federated localization across remote servers is feasible with acceptable query latencies. To highlight the potential of the system, we developed an augmented reality navigation application for a large indoor space, showing that OpenFLAME can successfully power location-based applications.

We develop a robust translation model for four low-resource Indic languages: Khasi, Mizo, Manipuri, and Assamese. Our approach includes a comprehensive pipeline from data collection and preprocessing to training and evaluation, leveraging data from WMT task datasets, BPCC, PMIndia, and OpenLanguageData. To address the scarcity of bilingual data, we use back-translation techniques on monolingual datasets for Mizo and Khasi, significantly expanding our training corpus. We fine-tune the pre-trained NLLB 3.3B model for Assamese, Mizo, and Manipuri, achieving improved performance over the baseline. For Khasi, which is not supported by the NLLB model, we introduce special tokens and train the model on our Khasi corpus. Our training involves masked language modelling, followed by fine-tuning for English-to-Indic and Indic-to-English translations., Comment: To be published in WMT 2024. Low-Resource Indic Language Translation Shared Task

We consider the problems of testing and learning an unknown $n$-qubit Hamiltonian $H$ from queries to its evolution operator $e^{-iHt}$ under the normalized Frobenius norm. We prove: 1. Local Hamiltonians: We give a tolerant testing protocol to decide if $H$ is $\epsilon_1$-close to $k$-local or $\epsilon_2$-far from $k$-local, with $O(1/(\epsilon_2-\epsilon_1)^{4})$ queries, solving open questions posed in a recent work by Bluhm et al. For learning a $k$-local $H$ up to error $\epsilon$, we give a protocol with query complexity $\exp(O(k^2+k\log(1/\epsilon)))$ independent of $n$, by leveraging the non-commutative Bohnenblust-Hille inequality. 2. Sparse Hamiltonians: We give a protocol to test if $H$ is $\epsilon_1$-close to being $s$-sparse (in the Pauli basis) or $\epsilon_2$-far from being $s$-sparse, with $O(s^{6}/(\epsilon_2^2-\epsilon_1^2)^{6})$ queries. For learning up to error $\epsilon$, we show that $O(s^{4}/\epsilon^{8})$ queries suffice. 3. Learning without memory: The learning results stated above have no dependence on $n$, but require $n$-qubit quantum memory. We give subroutines that allow us to learn without memory; increasing the query complexity by a $(\log n)$-factor in the local case and an $n$-factor in the sparse case. 4. Testing without memory: We give a new subroutine called Pauli hashing, which allows one to tolerantly test $s$-sparse Hamiltonians with $O(s^{14}/(\epsilon_2^2-\epsilon_1^2)^{18})$ queries. A key ingredient is showing that $s$-sparse Pauli channels can be tolerantly tested under the diamond norm with $O(s^2/(\epsilon_2-\epsilon_1)^6)$ queries. Along the way, we prove new structural theorems for local and sparse Hamiltonians. We complement our learning results with polynomially weaker lower bounds. Furthermore, our algorithms use short time evolutions and do not assume prior knowledge of the terms in the support of the Pauli spectrum of $H$., Comment: 45 pages. This work subsumes a prior work by the third author (arXiv:2404.06282)

Eye tracking technology offers great potential for improving road safety. It is already being built into vehicles, namely cars and trucks. When this technology is integrated into transit service vehicles, employees, i.e., bus drivers, will be subject to being eye tracked on their job. Although there is much research effort advancing algorithms for eye tracking in transportation, less is known about how end users perceive this technology, especially when interacting with it in an employer-mandated context. In this first study of its kind, we investigated transit bus operators' perceptions of eye tracking technology. From a methodological perspective, we introduce a mixed methods approach where participants experience the technology first-hand and then reflect on their experience while viewing a playback of the recorded data. Thematic analysis of the interview transcripts reveals interesting potential uses of eye tracking in this work context and surfaces transit operators' fears and concerns about this technology.

This work explores conditions under which multi-finger grasping algorithms can attain robust sim-to-real transfer. While numerous large datasets facilitate learning generative models for multi-finger grasping at scale, reliable real-world dexterous grasping remains challenging, with most methods degrading when deployed on hardware. An alternate strategy is to use discriminative grasp evaluation models for grasp selection and refinement, conditioned on real-world sensor measurements. This paradigm has produced state-of-the-art results for vision-based parallel-jaw grasping, but remains unproven in the multi-finger setting. In this work, we find that existing datasets and methods have been insufficient for training discriminitive models for multi-finger grasping. To train grasp evaluators at scale, datasets must provide on the order of millions of grasps, including both positive and negative examples, with corresponding visual data resembling measurements at inference time. To that end, we release a new, open-source dataset of 3.5M grasps on 4.3K objects annotated with RGB images, point clouds, and trained NeRFs. Leveraging this dataset, we train vision-based grasp evaluators that outperform both analytic and generative modeling-based baselines on extensive simulated and real-world trials across a diverse range of objects. We show via numerous ablations that the key factor for performance is indeed the evaluator, and that its quality degrades as the dataset shrinks, demonstrating the importance of our new dataset. Project website at: https://sites.google.com/view/get-a-grip-dataset.

The power of foundation models (FMs) lies in their capacity to learn highly expressive representations that can be adapted to a broad spectrum of tasks. However, these pretrained models require multiple stages of fine-tuning to become effective for downstream applications. Conventionally, the model is first retrained on the aggregate of a diverse set of tasks of interest and then adapted to specific low-resource downstream tasks by utilizing a parameter-efficient fine-tuning (PEFT) scheme. While this two-phase procedure seems reasonable, the independence of the retraining and fine-tuning phases causes a major issue, as there is no guarantee the retrained model will achieve good performance post-fine-tuning. To explicitly address this issue, we introduce a meta-learning framework infused with PEFT in this intermediate retraining stage to learn a model that can be easily adapted to unseen tasks. For our theoretical results, we focus on linear models using low-rank adaptations. In this setting, we demonstrate the suboptimality of standard retraining for finding an adaptable set of parameters. Further, we prove that our method recovers the optimally adaptable parameters. We then apply these theoretical insights to retraining the RoBERTa model to predict the continuation of conversations between different personas within the ConvAI2 dataset. Empirically, we observe significant performance benefits using our proposed meta-learning scheme during retraining relative to the conventional approach., Comment: Preprint

We show an improved inverse theorem for the Gowers-$3$ norm of $n$-qubit quantum states $|\psi\rangle$ which states that: for every $\gamma\geq 0$, if the $\textsf{Gowers}(|\psi \rangle,3)^8 \geq \gamma$ then the stabilizer fidelity of $|\psi\rangle$ is at least $\gamma^C$ for some constant $C>1$. This implies a constant-sample polynomial-time tolerant testing algorithm for stabilizer states which accepts if an unknown state is $\varepsilon_1$-close to a stabilizer state in fidelity and rejects when $|\psi\rangle$ is $\varepsilon_2 \leq \varepsilon_1^C$-far from all stabilizer states, promised one of them is the case., Comment: 6 pages

Pore-scale modeling of rock images based on information in 3D micro-computed tomography data is crucial for studying complex subsurface processes such as CO2 and brine multiphase flow during Geologic Carbon Storage (GCS). While deep learning models can generate 3D rock microstructures that match static rock properties, they have two key limitations: they don't account for the spatial distribution of rock properties that can have an important influence on the flow and transport characteristics (such as permeability and relative permeability) of the rock and they generate structures below the representative elementary volume (REV) scale for those transport properties. Addressing these issues is crucial for building a consistent workflow between pore-scale analysis and field-scale modeling. To address these challenges, we propose a two-stage modeling framework that combines a Vector Quantized Variational Autoencoder (VQVAE) and a transformer model for spatial upscaling and arbitrary-size 3D porous media reconstruction in an autoregressive manner. The VQVAE first compresses and quantizes sub-volume training images into low-dimensional tokens, while we train a transformer to spatially assemble these tokens into larger images following specific spatial order. By employing a multi-token generation strategy, our approach preserves both sub-volume integrity and spatial relationships among these sub-image patches. We demonstrate the effectiveness of our multi-token transformer generation approach and validate it using real data from a test well, showcasing its potential to generate models for the porous media at the well scale using only a spatial porosity model. The interpolated representative porous media that reflect field-scale geological properties accurately model transport properties, including permeability and multiphase flow relative permeability of CO2 and brine., Comment: 24 pages

Our interest is in constructing interactive systems involving a human-expert interacting with a machine learning engine on data analysis tasks. This is of relevance when addressing complex problems arising in areas of science, the environment, medicine and so on, which are not immediately amenable to the usual methods of statistical or mathematical modelling. In such situations, it is possible that harnessing human expertise and creativity to modern machine-learning capabilities of identifying patterns by constructing new internal representations of the data may provide some insight to possible solutions. In this paper, we examine the implementation of an abstract protocol developed for interaction between agents, each capable of constructing predictions and explanations. The \PXP protocol, described in [12] is motivated by the notion of ''two-way intelligibility'' and is specified using a pair of communicating finite-state machines. While the formalisation allows the authors to prove several properties about the protocol, no implementation was presented. Here, we address this shortcoming for the case in which one of the agents acts as a ''generator'' using a large language model (LLM) and the other is an agent that acts as a ''tester'' using either a human-expert, or a proxy for a human-expert (for example, a database compiled using human-expertise). We believe these use-cases will be a widely applicable form of interaction for problems of the kind mentioned above. We present an algorithmic description of general-purpose implementation, and conduct preliminary experiments on its use in two different areas (radiology and drug-discovery). The experimental results provide early evidence in support of the protocol's capability of capturing one- and two-way intelligibility in human-LLM in the manner proposed in [12].

This position paper proposes a novel approach to advancing NLP security by leveraging Large Language Models (LLMs) as engines for generating diverse adversarial attacks. Building upon recent work demonstrating LLMs' effectiveness in creating word-level adversarial examples, we argue for expanding this concept to encompass a broader range of attack types, including adversarial patches, universal perturbations, and targeted attacks. We posit that LLMs' sophisticated language understanding and generation capabilities can produce more effective, semantically coherent, and human-like adversarial examples across various domains and classifier architectures. This paradigm shift in adversarial NLP has far-reaching implications, potentially enhancing model robustness, uncovering new vulnerabilities, and driving innovation in defense mechanisms. By exploring this new frontier, we aim to contribute to the development of more secure, reliable, and trustworthy NLP systems for critical applications., Comment: 5 pages

Potential-driven steady-state flow in networks is an abstract problem which manifests in various engineering applications, such as transport of natural gas, water, electric power through infrastructure networks or flow through fractured rocks modeled as discrete fracture networks. The relevance of steady-state network flow to control systems and optimization, as well as the question of the existence of a solution for a particular class of flows, has been established in a prior article (IEEE Control Systems Letters (2024), doi:10.1109/LCSYS.2024.3394317). Building on that foundation, this article concerns itself with computation of such a solution for a large network since the problem while simple when restricted to a single edge of a network, ceases to be so for a large network. The resultant system of nonlinear equations depends on the network topology and in general there is no numerical algorithm that offers guaranteed convergence to the solution (assuming a solution exists). Some methods offer guarantees in cases where the network topology satisfies certain assumptions, but these methods fail for larger networks. On the other hand, the Newton-Raphson algorithm offers a convergence guarantee if the starting point lies close to the (unknown) solution. It would be advantageous to compute the solution of the large nonlinear system through the solution of smaller nonlinear sub-systems wherein the solution algorithms (Newton-Raphson or otherwise) are more likely to succeed. This article proposes and describes such a procedure, an hierarchical network partitioning algorithm that enables the solution of large nonlinear systems corresponding to potential-driven steady-state network flow equations.

Large language models (LLMs) such as ChatGPT, GPT-4, Claude-3, and Llama are being integrated across a variety of industries. Despite this rapid proliferation, experts are calling for caution in the interpretation and adoption of LLMs, owing to numerous associated ethical concerns. Research has also uncovered shortcomings in LLMs' reasoning and logical abilities, raising questions on the potential of LLMs as evaluation tools. In this paper, we investigate LLMs' self-evaluation capabilities on a novel proverb reasoning task. We introduce a novel proverb database consisting of 300 proverb pairs that are similar in intent but different in wordings, across topics spanning gender, wisdom, and society. We propose tests to evaluate textual consistencies as well as numerical consistencies across similar proverbs, and demonstrate the effectiveness of our method and dataset in identifying failures in LLMs' self-evaluation which in turn can highlight issues related to gender stereotypes and lack of cultural understanding in LLMs.

This paper proposes a nonlinear magnetics model for Permanent Magnet Synchronous Machines (PMSMs) that accurately captures the effects of magnetic saturation in the machine iron and variations in rotor temperature on the permanent magnet excitation. The proposed model considers the permanent magnet as a current source rather than the more commonly used flux-linkage source. A comparison of the two modelling approaches is conducted using Finite Element Analysis (FEA) for different machine designs as well as experimental validation, where it is shown that the proposed model has substantially better accuracy. The proposed model decouples magnetic saturation and rotor temperature effects in the current/flux-linkage relationship, allowing for adaptive estimation of the PM excitation.

Pedestrian dynamics simulates the fine-scaled trajectories of individuals in a crowd. It has been used to suggest public health interventions to reduce infection risk in important components of air travel, such as during boarding and in airport security lines. Due to inherent variability in human behavior, it is difficult to generalize simulation results to new geographic, cultural, or temporal contexts. A digital twin, relying on real-time data, such as video feeds, can resolve this limitation. This paper addresses the following critical gaps in knowledge required for a digital twin. (1) Pedestrian dynamics models currently lack accurate representations of collision avoidance behavior when two moving pedestrians try to avoid collisions. (2) It is not known whether data assimilation techniques designed for physical systems are effective for pedestrian dynamics. We address the first limitation by training a model with data from offline video feeds of collision avoidance to simulate these trajectories realistically, using symbolic regression to identify unknown functional forms. We address the second limitation by showing that pedestrian dynamics with data assimilation can predict pedestrian trajectories with sufficient accuracy. These results promise to enable the development of a digital twin for pedestrian movement in airports that can help with real-time crowd management to reduce health risks., Comment: Submitted to IEEE

We consider the task of distributed inner product estimation when allowed limited quantum communication. Here, Alice and Bob are given $k$ copies of an unknown $n$-qubit quantum states $\vert \psi \rangle,\vert \phi \rangle$ respectively. They are allowed to communicate $q$ qubits and unlimited classical communication, and their goal is to estimate $|\langle \psi|\phi\rangle|^2$ up to constant accuracy. We show that $k=\Theta(\sqrt{2^{n-q}})$ copies are essentially necessary and sufficient for this task (extending the work of Anshu, Landau and Liu (STOC'22) who considered the case when $q=0$). Additionally, we consider estimating $|\langle \psi|M|\phi\rangle|^2$, for arbitrary Hermitian $M$. For this task we show that certain norms on $M$ characterize the sample complexity of estimating $|\langle \psi|M|\phi\rangle|^2$ when using only classical~communication., Comment: 26 pages, 2 figures

Recent LLM (Large Language Models) advancements benefit many fields such as education and finance, but HR has hundreds of repetitive processes, such as access requests, medical claim filing and time-off submissions, which are unaddressed. We relate these tasks to the LLM agent, which has addressed tasks such as writing assisting and customer support. We present HR-Agent, an efficient, confidential, and HR-specific LLM-based task-oriented dialogue system tailored for automating repetitive HR processes such as medical claims and access requests. Since conversation data is not sent to an LLM during inference, it preserves confidentiality required in HR-related tasks.

Exploring the potential of quantum hardware for enhancing classical and real-world applications is an ongoing challenge. This study evaluates the performance of quantum and quantum-inspired methods compared to classical models for crack segmentation. Using annotated gray-scale image patches of concrete samples, we benchmark a classical mean Gaussian mixture technique, a quantum-inspired fermion-based method, Q-Seg a quantum annealing-based method, and a U-Net deep learning architecture. Our results indicate that quantum-inspired and quantum methods offer a promising alternative for image segmentation, particularly for complex crack patterns, and could be applied in near-future applications.

The first very long baseline interferometry (VLBI) detections at 870$\mu$m wavelength (345$\,$GHz frequency) are reported, achieving the highest diffraction-limited angular resolution yet obtained from the surface of the Earth, and the highest-frequency example of the VLBI technique to date. These include strong detections for multiple sources observed on inter-continental baselines between telescopes in Chile, Hawaii, and Spain, obtained during observations in October 2018. The longest-baseline detections approach 11$\,$G$\lambda$ corresponding to an angular resolution, or fringe spacing, of 19$\mu$as. The Allan deviation of the visibility phase at 870$\mu$m is comparable to that at 1.3$\,$mm on the relevant integration time scales between 2 and 100$\,$s. The detections confirm that the sensitivity and signal chain stability of stations in the Event Horizon Telescope (EHT) array are suitable for VLBI observations at 870$\mu$m. Operation at this short wavelength, combined with anticipated enhancements of the EHT, will lead to a unique high angular resolution instrument for black hole studies, capable of resolving the event horizons of supermassive black holes in both space and time., Comment: Corresponding author: S. Doeleman

A typical manipulation task consists of a manipulator equipped with a gripper to grasp and move an object with constraints on the motion of the hand-held object, which may be due to the nature of the task itself or from object-environment contacts. In this paper, we study the problem of computing joint torques and grasping forces for time-optimal motion of an object, while ensuring that the grasp is not lost and any constraints on the motion of the object, either due to dynamics, environment contact, or no-slip requirements, are also satisfied. We present a second-order cone program (SOCP) formulation of the time-optimal trajectory planning problem that considers nonlinear friction cone constraints at the hand-object and object-environment contacts. Since SOCPs are convex optimization problems that can be solved optimally in polynomial time using interior point methods, we can solve the trajectory optimization problem efficiently. We present simulation results on three examples, including a non-prehensile manipulation task, which shows the generality and effectiveness of our approach.

The development of tools and techniques to analyze and extract organizations data habits from privacy policies are critical for scalable regulatory compliance audits. Unfortunately, these tools are becoming increasingly limited in their ability to identify compliance issues and fixes. After all, most were developed using regulation-agnostic datasets of annotated privacy policies obtained from a time before the introduction of landmark privacy regulations such as EUs GDPR and Californias CCPA. In this paper, we describe the first open regulation-aware dataset of expert-annotated privacy policies, C3PA (CCPA Privacy Policy Provision Annotations), aimed to address this challenge. C3PA contains over 48K expert-labeled privacy policy text segments associated with responses to CCPA-specific disclosure mandates from 411 unique organizations. We demonstrate that the C3PA dataset is uniquely suited for aiding automated audits of compliance with CCPA-related disclosure mandates., Comment: 9 pages, EMNLP 2024

Existing automatic prompt engineering methods are typically designed for discriminative tasks, where new task prompts are iteratively refined with limited feedback from a single metric reflecting a single aspect. However, these approaches are suboptimal for generative tasks, which require more nuanced guidance beyond a single numeric metric to improve the prompt and optimize multiple aspects of the generated text. To address these challenges, we propose a novel multi-aspect Critique-Suggestion-guided automatic Prompt Optimization (CriSPO) approach. CriSPO introduces a critique-suggestion module as its core component. This module spontaneously discovers aspects, and compares generated and reference texts across these aspects, providing specific suggestions for prompt modification. These clear critiques and actionable suggestions guide a receptive optimizer module to make more substantial changes, exploring a broader and more effective search space. To further improve CriSPO with multi-metric optimization, we introduce an Automatic Suffix Tuning (AST) extension to enhance the performance of task prompts across multiple metrics. We evaluate CriSPO on 4 state-of-the-art LLMs across 4 summarization and 5 QA datasets. Extensive experiments show 3-4\% ROUGE score improvement on summarization and substantial improvement of various metrics on QA.

Retrieval Augmented Generation (RAG) is a widely used approach for leveraging external context in several natural language applications such as question answering and information retrieval. Yet, the exact nature in which a Language Model (LM) leverages this non-parametric memory or retrieved context isn't clearly understood. This paper mechanistically examines the RAG pipeline to highlight that LMs demonstrate a "shortcut'' effect and have a strong bias towards utilizing the retrieved context to answer questions, while relying minimally on model priors. We propose (a) Causal Mediation Analysis; for proving that parametric memory is minimally utilized when answering a question and (b) Attention Contributions and Knockouts for showing the last token residual stream do not get enriched from the subject token in the question, but gets enriched from tokens of RAG-context. We find this pronounced "shortcut'' behaviour to be true across both LLMs (e.g.,LlaMa) and SLMs (e.g., Phi), Comment: Accepted to Blackbox NLP @ EMNLP 2024

Large Language Models (LLMs) are poised to revolutionize healthcare. Ophthalmology-specific LLMs remain scarce and underexplored. We introduced an open-source, specialized LLM for ophthalmology, termed Language Enhanced Model for Eye (LEME). LEME was initially pre-trained on the Llama2 70B framework and further fine-tuned with a corpus of ~127,000 non-copyrighted training instances curated from ophthalmology-specific case reports, abstracts, and open-source study materials. We benchmarked LEME against eight other LLMs, namely, GPT-3.5, GPT-4, three Llama2 models (7B, 13B, 70B), PMC-LLAMA 13B, Meditron 70B, and EYE-Llama (another ophthalmology-specific LLM). Evaluations included four internal validation tasks: abstract completion, fill-in-the-blank, multiple-choice questions (MCQ), and short-answer QA. External validation tasks encompassed long-form QA, MCQ, patient EHR summarization, and clinical QA. Evaluation metrics included Rouge-L scores, accuracy, and expert evaluation of correctness, completeness, and readability. In internal validations, LEME consistently outperformed its counterparts, achieving Rouge-L scores of 0.20 in abstract completion (all p<0.05), 0.82 in fill-in-the-blank (all p<0.0001), and 0.22 in short-answer QA (all p<0.0001, except versus GPT-4). In external validations, LEME excelled in long-form QA with a Rouge-L of 0.19 (all p<0.0001), ranked second in MCQ accuracy (0.68; all p<0.0001), and scored highest in EHR summarization and clinical QA (ranging from 4.24 to 4.83 out of 5 for correctness, completeness, and readability). LEME's emphasis on robust fine-tuning and the use of non-copyrighted data represents a breakthrough in open-source ophthalmology-specific LLMs, offering the potential to revolutionize execution of clinical tasks while democratizing research collaboration.

Large Language Models (LLMs) have revolutionized natural language processing, yet they struggle with inconsistent reasoning, particularly in novel domains and complex logical sequences. This research introduces Proof of Thought, a framework that enhances the reliability and transparency of LLM outputs. Our approach bridges LLM-generated ideas with formal logic verification, employing a custom interpreter to convert LLM outputs into First Order Logic constructs for theorem prover scrutiny. Central to our method is an intermediary JSON-based Domain-Specific Language, which by design balances precise logical structures with intuitive human concepts. This hybrid representation enables both rigorous validation and accessible human comprehension of LLM reasoning processes. Key contributions include a robust type system with sort management for enhanced logical integrity, explicit representation of rules for clear distinction between factual and inferential knowledge, and a flexible architecture that allows for easy extension to various domain-specific applications. We demonstrate Proof of Thought's effectiveness through benchmarking on StrategyQA and a novel multimodal reasoning task, showing improved performance in open-ended scenarios. By providing verifiable and interpretable results, our technique addresses critical needs for AI system accountability and sets a foundation for human-in-the-loop oversight in high-stakes domains., Comment: 38th Conference on Neural Information Processing Systems (NeurIPS 2024) System 2 Reasoning At Scale Workshop

This paper presents a Dynamic Internal Predictive Power Scheduling (DIPPS) approach for optimizing power management in microgrids, particularly focusingon external power exchanges among diverse prosumers. DIPPS utilizes a dynamic objective function with a time-varying binary parameter to control the timing of power transfers to the external grid, facilitated by efficient usage of energy storage for surplus renewable power. The microgrid power scheduling problem is modeled as a mixed-integer nonlinear programmig (MINLP-PS) and subsequently transformed into a mixed-integer linear programming (MILP-PS) optimization through McCormick's relaxation to reduce the computational complexity. A predictive window with 6 data points is solved at an average of 0.92s, a 97.6% improvement over the 38.27s required for the MINLP-PS formulation, implying the numerical feasibility of the DIPPS approach for real-time implementation. Finally, the approach is validated against a static objective using real-world load data across three case studies with different time-varying parameters, demonstrationg the ability of DIPPS to optimize power exchanges and efficiently utilize distributed resources whie shifting the eexternal power transfers to specified time durations.

This paper presents the development of a novel plant communication application that allows plants to "talk" to humans using real-time sensor data and AI-powered language models. Utilizing soil sensors that track moisture, temperature, and nutrient levels, the system feeds this data into the Gemini API, where it is processed and transformed into natural language insights about the plant's health and "mood." Developed using Flutter, Firebase, and ThingSpeak, the app offers a seamless user experience with real-time interaction capabilities. By fostering human-plant connectivity, this system enhances plant care practices, promotes sustainability, and introduces innovative applications for AI and IoT technologies in both personal and agricultural contexts. The paper explores the technical architecture, system integration, and broader implications of AI-driven plant communication., Comment: Pre-print Version. Submitted to conference

Large language models (LLMs) have shown remarkable capabilities in various natural language tasks and are increasingly being applied in healthcare domains. This work demonstrates a new LLM-powered disease risk assessment approach via streaming human-AI conversation, eliminating the need for programming required by traditional machine learning approaches. In a COVID-19 severity risk assessment case study, we fine-tune pre-trained generative LLMs (e.g., Llama2-7b and Flan-t5-xl) using a few shots of natural language examples, comparing their performance with traditional classifiers (i.e., Logistic Regression, XGBoost, Random Forest) that are trained de novo using tabular data across various experimental settings. We develop a mobile application that uses these fine-tuned LLMs as its generative AI (GenAI) core to facilitate real-time interaction between clinicians and patients, providing no-code risk assessment through conversational interfaces. This integration not only allows for the use of streaming Questions and Answers (QA) as inputs but also offers personalized feature importance analysis derived from the LLM's attention layers, enhancing the interpretability of risk assessments. By achieving high Area Under the Curve (AUC) scores with a limited number of fine-tuning samples, our results demonstrate the potential of generative LLMs to outperform discriminative classification methods in low-data regimes, highlighting their real-world adaptability and effectiveness. This work aims to fill the existing gap in leveraging generative LLMs for interactive no-code risk assessment and to encourage further research in this emerging field.

We present ImPoster, a novel algorithm for generating a target image of a 'source' subject performing a 'driving' action. The inputs to our algorithm are a single pair of a source image with the subject that we wish to edit and a driving image with a subject of an arbitrary class performing the driving action, along with the text descriptions of the two images. Our approach is completely unsupervised and does not require any access to additional annotations like keypoints or pose. Our approach builds on a pretrained text-to-image latent diffusion model and learns the characteristics of the source and the driving image by finetuning the diffusion model for a small number of iterations. At inference time, ImPoster performs step-wise text prompting i.e. it denoises by first moving in the direction of the image manifold corresponding to the driving image followed by the direction of the image manifold corresponding to the text description of the desired target image. We propose a novel diffusion guidance formulation, image frequency guidance, to steer the generation towards the manifold of the source subject and the driving action at every step of the inference denoising. Our frequency guidance formulations are derived from the frequency domain properties of images. We extensively evaluate ImPoster on a diverse set of source-driving image pairs to demonstrate improvements over baselines. To the best of our knowledge, ImPoster is the first approach towards achieving both subject-driven as well as action-driven image personalization. Code and data is available at https://github.com/divyakraman/ImPosterDiffusion2024.

Understanding pedestrian behavior is crucial for the safe deployment of Autonomous Vehicles (AVs) in urban environments. Traditional pedestrian behavior models often fall into two categories: mechanistic models, which do not generalize well to complex environments, and machine-learned models, which generally overlook sensory-motor constraints influencing human behavior and thus prone to fail in untrained scenarios. We hypothesize that sensory-motor constraints, fundamental to how humans perceive and interact with their surroundings, are essential for realistic simulations. Thus, we introduce a constrained reinforcement learning (RL) model that simulates the crossing decision and locomotion of pedestrians. It was constrained to emulate human sensory mechanisms with noisy visual perception and looming aversion. Additionally, human motor constraint was incorporated through a bio-mechanical model of walking. We gathered data from a human-in-the-loop experiment to understand pedestrian behavior. The findings reveal several phenomena not addressed by existing pedestrian models, regarding how pedestrians adapt their walking speed to the kinematics and behavior of the approaching vehicle. Our model successfully captures these human-like walking speed patterns, enabling us to understand these patterns as a trade-off between time pressure and walking effort. Importantly, the model retains the ability to reproduce various phenomena previously captured by a simpler version of the model. Additionally, phenomena related to external human-machine interfaces and light conditions were also included. Overall, our results not only demonstrate the potential of constrained RL in modeling pedestrian behaviors but also highlight the importance of sensory-motor mechanisms in modeling pedestrian-vehicle interactions.

One of the standard composite insulation systems for cryogenic applications consists of a layer of hollow glass microspheres (HGMs) followed by a layer of variable density multilayer insulation (VDMLI) comprised of various internal combinations of reflectors and spacers. Microstructural effects of the HGM assembly and convection between the vessel and its surroundings are unaccounted for in existing calculations, which the current study incorporates in the unit cell-based HGM analytical model. Building on it, the insulation performance of the HGM layer with increasing thickness is studied for a Dewar insulation system containing liquid nitrogen (boiling point 78 K) based on pressures ranging from high vacuum (0.0133 Pa) to atmospheric (105 Pa) for two benchmark cryogenic vessel sizes. A suitable range of thickness for both systems is suggested. Further, the performance of four arithmetic progression-based VDMLI profiles in combination with an optimal HGM layer is studied. A VDMLI configuration with repeating spacer layers is found to perform best in high-pressure conditions. Finally, based on heat transfer performance within the given range of pressures and warm boundary temperatures up to 393 K, optimal thickness values of 120-130mm for the HGM layer and 125mm for the best VDMLI configuration are reported., Comment: 17 pages, 8 figures

We classify all solution triples with $k$-Fibonacci components to the equation $x^2+y^2+z^2=3xyz+m,$ where $m$ is a positive integer and $k\geq 2$. As a result, for $m=8$, we have the Markoff triples with Pell components $(F_2(2), F_2(2n), F_2(2n+2))$, for $n\geq 1$. For all other $m$ there exists at most one such ordered triple, except when $k=3,$ $a$ is odd, $b$ is even and $b\geq a+3$, where $(F_3(a),F_3(b),F_3(a+b))$ and $(F_3(a+1),F_3(b-1),F_3(a+b))$ share the same $m$., Comment: 15 pages, 1 figure

Micro-CT scanning of rocks significantly enhances our understanding of pore-scale physics in porous media. With advancements in pore-scale simulation methods, such as pore network models, it is now possible to accurately simulate multiphase flow properties, including relative permeability, from CT-scanned rock samples. However, the limited number of CT-scanned samples and the challenge of connecting pore-scale networks to field-scale rock properties often make it difficult to use pore-scale simulated properties in realistic field-scale reservoir simulations. Deep learning approaches to create synthetic 3D rock structures allow us to simulate variations in CT rock structures, which can then be used to compute representative rock properties and flow functions. However, most current deep learning methods for 3D rock structure synthesis don't consider rock properties derived from well observations, lacking a direct link between pore-scale structures and field-scale data. We present a method to construct 3D rock structures constrained to observed rock properties using generative adversarial networks (GANs) with conditioning accomplished through a gradual Gaussian deformation process. We begin by pre-training a Wasserstein GAN to reconstruct 3D rock structures. Subsequently, we use a pore network model simulator to compute rock properties. The latent vectors for image generation in GAN are progressively altered using the Gaussian deformation approach to produce 3D rock structures constrained by well-derived conditioning data. This GAN and Gaussian deformation approach enables high-resolution synthetic image generation and reproduces user-defined rock properties such as porosity, permeability, and pore size distribution. Our research provides a novel way to link GAN-generated models to field-derived quantities., Comment: 18 pages

There have been several modifications of how basic calculus has been taught, but very few of these modifications have considered the computational tools available at our disposal. Here, we present a few tools that are easy to develop and use. Doing so also addresses a different way to view calculus, and attempts to fill the gaps in students' understanding of both differentiation and integration. We will describe the basics of both these topics in a way that might be much more useful and relevant to students, and hence possible ways in refining calculus pedagogy to make calculus more accessible to them. For integration, an elementary development of Gaussian quadrature using basic linear algebra is presented. This numerical method can be extended to integrate functions over various domains in higher dimensions, a subject that is not currently well covered in multivariable calculus or other Mathematics courses. We also briefly discuss series that may be more useful than some of those taught in current calculus courses., Comment: 9 pages

Ni-Mo superalloys have emerged as materials of choice for a diverse array of applications owing to their superior mechanical properties, exceptional corrosion and oxidation resistance, electrocatalytic behavior, and surface stability. Understanding and optimizing the surface composition of Ni-Mo alloys is critical for enhancing their performance in practical applications. Traditional experimental surface analysis techniques, while informative, are often prohibitive in terms of cost and time. Likewise, theoretical approaches such as first-principle calculations demand substantial computational resources and it is difficult to simulate large structures. This study introduces an alternative approach utilizing hybrid Monte-Carlo / Molecular Dynamics (MC/MD) simulations to investigate the surface composition of Ni-Mo alloys. We report the development of an optimized Embedded-Atom Method (EAM) potential specifically for Ni-Mo alloys, carefully parameterized using empirical lattice constants and formation energies of elemental and face-centered cubic (FCC) Ni-Mo solid solution alloys. The reliability of the EAM potential is corroborated via the evaluation of equations of state, with a particular focus on reproducing structural properties. Utilizing this validated potential, MC/MD simulations were performed to understand the depth-wise variations in the compositions of Ni-Mo alloy nanoparticles and extended surfaces. These simulations reveal a preferential segregation of nickel on surface, and molybdenum in sub-surface layer. Due to this preferential segregation, it is imperative to consider surface segregation while tailoring the surface properties for targeted applications.

We consider the Weak Galerkin finite element approximation of the Singularly Perturbed Biharmonic elliptic problem on a unit square domain with clamped boundary conditions. Shishkin mesh is used for domain discretization as the solution exhibits boundary layers near the domain boundary. Error estimates in the equivalent $H^{2}-$ norm have been established and the uniform convergence of the proposed method has been proved. Numerical examples are presented corroborating our theoretical findings.

Stage IV large scale structure surveys are promising probes of gravity on cosmological scales. Due to the vast model-space in the modified gravity literature, model-independent parameterisations represent useful and scalable ways to test extensions of $\Lambda$CDM. In this work we use a recently validated approach of computing the non-linear $3\times 2$pt observables in modified gravity models with a time-varying effective gravitational constant $\mu$ and a gravitational slip $\eta$ that is binned in redshift to produce Fisher forecasts for an LSST Y10-like survey. We also include in our modelling an effective nulling scheme for weak-lensing by applying the BNT transformation that localises the weak-lensing kernel enabling well-informed scale cuts. We show that the combination of improved non-linear modelling and better control of the scales that are modelled/cut yields high precision constraints on the cosmological and modified gravity parameters. We find that 4 redshift bins for $\mu$ of width corresponding to equal incremental $\Lambda$CDM growth is optimal given the state-of-the-art modelling and show how the BNT transformation can be used to mitigate the impact of small-scale systematic effects, such as baryonic feedback., Comment: 27 pages, 12 figures A few typos corrected and a couple of small changes made to the text to improve presentation of results, added missing reference. Comments welcome!

State-of-the-art techniques for 3D reconstruction are largely based on volumetric scene representations, which require sampling multiple points to compute the color arriving along a ray. Using these representations for more general inverse rendering -- reconstructing geometry, materials, and lighting from observed images -- is challenging because recursively path-tracing such volumetric representations is expensive. Recent works alleviate this issue through the use of radiance caches: data structures that store the steady-state, infinite-bounce radiance arriving at any point from any direction. However, these solutions rely on approximations that introduce bias into the renderings and, more importantly, into the gradients used for optimization. We present a method that avoids these approximations while remaining computationally efficient. In particular, we leverage two techniques to reduce variance for unbiased estimators of the rendering equation: (1) an occlusion-aware importance sampler for incoming illumination and (2) a fast cache architecture that can be used as a control variate for the radiance from a high-quality, but more expensive, volumetric cache. We show that by removing these biases our approach improves the generality of radiance cache based inverse rendering, as well as increasing quality in the presence of challenging light transport effects such as specular reflections., Comment: Website: https://benattal.github.io/flash-cache/

Synchronization of oscillators is ubiquitous in nature. Often, the synchronized oscillators couple directly, yet in some cases synchronization can arise from their parametric interactions. Here, we theoretically predict and experimentally demonstrate the parametric synchronization of a dissipative Kerr soliton frequency comb. We specifically show that the parametric interaction between the soliton and two auxiliary lasers permits the entrainment of the frequency comb repetition rate. Besides representing the first prediction and demonstration of parametric synchronization of soliton frequency combs, our scheme offers significant flexibility for all-optical metrological-scale stabilization of the comb.

Over a century ago, Einstein formulated a precise mathematical model for describing Brownian motion. While this model adequately explains the diffusion of micron-sized particles in fluids, its limitations become apparent when applied to molecular self-diffusion in fluids. The foundational principles of Gaussianity and Markovianity, central to the Brownian diffusion paradigm, are insufficient for describing molecular diffusion, particularly in complex fluids characterized by intricate intermolecular interactions and hindered relaxation processes. This perspective delves into the nuanced behavior observed in diverse complex fluids, including molecular self-assembly, deep eutectic solvents, and ionic liquids, with a specific focus on modeling self-diffusion within these media. We explore the potential of extending diffusion models to incorporate non-Gaussian and non-Markovian effects by augmenting the Brownian model using non-local diffusion equations. Further, we validate the applicability of these models by utilizing them to describe results from quasielastic neutron scattering and MD simulations.

Multi-color solitons that are parametrically created in dual-pumped microresonators generate interleaved frequency combs that can be used to obtain combs at new frequencies and when synchronized can be used for low-noise microwave generation and potentially as an element in a chip-scale clockwork. Here, we first derive three-wave equations that describe multi-color solitons that appear in microresonators with a nearly quartic dispersion profile. These solitons are characterized by a single angular group velocity and multiple angular phase velocities. We then use these equations to explain the interleaved frequency combs that are observed at the output of the microresonator. Finally, we used these equations to describe the experimentally-observed soliton-OPO effect. In this effect, the pump frequency comb interacts nonlinearly with a signal frequency comb to create an idler frequency comb in a new frequency range, analogous to an optical parametric oscillation (OPO) process. We determine the conditions under which we expect this effect to occur. We anticipate that the three-wave equations and their extensions will be of use in designing new frequency comb systems and determining their stability and noise performance.

We consider the problem of independently, in a disentangled fashion, controlling the outputs of text-to-image diffusion models with color and style attributes of a user-supplied reference image. We present the first training-free, test-time-only method to disentangle and condition text-to-image models on color and style attributes from reference image. To realize this, we propose two key innovations. Our first contribution is to transform the latent codes at inference time using feature transformations that make the covariance matrix of current generation follow that of the reference image, helping meaningfully transfer color. Next, we observe that there exists a natural disentanglement between color and style in the LAB image space, which we exploit to transform the self-attention feature maps of the image being generated with respect to those of the reference computed from its L channel. Both these operations happen purely at test time and can be done independently or merged. This results in a flexible method where color and style information can come from the same reference image or two different sources, and a new generation can seamlessly fuse them in either scenario., Comment: 16 pages, 17 figures

The Prony method for approximating signals comprising sinusoidal/exponential components is known through the pioneering work of Prony in his seminal dissertation in the year 1795. However, the Prony method saw the light of real world application only upon the advent of the computational era, which made feasible the extensive numerical intricacies and labor which the method demands inherently. The Adaptive LMS Filter which has been the most pervasive method for signal filtration and approximation since its inception in 1965 does not provide a consistently assured level of highly precise results as the extended experiment in this work proves. As a remedy this study improvises upon the Prony method by observing that a better (more precise) computational approximation can be obtained under the premise that adjustment can be made for computational error , in the autoregressive model setup in the initial step of the Prony computation itself. This adjustment is in proportion to the deviation of the coefficients in the same autoregressive model. The results obtained by this improvisation live up to the expectations of obtaining consistency and higher value in the precision of the output (recovered signal) approximations as shown in this current work and as compared with the results obtained using the Adaptive LMS Filter.

We present Container Data Item (CDI), an abstract datatype that allows multiple containers to efficiently operate on a common data item while preserving their strong security and isolation semantics. Application developers can use CDIs to enable multiple containers to operate on the same data, synchronize execution among themselves, and control the ownership of the shared data item during runtime. These containers may reside on the same server or different servers. CDI is designed to support microservice based applications comprised of a set of interconnected microservices, each implemented by a separate dedicated container. CDI preserves the important isolation semantics of containers by ensuring that exactly one container owns a CDI object at any instant and the ownership of a CDI object may be transferred from one container to another only by the current CDI object owner. We present three different implementations of CDI that allow different containers residing on the same server as well containers residing on different servers to use CDI for efficiently operating on a common data item. The paper provides an extensive performance evaluation of CDI along with two representative applications, an augmented reality application and a decentralized workflow orchestrator.

Hand-crafting high quality prompts to optimize the performance of language models is a complicated and labor-intensive process. Furthermore, when migrating to newer, smaller, or weaker models (possibly due to latency or cost gains), prompts need to be updated to re-optimize the task performance. We propose Concept Distillation (CD), an automatic prompt optimization technique for enhancing weaker models on complex tasks. CD involves: (1) collecting mistakes made by weak models with a base prompt (initialization), (2) using a strong model to generate reasons for these mistakes and create rules/concepts for weak models (induction), and (3) filtering these rules based on validation set performance and integrating them into the base prompt (deduction/verification). We evaluated CD on NL2Code and mathematical reasoning tasks, observing significant performance boosts for small and weaker language models. Notably, Mistral-7B's accuracy on Multi-Arith increased by 20%, and Phi-3-mini-3.8B's accuracy on HumanEval rose by 34%. Compared to other automated methods, CD offers an effective, cost-efficient strategy for improving weak models' performance on complex tasks and enables seamless workload migration across different language models without compromising performance., Comment: 13 pages, 8 figures, conference

We introduce Imagen 3, a latent diffusion model that generates high quality images from text prompts. We describe our quality and responsibility evaluations. Imagen 3 is preferred over other state-of-the-art (SOTA) models at the time of evaluation. In addition, we discuss issues around safety and representation, as well as methods we used to minimize the potential harm of our models.