Search

This paper introduces a second-order hyperplane search, a novel optimization step that generalizes a second-order line search from a line to a $k$-dimensional hyperplane. This, combined with the forward-mode stochastic gradient method, yields a second-order optimization algorithm that consists of forward passes only, completely avoiding the storage overhead of backpropagation. Unlike recent work that relies on directional derivatives (or Jacobian--Vector Products, JVPs), we use hyper-dual numbers to jointly evaluate both directional derivatives and their second-order quadratic terms. As a result, we introduce forward-mode weight perturbation with Hessian information (FoMoH). We then use FoMoH to develop a novel generalization of line search by extending it to a hyperplane search. We illustrate the utility of this extension and how it might be used to overcome some of the recent challenges of optimizing machine learning models without backpropagation. Our code is open-sourced at https://github.com/SRI-CSL/fomoh., Comment: 14 pages, 8 figures

We resolve Stillman's conjecture for families of polynomial rings that are graded by any abelian group under mild conditions. Conversely, we show that these conditions are necessary for the existence of a Stillman bound. This has applications even for the well-known standard graded case., Comment: 7 pages

In applications such as personal assistants, large language models (LLMs) must consider the user's personal information and preferences. However, LLMs lack the inherent ability to learn from user interactions. This paper explores capturing personal information from user prompts using ontology and knowledge-graph approaches. We use a subset of the KNOW ontology, which models personal information, to train the language model on these concepts. We then evaluate the success of knowledge capture using a specially constructed dataset. Our code and datasets are publicly available at https://github.com/HaltiaAI/paper-PTODSKC, Comment: 7 pages, 5 figures

Augmenting large language models (LLMs) with user-specific knowledge is crucial for real-world applications, such as personal AI assistants. However, LLMs inherently lack mechanisms for prompt-driven knowledge capture. This paper investigates utilizing the existing LLM capabilities to enable prompt-driven knowledge capture, with a particular emphasis on knowledge graphs. We address this challenge by focusing on prompt-to-triple (P2T) generation. We explore three methods: zero-shot prompting, few-shot prompting, and fine-tuning, and then assess their performance via a specialized synthetic dataset. Our code and datasets are publicly available at https://github.com/HaltiaAI/paper-PTSKC., Comment: 8 pages, 5 figures, 1 table preprint. Under review

In this paper, a deep learning method for solving an improved one-dimensional Poisson-Nernst-Planck ion channel (PNPic) model, called the PNPic deep learning solver, is presented. In particular, it combines a novel local neural network scheme with an effective PNPic finite element solver. Since the input data of the neural network scheme only involves a small local patch of coarse grid solutions, which the finite element solver can quickly produce, the PNPic deep learning solver can be trained much faster than any corresponding conventional global neural network solvers. After properly trained, it can output a predicted PNPic solution in a much higher degree of accuracy than the low cost coarse grid solutions and can reflect different perturbation cases on the parameters, ion channel subregions, and interface and boundary values, etc. Consequently, the PNPic deep learning solver can generate a numerical solution with high accuracy for a family of PNPic models. As an initial study, two types of numerical tests were done by perturbing one and two parameters of the PNPic model, respectively, as well as the tests done by using a few perturbed interface positions of the model as training samples. These tests demonstrate that the PNPic deep learning solver can generate highly accurate PNPic numerical solutions., Comment: 17 pages, 4 figures, 5 tables

This article studies the vortex-wave system for the Surface Quasi-Geostrophic equation with parameter 0 < s < 1. We obtained local existence of classical solutions in H^4 under the standard "plateau hypothesis", H^2-stability of the solutions, and a blow-up criterion. In the sub-critical case s > 1/2 we established global existence of weak solutions. For the critical case s = 1/2, we introduced a weaker notion of solution (V-weak solutions) to give a meaning to the equation and prove global existence.

In this article, we study a non-Newtonian Stokes-Transport system. This set of PDEs was introduced as a model for describing the behavior of a cloud of particles in suspension in a Stokes fluid, and is a nonlinear coupling between a hyperbolic equation (Transport) and a nonlinear elliptic equation (non-Newtonian Stokes), and as such can be considered as an active scalar equation. We prove the existence of global weak solutions with initial data in critical Lebesgue spaces. In order to overcome the difficulties introduced by the highly nonlinear aspect of this problem, we resort to a combination of DiPerna-Lions theory of transport equations and Minty's trick for elliptic equations.

This paper explores the feasibility and performance of on-device large language model (LLM) inference on various Apple iPhone models. Amidst the rapid evolution of generative AI, on-device LLMs offer solutions to privacy, security, and connectivity challenges inherent in cloud-based models. Leveraging existing literature on running multi-billion parameter LLMs on resource-limited devices, our study examines the thermal effects and interaction speeds of a high-performing LLM across different smartphone generations. We present real-world performance results, providing insights into on-device inference capabilities., Comment: 7 pages, 3 Figures

Maximum likelihood estimation (MLE) is a fundamental problem in statistics. Characteristics of the MLE problem for algebraic statistical models are reflected in the geometry of the \textit{likelihood correspondence}, a variety that ties together data and their maximum likelihood estimators. We construct the ideal of the likelihood correspondence for the large class of toric models and find a Gr\"{o}bner basis in the case of complete and joint independence models arising from multi-way contingency tables. These results provide insight into their properties and offer faster computational strategies for solving the MLE problem., Comment: 16 pages

We introduce a new amortized likelihood ratio estimator for likelihood-free simulation-based inference (SBI). Our estimator is simple to train and estimates the likelihood ratio using a single forward pass of the neural estimator. Our approach directly computes the likelihood ratio between two competing parameter sets which is different from the previous approach of comparing two neural network output values. We refer to our model as the direct neural ratio estimator (DNRE). As part of introducing the DNRE, we derive a corresponding Monte Carlo estimate of the posterior. We benchmark our new ratio estimator and compare to previous ratio estimators in the literature. We show that our new ratio estimator often outperforms these previous approaches. As a further contribution, we introduce a new derivative estimator for likelihood ratio estimators that enables us to compare likelihood-free Hamiltonian Monte Carlo (HMC) with random-walk Metropolis-Hastings (MH). We show that HMC is equally competitive, which has not been previously shown. Finally, we include a novel real-world application of SBI by using our neural ratio estimator to design a quadcopter. Code is available at https://github.com/SRI-CSL/dnre., Comment: 12 Pages, 10 Figures, GitHub: https://github.com/SRI-CSL/dnre

Accelerated muon beams have been considered for next-generation studies of high-energy lepton-antilepton collisions and neutrino oscillations. However, high-brightness muon beams have not yet been produced. The main challenge for muon acceleration and storage stems from the large phase-space volume occupied by the beam, derived from the muon production mechanism through the decay of pions from proton collisions. Ionization cooling is the technique proposed to decrease the muon beam phase-space volume. Here we demonstrate a clear signal of ionization cooling through the observation of transverse emittance reduction in beams that traverse lithium hydride or liquid hydrogen absorbers in the Muon Ionization Cooling Experiment (MICE). The measurement is well reproduced by the simulation of the experiment and the theoretical model. The results shown here represent a substantial advance towards the realization of muon-based facilities that could operate at the energy and intensity frontiers., Comment: 23 pages and 5 figures

Atmospheric water harvesting is urgently needed given increasing global water scarcity. Current sorbent-based devices that cycle between water capture and release have low harvesting rates. We envision a radically different multi-material architecture with segregated and simultaneous capture and release. This way, proven fast-release mechanisms that approach theoretical limits can be incorporated; however, no capture mechanism exists to supply liquid adequately for release. Inspired by tree frogs and airplants, our capture approach transports water through a hydrogel membrane ``skin'' into a liquid desiccant. We report an extraordinarily high capture rate of 5.50 $\text{kg}\,\text{m}^{-2}\,\text{d}^{-1}$ at a low humidity of 35%, limited by the convection of air to the device. At higher humidities, we demonstrate up to 16.9 $\text{kg}\,\text{m}^{-2}\,\text{d}^{-1}$, exceeding theoretical limits for release. Simulated performance of a hypothetical one-square-meter device shows that water could be supplied to two to three people in dry environments. This work is a significant step toward providing new resources to water-scarce regions., Comment: 22 pages, 23 figures

Many downstream inference tasks for knowledge graphs, such as relation prediction, have been handled successfully by knowledge graph embedding techniques in the transductive setting. To address the inductive setting wherein new entities are introduced into the knowledge graph at inference time, more recent work opts for models which learn implicit representations of the knowledge graph through a complex function of a network's subgraph structure, often parametrized by graph neural network architectures. These come at the cost of increased parametrization, reduced interpretability and limited generalization to other downstream inference tasks. In this work, we bridge the gap between traditional transductive knowledge graph embedding approaches and more recent inductive relation prediction models by introducing a generalized form of harmonic extension which leverages representations learned through transductive embedding methods to infer representations of new entities introduced at inference time as in the inductive setting. This harmonic extension technique provides the best such approximation, can be implemented via an efficient iterative scheme, and can be employed to answer a family of conjunctive logical queries over the knowledge graph, further expanding the capabilities of transductive embedding methods. In experiments on a number of large-scale knowledge graph embedding benchmarks, we find that this approach for extending the functionality of transductive knowledge graph embedding models to perform knowledge graph completion and answer logical queries in the inductive setting is competitive with--and in some scenarios outperforms--several state-of-the-art models derived explicitly for such inductive tasks.

There is a growing awareness of the harmful effects of distribution shift on the performance of deployed machine learning models. Consequently, there is a growing interest in detecting these shifts before associated costs have time to accumulate. However, desiderata of crucial importance to the practicable deployment of sequential shift detectors are typically overlooked by existing works, precluding their widespread adoption. We identify three such desiderata, highlight existing works relevant to their satisfaction, and recommend impactful directions for future research., Comment: ICML 2022 Workshop on Principles of Distribution Shift (PODS)

We present AircraftVerse, a publicly available aerial vehicle design dataset. Aircraft design encompasses different physics domains and, hence, multiple modalities of representation. The evaluation of these cyber-physical system (CPS) designs requires the use of scientific analytical and simulation models ranging from computer-aided design tools for structural and manufacturing analysis, computational fluid dynamics tools for drag and lift computation, battery models for energy estimation, and simulation models for flight control and dynamics. AircraftVerse contains 27,714 diverse air vehicle designs - the largest corpus of engineering designs with this level of complexity. Each design comprises the following artifacts: a symbolic design tree describing topology, propulsion subsystem, battery subsystem, and other design details; a STandard for the Exchange of Product (STEP) model data; a 3D CAD design using a stereolithography (STL) file format; a 3D point cloud for the shape of the design; and evaluation results from high fidelity state-of-the-art physics models that characterize performance metrics such as maximum flight distance and hover-time. We also present baseline surrogate models that use different modalities of design representation to predict design performance metrics, which we provide as part of our dataset release. Finally, we discuss the potential impact of this dataset on the use of learning in aircraft design and, more generally, in CPS. AircraftVerse is accompanied by a data card, and it is released under Creative Commons Attribution-ShareAlike (CC BY-SA) license. The dataset is hosted at https://zenodo.org/record/6525446, baseline models and code at https://github.com/SRI-CSL/AircraftVerse, and the dataset description at https://aircraftverse.onrender.com/., Comment: The dataset is hosted at https://zenodo.org/record/6525446, baseline models and code at https://github.com/SRI-CSL/AircraftVerse, and the dataset description at https://aircraftverse.onrender.com/

Developmental psychologists have spent decades devising experiments to test the intelligence and knowledge of infants and children, tracing the origin of crucial concepts and capacities. Moreover, experimental techniques in developmental psychology have been carefully designed to discriminate the cognitive capacities that underlie particular behaviors. We propose that using classical experiments from child development is a particularly effective way to probe the computational abilities of AI models, in general, and LLMs in particular. First, the methodological techniques of developmental psychology, such as the use of novel stimuli to control for past experience or control conditions to determine whether children are using simple associations, can be equally helpful for assessing the capacities of LLMs. In parallel, testing LLMs in this way can tell us whether the information that is encoded in text is sufficient to enable particular responses, or whether those responses depend on other kinds of information, such as information from exploration of the physical world. In this work we adapt classical developmental experiments to evaluate the capabilities of LaMDA, a large language model from Google. We propose a novel LLM Response Score (LRS) metric which can be used to evaluate other language models, such as GPT. We find that LaMDA generates appropriate responses that are similar to those of children in experiments involving social understanding, perhaps providing evidence that knowledge of these domains is discovered through language. On the other hand, LaMDA's responses in early object and action understanding, theory of mind, and especially causal reasoning tasks are very different from those of young children, perhaps showing that these domains require more real-world, self-initiated exploration and cannot simply be learned from patterns in language input., Comment: 9 pages, 7 figures