Search

Dense Associative Memories are high storage capacity variants of the Hopfield networks that are capable of storing a large number of memory patterns in the weights of the network of a given size. Their common formulations typically require storing each pattern in a separate set of synaptic weights, which leads to the increase of the number of synaptic weights when new patterns are introduced. In this work we propose an alternative formulation of this class of models using random features, commonly used in kernel methods. In this formulation the number of network's parameters remains fixed. At the same time, new memories can be added to the network by modifying existing weights. We show that this novel network closely approximates the energy function and dynamics of conventional Dense Associative Memories and shares their desirable computational properties., Comment: Accepted to NeurIPS 2024

GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50\% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models. In line with our commitment to building AI safely and consistent with our voluntary commitments to the White House, we are sharing the GPT-4o System Card, which includes our Preparedness Framework evaluations. In this System Card, we provide a detailed look at GPT-4o's capabilities, limitations, and safety evaluations across multiple categories, focusing on speech-to-speech while also evaluating text and image capabilities, and measures we've implemented to ensure the model is safe and aligned. We also include third-party assessments on dangerous capabilities, as well as discussion of potential societal impacts of GPT-4o's text and vision capabilities.

Edge-based network models, especially those based on bond percolation methods, can be used to model disease transmission on complex networks and accommodate social heterogeneity while keeping tractability. Here we present an application of an edge-based network model to the spread of syphilis in the Kingston, Frontenac and Lennox & Addington (KFL&A) region of Southeastern Ontario, Canada. We compared the results of using a network-based susceptible-infectious-recovered (SIR) model to those generated from using a traditional mass action SIR model. We found that the network model yields very different predictions, including a much lower estimate of the final epidemic size. We also used the network model to estimate the potential impact of introducing a rapid syphilis point of care test (POCT) and treatment intervention strategy that has recently been implemented by the public health unit to mitigate syphilis transmission.

Batched sparse (BATS) code is a class of batched network code that can achieve a close-to-optimal rate when an optimal degree distribution is provided. We observed that most probability masses in this optimal distribution are very small, i.e., the distribution "looks" sparse. In this paper, we investigate the sparsity optimization of degree distribution for BATS codes that produces sparse degree distributions. There are many advantages to use a sparse degree distribution, say, it is robust to precision errors when sampling the degree distribution during encoding and decoding in practice. We discuss a few heuristics and also a way to obtain an exact sparsity solution. These approaches give a trade-off between computational time and achievable rate, thus give us the flexibility to adopt BATS codes in various scenarios, e.g., device with limited computational power, stable channel condition, etc., Comment: Full version of the conference version in ITW'24

Cosmic Explorer (CE) is a next-generation ground-based gravitational-wave observatory that is being designed in the 2020s and is envisioned to begin operations in the 2030s together with the Einstein Telescope in Europe. The CE concept currently consists of two widely separated L-shaped observatories in the United States, one with 40 km-long arms and the other with 20 km-long arms. This order of magnitude increase in scale with respect to the LIGO-Virgo-KAGRA observatories will, together with technological improvements, deliver an order of magnitude greater astronomical reach, allowing access to gravitational waves from remnants of the first stars and opening a wide discovery aperture to the novel and unknown. In addition to pushing the reach of gravitational-wave astronomy, CE endeavors to approach the lifecycle of large scientific facilities in a way that prioritizes mutually beneficial relationships with local and Indigenous communities. This article describes the (scientific, cost and access, and social) criteria that will be used to identify and evaluate locations that could potentially host the CE observatories., Comment: 12 pages, 1 figure

Batched network coding (BNC) is a solution to multi-hop transmission on networks with packet loss. To be compatible with the existing infrastructure, BNC is usually implemented over UDP. A single error bit will probably result in discarding the packet. UDP-Lite is a variant of UDP that supports partial checksums. As long as the data covered by the checksum is correct, damaged payload will be delivered. With UDP-Lite, we can cope with other techniques such as payload aggregation of BNC packets to reduce the protocol overhead, and forward error correction to combat against bit errors. Unlike traditional transmissions, BNC has a loss resilience feature and there are dependencies between BNC packets. In this paper, we conduct a preliminary investigation on BNC over UDP-Lite. We show that aggregating as much as we can is not always the best strategy, and a hop-by-hop distributed efficiency optimization approach may lead to a worse throughput compared with the scheme without aggregation in a long network. These unnatural results caution that a casual integration of techniques with BNC can be harmful, and give us hints on future research directions., Comment: Full version of the conference version in TENCON'24

Wireless relay network is a solution to extend the reach of a wireless connection by installing a relay node between the source node and the sink node. Due to the broadcast nature of wireless transmission, the sink node has a chance to receive part of the data sent by the source node. In this paper, we apply a network coding scheme called BATS codes on a wireless relay network where the relay node has a stable power supply, so that we can aim for the best decoding time instead of minimizing the number of transmissions for saving energy. We optimize the time efficiency that maximize the average decoding rate per unit time by some heuristics, and bring out a message that it is not optimal to set an average number of recoded packets per batch at the relay node equals the number of packets per batch sent by the source node., Comment: Full version of the conference version in TENCON'24

This paper proposes an alternative implementation of the MENT algorithm, an exact maximum-entropy algorithm used to infer a phase space distribution from its projections. A key step in the MENT algorithm is to compute the distribution's projections via numerical integration. In this approach, the run time scales quickly with the phase space dimension and measurement resolution. The proposed MENT implementation computes the projections via particle sampling, rather than numerical integration, eliminating the dependence on the measurement resolution. Furthermore, with the appropriate sampling algorithm, the particle-based approach scales to $N$-dimensions without computer memory limitations. Using synthetic data, we demonstrate MENT convergence in four dimensions using a grid-based sampling method and in six dimensions using Markov Chain Monte Carlo (MCMC) sampling., Comment: 7 pages, 2 figures, comments welcome

In this paper, we use one-dimensional measurements to infer the four-dimensional phase space density of an accumulated proton beam in the Spallation Neutron Source (SNS) accelerator. The reconstruction was performed by maximizing the distribution's entropy subject to the measurement constraints, and thus represents the most conservative inference from the data. The reconstructed distribution reproduces the measured profiles down to the noise level, and simulations indicate that the problem is reasonably well-constrained. Similar measurements could serve as benchmarks for beam dynamics simulations in the SNS or hadron accelerators., Comment: 9 pages, 9 figures, submitted to PRAB

Transformers have revolutionized machine learning, yet their inner workings remain opaque to many. We present Transformer Explainer, an interactive visualization tool designed for non-experts to learn about Transformers through the GPT-2 model. Our tool helps users understand complex Transformer concepts by integrating a model overview and enabling smooth transitions across abstraction levels of mathematical operations and model structures. It runs a live GPT-2 instance locally in the user's browser, empowering users to experiment with their own input and observe in real-time how the internal components and parameters of the Transformer work together to predict the next tokens. Our tool requires no installation or special hardware, broadening the public's education access to modern generative AI techniques. Our open-sourced tool is available at https://poloclub.github.io/transformer-explainer/. A video demo is available at https://youtu.be/ECR4oAwocjs., Comment: To be presented at IEEE VIS 2024

Food insecurity is widespread among college students in the United States. Food benefits delivered through the CalFresh program, California's version of the federal Supplemental Nutrition Assistance Program (SNAP), can reduce hunger by helping students pay for groceries, but may not reach all eligible students. To date, higher education systems have lacked good estimates of the share of their students who are eligible for CalFresh and the share who actually receive benefits. To address this information gap, the California Policy Lab (CPL) partnered with the California Community College (CCC) Chancellor's Office, the University of California Office of the President (UCOP), the California Department of Social Services (CDSS), and the California Student Aid Commission (CSAC) to build a linked database of student-level administrative data on college enrollment, financial aid, and CalFresh participation. This database covers all students enrolled at CCC or UC campuses from academic years 2010-11 through 2021-22, along with corresponding FAFSA submissions and CalFresh participation. Using these data, we are able to measure how many college students are likely eligible for CalFresh, and of those how many participate. CalFresh eligibility does not perfectly overlap with student need. Some students who are food insecure may not be eligible for benefits, while some eligible students may not be in great need. This in part reflects the rules of the program, which are designed to capture need but do not always do so perfectly. We attempt to measure eligibility according to the actual rules. Our estimates complement other work that uses survey data to measure students' basic needs (California Community Colleges League, 2022). We estimate that in Fall 2019, on the eve of the COVID pandemic, 16% of California community college students (256,000 students), 31% of UC undergraduate students (69,000 students), and 6% of UC graduate students (3,000 students) were likely eligible for CalFresh benefits. However, the majority of these students did not receive benefits -- only 30% of eligible community college students, 22% of eligible UC undergraduates, and 29% of eligible UC graduate students were actually enrolled in CalFresh. We emphasize that our eligibility determinations are estimates based on information available in existing data, which imperfectly capture some elements of the CalFresh eligibility determination process. They may somewhat overstate or understate student eligibility. However, extensive investigation led us to conclude that the errors are likely not large and that our estimates are a good approximation of the share of students who would be found eligible under individualized determinations. As we discuss below, the higher eligibility rate among UC undergraduates as compared to CCC students reflects program rules that make it easier for UC students to qualify than for CCC students with similar resources. We also find differences in eligibility across groups of students within each segment, reflecting both variation in need and program rules. Among both CCC and UC students, Black and Hispanic students, for example, are more likely to be eligible than are White or Asian American students, and students receiving federal, state, or institutional financial aid are much more likely to be eligible than are students not receiving aid. Benefits receipt generally mirrors this, although there are differences across groups in the share of eligible students who receive benefits. The take-up rate, which is the share of eligible students who participate in CalFresh, is higher for Black students (and, at the UC but not at CCCs, Hispanic students) than for White students, and for students on financial aid than for students who are not. Our report indicates there is much room to improve CalFresh participation among eligible students. Through detailed analyses of the paths to eligibility and of variation in participation rates, we hope to shed light on opportunities for policymakers, higher education administrators, community-based organizations, student groups, and advocates to better connect eligible students to benefits. [Additional support was provided by the Woven Foundation.]

"Pedagogical derivations for Nos\'e's dynamics can be developed in two different ways, (i) by starting with a temperature-dependent Hamiltonian in which the variable $s$ scales the time or the mass, or (ii) by requiring that the equations of motion generate the canonical distribution including a Gaussian distribution in the friction coefficient $\zeta$. Nos\'e's papers follow the former approach. Because the latter approach is not only constructive and simple, but also can be generalized to other forms of the equations of motion, we illustrate it here. We begin by considering the probability density $f(q,p,\zeta)$ in an extended phase space which includes $\zeta$ as well as all pairs of phase variables $q$ and $p$. This density $f(q,p,\zeta)$ satisfies the conservation of probability (Liouville's Continuity Equation)" $$(\partial f/\partial t) + \sum (\partial (\dot q f)/\partial q) + \sum (\partial (\dot p f)/\partial p) + \sum (\partial (\dot \zeta f)/\partial \zeta) = 0 \ . $$ The multi-authored ``review''\cite{b1} motivated our quoting the history of Nos\'e and Nos\'e-Hoover mechanics, aptly described on page 31 of Bill's 1986 {\it Molecular Dynamics} book, reproduced above\cite{b2}., Comment: Nine pages with three figures written for Computational Methods in Science and Technology

Particle accelerators generate charged-particle beams with tailored distributions in six-dimensional position-momentum space (phase space). Knowledge of the phase space distribution enables model-based beam optimization and control. In the absence of direct measurements, the distribution must be tomographically reconstructed from its projections. In this paper, we highlight that such problems can be severely underdetermined and that entropy maximization is the most conservative solution strategy. We leverage normalizing flows -- invertible generative models -- to extend maximum-entropy tomography to six-dimensional phase space and perform numerical experiments to validate the model's performance. Our numerical experiments demonstrate consistency with exact two-dimensional maximum-entropy solutions and the ability to fit complicated six-dimensional distributions to large measurement sets in reasonable time., Comment: 13 pages, 7 figures, submitted to PRResearch

We study watermarking schemes for language models with provable guarantees. As we show, prior works offer no robustness guarantees against adaptive prompting: when a user queries a language model more than once, as even benign users do. And with just a single exception (Christ and Gunn, 2024), prior works are restricted to zero-bit watermarking: machine-generated text can be detected as such, but no additional information can be extracted from the watermark. Unfortunately, merely detecting AI-generated text may not prevent future abuses. We introduce multi-user watermarks, which allow tracing model-generated text to individual users or to groups of colluding users, even in the face of adaptive prompting. We construct multi-user watermarking schemes from undetectable, adaptively robust, zero-bit watermarking schemes (and prove that the undetectable zero-bit scheme of Christ, Gunn, and Zamir (2024) is adaptively robust). Importantly, our scheme provides both zero-bit and multi-user assurances at the same time. It detects shorter snippets just as well as the original scheme, and traces longer excerpts to individuals. The main technical component is a construction of message-embedding watermarks from zero-bit watermarks. Ours is the first generic reduction between watermarking schemes for language models. A challenge for such reductions is the lack of a unified abstraction for robustness -- that marked text is detectable even after edits. We introduce a new unifying abstraction called AEB-robustness. AEB-robustness provides that the watermark is detectable whenever the edited text "approximates enough blocks" of model-generated output., Comment: 39 pages

Batched sparse (BATS) code is a network coding solution for multi-hop wireless networks with packet loss. Achieving a close-to-optimal rate relies on an optimal degree distribution. Technical challenges arise from the sensitivity of this distribution to the often empirically obtained rank distribution at the destination node. Specifically, if the empirical distribution overestimates the channel, BATS codes experience a significant rate degradation, leading to unstable rates across different runs and hence unpredictable transmission costs. Confronting this unresolved obstacle, we introduce a formulation for distributionally robust optimization in degree optimization. Deploying the resulting degree distribution resolves the instability of empirical rank distributions, ensuring a close-to-optimal rate, and unleashing the potential of applying BATS codes in real-world scenarios., Comment: 8 pages, accepted by 2024 IEEE International Symposium on Information Theory

Diffusion-based generative models' impressive ability to create convincing images has garnered global attention. However, their complex internal structures and operations often pose challenges for non-experts to grasp. We introduce Diffusion Explainer, the first interactive visualization tool designed to elucidate how Stable Diffusion transforms text prompts into images. It tightly integrates a visual overview of Stable Diffusion's complex components with detailed explanations of their underlying operations. This integration enables users to fluidly transition between multiple levels of abstraction through animations and interactive elements. Offering real-time hands-on experience, Diffusion Explainer allows users to adjust Stable Diffusion's hyperparameters and prompts without the need for installation or specialized hardware. Accessible via users' web browsers, Diffusion Explainer is making significant strides in democratizing AI education, fostering broader public access. More than 7,200 users spanning 113 countries have used our open-sourced tool at https://poloclub.github.io/diffusion-explainer/. A video demo is available at https://youtu.be/MbkIADZjPnA., Comment: 4 pages, 3 figures. arXiv admin note: substantial text overlap with arXiv:2305.03509

This paper presents a reliability life analysis and preventive maintenance schedule for ducted wind turbines. Ducted wind turbines (DWT) are an emerging segment of the renewable energy industry with innovations that promise reliable, efficient, low-cost energy for consumer and small business markets. Many attempts have been made to build viable ducted turbines over the last century, but until recently none have succeeded commercially. Optimal shroud and blade designs are the focus of most engineering research to improve performance and efficiency, however, we hypothesize that an equally important key to the long-term success of small wind innovations is reliability analysis. For consumers and companies who want to efficiently maximize the lifespan of DWTs, this has significant ramifications. Operating beyond service life can result in catastrophic component failure and high replacement costs, making the technology economically infeasible. Our approach is focused on the analysis of 3.5 kW D3 turbines manufactured by Ducted Wind Turbines, Inc. We develop a component-level reliability analysis using ASTM E3159 and a consumer-level preventative maintenance schedule including failure modes and life estimates. Future research can use these findings to guide options for DWT life extension as well as localized maintenance solutions meant to reduce operational costs while preserving energy output.

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

In this paper, we give part-preserving bijections between three fundamental families of objects that serve as natural framework for many problems in enumerative combinatorics. Specifically, we consider compositions, Dyck paths, and partitions of a convex polygon, and identify suitable building blocks that are then appropriately decorated to achieve matching cardinalities. Our bijections are constructive and apply for the general case where the building blocks are allowed to come in different colors., Comment: 10 pages. Submitted for publication

In a ring-signature-based anonymous cryptocurrency, signers of a transaction are hidden among a set of potential signers, called a ring, whose size is much smaller than the number of all users. The ring-membership relations specified by the sets of transactions thus induce bipartite transaction graphs, whose distribution is in turn induced by the ring sampler underlying the cryptocurrency. Since efficient graph analysis could be performed on transaction graphs to potentially deanonymise signers, it is crucial to understand the resistance of (the transaction graphs induced by) a ring sampler against graph analysis. Of particular interest is the class of partitioning ring samplers. Although previous works showed that they provide almost optimal local anonymity, their resistance against global, e.g. graph-based, attacks were unclear. In this work, we analyse transaction graphs induced by partitioning ring samplers. Specifically, we show (partly analytically and partly empirically) that, somewhat surprisingly, by setting the ring size to be at least logarithmic in the number of users, a graph-analysing adversary is no better than the one that performs random guessing in deanonymisation up to constant factor of 2.

Motivated by the factorization inherent in the original fast multipole method and the improved fast Gauss transform we introduce a factorable form of attention that operates efficiently in high dimensions. This approach reduces the computational and memory complexity of the attention mechanism in transformers from $O(N^2)$ to $O(N)$. In comparison to previous attempts, our work presents a linearly scaled attention mechanism that maintains the full representation of the attention matrix without compromising on sparsification and incorporates the all-to-all relationship between tokens. We explore the properties of our new attention metric and conduct tests in various standard settings. Results indicate that our attention mechanism has a robust performance and holds significant promise for diverse applications where self-attention is used.

Food insecurity is widespread among college students in the United States. CalFresh food benefits, known federally as the Supplemental Nutrition Assistance Program, or SNAP, can help students in California pay for food, but may not reach all eligible students. To better measure student participation in CalFresh, the California Policy Lab (CPL) partners with the California Community College system (CCC), the University of California Office of the President (UCOP), the California Department of Social Services (CDSS), and the California Student Aid Commission (CSAC). Through this partnership, CPL has created a linked database of student-level administrative data on college enrollment, financial aid, and CalFresh participation that makes this analysis possible.