Search

Attention mechanisms, particularly softmax attention, have been instrumental in the success of transformer-based models such as GPT. However, the quadratic memory complexity of softmax attention with respect to sequence length poses significant challenges for processing longer sequences. We introduce Cottention, a novel attention mechanism that replaces the softmax operation with cosine similarity. By leveraging the properties of cosine similarity and rearranging the attention equation, Cottention achieves native linear memory complexity with respect to sequence length, making it inherently more memory-efficient than softmax attention. We demonstrate that Cottention can be reformulated as a recurrent neural network (RNN) with a finite hidden state, allowing for constant memory usage during inference. We evaluate Cottention on both the bidirectional BERT and causal GPT tasks, demonstrating comparable performance to softmax attention while significantly reducing memory requirements. To ensure efficient computation, we develop a custom CUDA kernel for Cottention. Our results show that Cottention is a promising alternative to softmax attention, enabling the processing of longer sequences without sacrificing performance, due to its native linear memory complexity and ability to maintain a constant memory footprint during inference., Comment: 12 pages, 5 figures

We address three key challenges in learning continuous kernel representations: computational efficiency, parameter efficiency, and spectral bias. Continuous kernels have shown significant potential, but their practical adoption is often limited by high computational and memory demands. Additionally, these methods are prone to spectral bias, which impedes their ability to capture high-frequency details. To overcome these limitations, we propose a novel approach that leverages sparse learning in the Fourier domain. Our method enables the efficient scaling of continuous kernels, drastically reduces computational and memory requirements, and mitigates spectral bias by exploiting the Gibbs phenomenon.

Spiking neural networks (SNNs) are the third generation of neural networks that are biologically inspired to process data in a fashion that emulates the exchange of signals in the brain. Within the Computer Vision community SNNs have garnered significant attention due in large part to the availability of event-based sensors that produce a spatially resolved spike train in response to changes in scene radiance. SNNs are used to process event-based data due to their neuromorphic nature. The proposed work examines the neuromorphic advantage of fusing multiple sensory inputs in classification tasks. Specifically we study the performance of a SNN in digit classification by passing in a visual modality branch (Neuromorphic-MNIST [N-MNIST]) and an auditory modality branch (Spiking Heidelberg Digits [SHD]) from datasets that were created using event-based sensors to generate a series of time-dependent events. It is observed that multi-modal SNNs outperform unimodal visual and unimodal auditory SNNs. Furthermore, it is observed that the process of sensory fusion is insensitive to the depth at which the visual and auditory branches are combined. This work achieves a 98.43% accuracy on the combined N-MNIST and SHD dataset using a multimodal SNN that concatenates the visual and auditory branches at a late depth., Comment: 4 pages, 2 figures, submitted to 2025 IEEE International Conference on Acoustics, Speech, and Signal Processing

The interpolation problem is a natural and fundamental question whose roots trace back to ancient Greece. The story is long and rich, with many chapters, and a complete solution has been obtained only recently. Exploring it leads us on a tour through a number of general themes in geometry. This concrete problem motivates fundamental concepts such as moduli spaces and their properties, deformation theory, normal bundles, and more. Questions about smooth objects lead us to consider singular (non-smooth) objects, and in fact these smooth objects are studied by instead focusing on somehow simpler "non-smooth" objects, and then deforming them., Comment: Expository article, to appear in Bulletin of the AMS

In projective space over fields of characteristic different from 2, the normal bundle of a general nondegenerate rational curve is balanced. The corresponding statement for rational curves in other Grassmannians can fail. Nevertheless, we prove that the normal bundle of a general rational curve in a Grassmannian decomposes into a direct sum of line bundles whose degrees are at most 2 apart.

Physically unclonable functions (PUFs) are designed to act as device 'fingerprints.' Given an input challenge, the PUF circuit should produce an unpredictable response for use in situations such as root-of-trust applications and other hardware-level cybersecurity applications. PUFs are typically subcircuits present within integrated circuits (ICs), and while conventional IC PUFs are well-understood, several implementations have proven vulnerable to malicious exploits, including those perpetrated by machine learning (ML)-based attacks. Such attacks can be difficult to prevent because they are often designed to work even when relatively few challenge-response pairs are known in advance. Hence the need for both more resilient PUF designs and analysis of ML-attack susceptibility. Previous work has developed a PUF for photonic integrated circuits (PICs). A PIC PUF not only produces unpredictable responses given manufacturing-introduced tolerances, but is also less prone to electromagnetic radiation eavesdropping attacks than a purely electronic IC PUF. In this work, we analyze the resilience of the proposed photonic PUF when subjected to ML-based attacks. Specifically, we describe a computational PUF model for producing the large datasets required for training ML attacks; we analyze the quality of the model; and we discuss the modeled PUF's susceptibility to ML-based attacks. We find that the modeled PUF generates distributions that resemble uniform white noise, explaining the exhibited resilience to neural-network-based attacks designed to exploit latent relationships between challenges and responses. Preliminary analysis suggests that the PUF exhibits similar resilience to generative adversarial networks, and continued development will show whether more-sophisticated ML approaches better compromise the PUF and -- if so -- how design modifications might improve resilience., Comment: 14 pages, 8 figures

Most scientists need software to perform their research (Barker et al., 2020; Carver et al., 2022; Hettrick, 2014; Hettrick et al., 2014; Switters and Osimo, 2019), and neuroscientists are no exception. Whether we work with reaction times, electrophysiological signals, or magnetic resonance imaging data, we rely on software to acquire, analyze, and statistically evaluate the raw data we obtain - or to generate such data if we work with simulations. In recent years there has been a shift toward relying on free, open-source scientific software (FOSSS) for neuroscience data analysis (Poldrack et al., 2019), in line with the broader open science movement in academia (McKiernan et al., 2016) and wider industry trends (Eghbal, 2016). Importantly, FOSSS is typically developed by working scientists (not professional software developers) which sets up a precarious situation given the nature of the typical academic workplace (wherein academics, especially in their early careers, are on short and fixed term contracts). In this paper, we will argue that the existing ecosystem of neuroscientific open source software is brittle, and discuss why and how the neuroscience community needs to come together to ensure a healthy growth of our software landscape to the benefit of all.

We present design methods and insights for CO2 capture, transport, and storage systems for clusters of industrial facilities, with a case-study focus on the state of Louisiana. Our analytical framework includes: (1) evaluating the scale and concentration of capturable CO2 emissions at individual facilities for the purpose of estimating the cost of CO2 capture retrofits, (2) a screening method to identify potential CO2 storage sites and estimate their storage capacities, injectivities, and costs; and (3) an approach for cost-minimized design of pipeline infrastructure connecting CO2 capture plants with storage sites that considers land use patterns, existing rights-of-way, demographics, and a variety of social and environmental justice factors. In applying our framework to Louisiana, we estimate up to 50 million tCO2/y of industrial emissions (out of today's total emissions of 130 MtCO2/y) can be captured at under 100 USD/tCO2, and up to 100 MtCO2/y at under 120 USD/tCO2. We identified 98 potential storage sites with estimated aggregate total injectivity between 330 and 730 MtCO2/yr and storage costs ranging from 8 to 17 USD/tCO2. We find dramatic reductions in the aggregate pipeline length and CO2 transport cost per tonne when groups of capture plants share pipeline infrastructure rather than build dedicated single-user pipelines. Smaller facilities (emitting less than 1 MtCO2/y), which account for a quarter of Louisiana's industrial emissions, see the largest transport cost benefits from sharing of infrastructure. Pipeline routes designed to avoid disadvantaged communities (social and environmental justice) so as not to reinforce historical practices of disenfranchisement involve only modestly higher pipeline lengths and costs.

Physically unclonable functions (PUFs) identify integrated circuits using nonlinearly-related challenge-response pairs (CRPs). Ideally, the relationship between challenges and corresponding responses is unpredictable, even if a subset of CRPs is known. Previous work developed a photonic PUF offering improved security compared to non-optical counterparts. Here, we investigate this PUF's susceptibility to Multiple-Valued-Logic-based machine learning attacks. We find that approximately 1,000 CRPs are necessary to train models that predict response bits better than random chance. Given the significant challenge of acquiring a vast number of CRPs from a photonic PUF, our results demonstrate photonic PUF resilience against such attacks., Comment: 6 pages, 4 figures

The quantum computer has become contemporary reality, with the first two-qubit machine of mere decades ago transforming into cloud-accessible devices with tens, hundreds, or -- in a few cases -- even thousands of qubits. While such hardware is noisy and still relatively small, the increasing number of operable qubits raises another challenge: how to develop the now-sizeable quantum circuits executable on these machines. Preparing circuits manually for specifications of any meaningful size is at best tedious and at worst impossible, creating a need for automation. This article describes an automated quantum-software toolkit for synthesis, compilation, and optimization, which transforms classically-specified, irreversible functions into both technology-independent and technology-dependent quantum circuits. We also describe and analyze the toolkit's application to three situations -- quantum read-only memories, quantum random number generators, and quantum oracles -- and illustrate the toolkit's start-to-finish features, from the input of classical functions to the output of technology-dependent quantum circuits. Furthermore, we illustrate how the toolkit enables research beyond circuit synthesis, including comparison of synthesis and optimization methods and deeper understanding of even well-studied quantum algorithms. As quantum hardware continues to develop, such quantum circuit toolkits will play a critical role in realizing its potential., Comment: 48 pages, 23 figures, 20 tables

Let $C \subset \mathbb{P}^3$ be a general Brill--Noether curve. A classical problem is to determine when $H^0(N_C(-2)) = 0$, which controls the quadric section of $C$. So far this problem has only been solved in characteristic zero, in which case $H^0(N_C(-2)) = 0$ with finitely many exceptions. In this note, we extend these results to positive characteristic, uncovering a wealth of new exceptions in characteristic 2.

Let $\alpha: X \to Y$ be a finite cover of smooth curves. Beauville conjectured that the pushforward of a general vector bundle under $\alpha$ is semistable if the genus of $Y$ is at least $1$ and stable if the genus of $Y$ is at least $2$. We prove this conjecture if the map $\alpha$ is general in any component of the Hurwitz space of covers of an arbitrary smooth curve $Y$.

The Inflation Reduction Act (IRA) in the United States provides unprecedented incentives for deploying low-carbon hydrogen and liquid fuels, among other low greenhouse gas (GHG) emissions technologies. To better understand the prospective competitiveness of low-carbon or negative-carbon hydrogen and liquid fuels under the IRA in the early 2030s, we examine the impacts of IRA provisions on costs of producing hydrogen and synthetic liquid fuel made from natural gas, electricity, short-cycle biomass (agricultural residues), and corn-ethanol. With IRA credits (45V or 45Q), but excluding incentives provided by other national or state policies, hydrogen produced by electrolysis using carbon-free electricity (green H2) and natural gas reforming with carbon capture and storage (CCS) (blue H2) are cost-competitive with the carbon-intensive benchmark gray H2 from steam methane reforming. Biomass-derived H2 with or without CCS is not cost-completive under current IRA provisions. However, if IRA allowed biomass gasification with CCS to claim a 45V credit for carbon-neutral H2 and a 45Q credit for negative biogenic-CO2 emissions, this pathway would be less costly than gray H2. The IRA credit for clean fuels (45Z), currently stipulated to end in 2027, would need to be extended, or similar policy support provided by other national or state policies, for clean synthetic liquid fuel to be cost-competitive with petroleum-derived liquid fuels. Levelized IRA subsidies per unit of CO2 mitigated for all hydrogen and synthetic liquid fuel production pathways, except electricity-derived synthetic liquid fuel, range from 65 to 384 $/t CO2, which is within or below the range in U.S. federal government estimates of the Social Cost of Carbon (SCC) in the 2030 to 2040 timeframe.

We propose a method to improve steganography by increasing the resilience of stego-media to discovery through steganalysis. Our approach enhances a class of steganographic approaches through the inclusion of a steganographic assistant convolutional neural network (SA-CNN). Previous research showed success in discovering the presence of hidden information within stego-images using trained neural networks as steganalyzers that are applied to stego-images. Our results show that such steganalyzers are less effective when SA-CNN is employed during the generation of a stego-image. We also explore the advantages and disadvantages of representing all the possible outputs of our SA-CNN within a smaller, discrete space, rather than a continuous space. Our SA-CNN enables certain classes of parametric steganographic algorithms to be customized based on characteristics of the cover media in which information is to be embedded. Thus, SA-CNN is adaptive in the sense that it enables the core steganographic algorithm to be especially configured for each particular instance of cover media. Experimental results are provided that employ a recent steganographic technique, S-UNIWARD, both with and without the use of SA-CNN. We then apply both sets of stego-images, those produced with and without SA-CNN, to an exmaple steganalyzer, Yedroudj-Net, and we compare the results. We believe that this approach for the integration of neural networks with hand-crafted algorithms increases the reliability and adaptability of steganographic algorithms., Comment: 6 pages, 6 figures

Random number generators (RNG) are essential elements in many cryptographic systems. True random number generators (TRNG) rely upon sources of randomness from natural processes such as those arising from quantum mechanics phenomena. We demonstrate that a quantum computer can serve as a high-quality, weakly random source for a generalized user-defined probability mass function (PMF). Specifically, QC measurement implements the process of variate sampling according to a user-specified PMF resulting in a word comprised of electronic bits that can then be processed by an extractor function to address inaccuracies due to non-ideal quantum gate operations and other system biases. We introduce an automated and flexible method for implementing a TRNG as a programmed quantum circuit that executes on commercially-available, gate-model quantum computers. The user specifies the desired word size as the number of qubits and a definition of the desired PMF. Based upon the user specification of the PMF, our compilation tool automatically synthesizes the desired TRNG as a structural OpenQASM file containing native gate operations that are optimized to reduce the circuit's quantum depth. The resulting TRNG provides multiple bits of randomness for each execution/measurement cycle; thus, the number of random bits produced in each execution is limited only by the size of the QC. We provide experimental results to illustrate the viability of this approach., Comment: 15 pages, 7 figures, SPIE Defense + Commercial Sensing: Quantum Information Science, Sensing, and Computation XV

We generalize the Embedding Theorem of Eisenbud-Harris from classical Brill-Noether theory to the setting of Hurwitz-Brill-Noether theory. More precisely, in classical Brill-Noether theory, the embedding theorem states that a general linear series of degree d and rank r on a general curve of genus g is an embedding if r is at least 3. If \(f \colon C \to \mathbb{P}^1\) is a general cover of degree k, and L is a line bundle on C, recent work of the authors shows that the splitting type of \(f_* L\) provides the appropriate generalization of the pair (r, d) in classical Brill--Noether theory. In the context of Hurwitz-Brill-Noether theory, the condition that r is at least 3 is no longer sufficient to guarantee that a general such linear series is an embedding. We show that the additional condition needed to guarantee that a general linear series |L| is an embedding is that the splitting type of \(f_* L\) has at least three nonnegative parts. This new extra condition reflects the unique geometry of k-gonal curves, which lie on scrolls in \(\mathbb{P}^r\).

Automatic modulation classification is a desired feature in many modern software-defined radios. In recent years, a number of convolutional deep learning architectures have been proposed for automatically classifying the modulation used on observed signal bursts. However, a comprehensive analysis of these differing architectures and importance of each design element has not been carried out. Thus it is unclear what tradeoffs the differing designs of these convolutional neural networks might have. In this research, we investigate numerous architectures for automatic modulation classification and perform a comprehensive ablation study to investigate the impacts of varying hyperparameters and design elements on automatic modulation classification performance. We show that a new state of the art in performance can be achieved using a subset of the studied design elements. In particular, we show that a combination of dilated convolutions, statistics pooling, and squeeze-and-excitation units results in the strongest performing classifier. We further investigate this best performer according to various other criteria, including short signal bursts, common misclassifications, and performance across differing modulation categories and modes.

The 24-hour activity cycle (24HAC) is a new paradigm for studying activity behaviors in relation to health outcomes. This approach captures the interrelatedness of the daily time spent in physical activity (PA), sedentary behavior (SB), and sleep. We illustrate and compare the use of three popular approaches, namely isotemporal substitution model (ISM), compositional data analysis (CoDA), and latent profile analysis (LPA) for modeling outcome associations with the 24HAC. We apply these approaches to assess an association with a cognitive outcome, measured by CASI item response theory (IRT) score, in a cohort of 1034 older adults (mean [range] age = 77 [65-100]; 55.8% female; 90% White) who were part of the Adult Changes in Thought (ACT) Activity Monitoring (ACT-AM) sub-study. PA and SB were assessed with thigh-worn activPAL accelerometers for 7 days. We highlight differences in assumptions between the three approaches, discuss statistical challenges, and provide guidance on interpretation and selecting an appropriate approach. ISM is easiest to apply and interpret; however, the typical ISM model assumes a linear association. CoDA specifies a non-linear association through isometric logratio transformations that are more challenging to apply and interpret. LPA can classify individuals into groups with similar time-use patterns. Inference on associations of latent profiles with health outcomes need to account for the uncertainty of the LPA classifications which is often ignored. The selection of the most appropriate method should be guided by the scientific questions of interest and the applicability of each model's assumptions. The analytic results did not suggest that less time spent on SB and more in PA was associated with better cognitive function. Further research is needed into the health implications of the distinct 24HAC patterns identified in this cohort., Comment: 51 pages, 11 tables, 8 figures

We present an essentially complete solution to the Minimal Resolution Conjecture for general curves, determining the shape of the minimal resolution of general sets of points on a general curve C of degree d>2r-1 in P^r. Our methods also provide a proof (valid in arbitrary characteristic) of the strong version of Butler's Conjecture on the stability of syzygy bundles on a general curve of every genus at least 3, as well as of the Frobenius semistability in positive characteristic of the syzygy bundle of a general curve in the range d>2r-1., Comment: 25 pages. Final version, to appear in Annales Sci. Ecole Normale Superieure

Optically-pumped magnetometers (OPM) -- next-generation magnetoencephalography (MEG) sensors -- may be placed directly on the head, unlike the more commonly used superconducting quantum interference device (SQUID) sensors, which must be placed a few centimeters away. This allows for signals of higher spatial resolution to be captured, resulting in potentially more accurate source localization. In this paper, we show that in the noiseless and high signal-to-noise ratio (SNR) case of approximately $\geq 6$ dB, inaccuracies in boundary element method (BEM) head conductor models (or equivalently, inaccurate volume current models) lead to increased signal and equivalent current dipole (ECD) source localization inaccuracies when sensor arrays are placed closer to the head. This is true especially in the case of deep and superficial sources where volume current contributions are high. In the noisy case however, the higher SNR for closer sensor arrays allows for an improved ECD fit and outweighs the effects of head geometry inaccuracies. This calls for an increase in emphasis in head modeling to reduce inverse modeling errors, especially as the field of MEG strives for closer sensor arrays and cleaner signals. An analytical form to obtain the magnetic field errors for small perturbations in the BEM head geometry is also provided.

Let $\alpha : X \to Y$ be a general degree $r$ primitive map of nonsingular, irreducible, projective curves over an algebraically closed field of characteristic zero or larger than $r$. We prove that the Tschirnhausen bundle of $\alpha$ is semistable if $g(Y) \geq 1$ and stable if $g(Y) \geq 2$., Comment: To appear in International Math Research Notices

Molecular dynamics (MD) simulation is widely used to study protein conformations and dynamics. However, conventional simulation suffers from being trapped in some local energy minima that are hard to escape. Thus, most computational time is spent sampling in the already visited regions. This leads to an inefficient sampling process and further hinders the exploration of protein movements in affordable simulation time. The advancement of deep learning provides new opportunities for protein sampling. Variational autoencoders are a class of deep learning models to learn a low-dimensional representation (referred to as the latent space) that can capture the key features of the input data. Based on this characteristic, we proposed a new adaptive sampling method, latent space assisted adaptive sampling for protein trajectories (LAST), to accelerate the exploration of protein conformational space. This method comprises cycles of (i) variational autoencoders training, (ii) seed structure selection on the latent space and (iii) conformational sampling through additional MD simulations. The proposed approach is validated through the sampling of four structures of two protein systems: two metastable states of E. Coli adenosine kinase (ADK) and two native states of Vivid (VVD). In all four conformations, seed structures were shown to lie on the boundary of conformation distributions. Moreover, large conformational changes were observed in a shorter simulation time when compared with conventional MD (cMD) simulations in both systems. In metastable ADK simulations, LAST explored two transition paths toward two stable states while cMD became trapped in an energy basin. In VVD light state simulations, LAST was three times faster than cMD simulation with a similar conformational space.

Let $C$ be a general canonical curve of genus $g$ defined over an algebraically closed field of arbitrary characteristic. We prove that if $g \notin \{4,6\}$, then the normal bundle of $C$ is semistable. In particular, if $g \equiv 1$ or $3$ mod $6$, then the normal bundle is stable., Comment: To appear in Journal of the European Math Society

In this paper we determine the number of general points through which a Brill--Noether curve of fixed degree and genus in any projective space can be passed.

Knowledge Tracing is the process of tracking mastery level of different skills of students for a given learning domain. It is one of the key components for building adaptive learning systems and has been investigated for decades. In parallel with the success of deep neural networks in other fields, we have seen researchers take similar approaches in the learning science community. However, most existing deep learning based knowledge tracing models either: (1) only use the correct/incorrect response (ignoring useful information from other modalities) or (2) design their network architectures through domain expertise via trial and error. In this paper, we propose a sequential model based optimization approach that combines multimodal fusion and neural architecture search within one framework. The commonly used neural architecture search technique could be considered as a special case of our proposed approach when there is only one modality involved. We further propose to use a new metric called time-weighted Area Under the Curve (weighted AUC) to measure how a sequence model performs with time. We evaluate our methods on two public real datasets showing the discovered model is able to achieve superior performance. Unlike most existing works, we conduct McNemar's test on the model predictions and the results are statistically significant.