Search

Heterogeneous graph neural networks have recently gained attention for long document summarization, modeling the extraction as a node classification task. Although effective, these models often require external tools or additional machine learning models to define graph components, producing highly complex and less intuitive structures. We present GraphLSS, a heterogeneous graph construction for long document extractive summarization, incorporating Lexical, Structural, and Semantic features. It defines two levels of information (words and sentences) and four types of edges (sentence semantic similarity, sentence occurrence order, word in sentence, and word semantic similarity) without any need for auxiliary learning models. Experiments on two benchmark datasets show that GraphLSS is competitive with top-performing graph-based methods, outperforming recent non-graph models. We release our code on GitHub., Comment: Short paper submitted to ACL ARR November cycle

Lattice gauge theories (LGTs) describe a broad range of phenomena in condensed matter and particle physics. A prominent example is confinement, responsible for bounding quarks inside hadrons such as protons or neutrons. When quark-antiquark pairs are separated, the energy stored in the string of gluon fields connecting them grows linearly with their distance, until there is enough energy to create new pairs from the vacuum and break the string. While such phenomena are ubiquitous in LGTs, simulating the resulting dynamics is a challenging task. Here, we report the observation of string breaking in synthetic quantum matter using a programmable quantum simulator based on neutral atom arrays. We show that a (2+1)D LGT with dynamical matter can be efficiently implemented when the atoms are placed on a Kagome geometry, with a local U(1) symmetry emerging from the Rydberg blockade, while long-range Rydberg interactions naturally give rise to a linear confining potential for a pair of charges, allowing us to tune both their masses as well as the string tension. We experimentally map out the corresponding phase diagram by adiabatically preparing the ground state of the atom array in the presence of defects, and observe substructure of the confined phase, distinguishing regions dominated by fluctuating strings or by broken string configurations. Finally, by harnessing local control over the atomic detuning, we quench string states and observe string breaking dynamics exhibiting a many-body resonance phenomenon. Our work paves a way to explore phenomena in high-energy physics using programmable quantum simulators.

Modern high performance computers are massively parallel; for many PDE applications spatial parallelism saturates long before the computer's capability is reached. Parallel-in-time methods enable further speedup beyond spatial saturation by solving multiple timesteps simultaneously to expose additional parallelism. ParaDiag is a particular approach to parallel-in-time based on preconditioning the simultaneous timestep system with a perturbation that allows block diagonalisation via a Fourier transform in time. In this article, we introduce asQ, a new library for implementing ParaDiag parallel-in-time methods, with a focus on applications in the geosciences, especially weather and climate. asQ is built on Firedrake, a library for the automated solution of finite element models, and the PETSc library of scalable linear and nonlinear solvers. This enables asQ to build ParaDiag solvers for general finite element models and provide a range of solution strategies, making testing a wide array of problems straightforward. We use a quasi-Newton formulation that encompasses a range of ParaDiag methods, and expose building blocks for constructing more complex methods. The performance and flexibility of asQ is demonstrated on a hierarchy of linear and nonlinear atmospheric flow models. We show that ParaDiag can offer promising speedups and that asQ is a productive testbed for further developing these methods., Comment: 30 pages, 15 figures. To be submitted to journal

Physics, like many scientific disciplines, has long struggled with attracting and retaining a diverse population and fostering inclusivity. While there have been improvements in addressing equity issues within physics, significant challenges remain. Faculty members play a crucial role as change agents in promoting cultural transformation within academic environments. Empathy, a fundamental component of effective teaching, mentoring, and collegiality, is essential for fostering a student-centered and holistic approach in academia. However, understanding how empathy functions within the specific context of physics, including its interaction with power dynamics and other contextual factors, remains underexplored.This study presents a theoretical model of empathy development among physics faculty as they engage with students and colleagues. Conducted at a private R2 institution, the study involved four rounds of interviews from summer 2023 to spring 2024. The initial two rounds (summer 2023) included eight participants, followed by 19 participants in fall 2023, and nine participants chose to follow up with us in spring 2024. All participants were physics faculty members, either in teaching or tenure-track positions. The developed model builds on previous research by introducing new complexities in the understanding of empathy. It identifies key mediators, including reflective witnessing, personal experiences, and empathetic concern, as well as moderators such as individual experience, emotions, motivation, values, and situational information. The model delineates both cognitive and affective pathways of empathy, providing a nuanced framework for understanding how empathy develops and influences faculty interactions in the physics discipline.

In this work, we introduce the concept of controllability and observability to the Mamba SSM's architecture in our Sparse-Mamba (S-Mamba) for natural language processing (NLP) applications. The structured state space model (SSM) development in recent studies, such as Mamba and Mamba2, outperformed and solved the computational inefficiency of transformers and large language models at small to medium scale. The Mamba SSMs architecture drops the need for attention layers or multilayer perception blocks in transformers. However, current Mamba models lack reinforcement of controllability in state-space equations for computing the $A$, $B$, $C$, and $D$ matrices at each time step, leading to increased complexity and computational costs. In this paper, we demonstrate a reduction of parameters in comparison to the first published Mamba and Mamba2. We showcase an improvement in perplexity by 5\% and a decrease in training time by 3\% after reinforcing controllability and observability on the original Mamba architecture in our proposed S-Mamba. The controllable $n \times n$ state matrix $A$ is sparse and it has only $n$ free parameters. Our novel approach will ensure a controllable system which will be the gate key for Mamba3.

The classification of IoT traffic is important to improve the efficiency and security of IoT-based networks. As the state-of-the-art classification methods are based on Deep Learning, most of the current results require a large amount of data to be trained. Thereby, in real-life situations, where there is a scarce amount of IoT traffic data, the models would not perform so well. Consequently, these models underperform outside their initial training conditions and fail to capture the complex characteristics of network traffic, rendering them inefficient and unreliable in real-world applications. In this paper, we propose IoT Traffic Classification Transformer (ITCT), a novel approach that utilizes the state-of-the-art transformer-based model named TabTransformer. ITCT, which is pre-trained on a large labeled MQTT-based IoT traffic dataset and may be fine-tuned with a small set of labeled data, showed promising results in various traffic classification tasks. Our experiments demonstrated that the ITCT model significantly outperforms existing models, achieving an overall accuracy of 82%. To support reproducibility and collaborative development, all associated code has been made publicly available., Comment: Updated version of: B. Bazaluk, M. Hamdan, M. Ghaleb, M. S. M. Gismalla, F. S. Correa da Silva and D. M. Batista, "Towards a Transformer-Based Pre-trained Model for IoT Traffic Classification," NOMS 2024-2024 IEEE Network Operations and Management Symposium, Seoul, Korea, Republic of, 2024, pp. 1-7, doi: 10.1109/NOMS59830.2024.10575448

The choice of representation plays a key role in self-driving. Bird's eye view (BEV) representations have shown remarkable performance in recent years. In this paper, we propose to learn object-centric representations in BEV to distill a complex scene into more actionable information for self-driving. We first learn to place objects into slots with a slot attention model on BEV sequences. Based on these object-centric representations, we then train a transformer to learn to drive as well as reason about the future of other vehicles. We found that object-centric slot representations outperform both scene-level and object-level approaches that use the exact attributes of objects. Slot representations naturally incorporate information about objects from their spatial and temporal context such as position, heading, and speed without explicitly providing it. Our model with slots achieves an increased completion rate of the provided routes and, consequently, a higher driving score, with a lower variance across multiple runs, affirming slots as a reliable alternative in object-centric approaches. Additionally, we validate our model's performance as a world model through forecasting experiments, demonstrating its capability to predict future slot representations accurately. The code and the pre-trained models can be found at https://kuis-ai.github.io/CarFormer/., Comment: Accepted to ECCV 2024, code and the pre-trained models can be found at https://kuis-ai.github.io/CarFormer/

We present a unified approach to obtain scaling limits of neural networks using the genus expansion technique from random matrix theory. This approach begins with a novel expansion of neural networks which is reminiscent of Butcher series for ODEs, and is obtained through a generalisation of Fa\`a di Bruno's formula to an arbitrary number of compositions. In this expansion, the role of monomials is played by random multilinear maps indexed by directed graphs whose edges correspond to random matrices, which we call operator graphs. This expansion linearises the effect of the activation functions, allowing for the direct application of Wick's principle to compute the expectation of each of its terms. We then determine the leading contribution to each term by embedding the corresponding graphs onto surfaces, and computing their Euler characteristic. Furthermore, by developing a correspondence between analytic and graphical operations, we obtain similar graph expansions for the neural tangent kernel as well as the input-output Jacobian of the original neural network, and derive their infinite-width limits with relative ease. Notably, we find explicit formulae for the moments of the limiting singular value distribution of the Jacobian. We then show that all of these results hold for networks with more general weights, such as general matrices with i.i.d. entries satisfying moment assumptions, complex matrices and sparse matrices., Comment: v4: minor changes to presentation

Quantum machine learning has gained considerable attention as quantum technology advances, presenting a promising approach for efficiently learning complex data patterns. Despite this promise, most contemporary quantum methods require significant resources for variational parameter optimization and face issues with vanishing gradients, leading to experiments that are either limited in scale or lack potential for quantum advantage. To address this, we develop a general-purpose, gradient-free, and scalable quantum reservoir learning algorithm that harnesses the quantum dynamics of neutral-atom analog quantum computers to process data. We experimentally implement the algorithm, achieving competitive performance across various categories of machine learning tasks, including binary and multi-class classification, as well as timeseries prediction. Effective and improving learning is observed with increasing system sizes of up to 108 qubits, demonstrating the largest quantum machine learning experiment to date. We further observe comparative quantum kernel advantage in learning tasks by constructing synthetic datasets based on the geometric differences between generated quantum and classical data kernels. Our findings demonstrate the potential of utilizing classically intractable quantum correlations for effective machine learning. We expect these results to stimulate further extensions to different quantum hardware and machine learning paradigms, including early fault-tolerant hardware and generative machine learning tasks., Comment: 10 + 14 pages, 4 + 7 figures

Central to the Transformer architectures' effectiveness is the self-attention mechanism, a function that maps queries, keys, and values into a high-dimensional vector space. However, training the attention weights of queries, keys, and values is non-trivial from a state of random initialization. In this paper, we propose two methods. (i) We first address the initialization problem of Vision Transformers by introducing a simple, yet highly innovative, initialization approach utilizing Discrete Cosine Transform (DCT) coefficients. Our proposed DCT-based attention initialization marks a significant gain compared to traditional initialization strategies; offering a robust foundation for the attention mechanism. Our experiments reveal that the DCT-based initialization enhances the accuracy of Vision Transformers in classification tasks. (ii) We also recognize that since DCT effectively decorrelates image information in the frequency domain, this decorrelation is useful for compression because it allows the quantization step to discard many of the higher-frequency components. Based on this observation, we propose a novel DCT-based compression technique for the attention function of Vision Transformers. Since high-frequency DCT coefficients usually correspond to noise, we truncate the high-frequency DCT components of the input patches. Our DCT-based compression reduces the size of weight matrices for queries, keys, and values. While maintaining the same level of accuracy, our DCT compressed Swin Transformers obtain a considerable decrease in the computational overhead.

We derive precise upper bounds for the maximum of the Riemann zeta function on short intervals on the critical line, showing for any $\theta\in(-1,0]$, the set of $t\in [T,2T]$ for which $$\max_{|h|\leq \log^\theta T}|\zeta(\tfrac{1}{2}+it+ih)|>\exp\bigg({y+\sqrt{(\log\log T)|\theta|/2}\cdot {S}}\bigg)\frac{(\log T)^{(1+\theta)}}{(\log\log T)^{3/4}}$$ is bounded above by $Cy\exp({-2y-y^2/((1+\theta)\log\log T)})$ (where $S$ is a random variable that is approximately a standard Gaussian as $T$ tends to infinity). This settles a strong form of a conjecture of Fyodorov--Hiary--Keating in mesoscopic intervals which was only known in the leading order. Using similar techniques, we also derive upper bounds for the second moment of the zeta function on such intervals. Conditioning on the value of $S$, we show that for all $t\in[T,2T]$ outside a set of order $o(T)$, $$\frac{1}{\log^\theta T}\int_{|h|\in \log^\theta T} |\zeta(\tfrac{1}{2}+it+ih)|^2\mathrm{d}h \ll e^{2S}\cdot \left(\frac{(\log T)^{(1+\theta)}}{\sqrt{\log\log T}}\right).$$ This proves a weak form of another conjecture of Fyodorov-Keating and generalizes a result of Harper, which is recovered at $\theta = 0$ (in which case $S$ is defined to be zero). Our main tool is an adaptation of the recursive scheme introduced by one of the authors, Bourgade and Radziwi{\l}{\l} to mesoscopic intervals., Comment: 32 pages, 2 figures

The ability to emotionally or intellectually understand another person's thoughts and feelings-empathy-can foster critical connections that facilitate learning and collaboration. We present a case study of physics faculty that examines their experiences empathizing with students, both in and outside of the classroom. We expand on frameworks for understanding the empathy process by identifying key mediating factors, and note various barriers that faculty express as preventing them from taking empathetic action. Our analysis unpacks the mechanisms of communication and contextual information, which play key roles in the empathetic process, with implications for programs that rely on empathy to develop more inclusive STEM academic spaces., Comment: ISLS owns the copyright of this paper as it will appear in the upcoming 2024 Conference Proceedings. We have obtained permission to share this preprint

This study investigates how faculty acquire contextual information about students, examining mechanisms and motivations used when sharing their identity to facilitate empathy. Empathy is "the ability and tendency to share and understand others' internal state" (Zaki and Ochsner, 2012), and is a critical factor in both motivating faculty to enact large scale change and take immediate, smaller actions. This study explores the impact identity sharing has on obtaining contextual information that motivates empathetic action. Nineteen semi structured interviews with physics faculty explored participant identities and interactions across various contexts. Employing emergent thematic coding, we crafted four personas around faculty sharing, teaching values, and student reciprocity. Brooke, the Trust Builder, prioritizes creating an environment of trust by openly discussing their identity, aiming to foster student openness. Nour, the Identity Navigator, shares personal experiences to assist others in navigating their own identities, acknowledging the challenges of college years. Brooke and Nour had more students approaching them with personal issues, indicating a correlation between faculty identity sharing and student openness. Casey, the Cautious Sharer, expresses concerns about potential alienation or backlash, approaching personal sharing with caution. Wray, adopting a Walled Off approach, separates personal and professional life due to past negative experiences or a belief in the importance of that division. Among the faculty interviewed, 15 who were explicitly open about their identities reported that students approached them with personal issues. This study outlines mechanisms influencing when and what faculty share about themselves in different contexts. Our findings underscore the significance of fostering dialogue as the initial step in empathy development.

To complete the study of Fredholm type operators of [10] and [11], we define in this paper the classes of left and right semi-B-Fredholm operators (Definition 3.1). Then, we prove that an operator $T \in L(X), X$ being a Banach space, is a left( resp. right) semi-B-Fredholm operator if and only if $T$ is the direct sum of left(resp. right) semi-Fredholm operator and a nilpotent one. This result extend the earlier characterization of B-Fredholm operators as the direct sum of a Fredholm operator and a nilpotent one obtained in [4,Theorem 2.7]., Comment: There is no difference with version 2, only an author whose name was forgotten in version 2 is added. arXiv admin note: text overlap with arXiv:2401.01390, arXiv:2401.01061

While in \cite{HB} we studied classes of Fredholm-type operators defined by the homomorphism $\Pi$ from $L(X)$ onto the Calkin algebra $\mathcal{C}(X)$, $X$ being a Banach space, we study in this paper two classes of Fredholm-type operators defined by the homomorphism $\pi$ from $L(X)$ onto the algebra $\mathcal{C}_0(X)= L(X)/F_0(X),$ where $F_0(X)$ is the ideal of finite rank operators in $L(X).$ Then we define an index for Fredholm-type operators and we show that this new index satisfies similar properties as the usual Fredholm index., Comment: arXiv admin note: text overlap with arXiv:2401.01061

Given an idempotent $p$ in a Banach algebra and following the study in \cite{P50} of p-invertibility, we consider here left p-invertibility, right p-invertibility and p-invertibility in the Calkin Algebra $\mathcal{C}(X),$ where $X$ is a Banach space. Then we define and study left and right generalized Drazin invertibility and we characterize left and right Drazin invertible elements in the Calkin algebra. Globally, this leads to define and characterize the classes of P-Fredholm, pseudo B-Fredholm and weak B-Fredholm operators.

As the digital world evolves, so do the threats to our security do too. Keyloggers were once a large threat to the cyber world. Though undergoing many transformations alongside the technological advancements of today, it is important to raise questions about the importance of Anti-Keyloggers in our current state of cyber security. This survey dives into the historical evolution of Keyloggers and investigates their current day forms. Within this inspection of Keyloggers, we must propose whether Anti-Keyloggers serve a purpose to this ever-changing landscape before us or if emerging strategies have rendered them obsolete.