Search

Large Language Models (LLMs) are prevalent in modern applications but often memorize training data, leading to privacy breaches and copyright issues. Existing research has mainly focused on posthoc analyses, such as extracting memorized content or developing memorization metrics, without exploring the underlying architectural factors that contribute to memorization. In this work, we investigate memorization from an architectural lens by analyzing how attention modules at different layers impact its memorization and generalization performance. Using attribution techniques, we systematically intervene in the LLM architecture by bypassing attention modules at specific blocks while keeping other components like layer normalization and MLP transformations intact. We provide theorems analyzing our intervention mechanism from a mathematical view, bounding the difference in layer outputs with and without our attributions. Our theoretical and empirical analyses reveal that attention modules in deeper transformer blocks are primarily responsible for memorization, whereas earlier blocks are crucial for the models generalization and reasoning capabilities. We validate our findings through comprehensive experiments on different LLM families (Pythia and GPTNeo) and five benchmark datasets. Our insights offer a practical approach to mitigate memorization in LLMs while preserving their performance, contributing to safer and more ethical deployment in real world applications.

We investigate the special case of the chiral magnet with vanishing Heisenberg exchange energy, whose axisymmetric Skyrmion solution has previously been found. The dynamical equations of this model look like inviscid fluid flow, and by investigating path-lines of this flow we can construct explicit static and dynamic solutions. We find an infinite-dimensional family of static Skyrmions that are related to the axisymmetric Skyrmion by co-ordinate transformations thus discovering a new moduli space, and further infinite-dimensional families of axisymmetric and non-axisymmetric breather-like supercompactons. We call the maps generating such moduli space "contour-preserving" maps., Comment: LaTeX: 20 pages, 6 figures

The advent of multimodal learning has brought a significant improvement in document AI. Documents are now treated as multimodal entities, incorporating both textual and visual information for downstream analysis. However, works in this space are often focused on the textual aspect, using the visual space as auxiliary information. While some works have explored pure vision based techniques for document image understanding, they require OCR identified text as input during inference, or do not align with text in their learning procedure. Therefore, we present a novel image-text alignment technique specially designed for leveraging the textual information in document images to improve performance on visual tasks. Our document encoder model DoPTA - trained with this technique demonstrates strong performance on a wide range of document image understanding tasks, without requiring OCR during inference. Combined with an auxiliary reconstruction objective, DoPTA consistently outperforms larger models, while using significantly lesser pre-training compute. DoPTA also sets new state-of-the art results on D4LA, and FUNSD, two challenging document visual analysis benchmarks.

The processing unit of a solid-state quantum computer consists in an array of coupled qubits, each locally driven with on-chip microwave lines that route carefully-engineered control signals to the qubits in order to perform logical operations. This approach to quantum computing comes with two major problems. On the one hand, it greatly hampers scalability towards fault-tolerant quantum computers, which are estimated to need a number of qubits -- and, therefore driving lines -- on the order of $10^6$. On the other hand, these lines are a source of electromagnetic noise, exacerbating frequency crowding and crosstalk, while also contributing to power dissipation inside the dilution fridge. We here tackle these two overwhelming challenges by presenting a novel quantum processing unit (QPU) for a universal quantum computer which is globally (rather than locally) driven. Our QPU relies on a string of superconducting qubits with always-on ZZ interactions, enclosed into a closed geometry, which we dub ``conveyor belt''. Strikingly, this architecture requires only $\mathcal{O}(N)$ physical qubits to run a computation on $N$ computational qubits, in contrast to previous $\mathcal{O}(N^2)$ proposals for global quantum computation. Additionally, universality is achieved via the implementation of single-qubit gates and a one-shot Toffoli gate. The ability to perform multi-qubit operations in a single step could vastly improve the fidelity and execution time of many algorithms., Comment: 7 pages, 2 figures and supplemental material

We propose a generalization of the theory of magnetic Skyrmions in chiral magnets in two dimensions to a higher-dimensional theory with magnetic Skyrmions in three dimensions and an $S^3$ target space, requiring a 4-dimensional magnetization vector. A physical realization of our theory necessitates the use of a synthetic dimension, recently promoted and realized in condensed matter physics. In the simplest incarnation of the theory, we find a Skyrmion and a sphaleron - the latter being an unstable soliton. Including also the Skyrme term in theory enriches the spectrum to a small metastable Skyrmion, an unstable sphaleron and a large stable Skyrmion., Comment: LaTeX: 40 pages, 14 figures

Modern Text-to-Image (T2I) Diffusion models have revolutionized image editing by enabling the generation of high-quality photorealistic images. While the de facto method for performing edits with T2I models is through text instructions, this approach non-trivial due to the complex many-to-many mapping between natural language and images. In this work, we address exemplar-based image editing -- the task of transferring an edit from an exemplar pair to a content image(s). We propose ReEdit, a modular and efficient end-to-end framework that captures edits in both text and image modalities while ensuring the fidelity of the edited image. We validate the effectiveness of ReEdit through extensive comparisons with state-of-the-art baselines and sensitivity analyses of key design choices. Our results demonstrate that ReEdit consistently outperforms contemporary approaches both qualitatively and quantitatively. Additionally, ReEdit boasts high practical applicability, as it does not require any task-specific optimization and is four times faster than the next best baseline., Comment: First three authors contributed equally to this work

We propose a platform for implementing a universal, globally driven quantum computer based on a 2D ladder hosting three different species of superconducting qubits. In stark contrast with the existing literature, our scheme exploits the always-on longitudinal ZZ coupling. The latter, combined with specific driving frequencies, enables the reach of a blockade regime, which plays a pivotal role in the computing scheme., Comment: Typo corrected in the supplemental material

The magnetic Skyrmion is described by one control parameter and one length scale. We study the two extreme limits of the control parameter - infinitely large and vanishing - and find that the magnetic Skyrmion becomes a "restricted" magnetic Skyrmion and an O(3) sigma model lump, respectively. Depending on the potential under consideration, the restricted limit manifests differently. In the case of the Zeeman term, the restricted magnetic Skyrmion becomes a "supercompacton" that develops a discontinuity, whereas for the Zeeman term to the power 3/2 it becomes a normal compacton. In both the lump and the restricted limit the solution is given in exact explicit form. We observe that the case of the Zeeman term squared, which can also be understood as a special combination of the Zeeman term and the easy-plane potential - realizable in the laboratory, the analytically exact solution for all values of the coupling - including the Bogomol'nyi-Prasad-Sommerfield (BPS) case - is also of the lump type. Finally, we notice that certain materials (e.g. Fe$_{1-x}$Co$_x$Si or Mn$_{1-x}$Fe$_x$Ge) have a rather large control parameter $\epsilon$ of order 100, making the restricted limit a suitable rough approximation., Comment: LaTeX: 28 pages, 8 figures; V2: comments and references added, figures updated; V3: clarified the discussion of the boundary terms

Quantum states of a spin $1/2$ (a qubit) are parametrized by the space $CP^1 \sim S^2$, the Bloch sphere. A spin j (a 2j+1 -state system) for generic j is represented instead by a point of a larger space, $CP^{2j}$. Here we study the angular momentum/spin in the limit, $j \to \infty$. The state, $(J \cdot n) | j, n\rangle = j |j, n \rangle $, where $J$ is the angular momentum operator and $n$ stands for a generic unit vector in $R^3$, is found to behave as a classical angular momentum, $ j n $. We discuss this phenomenon, by analysing the Stern-Gerlach experiments, the angular-momentum composition rule, and the rotation matrix. This problem arose from the consideration of a macroscopic body under an inhomogeneous magnetic field. Our observations help to explain how classical mechanics (with unique particle trajectories) emerges naturally from quantum mechanics in this context, and at the same time, make the widespread idea that large spins somehow become classical, a more precise one., Comment: 19 pages, 5 figures

Neural cellular automata represent an evolution of the traditional cellular automata model, enhanced by the integration of a deep learning-based transition function. This shift from a manual to a data-driven approach significantly increases the adaptability of these models, enabling their application in diverse domains, including content generation and artificial life. However, their widespread application has been hampered by significant computational requirements. In this work, we introduce the Latent Neural Cellular Automata (LNCA) model, a novel architecture designed to address the resource limitations of neural cellular automata. Our approach shifts the computation from the conventional input space to a specially designed latent space, relying on a pre-trained autoencoder. We apply our model in the context of image restoration, which aims to reconstruct high-quality images from their degraded versions. This modification not only reduces the model's resource consumption but also maintains a flexible framework suitable for various applications. Our model achieves a significant reduction in computational requirements while maintaining high reconstruction fidelity. This increase in efficiency allows for inputs up to 16 times larger than current state-of-the-art neural cellular automata models, using the same resources.

Dysarthria is a speech disorder that hinders communication due to difficulties in articulating words. Detection of dysarthria is important for several reasons as it can be used to develop a treatment plan and help improve a person's quality of life and ability to communicate effectively. Much of the literature focused on improving ASR systems for dysarthric speech. The objective of the current work is to develop models that can accurately classify the presence of dysarthria and also give information about the intelligibility level using limited data by employing a few-shot approach using a transformer model. This work also aims to tackle the data leakage that is present in previous studies. Our whisper-large-v2 transformer model trained on a subset of the UASpeech dataset containing medium intelligibility level patients achieved an accuracy of 85%, precision of 0.92, recall of 0.8 F1-score of 0.85, and specificity of 0.91. Experimental results also demonstrate that the model trained using the 'words' dataset performed better compared to the model trained on the 'letters' and 'digits' dataset. Moreover, the multiclass model achieved an accuracy of 67%., Comment: Paper has been presented at ICCCNT 2023 and the final version will be published in IEEE Digital Library Xplore

This research paper explores the classification of knee osteoarthritis (OA) severity levels using advanced computer vision models and augmentation techniques. The study investigates the effectiveness of data preprocessing, including Contrast-Limited Adaptive Histogram Equalization (CLAHE), and data augmentation using diffusion models. Three experiments were conducted: training models on the original dataset, training models on the preprocessed dataset, and training models on the augmented dataset. The results show that data preprocessing and augmentation significantly improve the accuracy of the models. The EfficientNetB3 model achieved the highest accuracy of 84\% on the augmented dataset. Additionally, attention visualization techniques, such as Grad-CAM, are utilized to provide detailed attention maps, enhancing the understanding and trustworthiness of the models. These findings highlight the potential of combining advanced models with augmented data and attention visualization for accurate knee OA severity classification., Comment: Paper has been accepted to be presented at ICACECS 2023 and the final version will be published by Atlantis Highlights in Computer Science (AHCS) , Atlantis Press(part of Springer Nature)

As transfer learning techniques are increasingly used to transfer knowledge from the source model to the target task, it becomes important to quantify which source models are suitable for a given target task without performing computationally expensive fine tuning. In this work, we propose HASTE (HArd Subset TransfErability), a new strategy to estimate the transferability of a source model to a particular target task using only a harder subset of target data. By leveraging the internal and output representations of model, we introduce two techniques, one class agnostic and another class specific, to identify harder subsets and show that HASTE can be used with any existing transferability metric to improve their reliability. We further analyze the relation between HASTE and the optimal average log likelihood as well as negative conditional entropy and empirically validate our theoretical bounds. Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE modified metrics are consistently better or on par with the state of the art transferability metrics., Comment: First three authors contributed equally

A short review which focuses on the employment of a Superconducting Radio Frequency cavity for detection of Gravitational Waves. This first part is inspired by the works of Berlin et al.. In the second part, we analyze also the idea to use a cavity-qubit system for the GWs detection and the Dark Photons detection. In this case, the starting point is the work of Dixit et al. which will be extended for SQMS cavities., Comment: This version has been removed by arXiv administrators because the submitter did not have the authority to grant the license at the time of submission

The main challenge in controlling hybrid systems arises from having to consider an exponential number of sequences of future modes to make good long-term decisions. Model predictive control (MPC) computes a control action through a finite-horizon optimisation problem. A key ingredient in this problem is a terminal cost, to account for the system's evolution beyond the chosen horizon. A good terminal cost can reduce the horizon length required for good control action and is often tuned empirically by observing performance. We build on the idea of using $N$-step $Q$-functions $(\mathcal{Q}^{(N)})$ in the MPC objective to avoid having to choose a terminal cost. We present a formulation incorporating the system dynamics and constraints to approximate the optimal $\mathcal{Q}^{(N)}$-function and algorithms to train the approximation parameters through an exploration of the state space. We test the control policy derived from the trained approximations on two benchmark problems through simulations and observe that our algorithms are able to learn good $\mathcal{Q}^{(N)}$-approximations for high dimensional hybrid systems based on a relatively small dataset. Finally, we compare our controller's performance against that of Hybrid MPC in terms of computation time and closed-loop cost., Comment: 8 pages, 4 figures, 1 algorithm, 2 tables, 1 appendix with 5 matrices

This work explores the novel idea of learning a submodular scoring function to improve the specificity/selectivity of existing feature attribution methods. Submodular scores are natural for attribution as they are known to accurately model the principle of diminishing returns. A new formulation for learning a deep submodular set function that is consistent with the real-valued attribution maps obtained by existing attribution methods is proposed. The final attribution value of a feature is then defined as the marginal gain in the induced submodular score of the feature in the context of other highly attributed features, thus decreasing the attribution of redundant yet discriminatory features. Experiments on multiple datasets illustrate that the proposed attribution method achieves higher specificity along with good discriminative power. The implementation of our method is publicly available at https://github.com/Piyushi-0/SEA-NN.

For future learning systems, incremental learning is desirable because it allows for: efficient resource usage by eliminating the need to retrain from scratch at the arrival of new data; reduced memory usage by preventing or limiting the amount of data required to be stored -- also important when privacy limitations are imposed; and learning that more closely resembles human learning. The main challenge for incremental learning is catastrophic forgetting, which refers to the precipitous drop in performance on previously learned tasks after learning a new one. Incremental learning of deep neural networks has seen explosive growth in recent years. Initial work focused on task-incremental learning, where a task-ID is provided at inference time. Recently, we have seen a shift towards class-incremental learning where the learner must discriminate at inference time between all classes seen in previous tasks without recourse to a task-ID. In this paper, we provide a complete survey of existing class-incremental learning methods for image classification, and in particular, we perform an extensive experimental evaluation on thirteen class-incremental methods. We consider several new experimental scenarios, including a comparison of class-incremental methods on multiple large-scale image classification datasets, an investigation into small and large domain shifts, and a comparison of various network architectures., Comment: Paper accepted for publication at TPAMI 2022. Code publicly available at https://github.com/mmasana/FACIL

Ambient Backscatter Communication (AmBC) is an emerging communication technology that can enable green Internet-of-Things deployments. The widespread acceptance of this paradigm is limited by low Signal-to-Interference-Plus-Noise Ratio (SINR) of the signal impinging on the receiver antenna due to the strong direct path interference and unknown ambient signal. The adverse impact of these two factors can be mitigated by using non-coherent multi-antenna receivers, which is known to require higher SINR to reach Bit-Error-Rate (BER) performance of coherent receivers. However, in literature, coherent receivers for AmBC systems are little-studied because of unknown ambient signal, unknown location of AmBC tags, and varying channel conditions. In this paper, a coherent multi-antenna receiver, which does not require a prior information of the ambient signal, for decoding Binary-Phase-shift-Keying (BPSK) modulated signal is presented. The performance of the proposed receiver is compared with the ideal coherent receiver that has a perfect phase information, and also with the performance of non-coherent receiver, which assumes distributions for ambient signal and phase offset caused by excess length of the backscatter path. Comparative simulation results show the designed receiver can achieve the same BER-performance of the ideal coherent receiver with 1-dB more SINR, which corresponds to 5-dB or more gain with respect to non-coherent reception of On-Off-Keying modulated signals. Variation of the detection performance with the tag location shows that the coverage area is in the close vicinity of the transmitter and a larger region around the receiver, which is consistent with the theoretical results.

Humans are capable of learning new tasks without forgetting previous ones, while neural networks fail due to catastrophic forgetting between new and previously-learned tasks. We consider a class-incremental setting which means that the task-ID is unknown at inference time. The imbalance between old and new classes typically results in a bias of the network towards the newest ones. This imbalance problem can either be addressed by storing exemplars from previous tasks, or by using image replay methods. However, the latter can only be applied to toy datasets since image generation for complex datasets is a hard problem. We propose a solution to the imbalance problem based on generative feature replay which does not require any exemplars. To do this, we split the network into two parts: a feature extractor and a classifier. To prevent forgetting, we combine generative feature replay in the classifier with feature distillation in the feature extractor. Through feature generation, our method reduces the complexity of generative replay and prevents the imbalance problem. Our approach is computationally efficient and scalable to large datasets. Experiments confirm that our approach achieves state-of-the-art results on CIFAR-100 and ImageNet, while requiring only a fraction of the storage needed for exemplar-based continual learning. Code available at \url{https://github.com/xialeiliu/GFR-IL}., Comment: Accepted at CVPR2020: Workshop on Continual Learning in Computer Vision

Recent advances in unsupervised domain adaptation have shown the effectiveness of adversarial training to adapt features across domains, endowing neural networks with the capability of being tested on a target domain without requiring any training annotations in this domain. The great majority of existing domain adaptation models rely on image translation networks, which often contain a huge amount of domain-specific parameters. Additionally, the feature adaptation step often happens globally, at a coarse level, hindering its applicability to tasks such as semantic segmentation, where details are of crucial importance to provide sharp results. In this thesis, we present a novel architecture, which learns to adapt features across domains by taking into account per class information. To that aim, we design a conditional pixel-wise discriminator network, whose output is conditioned on the segmentation masks. Moreover, following recent advances in image translation, we adopt the recently introduced StarGAN architecture as image translation backbone, since it is able to perform translations across multiple domains by means of a single generator network. Preliminary results on a segmentation task designed to assess the effectiveness of the proposed approach highlight the potential of the model, improving upon strong baselines and alternative designs., Comment: Master thesis dissertation for the Master in Computer Vision (Barcelona). 11 pages main article and 3 pages appendices. Code available

In bi-static Ambient Backscatter Communications (AmBC) systems, the direct path from the ambient source to the receiver can be several orders of magnitude stronger than the scattered path modulated by the AmBC device. Because of the large power difference between these two signals, the receiver needs to operate at a large dynamic range. In this paper, we propose a novel analog-digital hybrid null-steering beamformer which allows the backscatter receiver to detect and decode the weak AmBC-modulated signal buried in the strong direct path signals and the noise without requiring the instantaneous channel state information. The analog cancellation of the strong signal components allow the receiver automatic gain control to adjust to the level of the weak AmBC signals. This hence allows common analog-to-digital converters to be used for sampling the signal. After cancelling the strong components, the ambient source signal appears as zero mean fast fading from the AmBC system point of view. We use the direct path signal component to track the phase of the unknown ambient signal. In order to avoid channel estimation, we propose AmBC to use orthogonal channelization codes., Comment: 6 pages

We consider the problem of optimizing the steady state of a dynamical system in closed loop. Conventionally, the design of feedback optimization control laws assumes that the system is stationary. However, in reality, the dynamics of the (slow) iterative optimization routines can interfere with the (fast) system dynamics. We provide a study of the stability and convergence of these feedback optimization setups in closed loop with the underlying plant, via a custom-tailored singular perturbation analysis result. Our study is particularly geared towards applications in power systems and the question whether recently developed online optimization schemes can be deployed without jeopardizing dynamic system stability.