Search

This pilot study explores the application of language models (LMs) to model game event sequences, treating them as a customized natural language. We investigate a popular mobile game, transforming raw event data into textual sequences and pretraining a Longformer model on this data. Our approach captures the rich and nuanced interactions within game sessions, effectively identifying meaningful player segments. The results demonstrate the potential of self-supervised LMs in enhancing game design and personalization without relying on ground-truth labels., Comment: published in Workshop on Customizable NLP at EMNLP 2024

We propose a method to enhance the performance of Large Language Models (LLMs) by integrating quantum computing and quantum-inspired techniques. Specifically, our approach involves replacing the weight matrices in the Self-Attention and Multi-layer Perceptron layers with a combination of two variational quantum circuits and a quantum-inspired tensor network, such as a Matrix Product Operator (MPO). This substitution enables the reproduction of classical LLM functionality by decomposing weight matrices through the application of tensor network disentanglers and MPOs, leveraging well-established tensor network techniques. By incorporating more complex and deeper quantum circuits, along with increasing the bond dimensions of the MPOs, our method captures additional correlations within the quantum-enhanced LLM, leading to improved accuracy beyond classical models while maintaining low memory overhead., Comment: 4 pages, 2 figures

The persistent challenge of bias in machine learning models necessitates robust solutions to ensure parity and equal treatment across diverse groups, particularly in classification tasks. Current methods for mitigating bias often result in information loss and an inadequate balance between accuracy and fairness. To address this, we propose a novel methodology grounded in bilevel optimization principles. Our deep learning-based approach concurrently optimizes for both accuracy and fairness objectives, and under certain assumptions, achieving proven Pareto optimal solutions while mitigating bias in the trained model. Theoretical analysis indicates that the upper bound on the loss incurred by this method is less than or equal to the loss of the Lagrangian approach, which involves adding a regularization term to the loss function. We demonstrate the efficacy of our model primarily on tabular datasets such as UCI Adult and Heritage Health. When benchmarked against state-of-the-art fairness methods, our model exhibits superior performance, advancing fairness-aware machine learning solutions and bridging the accuracy-fairness gap. The implementation of FairBiNN is available on https://github.com/yazdanimehdi/FairBiNN., Comment: Accepted to NeurIPS 2024

Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on six diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8\%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines.

The EXperiment for Cryogenic Large-Aperture Intensity Mapping (EXCLAIM) is a balloon-borne telescope designed to survey star formation over cosmological time scales using intensity mapping in the 420 - 540 GHz frequency range. EXCLAIM uses a fully cryogenic telescope coupled to six on-chip spectrometers featuring kinetic inductance detectors (KIDs) to achieve high sensitivity, allowing for fast integration in dark atmospheric windows. The telescope receiver is cooled to $\approx$ 1.7 K by immersion in a superfluid helium bath and enclosed in a superfluid-tight shell with a meta-material anti-reflection coated silicon window. In addition to the optics and the spectrometer package, the receiver contains the magnetic shielding, the cryogenic segment of the spectrometer readout, and the sub-Kelvin cooling system. A three-stage continuous adiabatic demagnetization refrigerator (CADR) keeps the detectors at 100 mK while a $^4$He sorption cooler provides a 900 mK thermal intercept for mechanical suspensions and coaxial cables. We present the design of the EXCLAIM receiver and report on the flight-like testing of major receiver components, including the superfluid-tight receiver window and the sub-Kelvin coolers., Comment: 13 pages, 7 figures, 1 table

The absence of security measures between DNS recursive resolvers and authoritative nameservers has been exploited by both inline and off-path attacks. While many security proposals have been made in practice and previous literature, they typically suffer from deployability barriers and/or inadequate security properties. The absence of a broadly adopted security solution between resolvers and nameservers motivates a new scheme that mitigates these issues in previous proposals. We present DNSSEC+, which addresses security and deployability downsides of DNSSEC, while retaining its benefits. DNSSEC+ takes advantage of the existent DNSSEC trust model and authorizes the nameservers within a zone for short intervals to serve the zone data securely, facilitating real-time security properties for DNS responses, without requiring long-term private keys to be duplicated (thus put at risk) on authoritative nameservers. Regarding name resolution latency, DNSSEC+ offers a performance comparable to less secure schemes. We define nine security, privacy, and deployability properties for name resolution, and show how DNSSEC+ fulfills these properties., Comment: 15 pages, 6 figures

This paper investigates the optimization of 2D and 3D composite structures using machine learning (ML) techniques, focusing on fracture toughness and crack propagation in the Double Cantilever Beam (DCB) test. By exploring the intricate relationship between microstructural arrangements and macroscopic properties of composites, the study demonstrates the potential of ML as a powerful tool to expedite the design optimization process, offering notable advantages over traditional finite element analysis. The research encompasses four distinct cases, examining crack propagation and fracture toughness in both 2D and 3D composite models. Through the application of ML algorithms, the study showcases the capability for rapid and accurate exploration of vast design spaces in composite materials. The findings highlight the efficiency of ML in predicting mechanical behaviors with limited training data, paving the way for broader applications in composite design and optimization. This work contributes to advancing the understanding of ML's role in enhancing the efficiency of composite material design processes.

From a fractal perspective, the entropy bound of gravitational systems undergoes changes. Furthermore, in the cosmological setting, the conservation law of a perfect fluid is also altered in such systems, affecting spatial elements like volume, area, and radius. By applying the first law of thermodynamics and deriving the Friedmann equations, we can gain insight into the evolution of such a fractal cosmos. However, observations continue to necessitate the existence of a dark energy source. To address this, in this article, we have created a novel fractal $\Lambda$CDM cosmological model and determined the fractal cosmological observables. We show that the spatial fractal dimension is two, and the age of the Universe is 13.91 Gyr, by fitting the model's parameters to cosmological data., Comment: 9 pages, 6 figures, to appear in Physics of the Dark Universe

Methods for learning latent user representations from historical behavior logs have gained traction for recommendation tasks in e-commerce, content streaming, and other settings. However, this area still remains relatively underexplored in video and mobile gaming contexts. In this work, we present a novel method for overcoming this limitation by extending a long-range Transformer model from the natural language processing domain to player behavior data. We discuss specifics of behavior tracking in games and propose preprocessing and tokenization approaches by viewing in-game events in an analogous way to words in sentences, thus enabling learning player representations in a self-supervised manner in the absence of ground-truth annotations. We experimentally demonstrate the efficacy of the proposed approach in fitting the distribution of behavior events by evaluating intrinsic language modeling metrics. Furthermore, we qualitatively analyze the emerging structure of the learned embedding space and show its value for generating insights into behavior patterns to inform downstream applications.

Convolutional neural networks (CNNs) are one of the most widely used neural network architectures, showcasing state-of-the-art performance in computer vision tasks. Although larger CNNs generally exhibit higher accuracy, their size can be effectively reduced by ``tensorization'' while maintaining accuracy, namely, replacing the convolution kernels with compact decompositions such as Tucker, Canonical Polyadic decompositions, or quantum-inspired decompositions such as matrix product states, and directly training the factors in the decompositions to bias the learning towards low-rank decompositions. But why doesn't tensorization seem to impact the accuracy adversely? We explore this by assessing how \textit{truncating} the convolution kernels of \textit{dense} (untensorized) CNNs impact their accuracy. Specifically, we truncated the kernels of (i) a vanilla four-layer CNN and (ii) ResNet-50 pre-trained for image classification on CIFAR-10 and CIFAR-100 datasets. We found that kernels (especially those inside deeper layers) could often be truncated along several cuts resulting in significant loss in kernel norm but not in classification accuracy. This suggests that such ``correlation compression'' (underlying tensorization) is an intrinsic feature of how information is encoded in dense CNNs. We also found that aggressively truncated models could often recover the pre-truncation accuracy after only a few epochs of re-training, suggesting that compressing the internal correlations of convolution layers does not often transport the model to a worse minimum. Our results can be applied to tensorize and compress CNN models more effectively., Comment: 40 pages, 21 images

As societal interactions increasingly involve both intentional and unintentional agents, understanding their effects on human cognition becomes paramount. This study investigates the neural correlates of interacting with intentional versus artificial agents in a simulated tennis game scenario. Employing functional near-infrared spectroscopy (fNIRS), brain activity in 50 male participants during gameplay against both types of opponents was analyzed. The used methodological approach ensures ecological validity by simulating real-world decision-making scenarios while participants undergo fNIRS scanning, avoiding the constraints of traditional neuroimaging methods. Six prefrontal cortex channels are focused on, leveraging the 10-20 system, to capture nuanced differences in brain activity. Wavelet analysis was utilized to dissect the data into frequency-specific differences, revealing subtle variations across different channels and frequency bands. Moreover, activity was quantified by comparing average data signals between rest and play modes across all points using Generalized Linear Model (GLM). The findings unveil significant differences in neural activation patterns, particularly in one specific channel and frequency range, suggesting distinct cognitive processing when interacting with intentional agents. These results align with previous neuroimaging studies and contribute to understanding the neural underpinnings of human-agent interactions in naturalistic settings. While acknowledging study limitations, including sample homogeneity and spatial accuracy constraints, the study's findings underscore the potential of fNIRS in exploring complex cognitive phenomena beyond laboratory confines.

Explainable AI (XAI) aids in deciphering 'black-box' models. While several methods have been proposed and evaluated primarily in the image domain, the exploration of explainability in the text domain remains a growing research area. In this paper, we delve into the applicability of XAI methods for the text domain. In this context, the 'Similarity Difference and Uniqueness' (SIDU) XAI method, recognized for its superior capability in localizing entire salient regions in image-based classification is extended to textual data. The extended method, SIDU-TXT, utilizes feature activation maps from 'black-box' models to generate heatmaps at a granular, word-based level, thereby providing explanations that highlight contextually significant textual elements crucial for model predictions. Given the absence of a unified standard for assessing XAI methods, this study applies a holistic three-tiered comprehensive evaluation framework: Functionally-Grounded, Human-Grounded and Application-Grounded, to assess the effectiveness of the proposed SIDU-TXT across various experiments. We find that, in sentiment analysis task of a movie review dataset, SIDU-TXT excels in both functionally and human-grounded evaluations, demonstrating superior performance through quantitative and qualitative analyses compared to benchmarks like Grad-CAM and LIME. In the application-grounded evaluation within the sensitive and complex legal domain of asylum decision-making, SIDU-TXT and Grad-CAM demonstrate comparable performances, each with its own set of strengths and weaknesses. However, both methods fall short of entirely fulfilling the sophisticated criteria of expert expectations, highlighting the imperative need for additional research in XAI methods suitable for such domains., Comment: Preprint submitted to Elsevier on Jan 5th, 2024

Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while keeping the number of neurons fixed. While these compression methods have been relatively successful in practice, there is no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model's correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with - or on top of - other compression techniques. As a benchmark, we demonstrate that a combination of CompactifAI with quantization allows to reduce a 93% the memory size of LlaMA 7B, reducing also 70% the number of parameters, accelerating 50% the training and 25% the inference times of the model, and just with a small accuracy drop of 2% - 3%, going much beyond of what is achievable today by other compression techniques. Our methods also allow to perform a refined layer sensitivity profiling, showing that deeper layers tend to be more suitable for tensor network compression, which is compatible with recent observations on the ineffectiveness of those layers for LLM performance. Our results imply that standard LLMs are, in fact, heavily overparametrized, and do not need to be large at all., Comment: 5 pages, 4 figures, 2 tables, and supplementary information of 2 pages and 1 figure. Revised version with new benchmarks for LlaMA2-7B

Health issues and pandemics remain paramount concerns in the contemporary era. Clostridioides Difficile Infection (CDI) stands out as a critical healthcare-associated infection with global implications. Effectively understanding the mechanisms of infection dissemination within healthcare units and hospitals is imperative to implement targeted containment measures. In this study, we address the limitations of prior research by Sulyok et al., where they delineated two distinct categories of surfaces as high-touch and low-touch fomites, and subsequently evaluated the viral spread contribution of each surface utilizing mathematical modeling and Ordinary Differential Equations (ODE). Acknowledging the indispensable role of spatial features and heterogeneity in the modeling of hospital and healthcare settings, we employ agent-based modeling to capture new insights. By incorporating spatial considerations and heterogeneous patients, we explore the impact of high-touch and low-touch surfaces on contamination transmission between patients. Furthermore, the study encompasses a comprehensive assessment of various cleaning protocols, with differing intervals and detergent cleaning efficacies, in order to identify the most optimal cleaning strategy and the most important factor amidst the array of alternatives. Our results indicate that, among various factors, the frequency of cleaning intervals is the most critical element for controlling the spread of CDI in a hospital environment., Comment: Accepted and presented at the Computational Social Science Society of the Americas Conference (CSS 2023)

The rapid and widespread dissemination of misinformation through social networks is a growing concern in today's digital age. This study focused on modeling fake news diffusion, discovering the spreading dynamics, and designing control strategies. A common approach for modeling the misinformation dynamics is SIR-based models. Our approach is an extension of a model called 'SBFC' which is a SIR-based model. This model has three states, Susceptible, Believer, and Fact-Checker. The dynamics and transition between states are based on neighbors' beliefs, hoax credibility, spreading rate, probability of verifying the news, and probability of forgetting the current state. Our contribution is to push this model to real social networks by considering different classes of agents with their characteristics. We proposed two main strategies for confronting misinformation diffusion. First, we can educate a minor class, like scholars or influencers, to improve their ability to verify the news or remember their state longer. The second strategy is adding fact-checker bots to the network to spread the facts and influence their neighbors' states. Our result shows that both of these approaches can effectively control the misinformation spread., Comment: Accepted at The Computational Social Science Society of the Americas (CSS) - 2023, Annual Conference

Drug-Target Interaction (DTI) prediction is vital for drug discovery, yet challenges persist in achieving model interpretability and optimizing performance. We propose a novel transformer-based model, FragXsiteDTI, that aims to address these challenges in DTI prediction. Notably, FragXsiteDTI is the first DTI model to simultaneously leverage drug molecule fragments and protein pockets. Our information-rich representations for both proteins and drugs offer a detailed perspective on their interaction. Inspired by the Perceiver IO framework, our model features a learnable latent array, initially interacting with protein binding site embeddings using cross-attention and later refined through self-attention and used as a query to the drug fragments in the drug's cross-attention transformer block. This learnable query array serves as a mediator and enables seamless information translation, preserving critical nuances in drug-protein interactions. Our computational results on three benchmarking datasets demonstrate the superior predictive power of our model over several state-of-the-art models. We also show the interpretability of our model in terms of the critical components of both target proteins and drug molecules within drug-target pairs., Comment: Accepted at the NeurIPS workshop (AI4D3) - 2023

We show how quantum-inspired 2d tensor networks can be used to efficiently and accurately simulate the largest quantum processors from IBM, namely Eagle (127 qubits), Osprey (433 qubits) and Condor (1121 qubits). We simulate the dynamics of a complex quantum many-body system -- specifically, the kicked Ising experiment considered recently by IBM in Nature 618, p. 500-505 (2023) -- using graph-based Projected Entangled Pair States (gPEPS), which was proposed by some of us in PRB 99, 195105 (2019). Our results show that simple tensor updates are already sufficient to achieve very large unprecedented accuracy with remarkably low computational resources for this model. Apart from simulating the original experiment for 127 qubits, we also extend our results to 433 and 1121 qubits, and for evolution times around 8 times longer, thus setting a benchmark for the newest IBM quantum machines. We also report accurate simulations for infinitely-many qubits. Our results show that gPEPS are a natural tool to efficiently simulate quantum computers with an underlying lattice-based qubit connectivity, such as all quantum processors based on superconducting qubits., Comment: 7 pages, 8 figures, revised version