Search

When predicting the next token in a sequence, vanilla transformers compute attention over all previous tokens, resulting in quadratic scaling of compute with sequence length. State-space models compress the entire sequence of tokens into a fixed-dimensional representation to improve efficiency, while other architectures achieve sub-quadratic complexity via low-rank projections or sparse attention patterns over the sequence. In this paper, we introduce Attamba, a novel architecture that uses state-space models to compress chunks of tokens and applies attention on these compressed key-value representations. We find that replacing key and value projections in a transformer with SSMs can improve model quality and enable flexible token chunking, resulting in 24% improved perplexity with transformer of similar KV-Cache and attention footprint, and ~4 times smaller KV-Cache and Attention FLOPs for 5% perplexity trade-off. Attamba can perform attention on chunked-sequences of variable length, enabling a smooth transition between quadratic and linear scaling, offering adaptable efficiency gains.

The Jansen-Rit model of a cortical column in the cerebral cortex is widely used to simulate spontaneous brain activity (EEG) and event-related potentials. It couples a pyramidal cell population with two interneuron populations, of which one is fast and excitatory and the other slow and inhibitory. Our paper studies the transition between alpha and delta oscillations produced by the model. Delta oscillations are slower than alpha oscillations and have a more complex relaxation-type time profile. In the context of neuronal population activation dynamics, a small threshold means that neurons begin to activate with small input or stimulus, indicating high sensitivity to incoming signals. A steep slope signifies that activation increases sharply as input crosses the threshold. Accordingly in the model the excitatory activation thresholds are small and the slopes are steep. Hence, a singular limit replacing the excitatory activation function with all-or-nothing switches, eg. a Heaviside function, is appropriate. In this limit we identify the transition between alpha and delta oscillations as a discontinuity-induced grazing bifurcation. At the grazing the minimum of the pyramidal-cell output equals the threshold for switching off the excitatory interneuron population, leading to a collapse in excitatory feedback.

Novel reconfigurable intelligent surface (RIS) architectures, known as beyond diagonal RISs (BD-RISs), have been proposed to enhance reflection efficiency and expand RIS capabilities. However, their passive nature, non-diagonal reflection matrix, and the large number of coupled reflecting elements complicate the channel state information (CSI) estimation process. The challenge further escalates in scenarios with fast-varying channels. In this paper, we address this challenge by proposing novel joint channel estimation and prediction strategies with low overhead and high accuracy for two different RIS architectures in a BD-RIS-assisted multiple-input multiple-output system under correlated fast-fading environments with channel aging. The channel estimation procedure utilizes the Tucker2 decomposition with bilinear alternative least squares, which is exploited to decompose the cascade channels of the BD-RIS-assisted system into effective channels of reduced dimension. The channel prediction framework is based on a convolutional neural network combined with an autoregressive predictor. The estimated/predicted CSI is then utilized to optimize the RIS phase shifts aiming at the maximization of the downlink sum rate. Insightful simulation results demonstrate that our proposed approach is robust to channel aging, and exhibits a high estimation accuracy. Moreover, our scheme can deliver a high average downlink sum rate, outperforming other state-of-the-art channel estimation methods. The results also reveal a remarkable reduction in pilot overhead of up to 98\% compared to baseline schemes, all imposing low computational complexity., Comment: arXiv admin note: text overlap with arXiv:2406.07387

Our future society will be increasingly digitalised, hyper-connected and globally data driven. The sixth generation (6G) and beyond 6G wireless networks are expected to bridge the digital and physical worlds by providing wireless connectivity as a service to different vertical sectors, including industries, smart cities, eHealth and autonomous transportation. Such far reaching integration will render the society increasingly reliant on wireless networks. While this has the potential to greatly enhance our quality and ease of life, any disruption to these networks would also have significant impact with overreaching consequences. Disruptions can happen due to a variety of reasons, including planned outages, failures due to the nature of wireless propagation, natural disasters, and deliberate cybersecurity attacks. Hence, 6G and beyond 6G networks should not only provide near instant and virtually unlimited connectivity, but also be resilient against internal and external disruptions. This paper proposes a resilient-by-design framework towards this end. First, we provide an overview of the disruption landscape. Thereafter, we comprehensively outline the main features of the proposed concept. Finally, we detail the four key steps of the framework, namely predict, preempt, protect and progress. A simple but illustrative preliminary simulation result is also presented to highlight the potential advantages and efficiency of the proposed approach in addressing outages., Comment: Submitted to IEEE Communications Magazine

With the rapid growth of technology, emerging IT professionals increasingly require mentorship to secure positions in the field. Recognizing this need, ReCoded has enhanced the skill set of their tech Bootcamp graduates by introducing the community platform -- "ReCoded's Mentorship Platform". To improve the user experience of the volunteer mentors at ReCoded, this thesis investigates optimizing mentors' interactions with digital mentorship platforms and provides suggestions for enhancing these interactions. Multiple third-party collaborators have powered the mentorship platform at ReCoded. This thesis examines the platform powered by StellarUp as a case study. The insights obtained may inform the UX design of any subsequent mentorship platforms considered by ReCoded. This thesis adopted a user-centric approach to solving a UX question. The study identified challenges in the mentors' user journey and their needs by engaging with users via interviews, usability tests, and eye-tracking methods. Three principal issues emerged: platform navigation, the onboarding process, and the seamless integration of external tools. Solutions were derived from desk research for onboarding, card sorting techniques for navigation, and competitive analyses for tool integration. Throughout the research, feedback from 23 participants was gathered, ensuring a holistic understanding and actionable recommendations for developing a user-friendly and efficient mentorship platform., Comment: Master's thesis

Vocabulary adaptation, which integrates new vocabulary into pre-trained language models (LMs), enables expansion to new languages and mitigates token over-fragmentation. However, existing approaches are limited by their reliance on heuristic or external embeddings. We propose VocADT, a novel method for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model's weights fixed. VocADT offers a flexible and scalable solution without requiring external resources or language constraints. Across 11 languages-with various scripts, resource availability, and fragmentation-we demonstrate that VocADT outperforms the original Mistral model and other baselines across various multilingual tasks. We find that Latin-script languages and highly fragmented languages benefit the most from vocabulary adaptation. We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adaptation is still beneficial after fine-tuning and that VocADT is the most effective method.

We evaluate the performance of the LoRaWAN Long-Range Frequency Hopping Spread Spectrum (LR-FHSS) technique using a device-level probabilistic strategy for code rate and header replica allocation. Specifically, we investigate the effects of different header replica and code rate allocations at each end-device, guided by a probability distribution provided by the network server. As a benchmark, we compare the proposed strategy with the standardized LR-FHSS data rates DR8 and DR9. Our numerical results demonstrate that the proposed strategy consistently outperforms the DR8 and DR9 standard data rates across all considered scenarios. Notably, our findings reveal that the optimal distribution rarely includes data rate DR9, while data rate DR8 significantly contributes to the goodput and energy efficiency optimizations.

Large language models (LLMs) have achieved remarkable success across various NLP tasks, yet their focus has predominantly been on English due to English-centric pre-training and limited multilingual data. While some multilingual LLMs claim to support for hundreds of languages, models often fail to provide high-quality response for mid- and low-resource languages, leading to imbalanced performance heavily skewed in favor of high-resource languages like English and Chinese. In this paper, we prioritize quality over scaling number of languages, with a focus on multilingual machine translation task, and introduce X-ALMA, a model designed with a commitment to ensuring top-tier performance across 50 diverse languages, regardless of their resource levels. X-ALMA surpasses state-of-the-art open-source multilingual LLMs, such as Aya-101 and Aya-23, in every single translation direction on the FLORES and WMT'23 test datasets according to COMET-22. This is achieved by plug-and-play language-specific module architecture to prevent language conflicts during training and a carefully designed training regimen with novel optimization methods to maximize the translation performance. At the final stage of training regimen, our proposed Adaptive Rejection Preference Optimization (ARPO) surpasses existing preference optimization methods in translation tasks.

Crack detection, particularly from pavement images, presents a formidable challenge in the domain of computer vision due to several inherent complexities such as intensity inhomogeneity, intricate topologies, low contrast, and noisy backgrounds. Automated crack detection is crucial for maintaining the structural integrity of essential infrastructures, including buildings, pavements, and bridges. Existing lightweight methods often face challenges including computational inefficiency, complex crack patterns, and difficult backgrounds, leading to inaccurate detection and impracticality for real-world applications. To address these limitations, we propose EfficientCrackNet, a lightweight hybrid model combining Convolutional Neural Networks (CNNs) and transformers for precise crack segmentation. EfficientCrackNet integrates depthwise separable convolutions (DSC) layers and MobileViT block to capture both global and local features. The model employs an Edge Extraction Method (EEM) and for efficient crack edge detection without pretraining, and Ultra-Lightweight Subspace Attention Module (ULSAM) to enhance feature extraction. Extensive experiments on three benchmark datasets Crack500, DeepCrack, and GAPs384 demonstrate that EfficientCrackNet achieves superior performance compared to existing lightweight models, while requiring only 0.26M parameters, and 0.483 FLOPs (G). The proposed model offers an optimal balance between accuracy and computational efficiency, outperforming state-of-the-art lightweight models, and providing a robust and adaptable solution for real-world crack segmentation.

Selecting an automatic metric that best emulates human annotators is often non-trivial, because there is no clear definition of "best emulates." A meta-metric is required to compare the human judgments to the automatic metric scores, and metric rankings depend on the choice of meta-metric. We propose Soft Pairwise Accuracy (SPA), a new meta-metric that builds on Pairwise Accuracy (PA) but incorporates the statistical significance of both the human judgments and the metric scores. We show that SPA is more stable than PA with respect to changes in the number of systems/segments used for evaluation. We also show that PA can only assign a small set of distinct output values to metrics, and this results in many metrics being artificially assigned the exact same PA score. We demonstrate that SPA fixes this issue. Finally, we show that SPA is more discriminative than PA, producing more statistically significant comparisons between metrics. SPA was selected as the official system-level metric for the 2024 WMT Metrics Shared Task., Comment: Accepted at WMT 2024

In this paper, we present an improved version of the digital signature scheme proposed by Sharafi and Daghigh based on Module-LWE and Module-SIS problems. Our proposed signature scheme has a notably higher security level and smaller decoding failure probability, than the ones in the Sharaf-Daghigh scheme, at the expense of enlarging the module of the underlying basic ring.

In this paper, we intend to investigate the dynamics of the Circular Restricted Three-Body Problem. Here we assumed the primaries as the source of radiation and have variable mass. The gravitational perturbation from disk-like structure are also considered in this study. There exist five equilibrium points in this system. By considering the combined effect from disk-like structure and the mass transfer, we found that the classical collinear equilibrium points depart from x-axis. Meanwhile, this combined effect also breaks the symmetry of tringular equlibrium point positions. We noted that the quasi-equilibrium points are unstable whereas the triangular equilibrium points are stable if the mass ratio $\mu$ smaller than critical mass $\mu_c$. It shows that the stability of triangular equilibrium points depends on time., Comment: 15 pages, 5 figures, 2 tables. Accepted for publication in Romanian Astronomical Journal (RoAJ)

The high power consumption and latency-sensitive deployments of large language models (LLMs) have motivated efficiency techniques like quantization and sparsity. Contextual sparsity, where the sparsity pattern is input-dependent, is crucial in LLMs because the permanent removal of attention heads or neurons from LLMs can significantly degrade accuracy. Prior work has attempted to model contextual sparsity using neural networks trained to predict activation magnitudes, which can be used to dynamically prune structures with low predicted activation magnitude. In this paper, we look beyond magnitude-based pruning criteria to assess attention head and neuron importance in LLMs. We develop a novel predictor called ShadowLLM, which can shadow the LLM behavior and enforce better sparsity patterns, resulting in over 15% improvement in end-to-end accuracy compared to prior methods. In addition, ShadowLLM achieves up to a 20% speed-up over the state-of-the-art DejaVu framework. These enhancements are validated on Llama-2 and OPT models with up to 30 billion parameters. Our code is available at \href{https://github.com/abdelfattah-lab/shadow_llm/}{ShadowLLM}., Comment: Accepted to EMNLP 2024 (Main, Long Paper)

Reconfigurable intelligent surfaces (RISs) have emerged as a promising technology to enhance the performance of sixth-generation (6G) and beyond communication systems. The passive nature of RISs and their large number of reflecting elements pose challenges to the channel estimation process. The associated complexity further escalates when the channel coefficients are fast-varying as in scenarios with user mobility. In this paper, we propose an extended channel estimation framework for RIS-assisted multiple-input multiple-output (MIMO) systems based on a convolutional neural network (CNN) integrated with an autoregressive (AR) predictor. The implemented framework is designed for identifying the aging pattern and predicting enhanced estimates of the wireless channels in correlated fast-fading environments. Insightful simulation results demonstrate that our proposed CNN-AR approach is robust to channel aging, exhibiting a high-precision estimation accuracy. The results also show that our approach can achieve high spectral efficiency and low pilot overhead compared to traditional methods.

Recent advancements in Artificial Intelligence (AI) and machine learning have demonstrated transformative capabilities across diverse domains. This progress extends to the field of patent analysis and innovation, where AI-based tools present opportunities to streamline and enhance important tasks in the patent cycle such as classification, retrieval, and valuation prediction. This not only accelerates the efficiency of patent researchers and applicants but also opens new avenues for technological innovation and discovery. Our survey provides a comprehensive summary of recent AI tools in patent analysis from more than 40 papers from 26 venues between 2017 and 2023. Unlike existing surveys, we include methods that work for patent image and text data. Furthermore, we introduce a novel taxonomy for the categorization based on the tasks in the patent life cycle as well as the specifics of the AI methods. This interdisciplinary survey aims to serve as a resource for researchers and practitioners who are working at the intersection of AI and patent analysis as well as the patent offices that are aiming to build efficient patent systems.