Search

Large Language Models (LLMs) are pre-trained on large-scale corpora and excel in numerous general natural language processing (NLP) tasks, such as question answering (QA). Despite their advanced language capabilities, when it comes to domain-specific and knowledge-intensive tasks, LLMs suffer from hallucinations, knowledge cut-offs, and lack of knowledge attributions. Additionally, fine tuning LLMs' intrinsic knowledge to highly specific domains is an expensive and time consuming process. The retrieval-augmented generation (RAG) process has recently emerged as a method capable of optimization of LLM responses, by referencing them to a predetermined ontology. It was shown that using a Knowledge Graph (KG) ontology for RAG improves the QA accuracy, by taking into account relevant sub-graphs that preserve the information in a structured manner. In this paper, we introduce SMART-SLIC, a highly domain-specific LLM framework, that integrates RAG with KG and a vector store (VS) that store factual domain specific information. Importantly, to avoid hallucinations in the KG, we build these highly domain-specific KGs and VSs without the use of LLMs, but via NLP, data mining, and nonnegative tensor factorization with automatic model selection. Pairing our RAG with a domain-specific: (i) KG (containing structured information), and (ii) VS (containing unstructured information) enables the development of domain-specific chat-bots that attribute the source of information, mitigate hallucinations, lessen the need for fine-tuning, and excel in highly domain-specific question answering tasks. We pair SMART-SLIC with chain-of-thought prompting agents. The framework is designed to be generalizable to adapt to any specific or specialized domain. In this paper, we demonstrate the question answering capabilities of our framework on a corpus of scientific publications on malware analysis and anomaly detection., Comment: 9 pages 7 figures, 1 table, 1 cypher code Accepted to ICMLA 2024

As Machine Learning (ML) applications rapidly grow, concerns about adversarial attacks compromising their reliability have gained significant attention. One unsupervised ML method known for its resilience to such attacks is Non-negative Matrix Factorization (NMF), an algorithm that decomposes input data into lower-dimensional latent features. However, the introduction of powerful computational tools such as Pytorch enables the computation of gradients of the latent features with respect to the original data, raising concerns about NMF's reliability. Interestingly, naively deriving the adversarial loss for NMF as in the case of ML would result in the reconstruction loss, which can be shown theoretically to be an ineffective attacking objective. In this work, we introduce a novel class of attacks in NMF termed Latent Feature Attacks (LaFA), which aim to manipulate the latent features produced by the NMF process. Our method utilizes the Feature Error (FE) loss directly on the latent features. By employing FE loss, we generate perturbations in the original data that significantly affect the extracted latent features, revealing vulnerabilities akin to those found in other ML techniques. To handle large peak-memory overhead from gradient back-propagation in FE attacks, we develop a method based on implicit differentiation which enables their scaling to larger datasets. We validate NMF vulnerabilities and FE attacks effectiveness through extensive experiments on synthetic and real-world data., Comment: LA-UR-24-26951

In this paper, we introduce a high-order tensor-train (TT) finite volume method for the Shallow Water Equations (SWEs). We present the implementation of the $3^{rd}$ order Upwind and the $5^{th}$ order Upwind and WENO reconstruction schemes in the TT format. It is shown in detail that the linear upwind schemes can be implemented by directly manipulating the TT cores while the WENO scheme requires the use of TT cross interpolation for the nonlinear reconstruction. In the development of numerical fluxes, we directly compute the flux for the linear SWEs without using TT rounding or cross interpolation. For the nonlinear SWEs where the TT reciprocal of the shallow water layer thickness is needed for fluxes, we develop an approximation algorithm using Taylor series to compute the TT reciprocal. The performance of the TT finite volume solver with linear and nonlinear reconstruction options is investigated under a physically relevant set of validation problems. In all test cases, the TT finite volume method maintains the formal high-order accuracy of the corresponding traditional finite volume method. In terms of speed, the TT solver achieves up to 124x acceleration of the traditional full-tensor scheme.

Topic modeling is a technique for organizing and extracting themes from large collections of unstructured text. Non-negative matrix factorization (NMF) is a common unsupervised approach that decomposes a term frequency-inverse document frequency (TF-IDF) matrix to uncover latent topics and segment the dataset accordingly. While useful for highlighting patterns and clustering documents, NMF does not provide explicit topic labels, necessitating subject matter experts (SMEs) to assign labels manually. We present a methodology for automating topic labeling in documents clustered via NMF with automatic model determination (NMFk). By leveraging the output of NMFk and employing prompt engineering, we utilize large language models (LLMs) to generate accurate topic labels. Our case study on over 34,000 scientific abstracts on Knowledge Graphs demonstrates the effectiveness of our method in enhancing knowledge management and document organization., Comment: Accepted to ACM Symposium on Document Engineering 2024 (DocEng 24), 2024

Spectral methods provide highly accurate numerical solutions for partial differential equations, exhibiting exponential convergence with the number of spectral nodes. Traditionally, in addressing time-dependent nonlinear problems, attention has been on low-order finite difference schemes for time discretization and spectral element schemes for spatial variables. However, our recent developments have resulted in the application of spectral methods to both space and time variables, preserving spectral convergence in both domains. Leveraging Tensor Train techniques, our approach tackles the curse of dimensionality inherent in space-time methods. Here, we extend this methodology to the nonlinear time-dependent convection-diffusion equation. Our discretization scheme exhibits a low-rank structure, facilitating translation to tensor-train (TT) format. Nevertheless, controlling the TT-rank across Newton's iterations, needed to deal with the nonlinearity, poses a challenge, leading us to devise the "Step Truncation TT-Newton" method. We demonstrate the exponential convergence of our methods through various benchmark examples. Importantly, our scheme offers significantly reduced memory requirement compared to the full-grid scheme.

In this study, we introduce a tensor-train (TT) finite difference WENO method for solving compressible Euler equations. In a step-by-step manner, the tensorization of the governing equations is demonstrated. We also introduce \emph{LF-cross} and \emph{WENO-cross} methods to compute numerical fluxes and the WENO reconstruction using the cross interpolation technique. A tensor-train approach is developed for boundary condition types commonly encountered in Computational Fluid Dynamics (CFD). The performance of the proposed WENO-TT solver is investigated in a rich set of numerical experiments. We demonstrate that the WENO-TT method achieves the theoretical $\text{5}^{\text{th}}$-order accuracy of the classical WENO scheme in smooth problems while successfully capturing complicated shock structures. In an effort to avoid the growth of TT ranks, we propose a dynamic method to estimate the TT approximation error that governs the ranks and overall truncation error of the WENO-TT scheme. Finally, we show that the traditional WENO scheme can be accelerated up to 1000 times in the TT format, and the memory requirements can be significantly decreased for low-rank problems, demonstrating the potential of tensor-train approach for future CFD application. This paper is the first study that develops a finite difference WENO scheme using the tensor-train approach for compressible flows. It is also the first comprehensive work that provides a detailed perspective into the relationship between rank, truncation error, and the TT approximation error for compressible WENO solvers.

Much of human knowledge in cybersecurity is encapsulated within the ever-growing volume of scientific papers. As this textual data continues to expand, the importance of document organization methods becomes increasingly crucial for extracting actionable insights hidden within large text datasets. Knowledge Graphs (KGs) serve as a means to store factual information in a structured manner, providing explicit, interpretable knowledge that includes domain-specific information from the cybersecurity scientific literature. One of the challenges in constructing a KG from scientific literature is the extraction of ontology from unstructured text. In this paper, we address this topic and introduce a method for building a multi-modal KG by extracting structured ontology from scientific papers. We demonstrate this concept in the cybersecurity domain. One modality of the KG represents observable information from the papers, such as the categories in which they were published or the authors. The second modality uncovers latent (hidden) patterns of text extracted through hierarchical and semantic non-negative matrix factorization (NMF), such as named entities, topics or clusters, and keywords. We illustrate this concept by consolidating more than two million scientific papers uploaded to arXiv into the cyber-domain, using hierarchical and semantic NMF, and by building a cyber-domain-specific KG., Comment: Accepted at IEEE ISDFS

National security is threatened by malware, which remains one of the most dangerous and costly cyber threats. As of last year, researchers reported 1.3 billion known malware specimens, motivating the use of data-driven machine learning (ML) methods for analysis. However, shortcomings in existing ML approaches hinder their mass adoption. These challenges include detection of novel malware and the ability to perform malware classification in the face of class imbalance: a situation where malware families are not equally represented in the data. Our work addresses these shortcomings with MalwareDNA: an advanced dimensionality reduction and feature extraction framework. We demonstrate stable task performance under class imbalance for the following tasks: malware family classification and novel malware detection with a trade-off in increased abstention or reject-option rate., Comment: Accepted at IEEE 12th International Symposium on Digital Forensics and Security (ISDFS), 2024. arXiv admin note: text overlap with arXiv:2309.01350

Emerging tensor network techniques for solutions of Partial Differential Equations (PDEs), known for their ability to break the curse of dimensionality, deliver new mathematical methods for ultrafast numerical solutions of high-dimensional problems. Here, we introduce a Tensor Train (TT) Chebyshev spectral collocation method, in both space and time, for solution of the time dependent convection-diffusion-reaction (CDR) equation with inhomogeneous boundary conditions, in Cartesian geometry. Previous methods for numerical solution of time dependent PDEs often use finite difference for time, and a spectral scheme for the spatial dimensions, which leads to slow linear convergence. Spectral collocation space-time methods show exponential convergence, however, for realistic problems they need to solve large four-dimensional systems. We overcome this difficulty by using a TT approach as its complexity only grows linearly with the number of dimensions. We show that our TT space-time Chebyshev spectral collocation method converges exponentially, when the solution of the CDR is smooth, and demonstrate that it leads to very high compression of linear operators from terabytes to kilobytes in TT-format, and tens of thousands times speedup when compared to full grid space-time spectral method. These advantages allow us to obtain the solutions at much higher resolutions.

Highly specific datasets of scientific literature are important for both research and education. However, it is difficult to build such datasets at scale. A common approach is to build these datasets reductively by applying topic modeling on an established corpus and selecting specific topics. A more robust but time-consuming approach is to build the dataset constructively in which a subject matter expert (SME) handpicks documents. This method does not scale and is prone to error as the dataset grows. Here we showcase a new tool, based on machine learning, for constructively generating targeted datasets of scientific literature. Given a small initial "core" corpus of papers, we build a citation network of documents. At each step of the citation network, we generate text embeddings and visualize the embeddings through dimensionality reduction. Papers are kept in the dataset if they are "similar" to the core or are otherwise pruned through human-in-the-loop selection. Additional insight into the papers is gained through sub-topic modeling using SeNMFk. We demonstrate our new tool for literature review by applying it to two different fields in machine learning., Comment: Accepted at 2023 IEEE ICMLA conference

Tensor network techniques, known for their low-rank approximation ability that breaks the curse of dimensionality, are emerging as a foundation of new mathematical methods for ultra-fast numerical solutions of high-dimensional Partial Differential Equations (PDEs). Here, we present a mixed Tensor Train (TT)/Quantized Tensor Train (QTT) approach for the numerical solution of time-independent Boltzmann Neutron Transport equations (BNTEs) in Cartesian geometry. Discretizing a realistic three-dimensional (3D) BNTE by (i) diamond differencing, (ii) multigroup-in-energy, and (iii) discrete ordinate collocation leads to huge generalized eigenvalue problems that generally require a matrix-free approach and large computer clusters. Starting from this discretization, we construct a TT representation of the PDE fields and discrete operators, followed by a QTT representation of the TT cores and solving the tensorized generalized eigenvalue problem in a fixed-point scheme with tensor network optimization techniques. We validate our approach by applying it to two realistic examples of 3D neutron transport problems, currently solved by the PARallel TIme-dependent SN (PARTISN) solver. We demonstrate that our TT/QTT method, executed on a standard desktop computer, leads to a yottabyte compression of the memory storage, and more than 7500 times speedup with a discrepancy of less than 1e-5 when compared to the PARTISN solution., Comment: 38 pages, 9 figures

Malware is one of the most dangerous and costly cyber threats to national security and a crucial factor in modern cyber-space. However, the adoption of machine learning (ML) based solutions against malware threats has been relatively slow. Shortcomings in the existing ML approaches are likely contributing to this problem. The majority of current ML approaches ignore real-world challenges such as the detection of novel malware. In addition, proposed ML approaches are often designed either for malware/benign-ware classification or malware family classification. Here we introduce and showcase preliminary capabilities of a new method that can perform precise identification of novel malware families, while also unifying the capability for malware/benign-ware classification and malware family classification into a single framework., Comment: Accepted at IEEE ISI 2023

As machine learning techniques become increasingly prevalent in data analysis, the threat of adversarial attacks has surged, necessitating robust defense mechanisms. Among these defenses, methods exploiting low-rank approximations for input data preprocessing and neural network (NN) parameter factorization have shown potential. Our work advances this field further by integrating the tensorization of input data with low-rank decomposition and tensorization of NN parameters to enhance adversarial defense. The proposed approach demonstrates significant defense capabilities, maintaining robust accuracy even when subjected to the strongest known auto-attacks. Evaluations against leading-edge robust performance benchmarks reveal that our results not only hold their ground against the best defensive methods available but also exceed all current defense strategies that rely on tensor factorizations. This study underscores the potential of integrating tensorization and low-rank decomposition as a robust defense against adversarial attacks in machine learning., Comment: Accepted at 2023 ICMLA Conference

Monitoring of industrial processes is a critical capability in industry and in government to ensure reliability of production cycles, quick emergency response, and national security. Process monitoring allows users to gauge the progress of an organization in an industrial process or predict the degradation or aging of machine parts in processes taking place at a remote location. Similar to many data science applications, we usually only have access to limited raw data, such as satellite imagery, short video clips, event logs, and signatures captured by a small set of sensors. To combat data scarcity, we leverage the knowledge of Subject Matter Experts (SMEs) who are familiar with the actions of interest. SMEs provide expert knowledge of the essential activities required for task completion and the resources necessary to carry out each of these activities. Various process mining techniques have been developed for this type of analysis; typically such approaches combine theoretical process models built based on domain expert insights with ad-hoc integration of available pieces of raw data. Here, we introduce a novel mathematically sound method that integrates theoretical process models (as proposed by SMEs) with interrelated minimal Hidden Markov Models (HMM), built via nonnegative tensor factorization. Our method consolidates: (a) theoretical process models, (b) HMMs, (c) coupled nonnegative matrix-tensor factorizations, and (d) custom model selection. To demonstrate our methodology and its abilities, we apply it on simple synthetic and real world process models., Comment: 19 pages, 8 figures, TOMACS

Nonlinear isolated and coupled oscillators are extensively studied as prototypical nonlinear dynamics models. Much attention has been devoted to oscillator synchronization or the lack thereof. Here, we study the synchronization and stability of coupled driven-damped Helmholtz-Duffing oscillators in bi-stability regimes. We find that despite the fact that the system parameters and the driving force are identical, the stability of the two states to spatially non-uniform perturbations is very different. Moreover, the final stable states, resulting from these spatial perturbations, are not solely dictated by the wavelength of the perturbing mode and take different spatial configurations in terms of the coupled oscillator phases.

As the amount of text data continues to grow, topic modeling is serving an important role in understanding the content hidden by the overwhelming quantity of documents. One popular topic modeling approach is non-negative matrix factorization (NMF), an unsupervised machine learning (ML) method. Recently, Semantic NMF with automatic model selection (SeNMFk) has been proposed as a modification to NMF. In addition to heuristically estimating the number of topics, SeNMFk also incorporates the semantic structure of the text. This is performed by jointly factorizing the term frequency-inverse document frequency (TF-IDF) matrix with the co-occurrence/word-context matrix, the values of which represent the number of times two words co-occur in a predetermined window of the text. In this paper, we introduce a novel distributed method, SeNMFk-SPLIT, for semantic topic extraction suitable for large corpora. Contrary to SeNMFk, our method enables the joint factorization of large documents by decomposing the word-context and term-document matrices separately. We demonstrate the capability of SeNMFk-SPLIT by applying it to the entire artificial intelligence (AI) and ML scientific literature uploaded on arXiv., Comment: Accepted at ACM Symposium on Document Engineering 2022 (DocEng 22), 2022

Traditional member-based two-step design approaches included in current structural codes for steel structures, as well as more recent system-based direct-design alternatives, require building rigorous structural reliability frameworks for the calibration of partial coefficients (resistance factors) to achieve specified target reliability requirements. Key design parameters affecting the strength of structures and their random variations are generally modelled by nominal or characteristic values in design standards, which are combined with partial coefficients that need to be calibrated from measurements on real samples. While the statistical characterization of material and geometric properties of structural steels has been consolidated over the last decades, information about the characterization of structural stainless steels is virtually non-existent due to the limited pool of available data. Thus, this paper presents the basic ingredient for developing reliability frameworks for stainless steel structures and components by statistically characterizing the main random parameters affecting their strength through a comprehensive database collected from the literature. Based on the collected data, appropriate probabilistic models are proposed for geometric properties, material properties, imperfections and residual stresses of different stainless steel alloys and cross-section or product types. The data is equally applicable to member-based reliability analyses as described in existing codes and system-based analyses targeted at the direct-design of stainless steel structures by advanced analysis.

Fractional Brownian motion (fBm) is a ubiquitous diffusion process in which the memory effects of the stochastic transport result in the mean squared particle displacement following a power law, $\langle {\Delta r}^2 \rangle \sim t^{\alpha}$, where the diffusion exponent $\alpha$ characterizes whether the transport is subdiffusive, ($\alpha<1$), diffusive ($\alpha = 1$), or superdiffusive, ($\alpha >1$). Due to the abundance of fBm processes in nature, significant efforts have been devoted to the identification and characterization of fBm sources in various phenomena. In practice, the identification of the fBm sources often relies on solving a complex and ill-posed inverse problem based on limited observed data. In the general case, the detected signals are formed by an unknown number of release sources, located at different locations and with different strengths, that act simultaneously. This means that the observed data is composed of mixtures of releases from an unknown number of sources, which makes the traditional inverse modeling approaches unreliable. Here, we report an unsupervised learning method, based on Nonnegative Matrix Factorization, that enables the identification of the unknown number of release sources as well the anomalous diffusion characteristics based on limited observed data and the general form of the corresponding fBm Green's function. We show that our method performs accurately for different types of sources and configurations with a predetermined number of sources with specific characteristics and introduced noise., Comment: published in physical review research

The dynamics of active particles is of interest at many levels and is the focus of theoretical and experimental research. There have been many attempts to describe the dynamics of particles affected by random active forces in terms of an effective temperature. This kind of description is tempting due to the similarities (or lack thereof) with systems in or near thermal equilibrium. However, the generality and validity of the effective temperature is not yet fully understood. Here, we studied the dynamics of trapped particles subjected to both thermal and active forces. The particles were not overdamped. Expressions for the effective temperature due to the potential and kinetic energies were derived, and they differ from each other. A third possible effective temperature can be derived from the escape time of the particle from the trap, using a Kramers-like expression for the mean escape time. We found that over a large fraction of the parameter space, the potential energy effective temperature is in agreement with the escape temperature, while the kinetic effective temperature only agrees with the former two in the overdamped limit. Moreover, we show that the specific implementation of the random active force, and not only its first two moments and the two point auto-correlation function, affects the escape time distribution.