Search

Phishing attacks remain a persistent threat to online security, demanding robust detection methods. This study investigates the use of machine learning to identify phishing URLs, emphasizing the crucial role of feature selection and model interpretability for improved performance. Employing Recursive Feature Elimination, the research pinpointed key features like "length_url," "time_domain_activation" and "Page_rank" as strong indicators of phishing attempts. The study evaluated various algorithms, including CatBoost, XGBoost, and Explainable Boosting Machine, assessing their robustness and scalability. XGBoost emerged as highly efficient in terms of runtime, making it well-suited for large datasets. CatBoost, on the other hand, demonstrated resilience by maintaining high accuracy even with reduced features. To enhance transparency and trustworthiness, Explainable AI techniques, such as SHAP, were employed to provide insights into feature importance. The study's findings highlight that effective feature selection and model interpretability can significantly bolster phishing detection systems, paving the way for more efficient and adaptable defenses against evolving cyber threats

We present MetaMetrics-MT, an innovative metric designed to evaluate machine translation (MT) tasks by aligning closely with human preferences through Bayesian optimization with Gaussian Processes. MetaMetrics-MT enhances existing MT metrics by optimizing their correlation with human judgments. Our experiments on the WMT24 metric shared task dataset demonstrate that MetaMetrics-MT outperforms all existing baselines, setting a new benchmark for state-of-the-art performance in the reference-based setting. Furthermore, it achieves comparable results to leading metrics in the reference-free setting, offering greater efficiency., Comment: Preprint

Code-switching, the phenomenon of alternating between two or more languages in a single conversation, presents unique challenges for Natural Language Processing (NLP). Most existing research focuses on either syntactic constraints or neural generation, with few efforts to integrate linguistic theory with large language models (LLMs) for generating natural code-switched text. In this paper, we introduce EZSwitch, a novel framework that combines Equivalence Constraint Theory (ECT) with LLMs to produce linguistically valid and fluent code-switched text. We evaluate our method using both human judgments and automatic metrics, demonstrating a significant improvement in the quality of generated code-switching sentences compared to baseline LLMs. To address the lack of suitable evaluation metrics, we conduct a comprehensive correlation study of various automatic metrics against human scores, revealing that current metrics often fail to capture the nuanced fluency of code-switched text. Additionally, we create CSPref, a human preference dataset based on human ratings and analyze model performance across ``hard`` and ``easy`` examples. Our findings indicate that incorporating linguistic constraints into LLMs leads to more robust and human-aligned generation, paving the way for scalable code-switching text generation across diverse language pairs.

In recent years, personalized diffusion-based text-to-image generative tasks have been a hot topic in computer vision studies. A robust diffusion model is determined by its ability to perform near-perfect reconstruction of certain product outcomes given few related input samples. Unfortunately, the current prominent diffusion-based finetuning technique falls short in maintaining the foreground object consistency while being constrained to produce diverse backgrounds in the image outcome. In the worst scenario, the overfitting issue may occur, meaning that the foreground object is less controllable due to the condition above, for example, the input prompt information is transferred ambiguously to both foreground and background regions, instead of the supposed background region only. To tackle the issues above, we proposed Hypnos, a highly precise foreground-focused diffusion finetuning technique. On the image level, this strategy works best for inanimate object generation tasks, and to do so, Hypnos implements two main approaches, namely: (i) a content-centric prompting strategy and (ii) the utilization of our additional foreground-focused discriminative module. The utilized module is connected with the diffusion model and finetuned with our proposed set of supervision mechanism. Combining the strategies above yielded to the foreground-background disentanglement capability of the diffusion model. Our experimental results showed that the proposed strategy gave a more robust performance and visually pleasing results compared to the former technique. For better elaborations, we also provided extensive studies to assess the fruitful outcomes above, which reveal how personalization behaves in regard to several training conditions., Comment: 26 pages, 12 figures, to appear on the Rich Media with Generative AI workshop in conjunction with Asian Conference on Computer Vision (ACCV) 2024

Large-scale molecular representation methods have revolutionized applications in material science, such as drug discovery, chemical modeling, and material design. With the rise of transformers, models now learn representations directly from molecular structures. In this study, we develop an encoder-decoder model based on BART that is capable of leaning molecular representations and generate new molecules. Trained on SELFIES, a robust molecular string representation, our model outperforms existing baselines in downstream tasks, demonstrating its potential in efficient and effective molecular data analysis and manipulation., Comment: NeurIPS AI4Mat 2024

Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English and in underrepresented cultural contexts. To evaluate their understanding of such knowledge, we introduce WorldCuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding. This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points, making it the largest multicultural VQA benchmark to date. It includes tasks for identifying dish names and their origins. We provide evaluation datasets in two sizes (12k and 60k instances) alongside a training dataset (1 million instances). Our findings show that while VLMs perform better with correct location context, they struggle with adversarial contexts and predicting specific regional cuisines and languages. To support future research, we release a knowledge base with annotated food entries and images along with the VQA data., Comment: Preprint

Machine learning (ML) course for undergraduates face challenges in assessing student learning and providing practical exposure. Group project-based learning, an increasingly popular form of experiential learning in CS education, encounters certain limitation in participation and non-participation from a few students. Studies also suggest that students find longer programming assignments and project-based assessments distracting and struggle to maintain focus when they coincide with other courses. To tackle these issues, we introduced FlashHack: a monitored, incremental, in-classroom micro Hackathon that combines project-based learning with Hackathon elements. Engaging 229 third year CS undergraduate students in teams of four, FlashHack prompted them to tackle predefined challenges using machine learning techniques within a set timeframe. Assessment criteria emphasized machine learning application, problem-solving, collaboration, and creativity. Our results indicate high student engagement and satisfaction, alongside simplified assessment processes for instructors. This experience report outlines the Hackathon design and implementation, highlights successes and areas for improvement making it feasible for replication by interested computing educators.

Recently, numerous preference optimization algorithms have been introduced as extensions to the Direct Preference Optimization (DPO) family. While these methods have successfully aligned models with human preferences, there is a lack of understanding regarding the contributions of their additional components. Moreover, fair and consistent comparisons are scarce, making it difficult to discern which components genuinely enhance downstream performance. In this work, we propose RainbowPO, a unified framework that demystifies the effectiveness of existing DPO methods by categorizing their key components into seven broad directions. We integrate these components into a single cohesive objective, enhancing the performance of each individual element. Through extensive experiments, we demonstrate that RainbowPO outperforms existing DPO variants. Additionally, we provide insights to guide researchers in developing new DPO methods and assist practitioners in their implementations.

Understanding the quality of a performance evaluation metric is crucial for ensuring that model outputs align with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce MetaMetrics, a calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. MetaMetrics optimizes the combination of existing metrics to enhance their alignment with human preferences. Our metric demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi-domain scenarios. MetaMetrics aligns closely with human preferences and is highly extendable and easily integrable into any application. This makes MetaMetrics a powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts., Comment: Preprint

Natural Language Explanation (NLE) aims to elucidate the decision-making process by providing detailed, human-friendly explanations in natural language. It helps demystify the decision-making processes of large vision-language models (LVLMs) through the use of language models. While existing methods for creating a Vision Question-Answering with Natural Language Explanation (VQA-NLE) datasets can provide explanations, they heavily rely on human annotations that are time-consuming and costly. In this study, we propose a novel approach that leverages LVLMs to efficiently generate high-quality synthetic VQA-NLE datasets. By evaluating our synthetic data, we showcase how advanced prompting techniques can lead to the production of high-quality VQA-NLE data. Our findings indicate that this proposed method achieves up to 20x faster than human annotation, with only a minimal decrease in qualitative metrics, achieving robust quality that is nearly equivalent to human-annotated data. Furthermore, we show that incorporating visual prompts significantly enhances the relevance of text generation. Our study paves the way for a more efficient and robust automated generation of multi-modal NLE data, offering a promising solution to the problem., Comment: Preprint

Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth exploration of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area., Comment: Survey paper

We perform a study to obtain the initial spin or the nondimensional Kerr parameter $a_{*}$ of primordial black holes (PBHs) created during the radiation dominated phase of the universe from not only nearly monochromatic but also broad curvature power spectra. Motivated by inflation and first-order phase transitions, we consider a power law shape for the curvature perturbation. Although we can naturally neglect the contribution from the length scales smaller than the scale of interest, that from the larger scales may potentially be significant for a broad power spectrum, for which the spin is sensitive to the width of the power spectrum. So, we introduce a width parameter $r_{k}$, the ratio of the largest scale to the length of interest. We find that the root mean square of $a_{*}$ is largest for PBHs created from locally nearly scale invariant curvature power spectra with $r_{k} \sim 3.5$. The upper limit is $\sim 1\times 10^{-4}$ for $M=10^{17}-10^{23}$ g, $\sim 1.7\times 10^{-4}$ for $M=1-100 M_{\odot}$ and $\sim 2.5\times 10^{-3}$ even for an incredibly large mass of $M=10^{50}$ g., Comment: 17 pages, 5 figures

This work addresses the problem of finding a consensus point in the state space ($\mathbb{R}^2$) for a multi-agent system that is comprised of $N$ identical double integrator agents. It is assumed that each agent operates under constrained control input (i.e., $|u_i(t)| \leq 1$ $\forall i = 1, \hdots N$). Further, a fixed fuel budget is also assumed i.e., the total amount of cumulative input that can be expended is limited by $\int_0^{t_f}|u(t)|dt \le \beta$. First, the attainable set $\mathcal{A}(t,x_0,\beta)$ at time $t$, which is the set of all states that an agent can attain starting from initial conditions $x_0$ under the fuel budget constraints at time $t$ is computed for every agent. This attainable set is a convex set for all $t\ge0$. Then the minimum time to consensus is the minimum time $\bar{t}$ at which attainable sets of all agents intersect, and the consensus point is the point of intersection. A closed-form expression for the minimum time consensus point is provided for the case of three agents. Then, using Helly's theorem, the intersection will be non-empty at a time when all the $N \choose 3$ triplets of agents have non-empty intersection. The computation of minimum time consensus for all $N \choose 3$ triplets is performed independently and can be distributed among all the $N$ agents. Finally, the overall minimum time to consensus is given by the triplet that has the highest minimum time to consensus. Further, the intersection of all the attainable sets of this triplet gives the minimum time consensus point for all $N$ agents.

Development of efficient and high-performing electrolytes is crucial for advancing energy storage technologies, particularly in batteries. Predicting the performance of battery electrolytes rely on complex interactions between the individual constituents. Consequently, a strategy that adeptly captures these relationships and forms a robust representation of the formulation is essential for integrating with machine learning models to predict properties accurately. In this paper, we introduce a novel approach leveraging a transformer-based molecular representation model to effectively and efficiently capture the representation of electrolyte formulations. The performance of the proposed approach is evaluated on two battery property prediction tasks and the results show superior performance compared to the state-of-the-art methods., Comment: Accepted in ML4LMS Workshop at ICML 2024

We propose a novel method to study the ultra-light scalars, where compact rotating objects undergo the phenomenon of superradiance to create gravitational waves and neutrino flux signals. The neutrino flux results from the 'right' coupling between the ultra-light scalars and the neutrinos. We study the intertwining of gravitational waves and neutrino flux signals produced from a single source and elaborate if and when the signals can be detected in existing and upcoming experiments in a direct manner. We also discuss an indirect way to test it by means of cosmic neutrino background which can be detected by upcoming PTOLEMY experiment., Comment: 16 pages, 5 figures, 1 table

We conduct an analysis of a $U(1)_{B-L}$ extended inert doublet model and obtained the parameter space allowing strong first order phase transitions. We show that a large part of the parameter space can cause double first-order phase transitions. Whereas both of these phase transitions can generate a detectable stochastic gravitational wave background, one of them can create primordial black holes with appreciable abundance. The primordial black holes generated at the high scale transition can account for the dark matter maintaining the correct relic abundance. We also show specific benchmark cases and their consequences from the aspect of primordial black holes and gravitational waves., Comment: 22 pages, 10 figures, 3 tables

Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the dominance of English training data, raising concerns about potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub that fills the resource gap by providing standardized corpora in nearly 1,000 SEA languages across three modalities. Through our SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous languages across 13 tasks, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in SEA., Comment: https://seacrowd.github.io/ Accepted in EMNLP 2024

Performance prediction is a method to estimate the performance of Language Models (LMs) on various Natural Language Processing (NLP) tasks, mitigating computational costs associated with model capacity and data for fine-tuning. Our paper introduces ProxyLM, a scalable framework for predicting LM performance using proxy models in multilingual tasks. These proxy models act as surrogates, approximating the performance of the LM of interest. By leveraging proxy models, ProxyLM significantly reduces computational overhead on task evaluations, achieving up to a 37.08x speedup compared to traditional methods, even with our smallest proxy models. Additionally, our methodology showcases adaptability to previously unseen languages in pre-trained LMs, outperforming the state-of-the-art performance by 1.89x as measured by root-mean-square error (RMSE). This framework streamlines model selection, enabling efficient deployment and iterative LM enhancements without extensive computational resources., Comment: Preprint

Recent developments in self-supervised learning (SSL) have made it possible to learn data representations without the need for annotations. Inspired by the non-contrastive SSL approach (SimSiam), we introduce a novel framework SIMSAM to compute the Semantic Affinity Matrix, which is significant for unsupervised image segmentation. Given an image, SIMSAM first extracts features using pre-trained DINO-ViT, then projects the features to predict the correlations of dense features in a non-contrastive way. We show applications of the Semantic Affinity Matrix in object segmentation and semantic segmentation tasks. Our code is available at https://github.com/chandagrover/SimSAM., Comment: 6 Pages-Main Paper , 6 figures, 6Tables (Main Paper), ICIP 2024, 8 Pages: Supplementary

Words have been represented in a high-dimensional vector space that encodes their semantic similarities, enabling downstream applications such as retrieving synonyms, antonyms, and relevant contexts. However, despite recent advances in multilingual language models (LMs), the effectiveness of these models' representations in semantic retrieval contexts has not been comprehensively explored. To fill this gap, this paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual LMs in semantic retrieval tasks, including bitext mining and classification via retrieval-augmented contexts. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages, including extremely low-resource languages in challenging cross-lingual and code-switching settings. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches, without requiring any fine-tuning., Comment: Accepted by EMNLP 2024 Findings

Effective evaluation of language models remains an open challenge in NLP. Researchers and engineers face methodological issues such as the sensitivity of models to evaluation setup, difficulty of proper comparisons across methods, and the lack of reproducibility and transparency. In this paper we draw on three years of experience in evaluating large language models to provide guidance and lessons for researchers. First, we provide an overview of common challenges faced in language model evaluation. Second, we delineate best practices for addressing or lessening the impact of these challenges on research. Third, we present the Language Model Evaluation Harness (lm-eval): an open source library for independent, reproducible, and extensible evaluation of language models that seeks to address these issues. We describe the features of the library as well as case studies in which the library has been used to alleviate these methodological concerns.

Realization of noncentrosymmetric magnetic Weyl metals is expected to exhibit anomalous transport properties stemming from the interplay of unusual bulk electronic topology and magnetism. Here, we present spin-valve-like magnetoresistance at room temperature in ferrimagneticWeyl metal Mn$_2$PdSn that crystallizes in the inverse Heusler structure. Anomalous magnetoresistance display dominant asymmetric component attributed to domain wall electron scattering, indicative of spin-valve-like behavior. Ab initio calculations confirm the topologically non-trivial nature of the band structure, with three pairs of Weyl nodes proximate to the Fermi level, providing deeper insights into the observed intrinsic Berry curvature mediated substantial anomalous Hall conductivity. Our results underscore the inverse Heusler compounds as promising platform to realize magnetic Weyl metals/semimetals and leverage emergent transport properties for electronic functionalities.

Understanding conditions for the synchronization of a network of interconnected oscillators is a challenging problem. Typically, only sufficient conditions are reported for the synchronization problem. Here, we adopted the Lyapunov-Floquet theory and the Master Stability Function approach in order to derive the synchronization conditions for a set of coupled nonlinear oscillators. We found that the positivity of the coupling constant is a necessary and sufficient condition for synchronizing linearly full-state coupled identical oscillators. Moreover, in the case of partial state coupling, the asymptotic convergence of volume in state space is ensured by a positive coupling constant. The numerical calculation of the Master Stability Function for a benchmark two-dimensional oscillator validates the synchronization corresponding to the positive coupling. The results are illustrated using numerical simulations and experimentation on benchmark oscillators., Comment: 6 pages, 7 figures, Journal

Large language models (LLMs) show remarkable human-like capability in various domains and languages. However, a notable quality gap arises in low-resource languages, e.g., Indonesian indigenous languages, rendering them ineffective and inefficient in such linguistic contexts. To bridge this quality gap, we introduce Cendol, a collection of Indonesian LLMs encompassing both decoder-only and encoder-decoder architectures across a range of model sizes. We highlight Cendol's effectiveness across a diverse array of tasks, attaining 20% improvement, and demonstrate its capability to generalize to unseen tasks and indigenous languages of Indonesia. Furthermore, Cendol models showcase improved human favorability despite their limitations in capturing indigenous knowledge and cultural values in Indonesia. In addition, we discuss the shortcomings of parameter-efficient tunings, such as LoRA, for language adaptation. Alternatively, we propose the usage of vocabulary adaptation to enhance efficiency. Lastly, we evaluate the safety of Cendol and showcase that safety in pre-training in one language such as English is transferable to low-resource languages, such as Indonesian, even without RLHF and safety fine-tuning., Comment: Cendol models are released under Apache 2.0 license and will be made publicly available soon

The topological phase transition in the Qi-Wu-Zhang model is studied using a real-space approach. An effective Hamiltonian for the topologically protected edge-modes in a finite-size system is developed. The topological phase transition is understood in terms of a global perturbation to the system which lifts the degeneracy of the edge-modes. The effective Hamiltonian method is also applied to a one-dimensional system with spatially varying hopping strengths to understand the impact of disorder on the edge-modes., Comment: Edits: revisions made, references added

The interpretation of human voices holds importance across various applications. This study ventures into predicting age, gender, and emotion from vocal cues, a field with vast applications. Voice analysis tech advancements span domains, from improving customer interactions to enhancing healthcare and retail experiences. Discerning emotions aids mental health, while age and gender detection are vital in various contexts. Exploring deep learning models for these predictions involves comparing single, multi-output, and sequential models highlighted in this paper. Sourcing suitable data posed challenges, resulting in the amalgamation of the CREMA-D and EMO-DB datasets. Prior work showed promise in individual predictions, but limited research considered all three variables simultaneously. This paper identifies flaws in an individual model approach and advocates for our novel multi-output learning architecture Speech-based Emotion Gender and Age Analysis (SEGAA) model. The experiments suggest that Multi-output models perform comparably to individual models, efficiently capturing the intricate relationships between variables and speech inputs, all while achieving improved runtime.