Search

Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks. These biases stem not only from language but also from the cultural knowledge required to interpret questions, reducing the practical utility of translated datasets like MMLU. Furthermore, translation often introduces artifacts that can distort the meaning or clarity of questions in the target language. A common practice in multilingual evaluation is to rely on machine-translated evaluation sets, but simply translating a dataset is insufficient to address these challenges. In this work, we trace the impact of both of these issues on multilingual evaluations and ensuing model performances. Our large-scale evaluation of state-of-the-art open and proprietary models illustrates that progress on MMLU depends heavily on learning Western-centric concepts, with 28% of all questions requiring culturally sensitive knowledge. Moreover, for questions requiring geographic knowledge, an astounding 84.9% focus on either North American or European regions. Rankings of model evaluations change depending on whether they are evaluated on the full portion or the subset of questions annotated as culturally sensitive, showing the distortion to model rankings when blindly relying on translated MMLU. We release Global-MMLU, an improved MMLU with evaluation coverage across 42 languages -- with improved overall quality by engaging with compensated professional and community annotators to verify translation quality while also rigorously evaluating cultural biases present in the original dataset. This comprehensive Global-MMLU set also includes designated subsets labeled as culturally sensitive and culturally agnostic to allow for more holistic, complete evaluation.

This study is a modification of the McKendrick equation into a growth model with fractional order to predict the abalone length growth. We have shown that the model is a special case of Taylor's series after it was analysed using Adomian decomposition method and Caputo fractional derivative. By simulating the series with some fractional orders, the results indicate that the greater the fractional order of the model, the series values generated are greater as well. Moreover, the series that is close to the real data is the one with a fractional order of $0.5$. Therefore, the growth model with a fractional order provides more accuracy than a classical integer order., Comment: 15 pages

This study investigates the existence and stability of localized states in the discrete nonlinear Schr\"odinger (DNLS) equation with quadratic and cubic nonlinearities, describing the so-called quantum droplets and bubbles. Those states exist within an interval known as the pinning region, as we vary a control parameter. Within the interval, multistable states are connected through multiple hysteresis, called homoclinic snaking. In particular, we explore its mechanism and consider two limiting cases of coupling strength: weak (anti-continuum) and strong (continuum) limits. We employ an asymptotic and a variational method for the weak and strong coupling limits, respectively, to capture the pinning region's width. The width exhibits an algebraic and an exponentially small dependence on the coupling constant for the weak and strong coupling, respectively. This finding is supported by both analytical and numerical results, which show excellent agreement. We also consider the modulational instability of spatially uniform solutions. Our work sheds light on the intricate interplay between multistability and homoclinic snaking in discrete quantum systems, paving the way for further exploration of complex nonlinear phenomena in this context., Comment: Chaos, Solitons and Fractals

We present MetaMetrics-MT, an innovative metric designed to evaluate machine translation (MT) tasks by aligning closely with human preferences through Bayesian optimization with Gaussian Processes. MetaMetrics-MT enhances existing MT metrics by optimizing their correlation with human judgments. Our experiments on the WMT24 metric shared task dataset demonstrate that MetaMetrics-MT outperforms all existing baselines, setting a new benchmark for state-of-the-art performance in the reference-based setting. Furthermore, it achieves comparable results to leading metrics in the reference-free setting, offering greater efficiency., Comment: Preprint

Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English and in underrepresented cultural contexts. To evaluate their understanding of such knowledge, we introduce WorldCuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding. This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points, making it the largest multicultural VQA benchmark to date. It includes tasks for identifying dish names and their origins. We provide evaluation datasets in two sizes (12k and 60k instances) alongside a training dataset (1 million instances). Our findings show that while VLMs perform better with correct location context, they struggle with adversarial contexts and predicting specific regional cuisines and languages. To support future research, we release a knowledge base with annotated food entries and images along with the VQA data., Comment: Preprint

Solving the three-dimensional (3D) Bratu equation is highly challenging due to the presence of multiple and sharp solutions. Research on this equation began in the late 1990s, but there are no satisfactory results to date. To address this issue, we introduce a symmetric finite difference method (SFDM) which embeds the symmetry properties of the solutions into a finite difference method (FDM). This SFDM is primarily used to obtain more accurate solutions and bifurcation diagrams for the 3D Bratu equation. Additionally, we propose modifying the Bratu equation by incorporating a new constraint that facilitates the construction of bifurcation diagrams and simplifies handling the turning points. The proposed method, combined with the use of sparse matrix representation, successfully solves the 3D Bratu equation on grids of up to $301^3$ points. The results demonstrate that SFDM outperforms all previously employed methods for the 3D Bratu equation. Furthermore, we provide bifurcation diagrams for the 1D, 2D, 4D, and 5D cases, and accurately identify the first turning points in all dimensions. All simulations indicate that the bifurcation diagrams of the Bratu equation on the cube domains closely resemble the well-established behavior on the ball domains described by Joseph and Lundgren [1]. Furthermore, when SFDM is applied to linear stability analysis, it yields the same largest real eigenvalue as the standard FDM despite having fewer equations and variables in the nonlinear system., Comment: Accepted for publication in Applied Mathematics and Computation

Understanding the quality of a performance evaluation metric is crucial for ensuring that model outputs align with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce MetaMetrics, a calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. MetaMetrics optimizes the combination of existing metrics to enhance their alignment with human preferences. Our metric demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi-domain scenarios. MetaMetrics aligns closely with human preferences and is highly extendable and easily integrable into any application. This makes MetaMetrics a powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts., Comment: Preprint

Multilingual large language models (LLMs) today may not necessarily provide culturally appropriate and relevant responses to its Filipino users. We introduce Kalahi, a cultural LLM evaluation suite collaboratively created by native Filipino speakers. It is composed of 150 high-quality, handcrafted and nuanced prompts that test LLMs for generations that are relevant to shared Filipino cultural knowledge and values. Strong LLM performance in Kalahi indicates a model's ability to generate responses similar to what an average Filipino would say or do in a given situation. We conducted experiments on LLMs with multilingual and Filipino language support. Results show that Kalahi, while trivial for Filipinos, is challenging for LLMs, with the best model answering only 46.0% of the questions correctly compared to native Filipino performance of 89.10%. Thus, Kalahi can be used to accurately and reliably evaluate Filipino cultural representation in LLMs., Comment: Accepted for presentation at Paclic 38, 2024

We consider reflectionless wave propagation in networks modeled in terms of the nonlocal nonlinear Schr\"odinger (NNLS) equation on metric graphs, for which transparent boundary conditions are imposed at the vertices. By employing the ``potential approach" previously used for the nonlinear Schr\"odinger equation, we derive transparent boundary conditions for the NNLS equation on metric graphs. These conditions eliminate backscattering at graph vertices, which is crucial for minimizing losses in signal, heat, and charge transfer in various applications such as optical fibers, optoelectronic networks, and low-dimensional materials., Comment: 13 pages, 6 figures

This research introduces an extended application of neural networks for solving nonlinear partial differential equations (PDEs). A neural network, combined with a pseudo-arclength continuation, is proposed to construct bifurcation diagrams from parameterized nonlinear PDEs. Additionally, a neural network approach is also presented for solving eigenvalue problems to analyze solution linear stability, focusing on identifying the largest eigenvalue. The effectiveness of the proposed neural network is examined through experiments on the Bratu equation and the Burgers equation. Results from a finite difference method are also presented as comparison. Varying numbers of grid points are employed in each case to assess the behavior and accuracy of both the neural network and the finite difference method. The experimental results demonstrate that the proposed neural network produces better solutions, generates more accurate bifurcation diagrams, has reasonable computational times, and proves effective for linear stability analysis., Comment: Accepted for publication in Applied Mathematics and Computation

The proliferation of large language models (LLMs) requires robust evaluation of their alignment with local values and ethical standards, especially as existing benchmarks often reflect the cultural, legal, and ideological values of their creators. \textsc{LocalValueBench}, introduced in this paper, is an extensible benchmark designed to assess LLMs' adherence to Australian values, and provides a framework for regulators worldwide to develop their own LLM benchmarks for local value alignment. Employing a novel typology for ethical reasoning and an interrogation approach, we curated comprehensive questions and utilized prompt engineering strategies to probe LLMs' value alignment. Our evaluation criteria quantified deviations from local values, ensuring a rigorous assessment process. Comparative analysis of three commercial LLMs by USA vendors revealed significant insights into their effectiveness and limitations, demonstrating the critical importance of value alignment. This study offers valuable tools and methodologies for regulators to create tailored benchmarks, highlighting avenues for future research to enhance ethical AI development.

Addressing school bullying effectively and promptly is crucial for the mental health of students. This study examined the potential of large language models (LLMs) to empower students by discerning between bullying and joking in school peer interactions. We employed ChatGPT-4, Gemini 1.5 Pro, and Claude 3 Opus, evaluating their effectiveness through human review. Our results revealed that not all LLMs were suitable for an agentic approach, with ChatGPT-4 showing the most promise. We observed variations in LLM outputs, possibly influenced by political overcorrectness, context window limitations, and pre-existing bias in their training data. ChatGPT-4 excelled in context-specific accuracy after implementing the agentic approach, highlighting its potential to provide continuous, real-time support to vulnerable students. This study underlines the significant social impact of using agentic AI in educational settings, offering a new avenue for reducing the negative consequences of bullying and enhancing student well-being.

Hate speech poses a significant threat to social harmony. Over the past two years, Indonesia has seen a ten-fold increase in the online hate speech ratio, underscoring the urgent need for effective detection mechanisms. However, progress is hindered by the limited availability of labeled data for Indonesian texts. The condition is even worse for marginalized minorities, such as Shia, LGBTQ, and other ethnic minorities because hate speech is underreported and less understood by detection tools. Furthermore, the lack of accommodation for subjectivity in current datasets compounds this issue. To address this, we introduce IndoToxic2024, a comprehensive Indonesian hate speech and toxicity classification dataset. Comprising 43,692 entries annotated by 19 diverse individuals, the dataset focuses on texts targeting vulnerable groups in Indonesia, specifically during the hottest political event in the country: the presidential election. We establish baselines for seven binary classification tasks, achieving a macro-F1 score of 0.78 with a BERT model (IndoBERTweet) fine-tuned for hate speech classification. Furthermore, we demonstrate how incorporating demographic information can enhance the zero-shot performance of the large language model, gpt-3.5-turbo. However, we also caution that an overemphasis on demographic information can negatively impact the fine-tuned model performance due to data fragmentation.

Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the dominance of English training data, raising concerns about potential cultural misrepresentation. To address these challenges, we introduce SEACrowd, a collaborative initiative that consolidates a comprehensive resource hub that fills the resource gap by providing standardized corpora in nearly 1,000 SEA languages across three modalities. Through our SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous languages across 13 tasks, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in SEA., Comment: https://seacrowd.github.io/ Accepted in EMNLP 2024

Weakly-supervised anomaly detection can outperform existing unsupervised methods with the assistance of a very small number of labeled anomalies, which attracts increasing attention from researchers. However, existing weakly-supervised anomaly detection methods are limited as these methods do not factor in the multimodel nature of the real-world data distribution. To mitigate this, we propose the Weakly-supervised Variational-mixture-model-based Anomaly Detector (WVAD). WVAD excels in multimodal datasets. It consists of two components: a deep variational mixture model, and an anomaly score estimator. The deep variational mixture model captures various features of the data from different clusters, then these features are delivered to the anomaly score estimator to assess the anomaly levels. Experimental results on three real-world datasets demonstrate WVAD's superiority., Comment: 5 pages, 3 figures. Accepted by 2024 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC)

Deep learning-based sequence models are extensively employed in Time Series Anomaly Detection (TSAD) tasks due to their effective sequential modeling capabilities. However, the ability of TSAD is limited by two key challenges: (i) the ability to model long-range dependency and (ii) the generalization issue in the presence of non-stationary data. To tackle these challenges, an anomaly detector that leverages the selective state space model known for its proficiency in capturing long-term dependencies across various domains is proposed. Additionally, a multi-stage detrending mechanism is introduced to mitigate the prominent trend component in non-stationary data to address the generalization issue. Extensive experiments conducted on realworld public datasets demonstrate that the proposed methods surpass all 12 compared baseline methods., Comment: Accepted by IEEE Signal Processing Letters. DOI:10.1109/LSP.2024.3438078

Large Language Models (LLMs) demonstrate strong machine translation capabilities on languages they are trained on. However, the impact of factors beyond training data size on translation performance remains a topic of debate, especially concerning languages not directly encountered during training. Our study delves into Llama2's translation capabilities. By modeling a linear relationship between linguistic feature distances and machine translation scores, we ask ourselves if there are potentially better central languages for LLMs other than English. Our experiments show that the 7B Llama2 model yields above 10 BLEU when translating into all languages it has seen, which rarely happens for languages it has not seen. Most translation improvements into unseen languages come from scaling up the model size rather than instruction tuning or increasing shot count. Furthermore, our correlation analysis reveals that syntactic similarity is not the only linguistic factor that strongly correlates with machine translation scores. Interestingly, we discovered that under specific circumstances, some languages (e.g. Swedish, Catalan), despite having significantly less training data, exhibit comparable correlation levels to English. These insights challenge the prevailing landscape of LLMs, suggesting that models centered around languages other than English could provide a more efficient foundation for multilingual applications., Comment: TDLE 2024

Acoustic scene classification (ASC) is a crucial research problem in computational auditory scene analysis, and it aims to recognize the unique acoustic characteristics of an environment. One of the challenges of the ASC task is the domain shift between training and testing data. Since 2018, ASC challenges have focused on the generalization of ASC models across different recording devices. Although this task, in recent years, has achieved substantial progress in device generalization, the challenge of domain shift between different geographical regions, involving discrepancies such as time, space, culture, and language, remains insufficiently explored at present. In addition, considering the abundance of unlabeled acoustic scene data in the real world, it is important to study the possible ways to utilize these unlabelled data. Therefore, we introduce the task Semi-supervised Acoustic Scene Classification under Domain Shift in the ICME 2024 Grand Challenge. We encourage participants to innovate with semi-supervised learning techniques, aiming to develop more robust ASC models under domain shift.

An rf superconducting quantum interference device (SQUID) consists of a superconducting ring interrupted by a Josephson junction (JJ). When driven by an alternating magnetic field, the induced supercurrents around the ring are determined by the JJ through the celebrated Josephson relations. This system exhibits rich nonlinear behavior, including chaotic effects. We study the dynamics of a pair of parametrically-driven coupled SQUIDs arranged in series. We take advantage of the weak damping that characterizes these systems to perform a multiple-scales analysis and obtain amplitude equations, describing the slow dynamics of the system. This picture allows us to expose the existence of homoclinic orbits in the dynamics of the integrable part of the slow equations of motion. Using high-dimensional Melnikov theory, we are able to obtain explicit parameter values for which these orbits persist in the full system, consisting of both Hamiltonian and non-Hamiltonian perturbations, to form so-called Silnikov orbits, indicating a loss of integrability and the existence of chaos., Comment: 4 pages. arXiv admin note: text overlap with arXiv:1007.3939 by other authors

We consider a lattice equation modelling one-dimensional metamaterials formed by a discrete array of nonlinear resonators. We focus on periodic travelling waves due to the presence of a periodic force. The existence and uniqueness results of periodic travelling waves of the system are presented. Our analytical results are found to be in good agreement with direct numerical computations, Comment: 21 pages

In this work, we study a model of a one-dimensional magnetic metamaterial formed by a discrete array of nonlinear resonators. We focus on periodic and localized traveling waves of the model, in the presence of loss and an external drive. Employing a Melnikov analysis we study the existence and persistence of such traveling waves, and study their linear stability. We show that, under certain conditions, the presence of dissipation and/or driving may stabilize or destabilize the solutions. Our analytical results are found to be in good agreement with direct numerical computations., Comment: 12 pages