Search

Large language models (LLMs) are commonly used as evaluators in tasks (e.g., reward modeling, LLM-as-a-judge), where they act as proxies for human preferences or judgments. This leads to the need for meta-evaluation: evaluating the credibility of LLMs as evaluators. However, existing benchmarks primarily focus on English, offering limited insight into LLMs' effectiveness as evaluators in non-English contexts. To address this, we introduce MM-Eval, a multilingual meta-evaluation benchmark that covers 18 languages across six categories. MM-Eval evaluates various dimensions, including language-specific challenges like linguistics and language hallucinations. Evaluation results show that both proprietary and open-source language models have considerable room for improvement. Further analysis reveals a tendency for these models to assign middle-ground scores to low-resource languages. We publicly release our benchmark and code., Comment: work in progress

Jigsaw puzzle solving is a challenging task for computer vision since it requires high-level spatial and semantic reasoning. To solve the problem, existing approaches invariably use color and/or shape information but in many real-world scenarios, such as in archaeological fresco reconstruction, this kind of clues is often unreliable due to severe physical and pictorial deterioration of the individual fragments. This makes state-of-the-art approaches entirely unusable in practice. On the other hand, in such cases, simple geometrical patterns such as lines or curves offer a powerful yet unexplored clue. In an attempt to fill in this gap, in this paper we introduce a new challenging version of the puzzle solving problem in which one deliberately ignores conventional color and shape features and relies solely on the presence of linear geometrical patterns. The reconstruction process is then only driven by one of the most fundamental principles of Gestalt perceptual organization, namely Wertheimer's {\em law of good continuation}. In order to tackle this problem, we formulate the puzzle solving problem as the problem of finding a Nash equilibrium of a (noncooperative) multiplayer game and use classical multi-population replicator dynamics to solve it. The proposed approach is general and allows us to deal with pieces of arbitrary shape, size and orientation. We evaluate our approach on both synthetic and real-world data and compare it with state-of-the-art algorithms. The results show the intrinsic complexity of our purely line-based puzzle problem as well as the relative effectiveness of our game-theoretic formulation., Comment: to be published in ACCV2024

Human actions are based on the mental perception of the environment. Even when all the aspects of an environment are not visible, humans have an internal mental model that can generalize the partially visible scenes to fully constructed and connected views. This internal mental model uses learned abstract representations of spatial and temporal aspects of the environments encountered in the past. Artificial agents in reinforcement learning environments also benefit by learning a representation of the environment from experience. It provides the agent with viewpoints that are not directly visible to it, helping it make better policy decisions. It can also be used to predict the future state of the environment. This project explores learning the top-down view of an RL environment based on the artificial agent's first-person view observations with a generative adversarial network(GAN). The top-down view is useful as it provides a complete overview of the environment by building a map of the entire environment. It provides information about the objects' dimensions and shapes along with their relative positions with one another. Initially, when only a partial observation of the environment is visible to the agent, only a partial top-down view is generated. As the agent explores the environment through a set of actions, the generated top-down view becomes complete. This generated top-down view can assist the agent in deducing better policy decisions. The focus of the project is to learn the top-down view of an RL environment. It doesn't deal with any Reinforcement Learning task.

Conventional tomographic reconstruction typically depends on centralized servers for both data storage and computation, leading to concerns about memory limitations and data privacy. Distributed reconstruction algorithms mitigate these issues by partitioning data across multiple nodes, reducing server load and enhancing privacy. However, these algorithms often encounter challenges related to memory constraints and communication overhead between nodes. In this paper, we introduce a decentralized Alternating Directions Method of Multipliers (ADMM) with configurable quantization. By distributing local objectives across nodes, our approach is highly scalable and can efficiently reconstruct images while adapting to available resources. To overcome communication bottlenecks, we propose two quantization techniques based on K-means clustering and JPEG compression. Numerical experiments with benchmark images illustrate the tradeoffs between communication efficiency, memory use, and reconstruction accuracy., Comment: 26 pages, 8 figures

One fundamental challenge of data-driven analysis in neuroscience is modeling causal interactions and exploring the connectivity between nodes in a brain network. Various statistical methods, using different perspectives and data modalities, have been developed to understand the causal structures in brain dynamics. This study introduces a novel statistical approach, TAR4C, to dissect causal interactions in multichannel EEG recordings. TAR4C uses the threshold autoregressive (TAR) model to describe causal interactions between nodes in a brain network from two perspectives. The first tests whether one node controls the dynamics of another. The controlling node, named the threshold variable, implies its causative role since it operates as a switching mechanism governing the instantaneous transitions between autoregressive structures. This concept is known as threshold non-linearity. Once verified between a node pair, the next step in TAR modeling is assessing the causal node's predictive ability on the other's activity, representing causal interactions in autoregressive terms, a concept underlying Granger (G) causality. TAR4C can discover non-linear, time-dependent causal interactions while maintaining the G-causality framework. The approach's efficacy is demonstrated through EEG data from a motor execution/imagery experiment. By comparing causal interactions during motor execution and imagery, TAR4C reveals key similarities and differences in brain connectivity across subjects.

This talk summarized work done recently to organize the steps for implementing TMD phenomenology in a way optimized for contexts where the extraction and interpretation of hadronic structures and nonperturbative effects is the primary driving motivation., Comment: 17 pages, proceedings of 7th International Workshop on Transverse phenomena in hard processes and the transverse structure of the proton (Transversity2024)

Neural network models for audio tasks, such as automatic speech recognition (ASR) and acoustic scene classification (ASC), are susceptible to noise contamination for real-life applications. To improve audio quality, an enhancement module, which can be developed independently, is explicitly used at the front-end of the target audio applications. In this paper, we present an end-to-end learning solution to jointly optimise the models for audio enhancement (AE) and the subsequent applications. To guide the optimisation of the AE module towards a target application, and especially to overcome difficult samples, we make use of the sample-wise performance measure as an indication of sample importance. In experiments, we consider four representative applications to evaluate our training paradigm, i.e., ASR, speech command recognition (SCR), speech emotion recognition (SER), and ASC. These applications are associated with speech and non-speech tasks concerning semantic and non-semantic features, transient and global information, and the experimental results indicate that our proposed approach can considerably boost the noise robustness of the models, especially at low signal-to-noise ratios (SNRs), for a wide range of computer audition tasks in everyday-life noisy environments.

This paper presents the design and technical progress of a precision X-Y stage for detector dithering and flexure compensation. The stage is being developed for use in the Magellan InfraRed Multi-Object Spectrograph, MIRMOS. MIRMOS is a very large Nasmyth mounted spectrograph containing a combination of refractive, reflective and diffractive optics mounted on a long cryogenic optical bench. The instrument utilizes five science cameras, each having a custom x-y stage to control the in-plane detector position within each camera, providing both dithering capability for improved sampling, and flexure compensation to correct for image motion that results from the gravity variant operation of the instrument. Designed to operate at 120~K, the stage will accurately control detector position in two orthogonal degrees of freedom, and have manual fine adjustment features to set detector tip, tilt and piston. The piezo-driven flexure stage provides high-resolution backlash-free motion of the detector and is very compact along the optical path, keeping camera length to a minimum. A magnetoresistive bridge provides position feedback in each degree of freedom, greatly reducing hysteresis, which is common in piezoelectric actuators. The system is designed to operate in open loop using a lookup table keyed to the Nasmyth rotator angle for flexure control. Here, the optomechanical design of the stage, electrical control system, and current performance results from early prototype efforts are presented and discussed., Comment: Proceedings of SPIE Astronomical Telescopes + Instrumentation 2024, 11 pages, 10 figures

The Magellan InfraRed Multi-Object Spectrograph (MIRMOS) is a planned next generation multi-object and integral field spectrograph for the 6.5m Magellan telescopes at Las Campanas Observatory in Chile. MIRMOS will perform R$\sim$3700 spectroscopy over a simultaneous wavelength range of 0.886 - 2.404$\mu$m (Y,J,H,K bands) in addition to imaging over the range of 0.7 - 0.886$\mu$m. The integral field mode of operation for MIRMOS will be achieved via an image slicer style integral field unit (IFU) located on a linear stage to facilitate movement into the beam during use or storage while operating in multi-object mode. The IFU will provide a $\rm \sim20"\times26"$ field of view (FoV) made up of $\rm0.84"\times26"$ slices. This will be the largest FoV IFS operating at these wavelengths from either the ground or space, making MIRMOS an ideal instrument for a wide range of science cases including studying the high redshift circumgalactic medium and emission line tracers from ionized and molecular gas in nearby galaxies. In order to achieve the desired image quality and FoV while matching the focal ratio to the multi-object mode, our slicer design makes use of novel freeform surfaces for the pupil mirrors, which require the use of high precision multi-axis diamond milling to manufacture. We present here the optical design and predicted performance of the MIRMOS IFU along with a conceptual design for the opto-mechanical system., Comment: Proceedings of SPIE Astronomical Telescopes + Instrumentation 2024, 9 pages, 6 figures

In this paper, we study the Cauchy problem for the linear plate equation with mass term and its applications to semilinear models. For the linear problem we obtain $L^p-L^q$ estimates for the solutions in the full range $1\leq p\leq q\leq \infty$, and we show that such estimates are optimal. In the sequel, we discuss the global in time existence of solutions to the associated semilinear problem with power nonlinearity $|u|^\alpha$. For low dimension space $n\leq 4$, and assuming $L^1$ regularity on the second datum, we were able to prove global existence for $\alpha> \max\{\alpha_c(n), \tilde\alpha_c(n)\}$ where $\alpha_c = 1+4/n$ and $\tilde \alpha_c = 2+2/n$. However, assuming initial data in $H^2(\mathbb{R}^n)\times L^2(\mathbb{R}^n)$, the presence of the mass term allows us to obtain global in time existence for all $1<\alpha \leq (n+4)/[n-4]_+$. We also show that the latter upper bound is optimal, since we prove that there exist data such that a non-existence result for local weak solutions holds when $\alpha > (n+4)/[n-4]_+$.

Let $\mathscr{X}$ be the affine flag manifold of Lie type $A_{n-1}^{(1)}$ where $n \geq 3$ and let $W_{\text{aff}}$ be the associated affine Weyl group. The moment graph for $\mathscr{X}$ encodes the torus fixed points (corresponding to elements of the affine Weyl group $W_{\text{aff}}$) and the torus stable curves in $\mathscr{X}$. Given a fixed point $u\in W_{\text{aff}}$ and a degree $\mathbf{d}=(d_0,d_1,...,d_{n-1})\in \mathbb{Z}_{\geq 0}^{n}$, the combinatorial curve neighborhood is the set of maximal elements in the moment graph of $\mathscr{X}$ which can be reached from $u'\leq u$ by a chain of curves of total degree $\leq \mathbf{d}$. In this paper we give combinatorial formulas and algorithms for calculating these elements in $\mathscr{X}$.

A routing algorithm is the most fundamental problem in complex network communication. In complex networks, the amount of computation increases as the number of nodes increases which reduces routing performance. In this paper, we propose a routing algorithm for software-defined networking (SDN) based on a box-covering (BC) algorithm. It is known that using the BC algorithm it is possible to increase performance in complex SDN. We partition the entire SDN network into subnets using three existing box-covering methods such as MEMB, GC, and CIEA, then we use Dijkstra\textquotesingle s algorithm to find the shortest path between subnets and within each subnet. We compared all box-covering algorithms and found that the GC algorithm has the highest performance for SDN routing., Comment: 2023 10th International Conference on Wireless Networks and Mobile Communications (WINCOM)

A novel approach is developed for discovering directed connectivity between specified pairs of nodes in a high-dimensional network (HDN) of brain signals. To accurately identify causal connectivity for such specified objectives, it is necessary to properly address the influence of all other nodes within the network. The proposed procedure herein starts with the estimation of a low-dimensional representation of the other nodes in the network utilizing (frequency-domain-based) spectral dynamic principal component analysis (sDPCA). The resulting scores can then be removed from the nodes of interest, thus eliminating the confounding effect of other nodes within the HDN. Accordingly, causal interactions can be dissected between nodes that are isolated from the effects of the network. Extensive simulations have demonstrated the effectiveness of this approach as a tool for causality analysis in complex time series networks. The proposed methodology has also been shown to be applicable to multichannel EEG networks.

Telling stories is an integral part of human communication which can evoke emotions and influence the affective states of the audience. Automatically modeling emotional trajectories in stories has thus attracted considerable scholarly interest. However, as most existing works have been limited to unsupervised dictionary-based approaches, there is no benchmark for this task. We address this gap by introducing continuous valence and arousal labels for an existing dataset of children's stories originally annotated with discrete emotion categories. We collect additional annotations for this data and map the categorical labels to the continuous valence and arousal space. For predicting the thus obtained emotionality signals, we fine-tune a DeBERTa model and improve upon this baseline via a weakly supervised learning approach. The best configuration achieves a Concordance Correlation Coefficient (CCC) of $.8221$ for valence and $.7125$ for arousal on the test set, demonstrating the efficacy of our proposed approach. A detailed analysis shows the extent to which the results vary depending on factors such as the author, the individual story, or the section within the story. In addition, we uncover the weaknesses of our approach by investigating examples that prove to be difficult to predict., Comment: Accepted to ACL 2024 Findings. arXiv admin note: text overlap with arXiv:2212.11382

In this work, we study the Kantorovich variant of max-min neural network operators, in which the operator kernel is defined in terms of sigmoidal functions. Our main aim is to demonstrate the $L^{p}$-convergence of these nonlinear operators for $1\leq p<\infty$, which makes it possible to obtain approximation results for functions that are not necessarily continuous. In addition, we will derive quantitative estimates for the rate of approximation in the $L^{p}$-norm. We will provide some explicit examples, studying the approximation of discontinuous functions with the max-min operator, and varying additionally the underlying sigmoidal function of the kernel. Further, we numerically compare the $L^{p}$-approximation error with the respective error of the Kantorovich variants of other popular neural network operators. As a final application, we show that the Kantorovich variant has advantages compared to the sampling variant of the max-min operator and Kantorovich variant of the max-product operator when it comes to approximate noisy functions as for instance biomedical ECG signals., Comment: 23 pages, 6 figures

Transformers have recently gained prominence in long time series forecasting by elevating accuracies in a variety of use cases. Regrettably, in the race for better predictive performance the overhead of model architectures has grown onerous, leading to models with computational demand infeasible for most practical applications. To bridge the gap between high method complexity and realistic computational resources, we introduce the Residual Cyclic Transformer, ReCycle. ReCycle utilizes primary cycle compression to address the computational complexity of the attention mechanism in long time series. By learning residuals from refined smoothing average techniques, ReCycle surpasses state-of-the-art accuracy in a variety of application use cases. The reliable and explainable fallback behavior ensured by simple, yet robust, smoothing average techniques additionally lowers the barrier for user acceptance. At the same time, our approach reduces the run time and energy consumption by more than an order of magnitude, making both training and inference feasible on low-performance, low-power and edge computing devices. Code is available at https://github.com/Helmholtz-AI-Energy/ReCycle, Comment: 9 pages, 3 figures, to be published at IEEE CAI 2024, Associated code available at https://github.com/Helmholtz-AI-Energy/ReCycle

Laboratory experiments have shown that communication plays an important role in solving social dilemmas. Here, by extending the AI-Economist, a mixed motive multi-agent reinforcement learning environment, I intend to find an answer to the following descriptive question: which governing system does facilitate the emergence and evolution of communication and teaching among agents? To answer this question, the AI-Economist is extended by a voting mechanism to simulate three different governing systems across individualistic-collectivistic axis, from full-libertarian to Full-Utilitarian governing systems. Moreover, the AI-Economist is further extended to include communication with possible misalignment, a variant of signalling game, by letting agents to build houses together if they are able to name mutually complement material resources by the same letter. Moreover, another extension is made to the AI-Economist to include teaching with possible misalignment, again a variant of signalling game, by letting half the agents as teachers who know how to use mutually complement material resources to build houses but are not capable of building actual houses, and the other half as students who do not have this information but are able to actually build those houses if teachers teach them. I found a strong evidence that collectivistic environment such as Full-Utilitarian system is more favourable for the emergence of communication and teaching, or more precisely, evolution of language alignment. Moreover, I found some evidence that evolution of language alignment through communication and teaching under collectivistic governing systems makes individuals more advantageously inequity averse. As a result, there is a positive correlation between evolution of language alignment and equality in the society., Comment: 20 pages, 14 figures

O-RAN systems and their deployment in virtualized general-purpose computing platforms (O-Cloud) constitute a paradigm shift expected to bring unprecedented performance gains. However, these architectures raise new implementation challenges and threaten to worsen the already-high energy consumption of mobile networks. This paper presents first a series of experiments which assess the O-Cloud's energy costs and their dependency on the servers' hardware, capacity and data traffic properties which, typically, change over time. Next, it proposes a compute policy for assigning the base station data loads to O-Cloud servers in an energy-efficient fashion; and a radio policy that determines at near-real-time the minimum transmission block size for each user so as to avoid unnecessary energy costs. The policies balance energy savings with performance, and ensure that both of them are dispersed fairly across the servers and users, respectively. To cater for the unknown and time-varying parameters affecting the policies, we develop a novel online learning framework with fairness guarantees that apply to the entire operation horizon of the system (long-term fairness). The policies are evaluated using trace-driven simulations and are fully implemented in an O-RAN compatible system where we measure the energy costs and throughput in realistic scenarios., Comment: to appear in ACM Sigmetrics 2024