Search

Dataset contamination, where evaluation datasets overlap with pre-training corpora, inflates performance metrics and undermines the reliability of model evaluations. Quantifying dataset contamination thus becomes essential to ensure that performance evaluations genuinely reflect a model's ability to generalize to unseen data, rather than relying on memorized examples. To address this problem, we propose Kernel Divergence Score (KDS), a novel method that quantifies dataset contamination by computing the divergence between the kernel similarity matrix of sample embeddings, before and after fine-tuning on the benchmark dataset. Leveraging the insight that fine-tuning affects unseen samples more significantly than seen ones, KDS provides a reliable measure of contamination. Through extensive experiments on controlled contamination scenarios, KDS demonstrates a near-perfect correlation with contamination levels and outperforms existing baselines. Additionally, we perform comprehensive ablation studies to analyze the impact of key design choices, providing deeper insights into the components and effectiveness of KDS. These ablations highlight the importance of leveraging fine-grained kernel-based information and confirm the reliability of the proposed framework across diverse datasets and settings.

CuInS2 quantum dots have been studied in a broad range of applications, but despite this, the fine details of their charge carrier dynamics remain a subject of intense debate. Two of the most relevant points of discussion are the hole dynamics and the influence of Cu:In synthesis stoichiometry on them. It has been proposed that Cu-deficiency leads to the formation of Cu2+, affecting the localization of holes into Cu defects. Importantly, it is precisely these confined hole states which are used to explain the interesting photoluminescence properties of CuInS2 quantum dots. We use static X-ray spectroscopy to reveal no evidence for a measurable amount of native Cu2+ states in Cu-deficient samples. Instead, the improved properties of these samples are explained by an increase of crystallinity, reducing the concentration of mid gap states. Furthermore, to understand the charge carrier dynamics, herein we employ ultrafast optical transient absorption, and fluorescence up-conversion spectroscopies in combination with ultrafast X-ray absorption spectroscopy using a hard X-ray free electron laser. We demonstrate that in non-passivated samples, holes are transferred from Cu atoms in sub-picosecond timescales. We assign this transfer to occur towards the thiol-based ligands. Finally, we observe that Cu-deficient samples are more robust against the photothermal heating effects of using higher laser fluences. This is not the case for the stoichiometric sample, where heating effects on the structure are directly observed.

We present the first circularly polarized Floquet engineering time-resolved Resonant Inelastic X-ray Scattering (tr-RIXS) experiment in H$_3$LiIr$_2$O$_6$, an iridium-based Kitaev system. Our calculations and experimental results are consistent with the modification of the low energy magnetic excitations in H$_3$LiIr$_2$O$_6$ only during illumination by the laser pulse, consistent with the Floquet engineering of the exchange interactions. However, the penetration length mismatch between the X-ray probe and laser pump and the intrinsic complexity of Kitaev magnets prevented us from unequivocally extracting towards which ground H$_3$LiIr$_2$O$_6$ was driven. We outline possible solutions to these challenges for Floquet stabilization and observation of the Kitaev Quantum Spin Liquid limit by RIXS.

Fine-tuning Large Language Models (LLMs) has emerged as a common practice for tailoring models to individual needs and preferences. The choice of datasets for fine-tuning can be diverse, introducing safety concerns regarding the potential inclusion of harmful data samples. Manually filtering or avoiding such samples, however, can be labor-intensive and subjective. To address these difficulties, we propose a novel Safety-Aware Fine-Tuning (SAFT) framework designed to automatically detect and remove potentially harmful data, by leveraging a scoring function that exploits the subspace information of harmful and benign samples. Experimental results demonstrate the efficacy of SAFT across different LLMs and varying contamination rates, achieving reductions in harmfulness of up to 27.8%. Going beyond, we delve into the mechanism of our approach and validate its versatility in addressing practical challenges in real-world scenarios., Comment: NeurIPS 2024 Workshop on Safe Generative AI

Large language models (LLMs) often show unwarranted preference for certain choice options when responding to multiple-choice questions, posing significant reliability concerns in LLM-automated systems. To mitigate this selection bias problem, previous solutions utilized debiasing methods to adjust the model's input and/or output. Our work, in contrast, investigates the model's internal representation of the selection bias. Specifically, we introduce a novel debiasing approach, Bias Node Pruning (BNP), which eliminates the linear layer parameters that contribute to the bias. Furthermore, we present Auxiliary Option Injection (AOI), a simple yet effective input modification technique for debiasing, which is compatible even with black-box LLMs. To provide a more systematic evaluation of selection bias, we review existing metrics and introduce Choice Kullback-Leibler Divergence (CKLD), which addresses the insensitivity of the commonly used metrics to label imbalance. Experiments show that our methods are robust and adaptable across various datasets when applied to three LLMs.

Extending geodetic and astrometric Very Long Baseline Interferometry (VLBI) observations from traditional centimeter wavebands to millimeter wavebands offers numerous scientific potentials and benefits. However, it was considered quite challenging due to various factors, including the increased effects of atmospheric opacity and turbulence at millimeter wavelengths. Here, we present the results of the first geodetic-mode VLBI experiment, simultaneously observing 82 sources at 22/43/88/132 GHz (K/Q/W/D bands) using the Korean VLBI Network (KVN). We introduced the frequency phase transfer (FPT) method to geodetic VLBI analysis, an approach for calibrating atmospheric phase fluctuations at higher frequencies by transferring phase solutions from lower frequencies. With a 2-minute scan, FPT improved the signal-to-noise ratio (SNR) of most fringes, some by over 100%, thereby enhancing the detection rate of weak sources at millimeter wavebands. Additionally, FPT reduced systematic errors in group delay and delay rate, with the weighted root-mean-squares (WRMS) of the post-fitting residuals decreasing from 25.0 ps to 20.5 ps at the W band and from 39.3 ps to 27.6 ps at the D band. There were no notable differences observed in calibrating atmospheric phase fluctuations at the K band (WRMS = 12.4 ps) and Q band (WRMS = 11.8 ps) with the KVN baselines. This experiment demonstrated that the millimeter waveband can be used for geodetic and astrometric applications with high precision., Comment: 17 pages, 6 figures, accepted by The Astronomical Journal. Your feedback to improve the manuscript would be greatly appreciated

Large Language Models (LLMs) are trained on massive text corpora, which are encoded with diverse personality traits. This triggers an interesting goal of eliciting a desired personality trait from the LLM, and probing its behavioral preferences. Accordingly, we formalize the persona elicitation task, aiming to customize LLM behaviors to align with a target persona. We present Persona In-Context Learning (PICLe), a novel persona elicitation framework grounded in Bayesian inference. At the core, PICLe introduces a new ICL example selection criterion based on likelihood ratio, which is designed to optimally guide the model in eliciting a specific target persona. We demonstrate the effectiveness of PICLe through extensive comparisons against baseline methods across three contemporary LLMs. Code is available at https://github.com/deeplearning-wisc/picle., Comment: ICML 2024

Multi-hop Knowledge Graph Question Answering (KGQA) is a task that involves retrieving nodes from a knowledge graph (KG) to answer natural language questions. Recent GNN-based approaches formulate this task as a KG path searching problem, where messages are sequentially propagated from the seed node towards the answer nodes. However, these messages are past-oriented, and they do not consider the full KG context. To make matters worse, KG nodes often represent proper noun entities and are sometimes encrypted, being uninformative in selecting between paths. To address these problems, we propose Neural Tree Search (NuTrea), a tree search-based GNN model that incorporates the broader KG context. Our model adopts a message-passing scheme that probes the unreached subtree regions to boost the past-oriented embeddings. In addition, we introduce the Relation Frequency-Inverse Entity Frequency (RF-IEF) node embedding that considers the global KG context to better characterize ambiguous KG nodes. The general effectiveness of our approach is demonstrated through experiments on three major multi-hop KGQA benchmark datasets, and our extensive analyses further validate its expressiveness and robustness. Overall, NuTrea provides a powerful means to query the KG with complex natural language questions. Code is available at https://github.com/mlvlab/NuTrea., Comment: Neural Information Processing Systems (NeurIPS) 2023

In recent years, patch-based image restoration approaches have demonstrated superior performance compared to conventional variational methods. This paper delves into the mathematical foundations underlying patch-based image restoration methods, with a specific focus on establishing restoration guarantees for patch-based image inpainting, leveraging the assumption of self-similarity among patches. To accomplish this, we present a reformulation of the image inpainting problem as structured low-rank matrix completion, accomplished by grouping image patches with potential overlaps. By making certain incoherence assumptions, we establish a restoration guarantee, given that the number of samples exceeds the order of $rlog^2(N)$, where $N\times N$ denotes the size of the image and $r > 0$ represents the sum of ranks for each group of image patches. Through our rigorous mathematical analysis, we provide valuable insights into the theoretical foundations of patch-based image restoration methods, shedding light on their efficacy and offering guidelines for practical implementation.

Foundation models have shown outstanding performance and generalization capabilities across domains. Since most studies on foundation models mainly focus on the pretraining phase, a naive strategy to minimize a single task-specific loss is adopted for fine-tuning. However, such fine-tuning methods do not fully leverage other losses that are potentially beneficial for the target task. Therefore, we propose MEta Loss TRansformer (MELTR), a plug-in module that automatically and non-linearly combines various loss functions to aid learning the target task via auxiliary learning. We formulate the auxiliary learning as a bi-level optimization problem and present an efficient optimization algorithm based on Approximate Implicit Differentiation (AID). For evaluation, we apply our framework to various video foundation models (UniVL, Violet and All-in-one), and show significant performance gain on all four downstream tasks: text-to-video retrieval, video question answering, video captioning, and multi-modal sentiment analysis. Our qualitative analyses demonstrate that MELTR adequately `transforms' individual loss functions and `melts' them into an effective unified loss. Code is available at https://github.com/mlvlab/MELTR., Comment: Accepted paper at CVPR 2023

Here, we optimized the annealing temperature of HZO/TiN thin film heterostructure via multiscale analysis of remnant polarization, crystallographic phase, minimum ferroelectric domain size, and average grain size. We found that the remnant polarization was closely related to the relative amount of the orthorhombic phase whereas the minimum domain size was to the relative amount of the monoclinic phase. The minimum domain size was obtained at the annealing temperature of 500$^\cird$C while the optimum remnant polarization and capacitance at the annealing temperature of 600$^\circ$C. We conclude that the minimum domain size is more important than the sheer magnitude of remnant polarization considering the retention and fatigue of switchable polarization in nanoscale ferroelectric devices. Our results are expected to contribute to the development of ultra-low-power logic transistors and next-generation non-volatile memory devices., Comment: 28 pages, 11 figures, 1 table

Effective photoinduced charge transfer makes molecular bimetallic assemblies attractive for applications as active light induced proton reduction systems. For a more sustainable future, development of competitive base metal dyads is mandatory. However, the electron transfer mechanisms from the photosensitizer to the proton reduction catalyst in base metal dyads remain so far unexplored. We study a Fe-Co dyad that exhibits photocatalytic H2 production activity using femtosecond X-ray emission spectroscopy, complemented by ultrafast optical spectroscopy and theoretical time-dependent DFT calculations, to understand the electronic and structural dynamics after photoexcitation and during the subsequent charge transfer process from the FeII photosensitizer to the cobaloxime catalyst. Using this novel approach, the simultaneous measurement of the transient Kalpha X-ray emission at the iron and cobalt K-edges in a two-colour experiment is enabled making it possible to correlate the excited state dynamics to the electron transfer processes. The methodology, therefore, provides a clear and direct spectroscopic evidence of the Fe->Co electron transfer responsible for the proton reduction activity.

Question Answering (QA) is a task that entails reasoning over natural language contexts, and many relevant works augment language models (LMs) with graph neural networks (GNNs) to encode the Knowledge Graph (KG) information. However, most existing GNN-based modules for QA do not take advantage of rich relational information of KGs and depend on limited information interaction between the LM and the KG. To address these issues, we propose Question Answering Transformer (QAT), which is designed to jointly reason over language and graphs with respect to entity relations in a unified manner. Specifically, QAT constructs Meta-Path tokens, which learn relation-centric embeddings based on diverse structural and semantic relations. Then, our Relation-Aware Self-Attention module comprehensively integrates different modalities via the Cross-Modal Relative Position Bias, which guides information exchange between relevant entites of different modalities. We validate the effectiveness of QAT on commonsense question answering datasets like CommonsenseQA and OpenBookQA, and on a medical question answering dataset, MedQA-USMLE. On all the datasets, our method achieves state-of-the-art performance. Our code is available at http://github.com/mlvlab/QAT., Comment: AAAI2023 (accepted)

Mixup is a commonly adopted data augmentation technique for image classification. Recent advances in mixup methods primarily focus on mixing based on saliency. However, many saliency detectors require intense computation and are especially burdensome for parameter-heavy transformer models. To this end, we propose TokenMixup, an efficient attention-guided token-level data augmentation method that aims to maximize the saliency of a mixed set of tokens. TokenMixup provides x15 faster saliency-aware data augmentation compared to gradient-based methods. Moreover, we introduce a variant of TokenMixup which mixes tokens within a single instance, thereby enabling multi-scale feature augmentation. Experiments show that our methods significantly improve the baseline models' performance on CIFAR and ImageNet-1K, while being more efficient than previous methods. We also reach state-of-the-art performance on CIFAR-100 among from-scratch transformer models. Code is available at https://github.com/mlvlab/TokenMixup., Comment: Accepted paper at NeurIPS 2022

In this paper, we analyze the error estimate of a wavelet frame based image restoration method from degraded and incomplete measurements. We present the error between the underlying original discrete image and the approximate solution which has the minimal $\ell_1$-norm of the canonical wavelet frame coefficients among all possible solutions. Then we further connect the error estimate for the discrete model to the approximation to the underlying function from which the underlying image comes., Comment: arXiv admin note: text overlap with arXiv:2207.07473

The finite-rate-of-innovation (FRI) framework which corresponds a signal/image to a structured low-rank matrix is emerging as an alternative to the traditional sparse regularization. This is because such an off-the-grid approach is able to alleviate the basis mismatch between the true support in the continuous domain and the discrete grid. In this paper, we propose a two-stage off-the-grid regularization model for the image restoration. Given that the discontinuities/edges of the image lie in the zero level set of a band-limited periodic function, we can derive that the Fourier samples of the gradient of the image satisfy an annihilation relation, resulting in a low-rank two-fold Hankel matrix. In addition, since the singular value decomposition of a low-rank Hankel matrix corresponds to an adaptive tight frame system which can represent the image with sparse canonical coefficients, our approach consists of the following two stages. The first stage learns the tight wavelet frame system from a given measurement, and the second stage restores the image via the analysis approach based sparse regularization. The numerical results are presented to demonstrate that the proposed approach is compared favorably against several popular discrete regularization approaches and structured low-rank matrix approaches., Comment: arXiv admin note: text overlap with arXiv:2012.06827

Total variation (TV) minimization is one of the most important techniques in modern signal/image processing, and has wide range of applications. While there are numerous recent works on the restoration guarantee of the TV minimization in the framework of compressed sensing, there are few works on the restoration guarantee of the restoration from partial observations. This paper is to analyze the error of TV based restoration from random entrywise samples. In particular, we estimate the error between the underlying original data and the approximate solution that interpolates (or approximates with an error bound depending on the noise level) the given data that has the minimal TV seminorm among all possible solutions. Finally, we further connect the error estimate for the discrete model to the sparse gradient restoration problem and to the approximation to the underlying function from which the underlying true data comes.

Human-Object Interaction detection is a holistic visual recognition task that entails object detection as well as interaction classification. Previous works of HOI detection has been addressed by the various compositions of subset predictions, e.g., Image -> HO -> I, Image -> HI -> O. Recently, transformer based architecture for HOI has emerged, which directly predicts the HOI triplets in an end-to-end fashion (Image -> HOI). Motivated by various inference paths for HOI detection, we propose cross-path consistency learning (CPC), which is a novel end-to-end learning strategy to improve HOI detection for transformers by leveraging augmented decoding paths. CPC learning enforces all the possible predictions from permuted inference sequences to be consistent. This simple scheme makes the model learn consistent representations, thereby improving generalization without increasing model capacity. Our experiments demonstrate the effectiveness of our method, and we achieved significant improvement on V-COCO and HICO-DET compared to the baseline models. Our code is available at https://github.com/mlvlab/CPChoi., Comment: CVPR2022 accepted

Sparse-Matrix Dense-Matrix multiplication (SpMM) is the key operator for a wide range of applications, including scientific computing, graph processing, and deep learning. Architecting accelerators for SpMM is faced with three challenges - (1) the random memory accessing and unbalanced load in processing because of random distribution of elements in sparse matrices, (2) inefficient data handling of the large matrices which can not be fit on-chip, and (3) anon-general-purpose accelerator design where one accelerator can only process a fixed-size problem. In this paper, we present Sextans, an accelerator for general-purpose SpMM processing. Sextans accelerator features (1) fast random access using on-chip memory, (2) streaming access to off-chip large matrices, (3) PE-aware non-zero scheduling for balanced workload with an II=1 pipeline, and (4) hardware flexibility to enable prototyping the hardware once to support SpMMs of different size as a general-purpose accelerator. We leverage high bandwidth memory (HBM) for the efficient accessing of both sparse and dense matrices. In the evaluation, we present an FPGA prototype Sextans which is executable on a Xilinx U280 HBM FPGA board and a projected prototype Sextans-P with higher bandwidth comparable to V100 and more frequency optimization. We conduct a comprehensive evaluation on 1,400 SpMMs on a wide range of sparse matrices including 50 matrices from SNAP and 150 from SuiteSparse. WecompareSextanswith NVIDIA K80 and V100 GPUs.Sextansachieves a 2.50x geomean speedup over K80 GPU andSextans-Pachieves a 1.14x geomean speedup over V100 GPU (4.94x over K80). The code is available at https://github.com/linghaosong/Sextans., Comment: To appear in FPGA'22