Search

Predictive models are often required to produce reliable predictions under statistical conditions that are not matched to the training data. A common type of training-testing mismatch is covariate shift, where the conditional distribution of the target variable given the input features remains fixed, while the marginal distribution of the inputs changes. Weighted conformal risk control (W-CRC) uses data collected during the training phase to convert point predictions into prediction sets with valid risk guarantees at test time despite the presence of a covariate shift. However, while W-CRC provides statistical reliability, its efficiency -- measured by the size of the prediction sets -- can only be assessed at test time. In this work, we relate the generalization properties of the base predictor to the efficiency of W-CRC under covariate shifts. Specifically, we derive a bound on the inefficiency of the W-CRC predictor that depends on algorithmic hyperparameters and task-specific quantities available at training time. This bound offers insights on relationships between the informativeness of the prediction sets, the extent of the covariate shift, and the size of the calibration and training sets. Experiments on fingerprinting-based localization validate the theoretical results.

This article investigates how well deep learning models can identify web registers -- text varieties such as news reports and discussion forums -- across 16 languages. We introduce the Multilingual CORE corpora, which contain 72,504 documents annotated with a hierarchical taxonomy of 25 registers designed to cover the entire open web. Our multilingual models achieve state-of-the-art results (79% F1 score) using multi-label classification. This performance matches or exceeds previous studies that used simpler classification schemes, showing that models can perform well even with a complex register scheme at a massively multilingual scale. However, we observe a consistent performance ceiling around 77-80% F1 score across all models and configurations. When we remove documents with uncertain labels through data pruning, performance increases to over 90% F1, suggesting that this ceiling stems from inherent ambiguity in web registers rather than model limitations. Analysis of hybrid documents -- texts combining multiple registers -- reveals that the main challenge is not in classifying hybrids themselves, but in distinguishing between hybrid and non-hybrid documents. Multilingual models consistently outperform monolingual ones, particularly helping languages with limited training data. While zero-shot performance drops by an average of 7% on unseen languages, this decrease varies substantially between languages (from 3% to 20%), indicating that while registers share many features across languages, they also maintain language-specific characteristics.

The paper reports an empirical study of the effect of human treatment of a robot on the social perception of the robot's behavior. The study employed an enacted interaction between an anthropomorphic "waiter" robot and two customers. The robot and one of the customers (acted out by a researcher) were following four different interaction scripts, representing all combinations of mutual politeness and impoliteness of the robot and the customer. The participants (N=24, within-subject design) were assigned the role of an "included observer", that is, a fellow customer who was present in the situation without being actively involved in the interactions. The participants assessed how they experienced the interaction scenarios by providing Likert scale scores and free-text responses. The results indicate that while impolite robots' behavior was generally assessed negatively, it was commonly perceived as more justifiable and fairer if the robot was treated impolitely by the human. Politeness reciprocity expectations in the context of the social perception of robots are discussed., Comment: 19 pages, 5 figures, 7 tables

This paper investigates over-the-air computation (AirComp) over multiple-access time-varying channels, where devices with high mobility transmit their sensing data to a fusion center (FC) for averaging. To combat the Doppler shift induced by time-varying channels, each device adopts orthogonal time frequency space (OTFS) modulation. Our objective is minimizing the mean squared error (MSE) for the target function estimation. Due to the multipath time-varying channels, the OTFS-based AirComp not only suffers from noise but also interference. Specifically, we propose three schemes, namely S1, S2, and S3, for the target function estimation. S1 directly estimates the target function under the impacts of noise and interference. S2 mitigates the interference by introducing a zero padding-assisted OTFS. In S3, we propose an iterative algorithm to estimate the function in a matrix form. In the numerical results, we evaluate the performance of S1, S2, and S3 from the perspectives of MSE and computational complexity, and compare them with benchmarks. Specifically, compared to benchmarks, S3 outperforms them with a significantly lower MSE but incurs a higher computational complexity. In contrast, S2 demonstrates a reduction in both MSE and computational complexity. Lastly, S1 shows superior error performance at small SNR and reduced computational complexity., Comment: 15 pages, 11 figures, accepted by IEEE Transactions on Wireless Communications. Conference version is available at arXiv:2403.11272

There could be a significant population of double white dwarf binaries (DWDs) inside globular clusters (GCs), however, these are often too faint to be individually observed. We have utilized a large number GC models evolved with the Monte Carlo Cluster Simulator (MOCCA) code, to create a large statistical dataset of DWDs. These models include multiple-stellar populations, resulting in two distinct initial populations: one dense and another less dense. Due to the lower density of one population, a large number of objects escape during the early GC evolution, leading to a high mass-loss rate. In this dataset we have analysed three main groups of DWDs, namely in-cluster binaries, escaped binaries, and isolated evolution of primordial binaries. We compared the properties of these groups to observations of close and wide binaries. We find that the number of escaping DWDs is significantly larger than the number of in-cluster binaries and those that form via the isolated evolution of all promiridial binaries in our GC models. This suggests that dynamics play an important role in the formation of DWDs. For close binaries, we found a good agreement in the separations of escaped binaries and isolated binaries, but in-cluster binaries showed slight differences. We could not reproduce the observed extremely low mass WDs due to the limitations of our stellar and binary evolution prescriptions. For wide binaries, we also found a good agreement in the separations and masses, after accounting for observational selection effects. We conclude that, even though the current observational samples of DWDs are extremely biased and incomplete, our results compare reasonably well with observations., Comment: 13 pages, 8 figures

The safe integration of machine learning modules in decision-making processes hinges on their ability to quantify uncertainty. A popular technique to achieve this goal is conformal prediction (CP), which transforms an arbitrary base predictor into a set predictor with coverage guarantees. While CP certifies the predicted set to contain the target quantity with a user-defined tolerance, it does not provide control over the average size of the predicted sets, i.e., over the informativeness of the prediction. In this work, a theoretical connection is established between the generalization properties of the base predictor and the informativeness of the resulting CP prediction sets. To this end, an upper bound is derived on the expected size of the CP set predictor that builds on generalization error bounds for the base predictor. The derived upper bound provides insights into the dependence of the average size of the CP set predictor on the amount of calibration data, the target reliability, and the generalization performance of the base predictor. The theoretical insights are validated using simple numerical regression and classification tasks.

We derive generic information-theoretic and PAC-Bayesian generalization bounds involving an arbitrary convex comparator function, which measures the discrepancy between the training and population loss. The bounds hold under the assumption that the cumulant-generating function (CGF) of the comparator is upper-bounded by the corresponding CGF within a family of bounding distributions. We show that the tightest possible bound is obtained with the comparator being the convex conjugate of the CGF of the bounding distribution, also known as the Cram\'er function. This conclusion applies more broadly to generalization bounds with a similar structure. This confirms the near-optimality of known bounds for bounded and sub-Gaussian losses and leads to novel bounds under other bounding distributions., Comment: AISTATS 2024

Pancreatic ductal adenocarcinoma (PDAC) is a highly aggressive cancer with limited treatment options. This research proposes a workflow and deep learning-based segmentation models to automatically assess tumor-vessel involvement, a key factor in determining tumor resectability. Correct assessment of resectability is vital to determine treatment options. The proposed workflow involves processing CT scans to segment the tumor and vascular structures, analyzing spatial relationships and the extent of vascular involvement, which follows a similar way of working as expert radiologists in PDAC assessment. Three segmentation architectures (nnU-Net, 3D U-Net, and Probabilistic 3D U-Net) achieve a high accuracy in segmenting veins, arteries, and the tumor. The segmentations enable automated detection of tumor involvement with high accuracy (0.88 sensitivity and 0.86 specificity) and automated computation of the degree of tumor-vessel contact. Additionally, due to significant inter-observer variability in these important structures, we present the uncertainty captured by each of the models to further increase insights into the predicted involvement. This result provides clinicians with a clear indication of tumor-vessel involvement and may be used to facilitate more informed decision-making for surgical interventions. The proposed method offers a valuable tool for improving patient outcomes, personalized treatment strategies and survival rates in pancreatic cancer., Comment: ICCV CVAMD 2023

Over-the-air computation (OAC) is a promising wireless communication method for aggregating data from many devices in dense wireless networks. The fundamental idea of OAC is to exploit signal superposition to compute functions of multiple simultaneously transmitted signals. However, the time- and phase-alignment of these superimposed signals have a significant effect on the quality of function computation. In this study, we analyze the OAC problem for a system with unknown random time delays and phase shifts. We show that the classical matched filter does not produce optimal results, and generates bias in the function estimates. To counteract this, we propose a new filter design and show that, under a bound on the maximum time delay, it is possible to achieve unbiased function computation. Additionally, we propose a Tikhonov regularization problem that produces an optimal filter given a tradeoff between the bias and noise-induced variance of the function estimates. When the time delays are long compared to the length of the transmitted pulses, our filter vastly outperforms the matched filter both in terms of bias and mean-squared error (MSE). For shorter time delays, our proposal yields similar MSE as the matched filter, while reducing the bias., Comment: 7 pages, 4 figures, conference paper accepted for IEEE GLOBECOM 2023

A fundamental question in theoretical machine learning is generalization. Over the past decades, the PAC-Bayesian approach has been established as a flexible framework to address the generalization capabilities of machine learning algorithms, and design new ones. Recently, it has garnered increased interest due to its potential applicability for a variety of learning algorithms, including deep neural networks. In parallel, an information-theoretic view of generalization has developed, wherein the relation between generalization and various information measures has been established. This framework is intimately connected to the PAC-Bayesian approach, and a number of results have been independently discovered in both strands. In this monograph, we highlight this strong connection and present a unified treatment of PAC-Bayesian and information-theoretic generalization bounds. We present techniques and results that the two perspectives have in common, and discuss the approaches and interpretations that differ. In particular, we demonstrate how many proofs in the area share a modular structure, through which the underlying ideas can be intuited. We pay special attention to the conditional mutual information (CMI) framework; analytical studies of the information complexity of learning algorithms; and the application of the proposed methods to deep learning. This monograph is intended to provide a comprehensive introduction to information-theoretic generalization bounds and their connection to PAC-Bayes, serving as a foundation from which the most recent developments are accessible. It is aimed broadly towards researchers with an interest in generalization and theoretical machine learning., Comment: 228 pages

This article presents results from an experiment in which 44 human subjects interact with a dynamic system to perform 40 trials of a command-following task. The reference command is unpredictable and different on each trial, but all subjects have the same sequence of reference commands for the 40 trials. The subjects are divided into 4 groups of 11 subjects. One group performs the command-following task without preview of the reference command, and the other 3 groups are given preview of the reference command for different time lengths into the future (0.5 s, 1 s, 1.5 s). A subsystem identification algorithm is used to obtain best-fit models of each subject's control behavior on each trial. The time- and frequency-domain performance, as well as the identified models of the control behavior for the 4 groups are examined to investigate the effects of reference-command preview. The results suggest that preview tends to improve performance by allowing the subjects to compensate for sensory time delay and approximate the inverse dynamics in feedforward. However, too much preview may decrease performance by degrading the ability to use the correct phase lead in feedforward., Comment: Preprint submitted to IEEE Transactions on Cybernetics

We apply a global sensitivity method, the Hilbert-Schmidt independence criterion (HSIC), to the reparameterization of a Zn/S/H ReaxFF force field to identify the most appropriate parameters for reparameterization. Parameter selection remains a challenge in this context as high dimensional optimizations are prone to overfitting and take a long time, but selecting too few parameters leads to poor quality force fields. We show that the HSIC correctly and quickly identifies the most sensitive parameters, and that optimizations done using a small number of sensitive parameters outperform those done using a higher dimensional reasonable-user parameter selection. Optimizations using only sensitive parameters: 1) converge faster, 2) have loss values comparable to those found with the naive selection, 3) have similar accuracy in validation tests, and 4) do not suffer from problems of overfitting. We demonstrate that an HSIC global sensitivity is a cheap optimization pre-processing step that has both qualitative and quantitative benefits which can substantially simplify and speedup ReaxFF reparameterizations., Comment: author accepted manuscript

We consider online prediction of a binary sequence with expert advice. For this setting, we devise label-efficient forecasting algorithms, which use a selective sampling scheme that enables collecting much fewer labels than standard procedures, while still retaining optimal worst-case regret guarantees. These algorithms are based on exponentially weighted forecasters, suitable for settings with and without a perfect expert. For a scenario where one expert is strictly better than the others in expectation, we show that the label complexity of the label-efficient forecaster scales roughly as the square root of the number of rounds. Finally, we present numerical experiments empirically showing that the normalized regret of the label-efficient forecaster can asymptotically match known minimax rates for pool-based active learning, suggesting it can optimally adapt to benign settings.

Recent work has established that the conditional mutual information (CMI) framework of Steinke and Zakynthinou (2020) is expressive enough to capture generalization guarantees in terms of algorithmic stability, VC dimension, and related complexity measures for conventional learning (Harutyunyan et al., 2021, Haghifam et al., 2021). Hence, it provides a unified method for establishing generalization bounds. In meta learning, there has so far been a divide between information-theoretic results and results from classical learning theory. In this work, we take a first step toward bridging this divide. Specifically, we present novel generalization bounds for meta learning in terms of the evaluated CMI (e-CMI). To demonstrate the expressiveness of the e-CMI framework, we apply our bounds to a representation learning setting, with $n$ samples from $\hat n$ tasks parameterized by functions of the form $f_i \circ h$. Here, each $f_i \in \mathcal F$ is a task-specific function, and $h \in \mathcal H$ is the shared representation. For this setup, we show that the e-CMI framework yields a bound that scales as $\sqrt{ \mathcal C(\mathcal H)/(n\hat n) + \mathcal C(\mathcal F)/n} $, where $\mathcal C(\cdot)$ denotes a complexity measure of the hypothesis class. This scaling behavior coincides with the one reported in Tripuraneni et al. (2020) using Gaussian complexity., Comment: NeurIPS 2022

We present a new family of information-theoretic generalization bounds, in which the training loss and the population loss are compared through a jointly convex function. This function is upper-bounded in terms of the disintegrated, samplewise, evaluated conditional mutual information (CMI), an information measure that depends on the losses incurred by the selected hypothesis, rather than on the hypothesis itself, as is common in probably approximately correct (PAC)-Bayesian results. We demonstrate the generality of this framework by recovering and extending previously known information-theoretic bounds. Furthermore, using the evaluated CMI, we derive a samplewise, average version of Seeger's PAC-Bayesian bound, where the convex function is the binary KL divergence. In some scenarios, this novel bound results in a tighter characterization of the population loss of deep neural networks than previous bounds. Finally, we derive high-probability versions of some of these average bounds. We demonstrate the unifying nature of the evaluated CMI bounds by using them to recover average and high-probability generalization bounds for multiclass classification with finite Natarajan dimension., Comment: NeurIPS 2022