Search

Previous work has cast doubt on the general framework of uniform convergence and its ability to explain generalization in neural networks. By considering a specific dataset, it was observed that a neural network completely misclassifies a projection of the training data (adversarial set), rendering any existing generalization bound based on uniform convergence vacuous. We provide an extensive theoretical investigation of the previously studied data setting through the lens of infinitely-wide models. We prove that the Neural Tangent Kernel (NTK) also suffers from the same phenomenon and we uncover its origin. We highlight the important role of the output bias and show theoretically as well as empirically how a sensible choice completely mitigates the problem. We identify sharp phase transitions in the accuracy on the adversarial set and study its dependency on the training sample size. As a result, we are able to characterize critical sample sizes beyond which the effect disappears. Moreover, we study decompositions of a neural network into a clean and noisy part by considering its canonical decomposition into its different eigenfunctions and show empirically that for too small bias the adversarial phenomenon still persists.

Recently, FGSM adversarial training is found to be able to train a robust model which is comparable to the one trained by PGD but an order of magnitude faster. However, there is a failure mode called catastrophic overfitting (CO) that the classifier loses its robustness suddenly during the training and hardly recovers by itself. In this paper, we find CO is not only limited to FGSM, but also happens in $\mbox{DF}^{\infty}$-1 adversarial training. Then, we analyze the geometric properties for both FGSM and $\mbox{DF}^{\infty}$-1 and find they have totally different decision boundaries after CO. For FGSM, a new decision boundary is generated along the direction of perturbation and makes the small perturbation more effective than the large one. While for $\mbox{DF}^{\infty}$-1, there is no new decision boundary generated along the direction of perturbation, instead the perturbation generated by $\mbox{DF}^{\infty}$-1 becomes smaller after CO and thus loses its effectiveness. We also experimentally analyze three hypotheses on potential factors causing CO. And then based on the empirical analysis, we modify the RS-FGSM by not projecting perturbation back to the $l_\infty$ ball. By this small modification, we could achieve $47.56 \pm 0.37\% $ PGD-50-10 accuracy on CIFAR10 with $\epsilon=8/255$ in contrast to $43.57 \pm 0.30\% $ by RS-FGSM and also further extend the working range of $\epsilon$ from 8/255 to 11/255 on CIFAR10 without CO occurring.

Important advances have recently been achieved in developing procedures yielding uniformly valid inference for a low dimensional causal parameter when high-dimensional nuisance models must be estimated. In this paper, we review the literature on uniformly valid causal inference and discuss the costs and benefits of using uniformly valid inference procedures. Naive estimation strategies based on regularisation, machine learning, or a preliminary model selection stage for the nuisance models have finite sample distributions which are badly approximated by their asymptotic distributions. To solve this serious problem, estimators which converge uniformly in distribution over a class of data generating mechanisms have been proposed in the literature. In order to obtain uniformly valid results in high-dimensional situations, sparsity conditions for the nuisance models need typically to be made, although a double robustness property holds, whereby if one of the nuisance model is more sparse, the other nuisance model is allowed to be less sparse. While uniformly valid inference is a highly desirable property, uniformly valid procedures pay a high price in terms of inflated variability. Our discussion of this dilemma is illustrated by the study of a double-selection outcome regression estimator, which we show is uniformly asymptotically unbiased, but is less variable than uniformly valid estimators in the numerical experiments conducted.

The underspecification of most machine learning pipelines means that we cannot rely solely on validation performance to assess the robustness of deep learning systems to naturally occurring distribution shifts. Instead, making sure that a neural network can generalize across a large number of different situations requires to understand the specific way in which it solves a task. In this work, we propose to study this problem from a geometric perspective with the aim to understand two key characteristics of neural network solutions in underspecified settings: how is the geometry of the learned function related to the data representation? And, are deep networks always biased towards simpler solutions, as conjectured in recent literature? We show that the way neural networks handle the underspecification of these problems is highly dependent on the data representation, affecting both the geometry and the complexity of the learned predictors. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems., Comment: Presented as a RobustML workshop paper at ICLR 2021

In this paper, we introduce SciGen, a new challenge dataset for the task of reasoning-aware data-to-text generation consisting of tables from scientific articles and their corresponding descriptions. Describing scientific tables goes beyond the surface realization of the table content and requires reasoning over table values. The unique properties of SciGen are that (1) tables mostly contain numerical values, and (2) the corresponding descriptions require arithmetic reasoning. SciGen is therefore the first dataset that assesses the arithmetic reasoning capabilities of generation models on complex input structures, i.e., tables from scientific articles. We study the effectiveness of state-of-the-art data-to-text generation models on SciGen and evaluate the results using common metrics as well as human evaluation. Our results and analyses show that (a) while humans like to reason for describing scientific tables, the ability of state-of-the-art models is severely limited on this task, (b) while adding more training data improves the results, it is not the solution for reasoning-aware text generation, and (c) one of the main bottlenecks for this task is the lack of proper automatic evaluation metrics. The data, code, and annotations for human evaluation will be available at https://github.com/UKPLab/SciGen. SciGen opens new avenues for future research in reasoning-aware text generation and evaluation.

Tomonaga-Luttinger liquids (TLLs) can be used to effectively describe one-dimensional quantum many-body systems such as ultracold atoms, charges in nanowires, superconducting circuits, and gapless spin chains. Their properties are given by two parameters, the propagation velocity and the Luttinger parameter. Here we study inhomogeneous TLLs where these are promoted to functions of position and demonstrate that they profoundly affect the dynamics: In general, besides curving the light cone, we show that propagation is no longer ballistically localized to the light-cone trajectories, different from standard homogeneous TLLs. Specifically, if the Luttinger parameter depends on position, the dynamics features pronounced spreading into the light cone, which cannot be understood via a simple superposition of waves as in the Huygens-Fresnel principle. This is the case for ultracold atoms in a parabolic trap, which serves as our main motivation, and we discuss possible experimental observations in such systems., Comment: 23 pages, RevTeX, 10 figures; final published version

The state-of-the-art on basic, single-antecedent anaphora has greatly improved in recent years. Researchers have therefore started to pay more attention to more complex cases of anaphora such as split-antecedent anaphora, as in Time-Warner is considering a legal challenge to Telecommunications Inc's plan to buy half of Showtime Networks Inc-a move that could lead to all-out war between the two powerful companies. Split-antecedent anaphora is rarer and more complex to resolve than single-antecedent anaphora; as a result, it is not annotated in many datasets designed to test coreference, and previous work on resolving this type of anaphora was carried out in unrealistic conditions that assume gold mentions and/or gold split-antecedent anaphors are available. These systems also focus on split-antecedent anaphors only. In this work, we introduce a system that resolves both single and split-antecedent anaphors, and evaluate it in a more realistic setting that uses predicted mentions. We also start addressing the question of how to evaluate single and split-antecedent anaphors together using standard coreference evaluation metrics., Comment: accepted at NAACL 2021

We study the impact of intrinsic strange (IS) component of nucleon sea on the global analysis of parton distribution functions (PDFs) considering a wide range of experimental data. To this aim, we consider two scenarios on the basis of BHPS model results for the IS distribution. In the first scenario, we apply the results presented through the BHPS model and in the second scenario we use its evolved distributions. For each scenarios, we present the limit of the IS probability $ {\cal P}_5^{s\bar{s}} $ for the standard tolerance criteria $ \Delta\chi^2=1 $ and $ 18.112 $ at $ 1\sigma $ and $ 4\sigma $ levels. Our results show that the experimental data can tolerate an IS component with a greater probability $ {\cal P}_5^{s\bar{s}} $ if one employs the second scenario. We obtain $ {\cal P}_5^{s\bar{s}}\approx 0.01 $ and $ {\cal P}_5^{s\bar{s}}\approx 0.025 $ for $ \Delta\chi^2=1 $ and $18.112$, respectively, at the $ 4\sigma $ level. We also calculate the ratio of strange-to-light sea-quark densities $r_s$ in the proton both including and excluding the IS component. Our results show that one can obtain a higher value for the ratio $r_s$ if the IS component is included., Comment: 12 pages, 5 figs , 2 tables

Identifying driving styles is the task of analyzing the behavior of drivers in order to capture variations that will serve to discriminate different drivers from each other. This task has become a prerequisite for a variety of applications, including usage-based insurance, driver coaching, driver action prediction, and even in designing autonomous vehicles; because driving style encodes essential information needed by these applications. In this paper, we present a deep-neural-network architecture, we term D-CRNN, for building high-fidelity representations for driving style, that combine the power of convolutional neural networks (CNN) and recurrent neural networks (RNN). Using CNN, we capture semantic patterns of driver behavior from trajectories (such as a turn or a braking event). We then find temporal dependencies between these semantic patterns using RNN to encode driving style. We demonstrate the effectiveness of these techniques for driver identification by learning driving style through extensive experiments conducted on several large, real-world datasets, and comparing the results with the state-of-the-art deep-learning and non-deep-learning solutions. These experiments also demonstrate a useful example of bias removal, by presenting how we preprocess the input data by sampling dissimilar trajectories for each driver to prevent spatial memorization. Finally, this paper presents an analysis of the contribution of different attributes for driver identification; we find that engine RPM, Speed, and Acceleration are the best combination of features., Comment: 12 pages, research on driving style representation

We study the inclusive production of bottom-flavored hadrons from semileptonic decays of polarized top quarks at next-to-leading order in QCD using fragmentation functions recently determined from a global fit to $e^+e^-$ data. We provide the relevant differential decay widths at parton level in analytic form. These results fill an important gap in the theoretical interpretation of recent measurements of the top-quark polarization and the $t\bar{t}$ spin correlations using dilepton final states in proton-proton collisions at the CERN Large Hadron Collider. As an illustration, we study the distributions in the scaled bottom-hadron energy of the polarized-top-quark decay widths for different $W$-boson helicities., Comment: 16 pages, 6 figures; to appear in Physical Review D

Financial regulators have long-standing concerns about fully decentralized exchanges that run 'on-chain' without any obvious regulatory hooks. The popularity of Uniswap, an automated market makers (AMM), made these concerns a reality. AMMs implement a lightweight dealer-based trading system, but they are unlike anything on Wall Street, require fees intrinsically, and are susceptible to front-running attacks. This leaves the following research questions we address in this paper: (1) are conventional (i.e., order books), secure (i.e., resistant to front-running and price manipulation) and fully decentralized exchanges feasible on a public blockchain like Ethereum, (2) what is the performance profile, and (3) how much do Layer 2 techniques (e.g., Arbitrum) increase performance? To answer these questions, we implement, benchmark, and experiment with an Ethereum-based call market exchange called Lissy. We confirm the functionality is too heavy for Ethereum today (you cannot expect to exceed a few hundred trade executions per block) but show it scales dramatically (99.88% gas cost reduction) on Arbitrum.

Coreference resolution is essential for natural language understanding and has been long studied in NLP. In recent years, as the format of Question Answering (QA) became a standard for machine reading comprehension (MRC), there have been data collection efforts, e.g., Dasigi et al. (2019), that attempt to evaluate the ability of MRC models to reason about coreference. However, as we show, coreference reasoning in MRC is a greater challenge than earlier thought; MRC datasets do not reflect the natural distribution and, consequently, the challenges of coreference reasoning. Specifically, success on these datasets does not reflect a model's proficiency in coreference reasoning. We propose a methodology for creating MRC datasets that better reflect the challenges of coreference reasoning and use it to create a sample evaluation set. The results on our dataset show that state-of-the-art models still struggle with these phenomena. Furthermore, we develop an effective way to use naturally occurring coreference phenomena from existing coreference resolution datasets when training MRC models. This allows us to show an improvement in the coreference reasoning abilities of state-of-the-art models. The code and the resulting dataset are available at https://github.com/UKPLab/coref-reasoning-in-qa., Comment: Accepted at ACL-IJCNLP 2021

We revisit the stability (instability) of the outer (inner) catenoid connecting two concentric circular rings and give an explicit new construction of the unstable mode of the inner catenoid by studying the spectrum of an exactly solvable one-dimensional Schr\"odinger operator with an asymmetric Darboux-P\"oschl-Teller potential., Comment: 9 pages, LaTeX, 2 figures; minor updates and corrections; final published version