Search

Define an optimizer as having memory $k$ if it stores $k$ dynamically changing vectors in the parameter space. Classical SGD has memory $0$, momentum SGD optimizer has $1$ and Adam optimizer has $2$. We address the following questions: How can optimizers make use of more memory units? What information should be stored in them? How to use them for the learning steps? As an approach to the last question, we introduce a general method called "Retrospective Learning Law Correction" or shortly RLLC. This method is designed to calculate a dynamically varying linear combination (called learning law) of memory units, which themselves may evolve arbitrarily. We demonstrate RLLC on optimizers whose memory units have linear update rules and small memory ($\leq 4$ memory units). Our experiments show that in a variety of standard problems, these optimizers outperform the above mentioned three classical optimizers. We conclude that RLLC is a promising framework for boosting the performance of known optimizers by adding more memory units and by making them more adaptive.

Richmond and Richmond (American Mathematical Monthly 104 (1997), 713--719) proved the following theorem: If, in a metric space with at least five points, all triangles are degenerate, then the space is isometric to a subset of the real line. We prove that the hypothesis is unnecessarily strong: In a metric space on $n$ points, fewer than $7n^2/6$ suitably placed degenerate triangles suffice. However, fewer than $n(n-1)/2$ degenerate triangles, no matter how cleverly placed, never suffice., Comment: Supersedes arXiv:2209.14361 [math.MG]

We explore element-wise convex combinations of two permutation-aligned neural network parameter vectors $\Theta_A$ and $\Theta_B$ of size $d$. We conduct extensive experiments by examining various distributions of such model combinations parametrized by elements of the hypercube $[0,1]^{d}$ and its vicinity. Our findings reveal that broad regions of the hypercube form surfaces of low loss values, indicating that the notion of linear mode connectivity extends to a more general phenomenon which we call mode combinability. We also make several novel observations regarding linear mode connectivity and model re-basin. We demonstrate a transitivity property: two models re-based to a common third model are also linear mode connected, and a robustness property: even with significant perturbations of the neuron matchings the resulting combinations continue to form a working model. Moreover, we analyze the functional and weight similarity of model combinations and show that such combinations are non-vacuous in the sense that there are significant functional differences between the resulting models.

We employ a toolset -- dubbed Dr. Frankenstein -- to analyse the similarity of representations in deep neural networks. With this toolset, we aim to match the activations on given layers of two trained neural networks by joining them with a stitching layer. We demonstrate that the inner representations emerging in deep convolutional neural networks with the same architecture but different initializations can be matched with a surprisingly high degree of accuracy even with a single, affine stitching layer. We choose the stitching layer from several possible classes of linear transformations and investigate their performance and properties. The task of matching representations is closely related to notions of similarity. Using this toolset, we also provide a novel viewpoint on the current line of research regarding similarity indices of neural network representations: the perspective of the performance on a task., Comment: To appear in the Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021)

We have developed the polyatomic extension of a recently established (M. Gustafsson, J. Chem. Phys, 138, 074308 (2013)) classical theory of radiative association in the absence of electronic transitions. The cross section of the process is calculated by a quasiclassical trajectory method combined with the classical Larmor formula which can provide the radiated power in collisions. We have also proposed a double-layered Monte Carlo scheme for efficient computation of ro-vibrationally quantum state resolved cross sections for radiative association. Besides the method development, we also calculated and fitted the global potential energy and dipole surface for the H + CN collisions to test our polyatomic semiclassical method., Comment: 13 pages, 8 figures

When processing data streams with highly skewed and nonstationary key distributions, we often observe overloaded partitions when the hash partitioning fails to balance data correctly. To avoid slow tasks that delay the completion of the whole stage of computation, it is necessary to apply adaptive, on-the-fly partitioning that continuously recomputes an optimal partitioner, given the observed key distribution. While such solutions exist for batch processing of static data sets and stateless stream processing, the task is difficult for long-running stateful streaming jobs where key distribution changes over time. Careful checkpointing and operator state migration is necessary to change the partitioning while the operation is running. Our key result is a lightweight on-the-fly Dynamic Repartitioning (DR) module for distributed data processing systems (DDPS), including Apache Spark and Flink, which improves the performance with negligible overhead. DR can adaptively repartition data during execution using our Key Isolator Partitioner (KIP). In our experiments with real workloads and power-law distributions, we reach a speedup of 1.5-6 for a variety of Spark and Flink jobs., Comment: 14 pages, 8 figures

Polarizability is a key response property of physical and chemical systems, which has an impact on intermolecular interactions, spectroscopic observables, and vacuum polarization. The calculation of polarizability for quantum systems involves an infinite sum over all excited (bound and continuum) states, concealing the physical interpretation of polarization mechanisms and complicating the derivation of efficient response models. Approximate expressions for the dipole polarizability, $\alpha$, rely on different scaling laws $\alpha \propto$ $R^3$, $R^4$, or $R^7$, for various definitions of the system radius $R$. Here, we consider a range of single-particle quantum systems of varying spatial dimensionality and having qualitatively different spectra, demonstrating that their polarizability follows a universal four-dimensional scaling law $\alpha = C (4 \mu q^2/\hbar^2)L^4$, where $\mu$ and $q$ are the (effective) particle mass and charge, $C$ is a dimensionless excitation-energy ratio, and the characteristic length $L$ is defined via the $\mathcal{L}^2$-norm of the position operator. %The applicability of this unified formula is demonstrated by accurately predicting the dipole polarizability of 36 atoms and 1641 small organic~molecules. This unified formula is also applicable to many-particle systems, as shown by} accurately predicting the dipole polarizability of 36 atoms, 1641 small organic \rrr{molecules, and Bloch electrons in periodic systems., Comment: 3 figures are included

We use a differential constitutive equation to model the flow of a viscoelastic flow in a cross-slot geometry, which is known to exhibit bistability above a critical flow rate. The novelty lies in two asymmetric modifications to the geometry, which causes a change in the bifurcation diagram such that one of the stable solutions becomes disconnected from the solution at low flow speeds. First we show that it is possible to mirror one of the modifications such that the system can be forced to the disconnected solution. Then we show that a slow decrease of the flow rate, can cause the system to go through a drastic change on a short time scale, also known as a catastrophe. The short time scale could lead to a precise and simple experimental measurement of the flow conditions at which the viscoelastic catastrophe occurs. Since the phenomena is intrinsically related to the extensional rheology of the fluid, we propose to exploit the phenomena for in-line extensional rheometry.

One way to study the combinatorics of finite metric spaces is to study the betweenness relation associated with the metric space. In the hypergraph metrization problem, one has to find and characterize metric betweennesses whose collinear triples (or alternatively, non-degenerate triangles) coincide with the edges of a given $3$-uniform hypergraph. Metrizability of different kinds of hypergraphs was investigated in the last decades. Chen showed that steiner triple systems are not metrizable, while Richmond and Richmond characterized linear betweennesses, i.e. metric betweennesses that realize the complete $3$-uniform hypergraph. The latter result was also generalized to almost-metric betweennesses by Beaudou et al. In this paper, we further extend this theory by characterizing the largest nonlinear almost-metric betweennesses that satisfy certain hereditary properties, as well as the ones that contain a small linear number of non-degenerate triangles., Comment: 36 pages, 7 figures; technical proofs are relegated to a separate appendix

The betweenness structure of a finite metric space $M = (X, d)$ is a pair $\mathcal{B}(M) = (X,\beta_M)$ where $\beta_M$ is the so-called betweenness relation of $M$ that consists of point triplets $(x, y, z)$ such that $d(x, z) = d(x, y) + d(y, z)$. The underlying graph of a betweenness structure $\mathcal{B} = (X,\beta)$ is the simple graph $G(\mathcal{B}) = (X, E)$ where the edges are pairs of distinct points with no third point between them. A connected graph $G$ is uniquely representable if there exists a unique metric betweenness structure with underlying graph $G$. It was implied by previous works that trees are uniquely representable. In this paper, we give a characterization of uniquely representable graphs by showing that they are exactly the block graphs. Further, we prove that two related classes of graphs coincide with the class of block graphs and the class of distance-hereditary graphs, respectively. We show that our results hold not only for metric but also for almost-metric betweenness structures., Comment: 16 pages (without references); 3 figures; major changes: simplified proofs, improved notations and namings, short overview of metric graph theory

In their paper published in 1997, Richmond and Richmond classified metric spaces in which all triangles are degenerate. That result was later reproved by Dovgoshei and Dordovskii in the finite case and it was generalized to finite pseudometric betweennesses by Beaudou et al. In this paper, we give a new, independent proof to the finite case of the original theorem which we reformulate in terms of linearity of betweenness structures., Comment: 10 pages, 3 figures Keywords: Finite metric space, Metric betweenness, Linearity, Degenerate triangles triangles

A semicalssical method based on surface-hopping techniques is developed to model the dynamics of radiative association with electronic transitions in arbitrary polyatomic systems. It can be proven that our method is an extension of the established semiclassical formula used in the characterization of diatomic molecule- formation. Our model is tested for diatomic molecules. It gives the same cross sections as the former semiclassical formula, but contrary to the former method it allows us to follow the fate of the trajectories after the emission of a photon. This means that we can characterize the rovibrational states of the stabilized molecules: using semiclassial quantization we can obtain quantum state resolved cross sections or emission spectra for the radiative association process. The calculated semiclassical state resolved spectra show good agreement with the result of quantum mechanical perturbation theory. Furthermore our surface-hopping model is not only applicable for the description of radiative association but it can be use for semiclassical characterization of any molecular process where spontaneous emission occurs.