Search

The manifold hypothesis says that natural high-dimensional data is actually supported on or around a low-dimensional manifold. Recent success of statistical and learning-based methods empirically supports this hypothesis, due to outperforming classical statistical intuition in very high dimensions. A natural step for analysis is thus to assume the manifold hypothesis and derive bounds that are independent of any embedding space. Theoretical implications in this direction have recently been explored in terms of generalization of ReLU networks and convergence of Langevin methods. We complement existing results by providing theoretical statistical complexity results, which directly relates to generalization properties. In particular, we demonstrate that the statistical complexity required to approximate a class of bounded Sobolev functions on a compact manifold is bounded from below, and moreover that this bound is dependent only on the intrinsic properties of the manifold. These provide complementary bounds for existing approximation results for ReLU networks on manifolds, which give upper bounds on generalization capacity.

Imaging is a standard example of an inverse problem, where the task of reconstructing a ground truth from a noisy measurement is ill-posed. Recent state-of-the-art approaches for imaging use deep learning, spearheaded by unrolled and end-to-end models and trained on various image datasets. However, many such methods require the availability of ground truth data, which may be unavailable or expensive, leading to a fundamental barrier that can not be bypassed by choice of architecture. Unsupervised learning presents an alternative paradigm that bypasses this requirement, as they can be learned directly on noisy data and do not require any ground truths. A principled Bayesian approach to unsupervised learning is to maximize the marginal likelihood with respect to the given noisy measurements, which is intrinsically linked to classical variational regularization. We propose an unsupervised approach using maximum marginal likelihood estimation to train a convex neural network-based image regularization term directly on noisy measurements, improving upon previous work in both model expressiveness and dataset size. Experiments demonstrate that the proposed method produces priors that are near competitive when compared to the analogous supervised training method for various image corruption operators, maintaining significantly better generalization properties when compared to end-to-end methods. Moreover, we provide a detailed theoretical analysis of the convergence properties of our proposed algorithm.

Under spherical symmetry, we show that the weak cosmic censorship holds for the gravitational collapse of the Einstein-Maxwell-charged scalar field system. Namely, for this system, with generic initial data, the formed spacetime singularities are concealed inside black-hole regions. This generalizes Christodoulou's celebrated results to the charged case. Due to the presence of charge $Q$ and the complexification of the scalar field $\phi$, multiple delicate features and miraculous monotonic properties of the Einstein-(real) scalar field system are not present. We develop a systematical approach to incorporate $Q$ and the complex-valued $\phi$ into the integrated arguments. For instance, we discover a new path, employing the reduced mass ratio, to establish the sharp trapped surface formation criterion for the charged case. Due to the complex structure and the absence of translational symmetry of $\phi$, we also carry out detailed modified scale-critical BV area estimates with renormalized quantities to deal with $Q$ and $\phi$. We present a new $C^1$ extension criterion by utilizing the Doppler exponent to elucidate the blueshift effect, analogous to the role of integrating vorticity in the Beale-Kato-Madja breakdown criterion for incompressible fluids. Furthermore, by utilizing only double-null foliations, we establish the desired first and second instability theorems for the charged scenarios and identify generic initial conditions for the non-appearance of naked singularities. Our instability argument requires intricate generalizations of the treatment for the uncharged case via analyzing the precise contribution of the charged terms and its connection to the reduced mass ratio., Comment: 148 pages

Quantifying the value of data within a machine learning workflow can play a pivotal role in making more strategic decisions in machine learning initiatives. The existing Shapley value based frameworks for data valuation in machine learning are computationally expensive as they require considerable amount of repeated training of the model to obtain the Shapley value. In this paper, we introduce an efficient data valuation framework EcoVal, to estimate the value of data for machine learning models in a fast and practical manner. Instead of directly working with individual data sample, we determine the value of a cluster of similar data points. This value is further propagated amongst all the member cluster points. We show that the overall value of the data can be determined by estimating the intrinsic and extrinsic value of each data. This is enabled by formulating the performance of a model as a \textit{production function}, a concept which is popularly used to estimate the amount of output based on factors like labor and capital in a traditional free economic market. We provide a formal proof of our valuation technique and elucidate the principles and mechanisms that enable its accelerated performance. We demonstrate the real-world applicability of our method by showcasing its effectiveness for both in-distribution and out-of-sample data. This work addresses one of the core challenges of efficient data valuation at scale in machine learning models. The code is available at \underline{https://github.com/respai-lab/ecoval}., Comment: KDD-2024

Blazars, a subclass of active galactic nuclei (AGN), are known to be bright $\gamma$-ray sources, frequently exhibiting active (flaring) periods. The blazar PKS~2155-304 is a high synchrotron-peaked BL Lac object located at redshift $z=0.116$. On 2006 July 28, an extremely remarkable outburst of VHE $\gamma$-ray emission from this blazar was reported by the H.E.S.S. experiment, with an average flux more than 10 times the low-state level. The variability timescale of this extraordinary flare was as short as approximately 200~s. In order to guarantee the transparency of the emission region for TeV photons, the fast variability demands an extremely high Doppler factor $\delta_{\rm D}>50$ of the jet within the classical one-zone model, leading to the so-called "Doppler factor crisis". Here we demonstrate that the stochastic dissipation model, which is a multi-blob scenario for blazars, can self-consistently explain the giant TeV flares of PKS~2155-304 and the low-state emission before and after the flares, in terms of both multi-wavelength spectral and variability characteristics. The required Doppler factor in this model can be as low as 20, which is a reasonable and typical value for blazar jets. The obtained model parameters may shed some light on the physical properties of the relativistic jet., Comment: 14 pages, 12 figures, 4 tables

Unsupervised deep learning approaches have recently become one of the crucial research areas in imaging owing to their ability to learn expressive and powerful reconstruction operators even when paired high-quality training data is scarcely available. In this chapter, we review theoretically principled unsupervised learning schemes for solving imaging inverse problems, with a particular focus on methods rooted in optimal transport and convex analysis. We begin by reviewing the optimal transport-based unsupervised approaches such as the cycle-consistency-based models and learned adversarial regularization methods, which have clear probabilistic interpretations. Subsequently, we give an overview of a recent line of works on provably convergent learned optimization algorithms applied to accelerate the solution of imaging inverse problems, alongside their dedicated unsupervised training schemes. We also survey a number of provably convergent plug-and-play algorithms (based on gradient-step deep denoisers), which are among the most important and widely applied unsupervised approaches for imaging problems. At the end of this survey, we provide an overview of a few related unsupervised learning frameworks that complement our focused schemes. Together with a detailed survey, we provide an overview of the key mathematical results that underlie the methods reviewed in the chapter to keep our discussion self-contained.

In this work, a time-dependent modeling is developed to study the emission properties of blazars in the low state. Motivated by various observations, we speculate and assume that numerous discrete radiation zones throughout the jet of a blazar contribute to the broadband emission. We model the temporal evolution of the electron spectrum in each emission zone taking into account the injection, cooling and escape of relativistic electrons. By doing so, we are able to calculate the multi-wavelength emission of each radiation zone. The observed emission of a blazar is then the superposition of the emission from all discrete radiation zones. We revisit the multi-wavelength spectral energy distributions, light curves and polarisation under the model, and discuss its potential to reproduce the flat radio spectra, the core-shift phenomena, the minute-scale gamma-ray variability, and the large polarisation-angle swings, which are difficult to explain under the conventional one-zone models simultaneously., Comment: 20 pages, 16 figures, 1 table; accepted by MNRAS

We consider the problem of sampling from a distribution governed by a potential function. This work proposes an explicit score based MCMC method that is deterministic, resulting in a deterministic evolution for particles rather than a stochastic differential equation evolution. The score term is given in closed form by a regularized Wasserstein proximal, using a kernel convolution that is approximated by sampling. We demonstrate fast convergence on various problems and show improved dimensional dependence of mixing time bounds for the case of Gaussian distributions compared to the unadjusted Langevin algorithm (ULA) and the Metropolis-adjusted Langevin algorithm (MALA). We additionally derive closed form expressions for the distributions at each iterate for quadratic potential functions, characterizing the variance reduction. Empirical results demonstrate that the particles behave in an organized manner, lying on level set contours of the potential. Moreover, the posterior mean estimator of the proposed method is shown to be closer to the maximum a-posteriori estimator compared to ULA and MALA in the context of Bayesian logistic regression. Additional examples demonstrate competitive performance for Bayesian neural network training.

Learning-to-optimize (L2O) is an emerging research area in large-scale optimization with applications in data science. Recently, researchers have proposed a novel L2O framework called learned mirror descent (LMD), based on the classical mirror descent (MD) algorithm with learnable mirror maps parameterized by input-convex neural networks. The LMD approach has been shown to significantly accelerate convex solvers while inheriting the convergence properties of the classical MD algorithm. This work proposes several practical extensions of the LMD algorithm, addressing its instability, scalability, and feasibility for high-dimensional problems. We first propose accelerated and stochastic variants of LMD, leveraging classical momentum-based acceleration and stochastic optimization techniques for improving the convergence rate and per-iteration computational complexity. Moreover, for the particular application of training neural networks, we derive and propose a novel and efficient parameterization for the mirror potential, exploiting the equivariant structure of the training problems to significantly reduce the dimensionality of the underlying problem. We provide theoretical convergence guarantees for our schemes under standard assumptions and demonstrate their effectiveness in various computational imaging and machine learning applications such as image inpainting, and the training of support vector machines and deep neural networks.

Plug-and-Play (PnP) methods are a class of efficient iterative methods that aim to combine data fidelity terms and deep denoisers using classical optimization algorithms, such as ISTA or ADMM, with applications in inverse problems and imaging. Provable PnP methods are a subclass of PnP methods with convergence guarantees, such as fixed point convergence or convergence to critical points of some energy function. Many existing provable PnP methods impose heavy restrictions on the denoiser or fidelity function, such as non-expansiveness or strict convexity, respectively. In this work, we propose a novel algorithmic approach incorporating quasi-Newton steps into a provable PnP framework based on proximal denoisers, resulting in greatly accelerated convergence while retaining light assumptions on the denoiser. By characterizing the denoiser as the proximal operator of a weakly convex function, we show that the fixed points of the proposed quasi-Newton PnP algorithm are critical points of a weakly convex function. Numerical experiments on image deblurring and super-resolution demonstrate 2--8x faster convergence as compared to other provable PnP methods with similar reconstruction quality.

Based on the first 13.4 yr of Fermi science data in the energy range from 300 MeV to 500 GeV, we discovered a bright GeV gamma-ray source with a $\sim$5.64 $\sigma$ detection, named Fermi J1242.5+3236, which has an offset of about 0fdg0996 from a nearby star-forming galaxy NGC 4631. When using the 12-yr data, the detection significance of Fermi J1242.5+3236 is about 4.72 $\sigma$. Fermi J1242.5+3236 is a steady point source without significant temporal variability and has a hard gamma-ray photon index of about -1.60 $\pm$0.24. The spatial offset and the hard gamma-ray spectrum disfavor this source as the diffuse gamma-ray emission from this galaxy. This new source might have a possible origin of an unidentified background blazar, which is more likely a high-synchrotron-peaked blazar for its hard gamma-ray photon index. A follow-up optical observation would help distinguish origin of Fermi J1242.5+3236., Comment: 8 pages,6 figures, published in Research in Astronomy and Astrophysics

Learning-to-optimize is an emerging framework that leverages training data to speed up the solution of certain optimization problems. One such approach is based on the classical mirror descent algorithm, where the mirror map is modelled using input-convex neural networks. In this work, we extend this functional parameterization approach by introducing momentum into the iterations, based on the classical accelerated mirror descent. Our approach combines short-time accelerated convergence with stable long-time behavior. We empirically demonstrate additional robustness with respect to multiple parameters on denoising and deconvolution experiments., Comment: Note inconsistency with ICASSP paper for step-size choice in (4c) and associated Alg. 1, this version is correct with step-size kt/r

Learning-to-optimize is an emerging framework that seeks to speed up the solution of certain optimization problems by leveraging training data. Learned optimization solvers have been shown to outperform classical optimization algorithms in terms of convergence speed, especially for convex problems. Many existing data-driven optimization methods are based on parameterizing the update step and learning the optimal parameters (typically scalars) from the available data. We propose a novel functional parameterization approach for learned convex optimization solvers based on the classical mirror descent (MD) algorithm. Specifically, we seek to learn the optimal Bregman distance in MD by modeling the underlying convex function using an input-convex neural network (ICNN). The parameters of the ICNN are learned by minimizing the target objective function evaluated at the MD iterate after a predetermined number of iterations. The inverse of the mirror map is modeled approximately using another neural network, as the exact inverse is intractable to compute. We derive convergence rate bounds for the proposed learned mirror descent (LMD) approach with an approximate inverse mirror map and perform extensive numerical evaluation on various convex problems such as image inpainting, denoising, learning a two-class support vector machine (SVM) classifier and a multi-class linear classifier on fixed features.