Search

We analytically investigate how over-parameterization of models in randomized machine learning algorithms impacts the information leakage about their training data. Specifically, we prove a privacy bound for the KL divergence between model distributions on worst-case neighboring datasets, and explore its dependence on the initialization, width, and depth of fully connected neural networks. We find that this KL privacy bound is largely determined by the expected squared gradient norm relative to model parameters during training. Notably, for the special setting of linearized network, our analysis indicates that the squared gradient norm (and therefore the escalation of privacy loss) is tied directly to the per-layer variance of the initialization distribution. By using this analysis, we demonstrate that privacy bound improves with increasing depth under certain initializations (LeCun and Xavier), while degrades with increasing depth under other initializations (He and NTK). Our work reveals a complex interplay between privacy and depth that depends on the chosen initialization distribution. We further prove excess empirical risk bounds under a fixed KL privacy budget, and show that the interplay between privacy utility trade-off and depth is similarly affected by the initialization.

This paper provides statistical sample complexity bounds for score-matching and its applications in causal discovery. We demonstrate that accurate estimation of the score function is achievable by training a standard deep ReLU neural network using stochastic gradient descent. We establish bounds on the error rate of recovering causal relationships using the score-matching-based causal discovery method of Rolland et al. [2022], assuming a sufficiently good estimation of the score function. Finally, we analyze the upper bound of score-matching estimation within the score-based generative modeling, which has been applied for causal discovery but is also of independent interest within the domain of generative models., Comment: Accepted in NeurIPS 2023

This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification while obtaining (nearly) zero-training error under the lazy training regime. For this purpose, we unify three interrelated concepts of overparameterization, benign overfitting, and the Lipschitz constant of DNNs. Our results indicate that interpolating with smoother functions leads to better generalization. Furthermore, we investigate the special case where interpolating smooth ground-truth functions is performed by DNNs under the Neural Tangent Kernel (NTK) regime for generalization. Our result demonstrates that the generalization error converges to a constant order that only depends on label noise and initialization noise, which theoretically verifies benign overfitting. Our analysis provides a tight lower bound on the normalized margin under non-smooth activation functions, as well as the minimum eigenvalue of NTK under high-dimensional settings, which has its own interest in learning theory., Comment: Accepted in ICML 2023

The equation of state of neutron-rich nuclear matter is of interest to both nuclear physics and astrophysics. We have demonstrated the consistency between laboratory and astrophysical nuclear matter in neutron stars by considering low-density nuclear physics constraints (from $^{208}$Pb neutron-skin thickness) and high-density astrophysical constraints (from neutron star global properties). We have used both quark-level and hadron-level models, taking the quark mean-field (QMF) model and the relativistic mean-field (RMF) model as examples, respectively. We have constrained the equation of states of neutron stars and some key nuclear matter parameters within the Bayesian statistical approach, using the first multi-messenger event GW170817/AT 2017gfo, as well as the mass-radius simultaneous measurements of PSR J0030+0451 and PSR J0740+6620 from NICER, and the neutron-skin thickness of $^{208}$Pb from both PREX-II measurement and ab initio calculations. Our results show that, compared with the RMF model, QMF model's direct coupling of quarks with mesons and gluons leads to the evolution of the in-medium nucleon mass with the quark mass correction. This feature enables QMF model a wider range of model applicability, as shown by a slow drop of the nucleon mass with density and a large value at saturation that is jointly constrained by nuclear physics and astronomy., Comment: 14 pages, 4 figures, 3 tables, Phys. Rev. C (2023) accepted

Automatic failure diagnosis is crucial for large microservice systems. Currently, most failure diagnosis methods rely solely on single-modal data (i.e., using either metrics, logs, or traces). In this study, we conduct an empirical study using real-world failure cases to show that combining these sources of data (multimodal data) leads to a more accurate diagnosis. However, effectively representing these data and addressing imbalanced failures remain challenging. To tackle these issues, we propose DiagFusion, a robust failure diagnosis approach that uses multimodal data. It leverages embedding techniques and data augmentation to represent the multimodal data of service instances, combines deployment data and traces to build a dependency graph, and uses a graph neural network to localize the root cause instance and determine the failure type. Our evaluations using real-world datasets show that DiagFusion outperforms existing methods in terms of root cause instance localization (improving by 20.9% to 368%) and failure type determination (improving by 11.0% to 169%).

The observations of optical and near-infrared counterparts of binary neutron star mergers not only enrich our knowledge about the abundance of heavy elements in the Universe, or help reveal the remnant object just after the merger as generally known, but also can effectively constrain dense nuclear matter properties and the equation of state (EOS) in the interior of the merging stars. Following the relativistic mean-field description of nuclear matter, we perform the Bayesian inference of the EOS and the nuclear matter properties using the first multi-messenger event GW170817/AT2017gfo, together with the NICER mass-radius measurements of pulsars. The kilonova is described by a radiation-transfer model with the dynamical ejecta, and light curves connect with the EOS through the quasi-universal relations between the ejecta properties (the ejected mass, velocity, opacity or electron fraction) and binary parameters (the mass ratio and reduced tidal deformability). It is found that the posterior distributions of the reduced tidal deformability from the AT2017gfo analysis display a bimodal structure, with the first peak enhanced by the GW170817 data, leading to slightly softened posterior EOSs, while the second peak cannot be achieved by a nuclear EOS with saturation properties in their empirical ranges. The inclusion of NICER data in our analyses results in stiffened EOS posterior because of the massive pulsar PSR J0740+6620. We give results at nuclear saturation density for the nuclear incompressibility, the symmetry energy and its slope, as well as the nucleon effective mass, from our analysis of those observational data., Comment: Added discussions of PREX-II and GW190814, accepted by APJ

Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks (PNNs). In this work, we derive the finite-width NTK formulation for a special class of NNs-Hp, i.e., polynomial neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK. Based on our results, we elucidate the separation of PNNs over standard neural networks with respect to extrapolation and spectral bias. Our two key insights are that when compared to standard neural networks, PNNs can fit more complicated functions in the extrapolation regime and admit a slower eigenvalue decay of the respective NTK, leading to a faster learning towards high-frequency functions. Besides, our theoretical results can be extended to other types of NNs-Hp, which expand the scope of our work. Our empirical results validate the separations in broader classes of NNs-Hp, which provide a good justification for a deeper understanding of neural architectures.

We study the average robustness notion in deep neural networks in (selected) wide and narrow, deep and shallow, as well as lazy and non-lazy training settings. We prove that in the under-parameterized setting, width has a negative effect while it improves robustness in the over-parameterized setting. The effect of depth closely depends on the initialization and the training mode. In particular, when initialized with LeCun initialization, depth helps robustness with the lazy training regime. In contrast, when initialized with Neural Tangent Kernel (NTK) and He-initialization, depth hurts the robustness. Moreover, under the non-lazy training regime, we demonstrate how the width of a two-layer ReLU network benefits robustness. Our theoretical developments improve the results by [Huang et al. NeurIPS21; Wu et al. NeurIPS21] and are consistent with [Bubeck and Sellke NeurIPS21; Bubeck et al. COLT21]., Comment: Accepted in NeurIPS 2022

Neural Architecture Search (NAS) has fostered the automatic discovery of state-of-the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generalization properties of NAS under a unifying framework enabling (deep) layer skip connection search and activation function search. To this end, we derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime using a certain search space including mixed activation functions, fully connected, and residual neural networks. We use the minimum eigenvalue to establish generalization error bounds of NAS in the stochastic gradient descent training. Importantly, we theoretically and experimentally show how the derived results can guide NAS to select the top-performing architectures, even in the case without training, leading to a train-free algorithm based on our theory. Accordingly, our numerical validation shed light on the design of computationally efficient methods for NAS. Our analysis is non-trivial due to the coupling of various architectures and activation functions under the unifying framework and has its own interest in providing the lower bound of the minimum eigenvalue of NTK in deep learning theory., Comment: Accepted in NeurIPS 2022

Based on the GW dispersion relation raised in [1], we investigate the possible reflection of gravitational wave (GW) by superfluidity (SF) in the neutron star, provided its high density and dissipationless properties. Following this scenario, an experimental proposal is raised to probe the expected SF in neutron star by means of GW detection. Two types of binary systems are considered, neutron star-black hole and binary neutron star systems, with weak gravitational field condition imposed. Non-negligible modulation on the total signal caused by the GW reflection is found, which contributes amplitude and phase variations distinguishable from the primitive sine signal. Furthermore, we show that it is possible for such modulations to be detected by the Cosmic Explorer and Einstein Telescope at $100\,\mbox{Mpc}$. Identification of those signals can evince the existence of the long-sought SF in neutron stars as well as the exotic superfluidity-induced GW reflection., Comment: 5 pages, 5 figures

While the class of Polynomial Nets demonstrates comparable performance to neural networks (NN), it currently has neither theoretical generalization characterization nor robustness guarantees. To this end, we derive new complexity bounds for the set of Coupled CP-Decomposition (CCP) and Nested Coupled CP-decomposition (NCP) models of Polynomial Nets in terms of the $\ell_\infty$-operator-norm and the $\ell_2$-operator norm. In addition, we derive bounds on the Lipschitz constant for both models to establish a theoretical certificate for their robustness. The theoretical results enable us to propose a principled regularization scheme that we also evaluate experimentally in six datasets and show that it improves the accuracy as well as the robustness of the models to adversarial perturbations. We showcase how this regularization can be combined with adversarial training, resulting in further improvements.

The hypothesis that strange quark matter is the true ground state of matter has been investigated for almost four decades, but only a few works have explored the dynamics of binary systems of quark stars. This is partly due to the numerical challenges that need to be faced when modelling the large discontinuities at the surface of these stars. We here present a novel technique in which the EOS of a quark star is suitably rescaled to produce a smooth change of the specific enthalpy across a very thin crust. The introduction of the crust has been carefully tested by considering the oscillation properties of isolated quark stars, showing that the response of the simulated quark stars matches accurately the perturbative predictions. Using this technique, we have carried out the first fully general-relativistic simulations of the merger of quark-star binaries finding several important differences between quark-star binaries and hadronic-star binaries with the same mass and comparable tidal deformability. In particular, we find that dynamical mass loss is significantly suppressed in quark-star binaries. In addition, quark-star binaries have merger and post-merger frequencies that obey the same quasi-universal relations derived from hadron stars if expressed in terms of the tidal deformability, but not when expressed in terms of the average stellar compactness. Hence, it may be difficult to distinguish the two classes of stars if no information on the stellar radius is available. Finally, differences are found in the distributions in velocity and entropy of the ejected matter, for which quark-stars have much smaller tails. Whether these differences in the ejected matter will leave an imprint in the electromagnetic counterpart and nucleosynthetic yields remains unclear, calling for the construction of an accurate model for the evaporation of the ejected quarks into nucleons., Comment: Revised estimates on ejection of matter; differences with neutron stars much smaller. PRD in press

We extend the quark mean-field (QMF) model for nuclear matter and study the possible presence of quark matter inside the cores of neutron stars. A sharp first-order hadron-quark phase transition is implemented combining the QMF for the hadronic phase with "constant-speed-of-sound" parametrization for the high-density quark phase. The interplay of the nuclear symmetry energy slope parameter, $L$, and the dimensionless phase transition parameters (the transition density $n_{\rm trans}/n_0$, the transition strength $\Delta\varepsilon/\varepsilon_{\rm trans}$, and the sound speed squared in quark matter $c^2_{\rm QM}$) are then systematically explored for the hybrid star proprieties, especially the maximum mass $M_{\rm max}$ and the radius and the tidal deformability of a typical $1.4 \,M_{\odot}$ star. We show the strong correlation between the symmetry energy slope $L$ and the typical stellar radius $R_{1.4}$, similar to that previously found for neutron stars without a phase transition. With the inclusion of phase transition, we obtain robust limits on the maximum mass ($M_{\rm max}< 3.6 \,M_{\odot}$) and the radius of $1.4 \,M_{\odot}$ stars ($R_{1.4}\gtrsim 9.6~\rm km$), and we find that a too-weak ($\Delta\varepsilon/\varepsilon_{\rm trans}\lesssim 0.2$) phase transition taking place at low densities $\lesssim 1.3-1.5 \, n_0$ is strongly disfavored. We also demonstrate that future measurements of the radius and tidal deformability of $\sim 1.4 \,M_{\odot}$ stars, as well as the mass measurement of very massive pulsars, can help reveal the presence and amount of quark matter in compact objects., Comment: 15 pages, 12 figures, 1 table, version accepted for publication in ApJ

Detection of gravitational waves (GWs) produced by coalescence of compact binaries provides a novel way to measure the luminosity distance of GW events. Combining their redshift, they can act as standard sirens to constrain cosmological parameters. For various GW detector networks in 2nd-generation (2G), 2.5G and 3G, we comprehensively analyze the method to constrain the equation-of-state (EOS) of binary neutron-stars (BNSs) and extract their redshifts through the imprints of tidal effects in GW waveforms. We find for these events, the observations of electromagnetic counterparts in low-redshift range $z < 0.1$ are important for constraining the tidal effects. Considering 17 different EOSs of NSs or quark-stars, we find GW observations have strong capability to determine the EOS. Applying the events as standard sirens, and considering the constraints of NS's EOS derived from low-redshift observations as prior, we can constrain the dark-energy EOS parameters $w_0$ and $w_a$. In 3G era, the potential constraints are $\Delta w_0\in (0.0006,0.004)$ and $\Delta w_a\in(0.004,0.02)$, which are 1-3 orders smaller than those from traditional methods, including Type Ia supernovas and baryon acoustic oscillations. The constraints are also 1 order smaller than the method of GW standard siren by fixing the redshifts through short-hard $\gamma$-ray bursts, due to more available GW events in this method. Therefore, GW standard sirens, based on the tidal effect measurement, provide a realizable and much more powerful tool in cosmology., Comment: 28 pages, 23 figures. Accepted for publication in ApJS

The evolution of the gravitational-wave phase in the signal produced by inspiralling binaries of compact stars is modified by the nonzero deformability of the two stars. Hence, the measurement of these corrections has the potential of providing important information on the equation of state of nuclear matter. Extensive work has been carried out over the last decade to quantify these corrections, but it has so far been restricted to stars with zero intrinsic magnetic fields. While the corrections introduced by the magnetic tension and magnetic pressure are expected to be subdominant, it is nevertheless useful to determine the precise conditions under which these corrections become important. To address this question, we have carried out a second-order perturbative analysis of the tidal deformability of magnetised compact stars under a variety of magnetic-field strengths and equations of state describing either neutron stars or quark stars. Overall, we find that magnetically induced corrections to the tidal deformability will produce changes in the gravitational-wave phase evolution that are unlikely to be detected for realistic magnetic field i.e., $B\sim 10^{10} - 10^{12}\,{\rm G}$. At the same time, if the magnetic field is unrealistically large, i.e., $B\sim 10^{16}\,{\rm G}$, these corrections would produce a sizeable contribution to the phase evolution, especially for quark stars. In the latter case, the induced phase differences would represent a unique tool to measure the properties of the magnetic fields, providing information that is otherwise hard to quantify., Comment: 21 pages, 6 figures

In many real-time applications, the deployment of deep neural networks is constrained by high computational cost and efficient lightweight neural networks are widely concerned. In this paper, we propose that depthwise convolution (DWC) is used to expand the number of channels in a bottleneck block, which is more efficient than 1 x 1 convolution. The proposed Pointwise-Standard-Depthwise network (PSDNet) based on channel expansion with DWC has fewer number of parameters, less computational cost and higher accuracy than corresponding ResNet on CIFAR datasets. To design more efficient lightweight concolutional neural netwok, Depthwise-Pointwise-Depthwise inverted bottleneck block (DPD block) is proposed and DPDNet is designed by stacking DPD block. Meanwhile, the number of parameters of DPDNet is only about 60% of that of MobileNetV2 for networks with the same number of layers, but can achieve approximated accuracy. Additionally, two hyperparameters of DPDNet can make the trade-off between accuracy and computational cost, which makes DPDNet suitable for diverse tasks. Furthermore, we find the networks with more DWC layers outperform the networks with more 1x1 convolution layers, which indicates that extracting spatial information is more important than combining channel information.

The phase state of dense matter in the intermediate density range ($\sim$1-10 times the nuclear saturation density) is both intriguing and unclear and could have important observable effects in the present gravitational wave era of neutron stars. As the matter density increases in compact stars, the sound velocity is expected to approach the conformal limit ($c_s/c=1/\sqrt{3}$) at high densities and should also fulfill the causality limit ($c_s/c<1$). However, its detailed behavior remains a hot topic of debate. It was suggested that the sound velocity of dense matter could be an important indicator for a deconfinement phase transition, where a particular shape might be expected for its density dependence. In this work, we explore the general properties of the sound velocity and the adiabatic index of dense matter in hybrid stars, as well as in neutron stars and quark stars. Various conditions are employed for hadron-quark phase transition with varying interface tension. We find that the expected behavior of the sound velocity can also be achieved by the nonperturbative properties of the quark phase, in addition to a deconfinement phase transition. And it leads to a more compact star with a similar mass. We then propose a new class of quark star equation of states, which could be tested by future high-precision radius measurements of pulsar-like objects., Comment: 10 pages, 5 figures, 1 table, review article to appear in Chinese Physics C