Search

Fine-tuning large language models (LLMs) with high parameter efficiency for downstream tasks has become a new paradigm. Low-Rank Adaptation (LoRA) significantly reduces the number of trainable parameters for fine-tuning. Although it has demonstrated commendable performance, updating parameters within a single scale may not be the optimal choice for complex downstream tasks.In this paper, we extend the LoRA to multiple scales, dubbed as LoRA$^2$. We first combine orthogonal projection theory to train a set of LoRAs in two mutually orthogonal planes. Then, we improve the importance score algorithm, which reduce parameter sensitivity score calculations by approximately 98.5\%. By pruning singular values with lower importance scores, thereby enhancing adaptability to various downstream tasks. Extensive experiments are conducted on two widely used pre-trained models to validate the effectiveness of LoRA$^2$. Results show that it significantly reduces the number of trainable parameters to just 0.72\% compared to full fine-tuning, while still delivering highly impressive performance. Even when the parameters are further reduced to 0.17M, it still achieves comparable results to the baseline with 8 times more parameters. Our code is available here: https://anonymous.4open.science/r/LoRA-2-5B4C

This paper studies privacy in the context of complex decision support queries composed of multiple conditions on different aggregate statistics combined using disjunction and conjunction operators. Utility requirements for such queries necessitate the need for private mechanisms that guarantee a bound on the false negative and false positive errors. This paper formally defines complex decision support queries and their accuracy requirements, and provides algorithms that proportion the existing budget to optimally minimize privacy loss while supporting a bounded guarantee on the accuracy. Our experimental results on multiple real-life datasets show that our algorithms successfully maintain such utility guarantees, while also minimizing privacy loss.

Recent research in tabular data synthesis has focused on single tables, whereas real-world applications often involve complex data with tens or hundreds of interconnected tables. Previous approaches to synthesizing multi-relational (multi-table) data fall short in two key aspects: scalability for larger datasets and capturing long-range dependencies, such as correlations between attributes spread across different tables. Inspired by the success of diffusion models in tabular data modeling, we introduce $\textbf{C}luster$ $\textbf{La}tent$ $\textbf{Va}riable$ $guided$ $\textbf{D}enoising$ $\textbf{D}iffusion$ $\textbf{P}robabilistic$ $\textbf{M}odels$ (ClavaDDPM). This novel approach leverages clustering labels as intermediaries to model relationships between tables, specifically focusing on foreign key constraints. ClavaDDPM leverages the robust generation capabilities of diffusion models while incorporating efficient algorithms to propagate the learned latent variables across tables. This enables ClavaDDPM to capture long-range dependencies effectively. Extensive evaluations on multi-table datasets of varying sizes show that ClavaDDPM significantly outperforms existing methods for these long-range dependencies while remaining competitive on utility metrics for single-table data.

The $K$-medoids problem is a challenging combinatorial clustering task, widely used in data analysis applications. While numerous algorithms have been proposed to solve this problem, none of these are able to obtain an exact (globally optimal) solution for the problem in polynomial time. In this paper, we present EKM: a novel algorithm for solving this problem exactly with worst-case $O\left(N^{K+1}\right)$ time complexity. EKM is developed according to recent advances in transformational programming and combinatorial generation, using formal program derivation steps. The derived algorithm is provably correct by construction. We demonstrate the effectiveness of our algorithm by comparing it against various approximate methods on numerous real-world datasets. We show that the wall-clock run time of our algorithm matches the worst-case time complexity analysis on synthetic datasets, clearly outperforming the exponential time complexity of benchmark branch-and-bound based MIP solvers. To our knowledge, this is the first, rigorously-proven polynomial time, practical algorithm for this ubiquitous problem.

The essence of atomic structure theory, quantum chemistry, and computational materials science is solving the multi-electron stationary Schr\"odinger equation. The Quantum Monte Carlo-based neural network wave function method has surpassed traditional post-Hartree-Fock methods in precision across various systems. However, its energy uncertainty is limited to 0.01%, posing challenges in accurately determining excited states and ionization energies, especially for elements beyond the fourth period. Using effective core potentials to account for inner electrons enhances the precision of vertical excitation and ionization energies. This approach has proved effective in computing ground state energies for elements like Lithium to Gallium and in calculating energy levels and wave functions for atoms and molecules with second and fourth period elements. Additionally, by integrating effective core potentials with Ferminet, we've achieved multiple excited state calculations with a precision comparable to experimental results, marking a significant advancement in practical applications and setting a new standard for theoretical excited state calculations., Comment: 23 pages, 6 figures

Despite several works that succeed in generating synthetic data with differential privacy (DP) guarantees, they are inadequate for generating high-quality synthetic data when the input data has missing values. In this work, we formalize the problems of DP synthetic data with missing values and propose three effective adaptive strategies that significantly improve the utility of the synthetic data on four real-world datasets with different types and levels of missing data and privacy requirements. We also identify the relationship between privacy impact for the complete ground truth data and incomplete data for these DP synthetic data generation algorithms. We model the missing mechanisms as a sampling process to obtain tighter upper bounds for the privacy guarantees to the ground truth data. Overall, this study contributes to a better understanding of the challenges and opportunities for using private synthetic data generation algorithms in the presence of missing data., Comment: 18 pages, 9 figures, 2 tables

Recent years have witnessed the adoption of differential privacy (DP) in practical database systems like PINQ, FLEX, and PrivateSQL. Such systems allow data analysts to query sensitive data while providing a rigorous and provable privacy guarantee. However, the existing design of these systems does not distinguish data analysts of different privilege levels or trust levels. This design can have an unfair apportion of the privacy budget among the data analyst if treating them as a single entity, or waste the privacy budget if considering them as non-colluding parties and answering their queries independently. In this paper, we propose DProvDB, a fine-grained privacy provenance framework for the multi-analyst scenario that tracks the privacy loss to each single data analyst. Under this framework, when given a fixed privacy budget, we build algorithms that maximize the number of queries that could be answered accurately and apportion the privacy budget according to the privilege levels of the data analysts., Comment: Full version of the paper at Proc. of the ACM on Management of Data (PACMMOD/SIGMOD'2024)

Algorithms for solving the linear classification problem have a long history, dating back at least to 1936 with linear discriminant analysis. For linearly separable data, many algorithms can obtain the exact solution to the corresponding 0-1 loss classification problem efficiently, but for data which is not linearly separable, it has been shown that this problem, in full generality, is NP-hard. Alternative approaches all involve approximations of some kind, including the use of surrogates for the 0-1 loss (for example, the hinge or logistic loss) or approximate combinatorial search, none of which can be guaranteed to solve the problem exactly. Finding efficient algorithms to obtain an exact i.e. globally optimal solution for the 0-1 loss linear classification problem with fixed dimension, remains an open problem. In research we report here, we detail the rigorous construction of a new algorithm, incremental cell enumeration (ICE), that can solve the 0-1 loss classification problem exactly in polynomial time. We prove correctness using concepts from the theory of hyperplane arrangements and oriented matroids. We demonstrate the effectiveness of this algorithm on synthetic and real-world datasets, showing optimal accuracy both in and out-of-sample, in practical computational time. We also empirically demonstrate how the use of approximate upper bound leads to polynomial time run-time improvements to the algorithm whilst retaining exactness. To our knowledge, this is the first, rigorously-proven polynomial time, practical algorithm for this long-standing problem., Comment: 19 pages, 3 figures

A universal method to solve the differential equations of light-like geodesics is developed. The validity of this method depends on a new theorem, which is introduced for light-like geodesics in analogy to Beltrami's "geometrical" method for time-like geodesics. we apply the method to the Schwarzschild and Kerr spacetime as two examples. The general solutions of the light-like geodesic equations in the two spacetimes are derived straightforwadly. After setting $\theta=\pi/2$, the general light-like geodesics in Schwarzschild spacetime reduce to the same expression as that in literatures. The method developed and results obtained in this paper may be useful in modeling dynamical phenomena in strong gravitaional fields like black holes since the solutions are expressed in terms of elliptic integrals, which can be calculated effectively., Comment: 12 pages

The hybrid architecture of convolutional neural networks (CNNs) and Transformer are very popular for medical image segmentation. However, it suffers from two challenges. First, although a CNNs branch can capture the local image features using vanilla convolution, it cannot achieve adaptive feature learning. Second, although a Transformer branch can capture the global features, it ignores the channel and cross-dimensional self-attention, resulting in a low segmentation accuracy on complex-content images. To address these challenges, we propose a novel hybrid architecture of convolutional neural networks hand in hand with vision Transformers (CiT-Net) for medical image segmentation. Our network has two advantages. First, we design a dynamic deformable convolution and apply it to the CNNs branch, which overcomes the weak feature extraction ability due to fixed-size convolution kernels and the stiff design of sharing kernel parameters among different inputs. Second, we design a shifted-window adaptive complementary attention module and a compact convolutional projection. We apply them to the Transformer branch to learn the cross-dimensional long-term dependency for medical images. Experimental results show that our CiT-Net provides better medical image segmentation results than popular SOTA methods. Besides, our CiT-Net requires lower parameters and less computational costs and does not rely on pre-training. The code is publicly available at https://github.com/SR0920/CiT-Net., Comment: 9 pages, 3 figures, 3 tables

Differential privacy (DP) allows data analysts to query databases that contain users' sensitive information while providing a quantifiable privacy guarantee to users. Recent interactive DP systems such as APEx provide accuracy guarantees over the query responses, but fail to support a large number of queries with a limited total privacy budget, as they process incoming queries independently from past queries. We present an interactive, accuracy-aware DP query engine, CacheDP, which utilizes a differentially private cache of past responses, to answer the current workload at a lower privacy budget, while meeting strict accuracy guarantees. We integrate complex DP mechanisms with our structured cache, through novel cache-aware DP cost optimization. Our thorough evaluation illustrates that CacheDP can accurately answer various workload sequences, while lowering the privacy loss as compared to related work., Comment: To appear in VLDB'23

Simply restricting the computation to non-sensitive part of the data may lead to inferences on sensitive data through data dependencies. Inference control from data dependencies has been studied in the prior work. However, existing solutions either detect and deny queries which may lead to leakage -- resulting in poor utility, or only protects against exact reconstruction of the sensitive data -- resulting in poor security. In this paper, we present a novel security model called full deniability. Under this stronger security model, any information inferred about sensitive data from non-sensitive data is considered as a leakage. We describe algorithms for efficiently implementing full deniability on a given database instance with a set of data dependencies and sensitive cells. Using experiments on two different datasets, we demonstrate that our approach protects against realistic adversaries while hiding only minimal number of additional non-sensitive cells and scales well with database size and sensitive data., Comment: Extended version of the paper accepted at 48th International Conference on Very Large Databases (VLDB 2022)

The recent observation of cyclotron resonance in optimally-doped La$_{2-x}$Sr$_x$CuO$_4$ using time-domain THz spectroscopy in high magnetic field has given new possibilities for the study of cuprate superconductors. One can measure the cyclotron mass in more disordered cuprates possesing short scattering times therefore expanding the study to materials and dopings in which quantum oscillations have not been observed. Here we present the measurement of the carrier mass of the hole doped cuprate La$_{2-x}$Sr$_x$CuO$_4$ across a a range of dopings spanning the slightly underdoped ($x=0.13$) to highly overdoped ($x=0.26$), near to the termination of the superconducting dome. These results reveal a systematic increase of $m_c$ with doping, up to values greater than thirteen times the bare electron mass. This is in contrast with those masses extracted from the heat capacity, which show a peak near the pseudogap critical point $p^*$ and/or Lifshitz transition. The cyclotron frequency is linear in field up to 31 T for all dopings giving no evidence for field induced Fermi surface reconstructions. The cyclotron mass is found to be positive for all dopings, but with a magnitude systematically below the heat capacity mass for under and optimally doped samples, while exceeding it for overdoped samples. Among other aspects, these results are surprising as photoemission reveals a Lifshitz transition in the middle of our doping range and the sign of the cyclotron mass determined from a finite frequency resonance is -- in conventional theories -- a {\it topological quantity} only sensitive to whether or not the Fermi surface is closed around holes or electrons. We see no sign of a divergence of the mass near $p^*$ nor near the Lifshitz transition, showing that any singularity -- if it exists -- is not strong enough to affect the cyclotron mass., Comment: 9 pages, 4 figures

Electronic anisotropy (or `nematicity') has been detected in all main families of cuprate superconductors by a range of experimental techniques -- electronic Raman scattering, THz dichroism, thermal conductivity, torque magnetometry, second-harmonic generation -- and was directly visualized by scanning tunneling microscope (STM) spectroscopy. Using angle-resolved transverse resistance (ARTR) measurements, a very sensitive and background-free technique that can detect 0.5$\%$ anisotropy in transport, we have observed it also in La$_{2-x}$Sr$_{x}$CuO$_{4}$ (LSCO) for $0.02 \leq x \leq 0.25$. Arguably the key enigma in LSCO is the rotation of the nematic director with temperature; this has not been seen before in any material. Here, we address this puzzle by measuring the angle-resolved transverse magnetoresistance (ARTMR) in LSCO. We report a discovery of colossal transverse magnetoresistance (CTMR) -- an order-of-magnitude drop in the transverse resistivity in the magnetic field of $6\,$T, while none is seen in the longitudinal resistivity. We show that the apparent rotation of the nematic director is caused by superconducting phase fluctuations, which are much more anisotropic than the normal-electron fluid, and their respective directors are not parallel. This qualitative conclusion is robust and follows straight from the raw experimental data. We quantify this by modelling the measured (magneto-)conductivity by a sum of two conducting channels that correspond to distinct anisotropic Drude and Cooper-pair effective mass tensors. Strikingly, the anisotropy of Cooper-pair stiffness is significantly larger than that of the normal electrons, and it grows dramatically on the underdoped side, where the fluctuations become effectively quasi-one dimensional., Comment: 27 pages, 10 figures and 4 tables