Search

Continual learning methods based on pre-trained models (PTM) have recently gained attention which adapt to successive downstream tasks without catastrophic forgetting. These methods typically refrain from updating the pre-trained parameters and instead employ additional adapters, prompts, and classifiers. In this paper, we from a novel perspective investigate the benefit of sparse orthogonal parameters for continual learning. We found that merging sparse orthogonality of models learned from multiple streaming tasks has great potential in addressing catastrophic forgetting. Leveraging this insight, we propose a novel yet effective method called SoTU (Sparse Orthogonal Parameters TUning). We hypothesize that the effectiveness of SoTU lies in the transformation of knowledge learned from multiple domains into the fusion of orthogonal delta parameters. Experimental evaluations on diverse CL benchmarks demonstrate the effectiveness of the proposed approach. Notably, SoTU achieves optimal feature representation for streaming data without necessitating complex classifier designs, making it a Plug-and-Play solution.

Large Language Models (LLMs) often suffer from catastrophic forgetting when learning multiple tasks sequentially, making continual learning (CL) essential for their dynamic deployment. Existing state-of-the-art (SOTA) methods, such as O-LoRA, typically focus on constructing orthogonality tasks to decouple parameter interdependence from various domains.In this paper, we reveal that building non-collision parameters is a more critical factor in addressing CL challenges. Our theoretical and experimental analyses demonstrate that non-collision parameters can provide better task orthogonality, which is a sufficient but unnecessary condition. Furthermore, knowledge from multiple domains will be preserved in non-collision parameter subspaces, making it more difficult to forget previously seen data. Leveraging this insight, we propose Non-collision Low-Rank Adaptation (N-LoRA), a simple yet effective approach leveraging low collision rates to enhance CL in LLMs. Experimental results on multiple CL benchmarks indicate that N-LoRA achieves superior performance (+2.9), higher task orthogonality (*4.1 times), and lower parameter collision (*58.1 times) than SOTA methods.

Existing large language models (LLMs) evaluation methods typically focus on testing the performance on some closed-environment and domain-specific benchmarks with human annotations. In this paper, we explore a novel unsupervised evaluation direction, utilizing peer-review mechanisms to measure LLMs automatically. In this setting, both open-source and closed-source LLMs lie in the same environment, capable of answering unlabeled questions and evaluating each other, where each LLM's response score is jointly determined by other anonymous ones. To obtain the ability hierarchy among these models, we assign each LLM a learnable capability parameter to adjust the final ranking. We formalize it as a constrained optimization problem, intending to maximize the consistency of each LLM's capabilities and scores. The key assumption behind is that high-level LLM can evaluate others' answers more accurately than low-level ones, while higher-level LLM can also achieve higher response scores. Moreover, we propose three metrics called PEN, CIN, and LIS to evaluate the gap in aligning human rankings. We perform experiments on multiple datasets with these metrics, validating the effectiveness of the proposed approach.

Large Language Models (LLMs), including GPT-3.5, LLaMA, and PaLM, seem to be knowledgeable and able to adapt to many tasks. However, we still cannot completely trust their answers, since LLMs suffer from \textbf{hallucination}\textemdash fabricating non-existent facts, deceiving users with or without their awareness. However, the reasons for their existence and pervasiveness remain unclear. In this paper, we demonstrate that nonsensical prompts composed of random tokens can also elicit the LLMs to respond with hallucinations. Moreover, we provide both theoretical and experimental evidence that transformers can be manipulated to produce specific pre-define tokens by perturbing its input sequence. This phenomenon forces us to revisit that \emph{hallucination may be another view of adversarial examples}, and it shares similar characteristics with conventional adversarial examples as a basic property of LLMs. Therefore, we formalize an automatic hallucination triggering method as the \textit{hallucination attack} in an adversarial way. Finally, we explore the basic properties of attacked adversarial prompts and propose a simple yet effective defense strategy. Our code is released on GitHub\footnote{https://github.com/PKU-YuanGroup/Hallucination-Attack}.

We develop a formalism to directly evaluate the matrix of force constants within a Quantum Monte Carlo calculation. We utilize the matrix of force constants to accurately relax the positions of atoms in molecules and determine their vibrational modes, using a combination of Variational and Diffusion Monte Carlo. The computed bond lengths differ by less than 0.007{\AA} from the experimental results for all four tested molecules. For hydrogen and hydrogen chloride, we obtain fundamental vibrational frequencies within 0.1% of experimental results and ~10 times more accurate than leading computational methods. For carbon dioxide and methane, the vibrational frequency obtained is on average within 1.1% of the experimental result, which is at least 3 times closer than results using Restricted Hartree-Fock and Density Functional Theory with a Perdew-Burke-Ernzerhof (PBE) functional and comparable or better than Density Functional Theory with a semi-empirical functional.

Using density-functional perturbation theory and the Gr\"uneisen formalism, we directly calculate the linear thermal expansion coefficients (TECs) of a hexagonal bulk system MoS$_2$ in the crystallographic $a$ and $c$ directions. The TEC calculation depends critically on the evaluation of a temperature-dependent quantity $I_i(T)$, which is the integral of the product of heat capacity and $\Gamma_i(\nu)$, of frequency $\nu$ and strain type $i$, where $\Gamma_i(\nu)$ is the phonon density of states weighted by the Gr\"uneisen parameters. We show that to determine the linear TECs we may use minimally two uniaxial strains in the $z$ direction, and either the $x$ or $y$ direction. However, a uniaxial strain in either the $x$ or $y$ direction drastically reduces the symmetry of the crystal from a hexagonal one to a base-centered orthorhombic one. We propose to use an efficient and accurate symmetry-preserving biaxial strain in the $xy$ plane to derive the same result for $\Gamma(\nu)$. We highlight that the Gr\"uneisen parameter associated with a biaxial strain may not be the same as the average of Gr\"uneisen parameters associated with two separate uniaxial strains in the $x$ and $y$ directions due to possible preservation of degeneracies of the phonon modes under a biaxial deformation. Large anisotropy of TECs is observed where the linear TEC in the $c$ direction is about $1.8$ times larger than that in the $a$ or $b$ direction at high temperatures. Our theoretical TEC results are compared with experiment. The symmetry-preserving approach adopted here may be applied to a broad class of two lattice-parameter systems such as hexagonal, trigonal, and tetragonal systems, which allows many complicated systems to be treated on a first-principles level., Comment: 5 figures