Search

With increasing concerns and regulations on data privacy, fine-tuning pretrained language models (PLMs) in federated learning (FL) has become a common paradigm for NLP tasks. Despite being extensively studied, the existing methods for this problem still face two primary challenges. First, the huge number of parameters in large-scale PLMs leads to excessive communication and computational overhead. Second, the heterogeneity of data and tasks across clients poses a significant obstacle to achieving the desired fine-tuning performance. To address the above problems, we propose FedMCP, a novel parameter-efficient fine-tuning method with model-contrastive personalization for FL. Specifically, FedMCP adds two lightweight adapter modules, i.e., the global adapter and the private adapter, to the frozen PLMs within clients. In a communication round, each client sends only the global adapter to the server for federated aggregation. Furthermore, FedMCP introduces a model-contrastive regularization term between the two adapters. This, on the one hand, encourages the global adapter to assimilate universal knowledge and, on the other hand, the private adapter to capture client-specific knowledge. By leveraging both adapters, FedMCP can effectively provide fine-tuned personalized models tailored to individual clients. Extensive experiments on highly heterogeneous cross-task, cross-silo datasets show that FedMCP achieves substantial performance improvements over state-of-the-art FL fine-tuning approaches for PLMs.

This short paper reports a study of the electronic dissociation energies, De, of water clusters from direct ab initio (mostly CCSD(T)) calculations and the q-AQUA and MB-pol potentials. These clusters range in size from 6-25 monomers. These are all in very good agreement with each other, as shown in a recent Perspective by Herman and Xantheas. To the best of our knowledge, we present for the first time results for the De per monomer. To our surprise this quantity appears to be converging to a value close to 12 kcal/mol. An estimate of 1.5 - 2 kcal/mol for the {\Delta}ZPE for these clusters puts the value of D0 at 10 to 10.5 kcal/mol. This value is remarkably (and probably fortuitously) close to the reported sublimation enthalpy of 10.2 kcal/mol at 10 K. However, given that these De energies correspond to dissociation of the cluster to N isolated monomers the interpretation of ``vaporization" of these ``solid" clusters is qualitatively reasonable., Comment: 7 pages 2 figures. Working draft

Conversational search supports multi-turn user-system interactions to solve complex information needs. Different from the traditional single-turn ad-hoc search, conversational search encounters a more challenging problem of context-dependent query understanding with the lengthy and long-tail conversational history context. While conversational query rewriting methods leverage explicit rewritten queries to train a rewriting model to transform the context-dependent query into a stand-stone search query, this is usually done without considering the quality of search results. Conversational dense retrieval methods use fine-tuning to improve a pre-trained ad-hoc query encoder, but they are limited by the conversational search data available for training. In this paper, we leverage both rewritten queries and relevance judgments in the conversational search data to train a better query representation model. The key idea is to align the query representation with those of rewritten queries and relevant documents. The proposed model -- Query Representation Alignment Conversational Dense Retriever, QRACDR, is tested on eight datasets, including various settings in conversational search and ad-hoc search. The results demonstrate the strong performance of QRACDR compared with state-of-the-art methods, and confirm the effectiveness of representation alignment., Comment: Accepted by CIKM 2024

Progress in machine learning has facilitated the development of potentials that offer both the accuracy of first-principles techniques and vast increases in the speed of evaluation. Recently,"$\Delta$-machine learning" has been used to elevate the quality of a potential energy surface (PES) based on low-level, e.g., density functional theory (DFT) energies and gradients to close to the gold-standard coupled cluster level of accuracy. We have demonstrated the success of this approach for molecules, ranging in size from H$_3$O$^+$ to 15-atom acetyl-acetone and tropolone. These were all done using the B3LYP functional. Here we investigate the generality of this approach for the PBE, M06, M06-2X, and PBE0+MBD functionals, using ethanol as the example molecule. Linear regression with permutationally invariant polynomials is used to fit both low-level and correction PESs. These PESs are employed for standard RMSE analysis for training and test datasets, and then general fidelity tests such as energetics of stationary points, normal mode frequencies, and torsional potentials are examined. We achieve similar improvements in all cases. Interestingly, we obtained significant improvement over DFT gradients where coupled cluster gradients were not used to correct the low-level PES. Finally, we present some results for correcting a recent molecular mechanics force field for ethanol and comment on the possible generality of this approach.

Machine learning potentials (MLPs) are widely applied as an efficient alternative way to represent potential energy surfaces (PES) in many chemical simulations. The MLPs are often evaluated with the root-mean-square errors on the test set drawn from the same distribution as the training data. Here, we systematically investigate the relationship between such test errors and the simulation accuracy with MLPs on an example of a full-dimensional, global PES for the glycine amino acid. Our results show that the errors in the test set do not unambiguously reflect the MLP performance in different simulation tasks such as relative conformer energies, barriers, vibrational levels, and zero-point vibrational energies. We also offer an easily accessible solution for improving the MLP quality in a simulation-oriented manner, yielding the most precise relative conformer energies and barriers. This solution also passed the stringent test by the diffusion Monte Carlo simulations.

Conversational search provides a more convenient interface for users to search by allowing multi-turn interaction with the search engine. However, the effectiveness of the conversational dense retrieval methods is limited by the scarcity of training data required for their fine-tuning. Thus, generating more training conversational sessions with relevant labels could potentially improve search performance. Based on the promising capabilities of large language models (LLMs) on text generation, we propose ConvSDG, a simple yet effective framework to explore the feasibility of boosting conversational search by using LLM for session data generation. Within this framework, we design dialogue/session-level and query-level data generation with unsupervised and semi-supervised learning, according to the availability of relevance judgments. The generated data are used to fine-tune the conversational dense retriever. Extensive experiments on four widely used datasets demonstrate the effectiveness and broad applicability of our ConvSDG framework compared with several strong baselines., Comment: Accepted by WWW 2024 Workshop

Here we assess two machine-learned potentials, one using the symmetric gradient domain machine learning (sGDML) method and one based on permutationally invariant polynomials (PIPs). These are successors to a PIP potential energy surface (PES) reported in 2004. We describe the details of both fitting methods and then compare the two PESs with respect to precision, properties, and speed of evaluation. While the precision of the potentials is similar, the PIP PES is much faster to evaluate for energies and energies plus gradient than the sGDML one. Diffusion Monte Carlo calculations of the ground vibrational state, using both potentials, produce similar large anharmonic downshift of the zero-point energy compared to the harmonic approximation the PIP and sGDML potentials. The computational time for these calculations using the sGDML PES is roughly 300 times greater than using the PIP one., Comment: 25 pages, 6 figures

Conversational search facilitates complex information retrieval by enabling multi-turn interactions between users and the system. Supporting such interactions requires a comprehensive understanding of the conversational inputs to formulate a good search query based on historical information. In particular, the search query should include the relevant information from the previous conversation turns. However, current approaches for conversational dense retrieval primarily rely on fine-tuning a pre-trained ad-hoc retriever using the whole conversational search session, which can be lengthy and noisy. Moreover, existing approaches are limited by the amount of manual supervision signals in the existing datasets. To address the aforementioned issues, we propose a History-Aware Conversational Dense Retrieval (HAConvDR) system, which incorporates two ideas: context-denoised query reformulation and automatic mining of supervision signals based on the actual impact of historical turns. Experiments on two public conversational search datasets demonstrate the improved history modeling capability of HAConvDR, in particular for long conversations with topic shifts., Comment: Accepted to Findings of ACL 2024

Assessments of machine-learned (ML) potentials are an important aspect of the rapid development of this field. We recently reported an assessment of the linear-regression permutationally invariant polynomial (PIP) method for ethanol, using the widely used (revised) MD17 dataset. We demonstrated that the PIP approach outperformed numerous other methods, e.g., ANI, PhysNet, sGDML, p-KRR, with respect to precision and notably with respect to speed [Houston $et$ $al$., $J. Chem. Phys.$ 2022, 156, 044120.]. Here we extend this assessment to the 21-atom aspirin molecule, using the rMD17 dataset. Both energies and forces are used for training and the precision of several PIPs is examined for both. Normal mode frequencies, the methyl torsional potential, and 1d vibrational energies for an OH stretch are presented. Overall, we show that the PIPs approach outperforms other ML methods, including sGDML, ANI, GAP, PhysNet, and ACE, as reported by Kov\'acs $et$ $al.$ in $J. Chem. Theory$ $Comput.$ 2021, 17, 7696-7711.

$\Delta$-Machine Learning ($\Delta$-ML) has been shown to effectively and efficiently bring a low-level ML potential energy surface to CCSD(T) quality. Here we propose extending this approach to general force fields, which implicitly or explicitly contain many-body effects. After describing this general approach, we illustrate it for the MB-pol water potential which contains CCSD(T) 2-body and 3-body interactions but relies on the TTM4-F 4-body and higher body interactions. The 4-body MB-pol (TTM4-F) interaction fails at a very short range and for the water hexamer errors up to 0.84 kcal/mol are seen for some isomers, owing mainly to 4-body errors. We apply $\Delta$-ML for the 4-body interaction, using a recent dataset of CCSD(T) 4-body energies that we used to develop a new water potential, q-AQUA. This 4-body correction is shown to improve the accuracy of the MB-pol potential for the relative energies of 8 isomers of the water hexamer as well as the harmonic frequencies. The new potential is robust in the very short range and so should be reliable for simulations at high pressure and/or high temperature.

Ethanol is a molecule of fundamental interest in combustion, astrochemistry, and condensed phase as a solvent. It is characterized by two methyl rotors and $trans$ ($anti$) and $gauche$ conformers, which are known to be very close in energy. Here we show that based on rigorous quantum calculations of the vibrational zero-point state, using a new ab initio potential energy surface (PES), the ground state resembles the $trans$ conformer but substantial delocalization to the $gauche$ conformer is present. This explains experimental issues about the identification and isolation of the two conformers. This "leak" effect is partially quenched when deuterating the OH group, which further demonstrates the need for a quantum mechanical approach. Diffusion Monte Carlo (DMC) and full-dimensional semiclassical dynamics calculations are employed. The new PES is obtained by means of a $\Delta$-Machine learning approach starting from a pre-existing low level (LL) density functional theory (DFT) surface. This surface is brought to the CCSD(T) level of theory using a relatively small number of $ab$ $initio$ CCSD(T) energies. Agreement between the corrected PES and direct $ab$ $initio$ results for standard fidelity tests is excellent. One- and two-dimensional discrete variable representation calculations focusing on the $trans$-$gauche$ torsional motion are also reported, in reasonable agreement with the experiment.

Dual encoders have been used for question-answering (QA) and information retrieval (IR) tasks with good results. Previous research focuses on two major types of dual encoders, Siamese Dual Encoder (SDE), with parameters shared across two encoders, and Asymmetric Dual Encoder (ADE), with two distinctly parameterized encoders. In this work, we explore different ways in which the dual encoder can be structured, and evaluate how these differences can affect their efficacy in terms of QA retrieval tasks. By evaluating on MS MARCO, open domain NQ and the MultiReQA benchmarks, we show that SDE performs significantly better than ADE. We further propose three different improved versions of ADEs by sharing or freezing parts of the architectures between two encoder towers. We find that sharing parameters in projection layers would enable ADEs to perform competitively with or outperform SDEs. We further explore and explain why parameter sharing in projection layer significantly improves the efficacy of the dual encoders, by directly probing the embedding spaces of the two encoder towers with t-SNE algorithm., Comment: Published in EMNLP 2022

Many model potential energy surfaces (PESs) have been reported for water; however, none are strictly from "first principles". Here we report such a potential, based on a many-body representation at the CCSD(T) level of theory up to the ultimate 4-body interaction. The new PES is benchmarked for the isomers of the water hexamer for dissociation energies, harmonic frequencies, and unrestricted diffusion Monte Carlo (DMC) calculations of the zero-point energies of the Prism, Book, and Cage isomers. Dissociation energies of several isomers of the 20-mer agree well with recent benchmark energies. Exploratory DMC calculations on this cluster verify the robustness of the new PES for quantum simulations. The accuracy and speed of the new PES are demonstrated for standard condensed phase properties, i.e., the radial distribution function and the self-diffusion constant. Quantum effects are shown to be substantial for these observables and also needed to bring theory into an excellent agreement with experiment.

It has been shown that dual encoders trained on one domain often fail to generalize to other domains for retrieval tasks. One widespread belief is that the bottleneck layer of a dual encoder, where the final score is simply a dot-product between a query vector and a passage vector, is too limited to make dual encoders an effective retrieval model for out-of-domain generalization. In this paper, we challenge this belief by scaling up the size of the dual encoder model {\em while keeping the bottleneck embedding size fixed.} With multi-stage training, surprisingly, scaling up the model size brings significant improvement on a variety of retrieval tasks, especially for out-of-domain generalization. Experimental results show that our dual encoders, \textbf{G}eneralizable \textbf{T}5-based dense \textbf{R}etrievers (GTR), outperform %ColBERT~\cite{khattab2020colbert} and existing sparse and dense retrievers on the BEIR dataset~\cite{thakur2021beir} significantly. Most surprisingly, our ablation study finds that GTR is very data efficient, as it only needs 10\% of MS Marco supervised data to achieve the best out-of-domain performance. All the GTR models are released at https://tfhub.dev/google/collections/gtr/1.

Permutationally invariant polynomial (PIP) regression has been used to obtain machine-learned (ML) potential energy surfaces, including analytical gradients, for many molecules and chemical reactions. Recently, the approach has been extended to moderate size molecules and applied to systems up to 15 atoms. The algorithm, including "purification of the basis", is computationally efficient for energies; however, we found that the recent extension to obtain analytical gradients, despite being a remarkable advance over previous methods, could be further improved. Here we report developments to compact further a purified basis and, more significantly, to use the reverse gradient approach to greatly speed up gradient evaluation. We demonstrate this for our recent 4-body water interaction potential. Comparisons of training and testing precision on the MD17 database of energies and gradients (forces) for ethanol against GP-SOAP, ANI, sGDML, PhysNet, pKREG, KRR, and other methods, which were recently assessed by Dral and co-workers, are given. The PIP fits are as precise as those using these methods, but the PIP computation time for energy and force evaluation is shown to be 10 to 1000 times faster. Finally, a new PIP PES is reported for ethanol based on a more extensive dataset of energies and gradients than in the MD17 database. Diffusion Monte Carlo calculations which fail on MD17-based PESs are successful using the new PES.

We report a permutationally invariant polynomial (PIP) potential energy surface for the water 4-body interaction. This 12-atom PES is a fit to 2119, symmetry-unique, CCSD(T)-F12a/haTZ (aug-cc-pVTZ basis for 'O' atom and cc-pVTZ basis for 'H' atom) 4-b interaction energies. These come from low-level, direct-dynamics calculations, tetramer fragments from an MD water simulation at 300 K, and from the water hexamer, heptamer, decamer, and 13-mer clusters. The PIP basis is purified to ensure that the 4-b potential goes rigorously to zero in monomer+trimer and dimer+dimer dissociations for all possible such fragments. The 4-b energies of isomers of the hexamer calculated with the new surface are shown to be in better agreement with benchmark CCSD(T) results than those from the MB-pol potential. Other tests validate the high-fidelity of the PES.

In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowledge, which in our case we define to be a large unstructured passage collection. We first conduct sparse retrieval with BM25 and study expanding the question with object names and image captions. We verify that visual clues play an important role and captions tend to be more informative than object names in sparse retrieval. We then construct a dual-encoder dense retriever, with the query encoder being LXMERT, a multi-modal pre-trained transformer. We further show that dense retrieval significantly outperforms sparse retrieval that uses object expansion. Moreover, dense retrieval matches the performance of sparse retrieval that leverages human-generated captions., Comment: Accepted to SIGIR'21 as a short paper

Privacy preservation remains a key challenge in data mining and Natural Language Understanding (NLU). Previous research shows that the input text or even text embeddings can leak private information. This concern motivates our research on effective privacy preservation approaches for pretrained Language Models (LMs). We investigate the privacy and utility implications of applying dx-privacy, a variant of Local Differential Privacy, to BERT fine-tuning in NLU applications. More importantly, we further propose privacy-adaptive LM pretraining methods and show that our approach can boost the utility of BERT dramatically while retaining the same level of privacy protection. We also quantify the level of privacy preservation and provide guidance on privacy configuration. Our experiments and findings lay the groundwork for future explorations of privacy-preserving NLU with pretrained LMs., Comment: Accepted to CIKM 2021

Machine-learned potential energy surfaces (PESs) for molecules with more than 10 atoms are typically forced to use lower-level electronic structure methods such as density functional theory and second-order Moller-Plesset perturbation theory (MP2). While these are efficient and realistic, they fall short of the accuracy of the ``gold standard'' coupled-cluster method, especially with respect to reaction and isomerization barriers. We report a major step forward in applying a $\Delta$-machine learning method to the challenging case of acetylacetone, whose MP2 barrier height for H-atom transfer is low by roughly 1.5 kcal/mol relative to the benchmark CCSD(T) barrier of 3.2 kcal/mol. From a database of 2151 local CCSD(T) energies, and training with as few as 430 energies, we obtain a new PES with a barrier of 3.49 kcal/mol in agreement with the LCCSD(T) one of 3.54 kcal/mol and close to the benchmark value. Tunneling splittings due to H-atom transfer are calculated using this new PES, providing improved estimates over previous ones obtained using an MP2-based PES.

Recent studies on Question Answering (QA) and Conversational QA (ConvQA) emphasize the role of retrieval: a system first retrieves evidence from a large collection and then extracts answers. This open-retrieval ConvQA setting typically assumes that each question is answerable by a single span of text within a particular passage (a span answer). The supervision signal is thus derived from whether or not the system can recover an exact match of this ground-truth answer span from the retrieved passages. This method is referred to as span-match weak supervision. However, information-seeking conversations are challenging for this span-match method since long answers, especially freeform answers, are not necessarily strict spans of any passage. Therefore, we introduce a learned weak supervision approach that can identify a paraphrased span of the known answer in a passage. Our experiments on QuAC and CoQA datasets show that the span-match weak supervisor can only handle conversations with span answers, and has less satisfactory results for freeform answers generated by people. Our method is more flexible as it can handle both span answers and freeform answers. Moreover, our method can be more powerful when combined with the span-match method which shows it is complementary to the span-match method. We also conduct in-depth analyses to show more insights on open-retrieval ConvQA under a weak supervision setting., Comment: Accepted to ECIR'21

The \kovats retention index is a dimensionless quantity that characterizes the rate at which a compound is processed through a gas chromatography column. This quantity is independent of many experimental variables and, as such, is considered a near-universal descriptor of retention time on a chromatography column. The \kovats retention indices of a large number of molecules have been determined experimentally. The "NIST 20: GC Method\slash Retention Index Library" database has collected and, more importantly, curated retention indices of a subset of these compounds resulting in a highly valued reference database. The experimental data in the library form an ideal data set for training machine learning models for the prediction of retention indices of unknown compounds. In this article, we describe the training of a graph neural network model to predict the \kovats retention index for compounds in the NIST library and compare this approach with previous work \cite{2019Matyushin}. We predict the \kovats retention index with a mean unsigned error of 28 index units as compared to 44, the putative best result using a convolutional neural network \cite{2019Matyushin}. The NIST library also incorporates an estimation scheme based on a group contribution approach that achieves a mean unsigned error of 114 compared to the experimental data. Our method uses the same input data source as the group contribution approach, making its application straightforward and convenient to apply to existing libraries. Our results convincingly demonstrate the predictive powers of systematic, data-driven approaches leveraging deep learning methodologies applied to chemical data and for the data in the NIST 20 library outperform previous models.

``$\Delta$-machine learning" refers to a machine learning approach to bring a property such as a potential energy surface (PES) based on low-level (LL) density functional theory (DFT) energies and gradients to close to a coupled cluster (CC) level of accuracy. Here we present such an approach that uses the permutationally invariant polynomial (PIP) method to fit high-dimensional PESs. The approach is represented by a simple equation, in obvious notation $V_{LL{\rightarrow}CC}=V_{LL}+\Delta{V_{CC-LL}}$, and demonstrated for \ce{CH4}, \ce{H3O+}, and $trans$ and $cis$-$N$-methyl acetamide (NMA), \ce{CH3CONHCH3}. For these molecules, the LL PES, $V_{LL}$, is a PIP fit to DFT/B3LYP/6-31+G(d) energies and gradients, and $\Delta{V_{CC-LL}}$ is a precise PIP fit obtained using a low-order PIP basis set and based on a relatively small number of CCSD(T) energies. For \ce{CH4} these are new calculations adopting an aug-cc-pVDZ basis, for \ce{H3O+} previous CCSD(T)-F12/aug-cc-pVQZ energies are used, while for NMA new CCSD(T)-F12/aug-cc-pVDZ calculations are performed. With as few as 200 CCSD(T) energies, the new PESs are in excellent agreement with benchmark CCSD(T) results for the small molecules, and for 12-atom NMA training is done with 4696 CCSD(T) energies.