Search

Though current long-context large language models (LLMs) have demonstrated impressive capacities in answering user questions based on extensive text, the lack of citations in their responses makes user verification difficult, leading to concerns about their trustworthiness due to their potential hallucinations. In this work, we aim to enable long-context LLMs to generate responses with fine-grained sentence-level citations, improving their faithfulness and verifiability. We first introduce LongBench-Cite, an automated benchmark for assessing current LLMs' performance in Long-Context Question Answering with Citations (LQAC), revealing considerable room for improvement. To this end, we propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs to automatically generate long-context QA instances with precise sentence-level citations, and leverage this pipeline to construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the LongCite-45k dataset, successfully enabling their generation of accurate responses and fine-grained sentence-level citations in a single output. The evaluation results on LongBench-Cite show that our trained models achieve state-of-the-art citation quality, surpassing advanced proprietary models including GPT-4o.

The evolution processes of complex systems carry key information in the systems' functional properties. Applying machine learning algorithms, we demonstrate that the historical formation process of various networked complex systems can be extracted, including protein-protein interaction, ecology, and social network systems. The recovered evolution process has demonstrations of immense scientific values, such as interpreting the evolution of protein-protein interaction network, facilitating structure prediction, and particularly revealing the key co-evolution features of network structures such as preferential attachment, community structure, local clustering, degree-degree correlation that could not be explained collectively by previous theories. Intriguingly, we discover that for large networks, if the performance of the machine learning model is slightly better than a random guess on the pairwise order of links, reliable restoration of the overall network formation process can be achieved. This suggests that evolution history restoration is generally highly feasible on empirical networks.

Program induction (PI) has become a promising paradigm for using knowledge bases (KBs) to help large language models (LLMs) answer complex knowledge-intensive questions. Nonetheless, PI typically relies on a large number of parallel question-program pairs to make the LLM aware of the schema of the given KB, and is thus challenging for many low-resourced KBs that lack annotated data. To this end, we propose KB-Plugin, a plug-and-play framework that enables LLMs to induce programs over any low-resourced KB. Firstly, KB-Plugin adopts self-supervised learning to encode the detailed schema information of a given KB into a pluggable module, namely schema plugin. Secondly, KB-Plugin utilizes abundant annotated data from a rich-resourced KB to train another pluggable module, namely PI plugin, which can help the LLM extract question-relevant schema information from the schema plugin of any KB and utilize this information to induce programs over this KB. Experiments on five heterogeneous KBQA datasets show that KB-Plugin achieves better or comparable performance with 25$\times$ smaller backbone LLM compared to SoTA PI methods for low-resourced KBs, and even approaches the performance of supervised methods. Our code and data are available at https://github.com/THU-KEG/KB-Plugin.

Knowledge distillation is one of the methods for model compression, and existing knowledge distillation techniques focus on how to improve the distillation algorithm so as to enhance the distillation efficdiency. This paper introduces dynamic incremental learning into knowledge distillation and proposes a distillation strategy for education distillation. Specifically, it is proposed to look at fragmented student models divided from the full student model as low models. As the grade level rises, fragmented student models deepen in conjunction with designed teaching reference layers, while learning and distilling from more teacher models. By moving from lower to higher grades, fragmented student models were gradually integrated into a complete target student model, and the performance of the student models gradually improved from lower to senior grades of the stage. Education distillation strategies combined with distillation algorithms outperform the results of single distillation algorithms on the public dataset CIFAR100,Caltech256, Food-101 dataset.

The presence of $1/f$ noise, also known as pink noise, is a well-established phenomenon in biological neural networks, and is thought to play an important role in information processing in the brain. In this study, we find that such $1/f$ noise is also found in deep neural networks trained on natural language, resembling that of their biological counterparts. Specifically, we trained Long Short-Term Memory (LSTM) networks on the `IMDb' AI benchmark dataset, then measured the neuron activations. The detrended fluctuation analysis (DFA) on the time series of the different neurons demonstrate clear $1/f$ patterns, which is absent in the time series of the inputs to the LSTM. Interestingly, when the neural network is at overcapacity, having more than enough neurons to achieve the learning task, the activation patterns deviate from $1/f$ noise and shifts towards white noise. This is because many of the neurons are not effectively used, showing little fluctuations when fed with input data. We further examine the exponent values in the $1/f$ noise in ``internal" and ``external" activations in the LSTM cell, finding some resemblance in the variations of the exponents in fMRI signals of the human brain. Our findings further supports the hypothesis that $1/f$ noise is a signature of optimal learning. With deep learning models approaching or surpassing humans in certain tasks, and being more ``experimentable'' than their biological counterparts, our study suggests that they are good candidates to understand the fundamental origins of $1/f$ noise.

Federated learning is widely used to perform decentralized training of a global model on multiple devices while preserving the data privacy of each device. However, it suffers from heterogeneous local data on each training device which increases the difficulty to reach the same level of accuracy as the centralized training. Supervised Contrastive Learning which outperform cross-entropy tries to minimizes the difference between feature space of points belongs to the same class and pushes away points from different classes. We propose Supervised Contrastive Federated Learning in which devices can share the learned class-wise feature spaces with each other and add the supervised-contrastive learning loss as a regularization term to foster the feature space learning. The loss tries to minimize the cosine similarity distance between the feature map and the averaged feature map from another device in the same class and maximizes the distance between the feature map and that in a different class. This new regularization term when added on top of the moon regularization term is found to outperform the other state-of-the-art regularization terms in solving the heterogeneous data distribution problem., Comment: 5 Pages, 2 figures, conference workshop

Self-supervised entity alignment (EA) aims to link equivalent entities across different knowledge graphs (KGs) without seed alignments. The current SOTA self-supervised EA method draws inspiration from contrastive learning, originally designed in computer vision based on instance discrimination and contrastive loss, and suffers from two shortcomings. Firstly, it puts unidirectional emphasis on pushing sampled negative entities far away rather than pulling positively aligned pairs close, as is done in the well-established supervised EA. Secondly, KGs contain rich side information (e.g., entity description), and how to effectively leverage those information has not been adequately investigated in self-supervised EA. In this paper, we propose an interactive contrastive learning model for self-supervised EA. The model encodes not only structures and semantics of entities (including entity name, entity description, and entity neighborhood), but also conducts cross-KG contrastive learning by building pseudo-aligned entity pairs. Experimental results show that our approach outperforms previous best self-supervised results by a large margin (over 9% average improvement) and performs on par with previous SOTA supervised counterparts, demonstrating the effectiveness of the interactive contrastive learning for self-supervised EA., Comment: Accepted by CIKM 2022

The success of deep neural networks in real-world problems has prompted many attempts to explain their training dynamics and generalization performance, but more guiding principles for the training of neural networks are still needed. Motivated by the edge of chaos principle behind the optimal performance of neural networks, we study the role of various hyperparameters in modern neural network training algorithms in terms of the order-chaos phase diagram. In particular, we study a fully analytical feedforward neural network trained on the widely adopted Fashion-MNIST dataset, and study the dynamics associated with the hyperparameters in back-propagation during the training process. We find that for the basic algorithm of stochastic gradient descent with momentum, in the range around the commonly used hyperparameter values, clear scaling relations are present with respect to the training time during the ordered phase in the phase diagram, and the model's optimal generalization power at the edge of chaos is similar across different training parameter combinations. In the chaotic phase, the same scaling no longer exists. The scaling allows us to choose the training parameters to achieve faster training without sacrificing performance. In addition, we find that the commonly used model regularization method - weight decay - effectively pushes the model towards the ordered phase to achieve better performance. Leveraging on this fact and the scaling relations in the other hyperparameters, we derived a principled guideline for hyperparameter determination, such that the model can achieve optimal performance by saturating it at the edge of chaos. Demonstrated on this simple neural network model and training algorithm, our work improves the understanding of neural network training dynamics, and can potentially be extended to guiding principles of more complex model architectures and algorithms.

The COVID-19 infection cases have surged globally, causing devastations to both the society and economy. A key factor contributing to the sustained spreading is the presence of a large number of asymptomatic or hidden spreaders, who mix among the susceptible population without being detected or quarantined. Here we propose an effective non-pharmacological intervention method of detecting the asymptomatic spreaders in contact-tracing networks, and validated it on the empirical COVID-19 spreading network in Singapore. We find that using pure physical spreading equations, the hidden spreaders of COVID-19 can be identified with remarkable accuracy. Specifically, based on the unique characteristics of COVID-19 spreading dynamics, we propose a computational framework capturing the transition probabilities among different infectious states in a network, and extend it to an efficient algorithm to identify asymptotic individuals. Our simulation results indicate that a screening method using our prediction outperforms machine learning algorithms, e.g. graph neural networks, that are designed as baselines in this work, as well as random screening of infection's closest contacts widely used by China in its early outbreak. Furthermore, our method provides high precision even with incomplete information of the contract-tracing networks. Our work can be of critical importance to the non-pharmacological interventions of COVID-19, especially with increasing adoptions of contact tracing measures using various new technologies. Beyond COVID-19, our framework can be useful for other epidemic diseases that also feature asymptomatic spreading, Comment: 14 pages, 4 figures

A large number of individuals are suffering from suicidal ideation in the world. There are a number of causes behind why an individual might suffer from suicidal ideation. As the most popular platform for self-expression, emotion release, and personal interaction, individuals may exhibit a number of symptoms of suicidal ideation on social media. Nevertheless, challenges from both data and knowledge aspects remain as obstacles, constraining the social media-based detection performance. Data implicitness and sparsity make it difficult to discover the inner true intentions of individuals based on their posts. Inspired by psychological studies, we build and unify a high-level suicide-oriented knowledge graph with deep neural networks for suicidal ideation detection on social media. We further design a two-layered attention mechanism to explicitly reason and establish key risk factors to individual's suicidal ideation. The performance study on microblog and Reddit shows that: 1) with the constructed personal knowledge graph, the social media-based suicidal ideation detection can achieve over 93% accuracy; and 2) among the six categories of personal factors, post, personality, and experience are the top-3 key indicators. Under these categories, posted text, stress level, stress duration, posted image, and ruminant thinking contribute to one's suicidal ideation detection., Comment: Accepted to IEEE Transaction on Multimedia

Percolation theory has been widely used to study phase transitions in complex networked systems. It has also successfully explained several macroscopic phenomena across different fields. Yet, the existent theoretical framework for percolation places the focus on the direct interactions among the system's components, while recent empirical observations have shown that indirect interactions are common in many systems like ecological and social networks, among others. Here, we propose a new percolation framework that accounts for indirect interactions, which allows to generalize the current theoretical body and understand the role of the underlying indirect influence of the components of a networked system on its macroscopic behavior. We report a rich phenomenology in which first-order, second-order or hybrid phase transitions are possible depending on whether the links of the substrate network are directed, undirected or a mix, respectively. We also present an analytical framework to characterize the proposed induced percolation, paving the way to further understand network dynamics with indirect interactions., Comment: 18 pages, including 10 figures and 2 tables

While current machine learning models have impressive performance over a wide range of applications, their large size and complexity render them unsuitable for tasks such as remote monitoring on edge devices with limited storage and computational power. A naive approach to resolve this on the model level is to use simpler architectures, but this sacrifices prediction accuracy and is unsuitable for monitoring applications requiring accurate detection of the onset of adverse events. In this paper, we propose an alternative solution to this problem by decomposing the predictive model as the sum of a simple function which serves as a local monitoring tool, and a complex correction term to be evaluated on the server. A sign requirement is imposed on the latter to ensure that the local monitoring function is safe, in the sense that it can effectively serve as an early warning system. Our analysis quantifies the trade-offs between model complexity and performance, and serves as a guidance for architecture design. We validate our proposed framework on a series of monitoring experiments, where we succeed at learning monitoring models with significantly reduced complexity that minimally violate the safety requirement. More broadly, our framework is useful for learning classifiers in applications where false negatives are significantly more costly compared to false positives.

Despite detection of suicidal ideation on social media has made great progress in recent years, people's implicitly and anti-real contrarily expressed posts still remain as an obstacle, constraining the detectors to acquire higher satisfactory performance. Enlightened by the hidden "tree holes" phenomenon on microblog, where people at suicide risk tend to disclose their inner real feelings and thoughts to the microblog space whose authors have committed suicide, we explore the use of tree holes to enhance microblog-based suicide risk detection from the following two perspectives. (1) We build suicide-oriented word embeddings based on tree hole contents to strength the sensibility of suicide-related lexicons and context based on tree hole contents. (2) A two-layered attention mechanism is deployed to grasp intermittently changing points from individual's open blog streams, revealing one's inner emotional world more or less. Our experimental results show that with suicide-oriented word embeddings and attention, microblog-based suicide risk detection can achieve over 91\% accuracy. A large-scale well-labelled suicide data set is also reported in the paper., Comment: Accepted to EMNLP 2019

It has long been suggested that the biological brain operates at some critical point between two different phases, possibly order and chaos. Despite many indirect empirical evidence from the brain and analytical indication on simple neural networks, the foundation of this hypothesis on generic non-linear systems remains unclear. Here we develop a general theory that reveals the exact edge of chaos is the boundary between the chaotic phase and the (pseudo)periodic phase arising from Neimark-Sacker bifurcation. This edge is analytically determined by the asymptotic Jacobian norm values of the non-linear operator and influenced by the dimensionality of the system. The optimality at the edge of chaos is associated with the highest information transfer between input and output at this point similar to that of the logistic map. As empirical validations, our experiments on the various deep learning models in computer vision demonstrate the optimality of the models near the edge of chaos, and we observe that the state-of-art training algorithms push the models towards such edge as they become more accurate. We further establishes the theoretical understanding of deep learning model generalization through asymptotic stability.

During an epidemic control, the containment of the disease is usually achieved through increasing devoted resource to shorten the duration of infectiousness. However, the impact of this resource expenditure has not been studied quantitatively. Using the well-documented cholera data, we observe empirically that the recovery rate which is related to the duration of infectiousness has a strong positive correlation with the average resource devoted to the infected individuals. By incorporating this relation we build a novel model and find that insufficient resource leads to an abrupt increase in the infected population size, which is in marked contrast with the continuous phase transitions believed previously. Counterintuitively, this abrupt phase transition is more pronounced in the less contagious diseases, which usually correspond to the most fatal ones. Furthermore, we find that even for a single infection source, public resource needs to meet a significant amount, which is proportional to the whole population size to ensure epidemic containment. Our findings provide a theoretical foundation for efficient epidemic containment strategies in the early stage., Comment: 12 pages, 5 figures

Re-finding electronic documents from a personal computer is a frequent demand to users. In a simple re-finding task, people can use many methods to retrieve a document, such as navigating directly to the document's folder, searching with a desktop search engine, or checking the Recent Files List. However, when encountering a difficult re-finding task, people usually cannot remember the attributes used by conventional re-finding methods, such as file path, file name, keywords etc., the re-finding would fail. We propose a new method to support difficult re-finding tasks. When a user is reading a document, we collect all kinds of possible memory pieces of the user about the document, such as number of pages, number of images, number of math formulas, cumulative reading time, reading frequency, printing experiences etc. If the user wants to re-find a document later, we use these collected attributes to filter out the target document. To alleviate the user's cognitive burden, we use a question and answer wizard interface and provide recommendations to the answers for the user, the recommendations are generated by analyzing the collected attributes of each document and the user's experiences about them.

Measuring and optimizing the influence of nodes in big-data online social networks are important for many practical applications, such as the viral marketing and the adoption of new products. As the viral spreading on social network is a global process, it is commonly believed that measuring the influence of nodes inevitably requires the knowledge of the entire network. Employing percolation theory, we show that the spreading process displays a nucleation behavior: once a piece of information spread from the seeds to more than a small characteristic number of nodes, it reaches a point of no return and will quickly reach the percolation cluster, regardless of the entire network structure, otherwise the spreading will be contained locally. Thus, we find that, without the knowledge of entire network, any nodes' global influence can be accurately measured using this characteristic number, which is independent of the network size. This motivates an efficient algorithm with constant time complexity on the long standing problem of best seed spreaders selection, with performance remarkably close to the true optimum., Comment: 6 pages, 5 figures, Proceedings of the National Academy of Sciences of the United States of America (PNAS), July 3, 2018

Interdependent networks are ubiquitous in our society, ranging from infrastructure to economics, and the study of their cascading behaviors using percolation theory has attracted much attention in the recent years. To analyze the percolation phenomena of these systems, different mathematical frameworks have been proposed including generating functions, eigenvalues among some others. These different frameworks approach the phase transition behaviors from different angles, and have been very successful in shaping the different quantities of interest including critical threshold, size of the giant component, order of phase transition and the dynamics of cascading. These methods also vary in their mathematical complexity in dealing with interdependent networks that have additional complexity in terms of the correlation among different layers of networks or links. In this work, we review a particular approach of simple self-consistent probability equations, and illustrate that it can greatly simplify the mathematical analysis for systems ranging from single layer network to various different interdependent networks. We give an overview on the detailed framework to study the nature of the critical phase transition, value of the critical threshold and size of the giant component for these different systems.

Although the many forms of modern social media have become major channels for the dissemination of information, they are becoming overloaded because of the rapidly-expanding number of information feeds. We analyze the expanding user-generated content in Sina Weibo, the largest micro-blog site in China, and find evidence that popular messages often follow a mechanism that differs from that found in the spread of disease, in contrast to common believe. In this mechanism, an individual with more friends needs more repeated exposures to spread further the information. Moreover, our data suggest that in contrast to epidemics, for certain messages the chance of an individual to share the message is proportional to the fraction of its neighbours who shared it with him/her. Thus the greater the number of friends an individual has the greater the number of repeated contacts needed to spread the message, which is a result of competition for attention. We model this process using a fractional susceptible infected recovered (FSIR) model, where the infection probability of a node is proportional to its fraction of infected neighbors. Our findings have dramatic implications for information contagion. For example, using the FSIR model we find that real-world social networks have a finite epidemic threshold. This is in contrast to the zero threshold that conventional wisdom derives from disease epidemic models. This means that when individuals are overloaded with excess information feeds, the information either reaches out the population if it is above the critical epidemic threshold, or it would never be well received, leading to only a handful of information contents that can be widely spread throughout the population., Comment: 11 pages, 5 figures

In this paper, we propose a 2D based partition method for solving the problem of Ranking under Team Context(RTC) on datasets without a priori. We first map the data into 2D space using its minimum and maximum value among all dimensions. Then we construct window queries with consideration of current team context. Besides, during the query mapping procedure, we can pre-prune some tuples which are not top ranked ones. This pre-classified step will defer processing those tuples and can save cost while providing solutions for the problem. Experiments show that our algorithm performs well especially on large datasets with correctness.

Hundreds of thousands of hashtags are generated every day on Twitter. Only a few become bursting topics. Among the few, only some can be predicted in real-time. In this paper, we take the initiative to conduct a systematic study of a series of challenging real-time prediction problems of bursting hashtags. Which hashtags will become bursting? If they do, when will the burst happen? How long will they remain active? And how soon will they fade away? Based on empirical analysis of real data from Twitter, we provide insightful statistics to answer these questions, which span over the entire lifecycles of hashtags.

Context-aware database has drawn increasing attention from both industry and academia recently by taking users' current situation and environment into consideration. However, most of the literature focus on individual context, overlooking the team users. In this paper, we investigate how to integrate team context into database query process to help the users' get top-ranked database tuples and make the team more competitive. We introduce naive and optimized query algorithm to select the suitable records and show that they output the same results while the latter is more computational efficient. Extensive empirical studies are conducted to evaluate the query approaches and demonstrate their effectiveness and efficiency.

Many applications motivate the distance measure between rankings, such as comparing top-k lists and rank aggregation for voting, and intrigue great interest to researchers. For example, for a search engine, the use of different ranking algorithms may return different ranking lists. The effect of a ranking algorithm can be estimated by computing the distance (similarity) between the result ranking it returns and the appropriate ranking people expect. People may be interested in only the first few items of result ranking, therefore the metric for measuring the distance should emphasize on the items in higher positions. Besides, in an extreme case, if a result ranking is the total reverse of the expected ranking, then it is considered to be the worst ranking with the maximum distance. Therefore, a metric is called for, which can satisfy both of the two intuitions. To address this problem, we present a weighted metric based on the classical Spearman's footrule metric to measure the distance between two permutations of n objects. This metric can be applied in rank aggregation problem with a polynomial time algorithm, and produces a 2-approximation for adopting the weighted Kendall's tau distance proposed by Farnoud et al., Comment: 10 pages, 1 figure